Slow Database Writes? Blame the Map.

Here are two pairs of cloud regions that sit almost exactly the same distance apart. Virginia to São Paulo is 7,480 km. São Paulo to Johannesburg is 7,429 km. Fifty kilometres between them, which is nothing.

Azure measures the first at 117 milliseconds and the second at 321.

Nearly three times the wait for the same distance, on the same provider’s backbone, in the same month. If you put a database quorum across the second pair instead of the first, every single write you ever do costs you an extra fifth of a second, and no amount of arguing about Raft versus Paxos in a design review will get you a millisecond of it back.

That is the argument. The protocol is the part everyone discusses and the part that barely matters. The map is the part nobody opens and the part that sets your p99. And it isn’t even the map you’re picturing.

The part of the bill that’s physics

Start with the number you genuinely cannot negotiate. Light in single-mode fibre travels at about c divided by a refractive index of 1.468, which is roughly 204,000 km per second. So any two points on earth have a minimum round trip that follows from the distance between them and nothing else.

Virginia to Amsterdam is 6,446 km of great circle, so the floor is 63 milliseconds. Azure measures 83.¹¹Round trips throughout this post are Azure’s published P50 between regions, from the table Microsoft refreshes every six to nine months (Microsoft). The dataset I’m reading covers the 30 days to 30 July 2026. Distances are mine, computed from the city each region names.

Twenty milliseconds of that 83 is somebody’s to fix: switching, routing, and the fact that no cable is laid along a geodesic. The other 63 belongs to the universe. You could buy the network, fire the network team, replace every router with something perfect, and Virginia to Amsterdam would still be 63 milliseconds, because that is how long light takes to get there and back through glass.

So the useful question about any link isn’t “how fast is it”. It’s “how many times the speed of light is it”. Drag the two ends around and watch that ratio move:

Now go back to São Paulo and Johannesburg. If you charged the whole 321 ms to distance, which isn’t fair because some of it is switching, you’d be paying for about 33,000 km of fibre one way. The planet is 40,075 km round. Whatever those packets are doing, they are not crossing the South Atlantic in any sense a globe would recognise.

That’s the first thing to internalise: the great circle sets a floor nobody beats, and how close you get to it is a property of the cable map, which looks nothing like the world map. Which is also why the link you’d expect to be practically free turns out to be the worst deal on the board.

The short hops are the worst value

Here’s the bit that surprised me, and it’s the opposite of what you’d guess.

The worst-value link in Azure’s table isn’t the one crossing the most water. Measured against its own floor, the worst deal available is London to Amsterdam: 358 km apart, a floor of 3.5 milliseconds, and a measured round trip of 11. That’s 3.1 times the speed of light, for two cities you could drive between in a day with a ferry in the middle.

Iowa to Virginia is 2.1 times. Washington State to Virginia is 2.1 times. Meanwhile Central US to Sydney, 14,383 km and most of an ocean, comes in at 1.25 times the floor. It is the best-value link on the board.

The reason is that the overheads don’t scale with distance. Every hop costs you a switch traversal whether the next hop is in the next rack or the next hemisphere, and a cable that detours 200 km to reach a landing station adds 200 km whether the journey is 358 km or 14,000. On a short link that fixed cost is most of the number. On a long haul it disappears into the noise.

On a long link you’re paying for physics. On a short link you’re paying for everything else, and the everything else is bigger than you think.

Which has a practical edge to it. If your three replicas are in one metro and the commit floor is 4 milliseconds, you are not close to the limit, you are about three times off it, and a serious network team might find you some of it. If your replicas are on three continents, there is nothing to find. You are already at 1.3 times physics and the remaining 0.3 is the cable going round Africa.

Either way, the number you can’t move is the one the protocol has to live inside. So what does the protocol actually get to decide?

The protocol only picks how many round trips

Consensus over a quorum requires at least one round trip to a majority of replicas. That isn’t an implementation detail, it’s the shape of the problem, and it has been proved as a lower bound rather than merely observed: Lamport shows consensus can’t be achieved in fewer than two message delays, which makes Paxos optimal rather than merely good (Lamport). Raft says the same thing from the other end, that in the common case a command completes as soon as a majority has answered a single round of RPCs (Ongaro and Ousterhout).²²Raft section 5.3 is the one to read. The leader appends the command, issues AppendEntries in parallel, and an entry counts as committed once it is on a majority of servers. Everything about latency follows from that one sentence.

Here is the protocol itself, running. Click a replica to kill it; kill the leader and the cluster has to notice through its failure detector before it can elect a new one. That pause is the thing the last section of this post is about.

The rule for the common case fits on a napkin, and it is not the slowest replica that matters:

def commit_latency_floor(leader, replicas, rtt):
    """Lower bound on commit latency for a majority quorum.

    rtt[a][b] is the measured round-trip time between sites.
    The leader commits once a majority (itself included) has acked,
    so it waits on the (n//2)th fastest peer, not the slowest.
    """
    peers = sorted(rtt[leader][r] for r in replicas if r != leader)
    quorum_peers = len(replicas) // 2      # leader is the +1
    return peers[quorum_peers - 1]

Run that over a real topology and it lands within a few milliseconds of what you’ll measure, because everything else (serialisation, fsync, the log append) is noise next to the time light takes to cross an ocean.

Protocol choice does move one number, to be fair to the design review. EPaxos commits in a single round trip in the common case, falling back to two when commands actually interfere with each other, which is a real saving when your workload rarely touches the same keys twice (Moraru et al.). It changes how many round trips you pay for. It does not change what one costs.

And since the count is fixed at one and the cost is fixed by geography, the only remaining variable is which replica the round trip starts from.

Where the leader sits is a product decision

Which makes leader placement uncomfortable for org charts. It isn’t an infrastructure setting. It decides which of your users get a fast write and which get a slow one, and that is a product question wearing an ops costume.

Every number in this table is the deciding ack from Azure’s published matrix, computed the way the listing above does it:

Look at rows one and two. Same three machines, same protocol, same code. Move the leadership from London to Virginia and every write goes from 11 milliseconds to 78, because the quorum now closes on Amsterdam instead of on a city 358 km away. That’s a seven-fold latency regression available to anyone with permission to run a failover, and in most shops nobody owns the decision.

Then look at the last row against the third. Adding Tokyo and Sydney to a Virginia-led cluster costs nothing at all. The floor stays at 83 milliseconds, because with five members the quorum closes on the second-fastest peer, and the second-fastest peer is still Amsterdam. Two extra continents of fault tolerance, free, and you only see that if you’ve done the arithmetic rather than assuming more replicas means more waiting.

Now the detail that makes this worse, and that I only noticed because Microsoft says it out loud. The latency table is directional. East US to East US 2 is 8 milliseconds; East US 2 to East US is 9. Same two buildings, different number, because the traffic can take a different path each way, and Microsoft tells you to use the value matching the direction your traffic flows.

A quorum only ever reads that table in one direction: outward from the leader. So a cluster you drew as a symmetric triangle isn’t one, and failing over to the “equivalent” replica can hand you a different commit latency for reasons that have nothing to do with the machines.

Pick a topology and a leader and watch the deciding replica change under you:

One row is conspicuously missing from that table, and it is the one people actually reach for when the numbers above come back too high.

The witness replica you have to ask permission for

The usual escape is a witness: a replica that votes but holds no data. Put two of those next to your leader and you get a cheap local quorum without paying to replicate the dataset three times. Every vendor offers some version of it.

MongoDB calls them arbiters, and MongoDB’s own documentation is the best argument against them I’ve read. If a secondary falls behind and the cluster is reconfigured, votes from multiple arbiters can elect the stale node, and the new primary won’t have writes the old configuration had already majority-committed. Their word for the outcome is data loss (MongoDB).

Here is the part worth repeating at a design review. Since MongoDB 5.3, you cannot build that configuration by accident. Multiple arbiters are disabled by default, and adding a second one fails with an error telling you to restart every node with allowMultipleArbiters=true.

A vendor made you type a flag to be allowed to lose committed writes. That is what an honest footgun looks like, and it is the standard I’d hold any witness deployment to: not “is this safe”, but “which specific pair of failures makes it unsafe, and is that sentence in the design doc or is it going to turn up in the postmortem”.

None of which, notice, is about steady state. Arbiters are fine on a Tuesday. They bite during a failure, and so does everything else worth worrying about here.

The timeout matters more than the protocol

Which is the part I’ve come to believe is most underrated. In steady state, every consensus implementation performs about the same, because they’re all paying the same round trip. The differences appear entirely during and after a failure, and that behaviour is governed not by the consensus algorithm but by the failure detector bolted to its side.

etcd is the best worked example because it publishes both the rule and the default. The rule: election timeouts must be at least ten times the round-trip time, to absorb ordinary network variance. The default: 1,000 milliseconds, with a 100 millisecond heartbeat (etcd).

Put those together and the shipped default is in spec up to exactly 100 milliseconds of round trip.

Virginia to Amsterdam is 83, so that just fits. Virginia to Singapore is 224, which needs 2,240 milliseconds, more than twice the default. The same page says the upper limit is 50 seconds and that it exists for globally distributed clusters. So etcd will happily start a cluster that its own tuning guide says is misconfigured, and the symptom won’t be an error, it’ll be an election every time the Pacific has a bad afternoon.³³The absolute numbers on that page are worth noticing too. etcd quotes 350 to 400 ms between the US and Japan; Azure measures 101 ms between Washington State and Tokyo. Both are right. One is describing the public internet and the other a private backbone, and which one you plan against changes your timeout by a factor of four. Measure your own.

A timeout is a policy decision wearing a constant’s clothing. Setting it to 500 ms is a claim: I would rather have a spurious election than 500 ms of unavailability. Setting it to 5 s is the opposite claim. Neither is correct in general, and the constant is usually chosen by whoever wrote the config file first.

// The honest version of a heartbeat timeout: it names the trade
// it is making, and it adapts to the link it is actually running on.
type Detector struct {
    baseline   time.Duration // p99 observed RTT to this peer
    slack      float64       // how much worse than p99 before we call it
    minTimeout time.Duration // floor, so a fast LAN link cannot flap
}

func (d *Detector) Timeout() time.Duration {
    t := time.Duration(float64(d.baseline) * d.slack)
    if t < d.minTimeout {
        return d.minTimeout
    }
    return t
}

Two things change once the timeout is derived rather than declared. Elections stop firing during ordinary network weather. More usefully, the value becomes something you can review: slack: 3.0 invites the question “why three?” in a way that heartbeat_timeout_ms: 1500 never does.

That question, asked out loud, in front of people, is most of what I actually want from any of this.

What I’d do about it

• Write the floor down first. Before choosing a protocol, compute the quorum RTT for the topology you’re actually allowed to deploy, and compute it in the direction the leader waits. If the floor already violates the SLO, no protocol will save you and you need a different consistency model.
• Check the ratio, not just the number. If you’re at three times the fibre floor, there is engineering available. If you’re at 1.3, there isn’t, and the next person who proposes a network optimisation should be shown the arithmetic instead.
• Make leader placement explicit and revisable. Pin it, measure who pays, and revisit when traffic shifts. It will shift.
• Derive timeouts from measurements. A constant is a claim about a network you haven’t measured, and the defaults ship in spec for a LAN.
• Specify the failure modes you accept. Especially with witnesses. The double failure that loses data should be a sentence in a document, not a discovery.

None of this is novel. All of it is routinely skipped, because the protocol is the interesting part and the geography is the part that feels like someone else’s job.

It is 63 milliseconds of someone else’s job, every write, forever.

Sources

Papers

• Lamport, L. Lower Bounds for Asynchronous Consensus, Distributed Computing 19, 2006.
• Ongaro, D. and Ousterhout, J. In Search of an Understandable Consensus Algorithm, USENIX ATC, 2014.
• Moraru, I., Andersen, D. G. and Kaminsky, M. There Is More Consensus in Egalitarian Parliaments, SOSP, 2013.

Documentation

• Microsoft. Azure network round-trip latency statistics.
• etcd. Tuning, v3.5 documentation.
• MongoDB. Replica Set Arbiter, server manual.