Write the Simulator First

For a couple of years I helped run a streaming pipeline that moved about a billion rows an hour. Kafka in front, ClickHouse behind, a few hundred consumers in the middle, and an error budget of 0.0001%, which we never came close to spending.

Every few weeks, a handful of rows would quietly fail to arrive.

Do the arithmetic and it looks like nothing. Three weeks at a billion rows an hour is 504 billion rows, so six missing is about one in eighty four billion, roughly five orders of magnitude inside a budget nobody was policing anyway. No alert fires. No SLO burns. Nobody gets paged.

It is still a bug: the pipeline’s one job is to not lose rows, and the customer on the far end was reconciling counts.

It took months to work out what it needed. A broker leader election had to land during a consumer group rebalance, while a slow disk write was still outstanding on the broker being elected away from. Each of those three things is ordinary. Elections happen, rebalances happen constantly, disks are slow all the time. The three of them inside the same few hundred milliseconds is not ordinary at all.

And here is the part that actually hurt: the one time you catch it, you cannot make it happen again. You ship a patch, the rows stop disappearing for six weeks, and you have learned precisely nothing, because they were only ever disappearing every six weeks. You are not debugging at that point. You are doing folk medicine.

I didn’t have a simulator then, and I have wanted one on every system I have touched since. The honest reason we never caught that bug isn’t carelessness. It is arithmetic, and the arithmetic is worth doing out loud.

How big is the room you’re searching

Take a deliberately tiny cluster. Five nodes, each delivering five messages, each keeping its own messages in its own fixed order. Twenty five deliveries in total.

The number of ways those twenty five deliveries can interleave globally is the multinomial coefficient 25! / (5!)⁵, which is about 620 trillion. That is with nothing going wrong: no crashes, no drops, no clock skew, no disk that takes 400ms to answer. Just twenty five successful messages arriving in different orders.

Now let the world misbehave. Give the scheduler four options at each delivery: deliver it, drop it, delay it, or hand back a torn write. That multiplies the space by 4²⁵, another 1.1 quadrillion, and the total for one twenty five message window of a five node cluster lands around 10³⁰.

Drag the sliders. The interesting control is the bottom one:

A week of production at a billion rows an hour walks through about 6.8 billion of those windows, far more than any simulator will run overnight. That is why people believe soak testing works.

Look at where all 6.8 billion of them land. Every one sits in the single fault mix where nothing went wrong, because in production nothing usually does. The other quadrillion mixes are the ones with your bug in them, and production visits them at their natural rate, which for a three way coincidence is approximately never.

Production isn’t a search. It’s one very long walk down one corridor of a building with a quadrillion corridors, and the corridor it walks down is the one marked everything is fine.

Which means running longer cannot fix it. You have to change what you are drawing from, and somebody did exactly that in 2009 by doing the work in an order almost nobody does.

FoundationDB wrote the simulator first

FoundationDB is a transactional key value store that underpins a good deal of Apple’s cloud infrastructure, and the SIGMOD paper its team published in 2021 is worth reading for the testing section alone.

The team started in 2009. Their first move was not a storage engine, a consensus protocol, or a query path. The paper puts it plainly: “before building the database itself, we built a deterministic database simulation framework”.¹¹Zhou, Xu, Shraer, Namasivayam et al., FoundationDB: A Distributed Unbundled Transactional Key Value Store, SIGMOD 2021. Section 4 is the one to read. Note the author list, which carries three affiliations at antithesis.com: the testing company grew out of the database.

Everything else is downstream of that. All the database code is deterministic: they avoided multithreaded concurrency entirely and deployed one node per core instead. They extended C++ with a language called Flow, which adds async/await style primitives and an actor model, so the runtime owns the scheduling rather than the kernel. Network, disk, time and the random number generator are all abstracted behind interfaces. A whole cluster, workloads included, runs inside one discrete event simulation in one process, and the production build is a thin shim over the real system calls.

The good material is a level below that summary, in the fault injection part.

Buggification. At many points in the codebase, the simulator is offered the chance to behave badly but legally: return an error from an operation that usually succeeds, insert a delay into something that is usually fast, choose a strange value for a tuning parameter. Randomising the tuning parameters has a second effect the paper calls out, and it is my favourite line in the section: it stops a performance tuning value from quietly becoming necessary for correctness.

Swarm testing. Every run randomises the cluster size, the configuration, the workload, the fault injection parameters, the tuning parameters, and which subset of buggification points is even switched on.²²The technique is from Groce et al., Swarm Testing, ISSTA 2012, and FoundationDB open sourced their harness for it as Joshua.

Coverage macros that tell you where you are blind. A developer who suspects a new branch rarely runs with a full buffer writes TEST(buffer.is_full()); into the code, and the analysis afterwards reports how many distinct runs hit that condition. If the answer is zero, that is not a passing test. It is a note telling you to add a fault that makes it happen.

Swizzle-clogging. There was, for a while, an informal contest to design the failure that found the nastiest bugs. The reigning champion, per FoundationDB’s own docs, is this: pick a random subset of nodes, stop each of their network connections one at a time over a few seconds, then bring them back one at a time in a different random order. That is the detail I have never managed to stop repeating.

One more advantage only shows up once you are inside a simulator. Discrete event simulation runs faster than real time when the simulated machines are idle, because there is nothing to wait for and the clock jumps to the next event. Distributed systems bugs are full of idle stretches, waiting out an election timeout or a lease. In production those bugs are slow and therefore expensive to hunt. In simulation they get cheaper.

The paper’s own evidence is more useful than the anecdote anyway. Early FoundationDB used Apache Zookeeper for coordination. Fault injection found two independent bugs in it around 2010, so they deleted the dependency and wrote their own Paxos implementation in Flow, and the paper reports no production bugs in it since. Apple’s CloudKit ran FoundationDB for more than half a million disk years without a single data corruption event.

None of which works without one property so plain it is easy to skim past: the entire cluster’s history has to be a function of a single number.

One seed is one Tuesday

That is the whole trick, and it’s worth stating irritatingly literally. Every decision the world would otherwise make for you, the seed makes instead. What time it is. Which message arrives first. Which disk is slow. Which process dies. All of it drawn from one seeded generator, inside one thread, with no way for the outside world to get a vote.

So the state of the simulated cluster at every step is a pure function of that seed. A failure stops being an event that happened to you once and becomes a value: a number you can put in a commit message, a ticket, or the repository next to the fix.

Here is a toy scheduler doing exactly that. The chips are seeds. Press the button as often as you like:

Seed 4,829,780 loses row 42 at t04, because a rebalance begins at t02, broker 1 starts a slow fsync of row 42 at t03, and the election lands on broker 1 at t04. Then the consumer cheerfully commits offset 42 at t05, which is the bit that makes it silent. That is my pipeline’s bug, in five lines, with a name.

The practical consequence isn’t really about finding bugs. It is that you get unlimited tries. Add a log line, add an assertion, add a print statement in the middle of the hot loop: the schedule doesn’t move. FoundationDB’s lessons section calls this out as the reason their debugging is cheap, and it is the whole difference: a print statement changes what you can see, not what happens.

Compare that with the usual experience, where you add a log line to a race and the race goes away. That log line is not a diagnostic. It’s a fix you can’t ship.

My six vanishing rows gave me maybe eight reproductions a year, all useless, because each was gone by the time I looked. A seed gives you eight thousand a minute and they are all the same one.

The catch is that the seed only owns the history if the seed owns everything, and the world has more ways to get a word in than you would guess.

Six taps, and all of them have to be shut

There are six places the nondeterminism gets back in. Five are the ones you would list if asked: the wall clock, the random number generator, thread scheduling, network delivery order, and the filesystem. Each needs an interface with two implementations, a real one for production and a seeded one for tests.

The sixth is the one that actually ships.

Go’s specification declines to promise anything at all about the order you get when you range over a map, including that two passes over the same map will agree (Go). Python salts the hashes of str and bytes objects per process, on by default, unless you pin PYTHONHASHSEED (Python). Neither of those is a bug. Both are deliberate, and both will make two runs of the same seed diverge with absolutely nothing in the diff to explain why. One for k := range m over a map of peers, one ordering decision downstream of it, and your reproducible failure is a flake again.

Toggle the taps and see what a partial job buys you:

Nothing, is the answer, and that is not rhetoric. It is the argument Antithesis makes about their own infrastructure: connect a deterministic system to one nondeterministic channel and the whole thing is nondeterministic again.

They took that seriously enough to write a hypervisor: a fork of FreeBSD’s bhyve with a lot of standard functionality deliberately removed, on the grounds that ordinary computers were never designed for end to end determinism. Read the clock from inside the guest by any route, TSC and HPET included, and what comes back is a number the hypervisor made up. They tried driving that number from the CPU’s instructions retired counter, which sounds ideal, and found it was miscounted roughly once in every trillion instructions, for reasons they can only guess at. Each instance is pinned to a single physical core, because parallelism turns out to be a time problem wearing a different hat.

One in a trillion instructions is the level of paranoia that owning the whole history actually requires.

You are not going to write a hypervisor. You can write an interface. Clock, Net, Rng, Storage: real implementations wired up in main, seeded ones wired up in tests. FoundationDB put all the difficulty on the test side of that boundary and left the production path a pass-through, which is the right shape: the cleverness lives in the test build, and the thing you ship is boring.

Which makes it sound solved. It isn’t, and the people who build the best simulators say so more loudly than their critics.

The part where this doesn’t work

TigerBeetle is a financial transactions database written in Zig, and the modern poster child for this approach. Its simulator, the VOPR, runs an entire cluster with simulated clock, disk and network, at roughly a thousand times speed, continuously on 1,024 cores. They even split it into a safety mode and a liveness mode, because uniformly random fault injection heals partitions on its own and therefore cannot catch a cluster that livelocks forever.³³The liveness work found a lovely resonance bug: replica A was missing ops 5 and 6, B had 6 but not 5, C had 5 but not 6, and A’s round robin repair counter alternated perfectly, so it asked B for the one thing B lacked and C for the one thing C lacked, forever (TigerBeetle).

Then they paid Kyle Kingsbury to run Jepsen against it, and he found a correctness bug in the query engine: a query intersecting two fields would sometimes return no results at all. That component was covered by four separate fuzzers.

TigerBeetle wrote up why all four missed it, and it is the most useful thing I have read about the limits of this technique. The VOPR pre-registered the queries it would run, giving each one a tidy set of field values, user_data_64 = index * 1_000_000 and so on, and every transfer it created matched exactly one of those queries. It did that for a sensible reason: it could then check a cheap property, the result count, without building a full model of the query engine.

The side effect was that the objects matching any given query were always consecutive in each secondary index. So the two index iterators were always in step. So the merge join never needed to skip one forward, which is the operation they call probing. And probing is the only thing that triggers the bug.

A fuzzer handed a tidy workload had quietly deleted the exact dimension the bug lived in. Four fuzzers over that code: two couldn’t reach it by construction, and the other two had been given a workload that made probing impossible. All four green. Their own conclusion is the one I would put on a wall: a fuzzer going quiet doesn’t mean the bugs have gone, it may only mean the fuzzer has run out of state space it knows how to reach.

FoundationDB’s paper is candid about its own limits too, in a paragraph most teams would have cut. Simulation cannot reliably detect performance problems, like a load balancing algorithm that is merely bad rather than wrong. It cannot test third party libraries, or even first party code not written in Flow, which is why they went to such lengths to avoid dependencies. And several of their bugs came from the real operating system contract being weaker than they believed.

That last one is the honest ceiling. A simulator is exactly as truthful as your model of the machine, and the machine lies.

So simulation and black box testing are not rivals, and picking one is a category error. One of them owns the orderings and gives you a failure you can replay at lunchtime. The other runs the real binary on a real kernel with a real disk underneath it, which is where your model was wrong. Neither one covers the other’s half.

All of which is excellent advice if you are starting a database in 2009, and much less obvious on a Monday morning with an existing service and a bug you cannot reproduce.

What I’d do on Monday

• Pick the interfaces before you pick a framework. Clock, network, randomness, storage. If nobody can say which modules read the wall clock, that is the whole project for this quarter, and worth doing on its own.
• Write the replay before you write the search. A harness that can re-run seed 4,829,780 is worth more on day one than a harness that can generate ten million seeds, because the second one is useless without the first.
• Put the seed in every failure output. Every assertion, every panic, every CI log line. A failure without its seed is a failure you have thrown away.
• Randomise the tuning parameters as well as the faults. FoundationDB’s reason is the good one: it stops a magic number from becoming load bearing for correctness without anybody choosing it.
• Test your determinism deliberately. Run the same seed twice, diff the event trace, and do it in CI, because nondeterminism creeps back in through a map range or a new dependency and nothing else will tell you.
• Count the fault types your current suite enables. For most soak tests the answer is zero. Put the fault slider in that first figure on zero and look at what happens to the space. That is what you own today.

None of this makes the bug rarer. A three way coincidence is as rare inside a simulator as in production, and anyone who says otherwise is selling something. What changes is what happens on the day it finally fires. It fires at a number, the number goes in the ticket, and the bug is now a thing you can summon at will with a debugger attached, for as long as the code lives.

I still think about those six rows. Somewhere there’s a seed for them.

Sources

Papers

• Zhou, J., Xu, M., Shraer, A., Namasivayam, B. et al. FoundationDB: A Distributed Unbundled Transactional Key Value Store, SIGMOD, 2021.
• Groce, A., Zhang, C., Eide, E., Chen, Y. and Regehr, J. Swarm Testing, ISSTA, 2012.

Documentation and reports

• FoundationDB. Simulation and Testing, project documentation.
• FoundationDB. Joshua, the open sourced swarm testing harness.
• Wilson, W. Is something bugging you?, Antithesis, 2024.
• Antithesis. So you think you want to write a deterministic hypervisor?
• TigerBeetle. Safety, project documentation.
• TigerBeetle. Simulation Testing For Liveness, 2023.
• TigerBeetle. Fuzzer Blind Spots (Meet Jepsen!), 2025.
• Kingsbury, K. Jepsen: TigerBeetle 0.16.11, 2025.
• Jepsen. Analyses, published reports index.
• The Go Authors. The Go Programming Language Specification.
• Python Software Foundation. Command line and environment, Python 3 documentation.

@article{ghosh2025write,
  title = {Write the Simulator First},
  author = {Ghosh, Krish},
  journal = {krishghosh.com},
  year = {2025},
  month = {September},
  url = "https://krishghosh.com/writing/write-the-simulator-first"
}