Testing Raft with a deterministic simulator

Most Raft implementations are tested by spinning up three processes, killing one, and watching the logs. That proves the happy path works on your laptop today. It proves nothing about the interleaving you didn't think of.

My key-value store takes a different route: nothing inside the cluster touches the real world directly. The clock, the random number generator and the RPC transport are all interfaces injected at construction time.

struct Clock { virtual TimePoint now() = 0; };
struct Rng   { virtual uint64_t next() = 0; };
struct RpcTransport { virtual void send(NodeId to, Message m) = 0; };

In production those are the wall clock, a seeded Mersenne twister and gRPC. In tests they are SimClock, SimRng and SimTransport, and the whole five-node cluster runs inside one process, one thread, in simulated time.

What the harness does

Twelve scenarios, each 50 chaos steps. A step picks one of: partition a subset of nodes, delay a link by 40 to 120 ms, drop 15 percent of AppendEntries, kill the leader, or heal everything. After every step the harness asserts the Raft safety properties over the full cluster state:

• Election safety — at most one leader per term.
• Log matching — if two logs agree at an index and term, they agree on every earlier entry.
• Leader completeness — a committed entry appears in every later leader's log.
• State-machine safety — no two nodes apply different commands at the same index.

That is 12 × 50 = 600 seeded state transitions, and every one of them is checked, not just the last.

The seed is the bug report

Because everything is deterministic, the seed is the reproduction. When an invariant fails, the harness prints the seed and the step number. You rerun with that seed and you get the identical interleaving, every time, in a debugger. No flaky-test folklore.

Linearizability is a separate question

Safety invariants say the cluster never contradicts itself. They do not say clients see a linearizable history. For that, the harness emits a JSON operation history from a real gRPC cluster, and a small Go program runs Porcupine's CheckOperations against a key-value model. Read-index reads are gated on a term no-op commit, and the client-side dedup by (client_id, seq) at apply time means retries do not double-apply.

What it cost

Designing around injected interfaces added a day. It also meant there is a documented lock order, no lock is ever held across an RPC send, and the ASan and TSan build targets stay green. I would not build a consensus implementation any other way now.