Deterministic Raft experiments with durable, replayable decisions.

d-raft is a research platform for turning distributed-systems failures into small, independently checkable execution artifacts. It combines a pure Raft reference state machine with virtual time, a faultable network, explicit persistence acknowledgements, crash/restart simulation, semantic decision recording, and package-separated safety checks with structured witnesses.

The project is created and maintained by Mohammadamin Khanbabaei (aminkbi).

Research status: the deterministic kernel, durable reference Raft model, cluster harness, safety checker, observational trace decoder, and exact semantic decision replay are implemented. Self-describing run artifacts and the research CLI are also usable. Bounded prefix exploration and fingerprint-preserving semantic minimization are implemented. Durable snapshots, safe log compaction, and snapshot-bearing run artifacts are also implemented. Joint-consensus membership changes and learners are implemented, including durable recovery and snapshot-aware configuration state. An experimental adapter for the production-used go.etcd.io/raft/v3 core is implemented for a declared fixed-membership capability subset. A versioned, portable binary KV application oracle now produces independently checkable state/history commitments in both adapters, including reference snapshot recovery. A versioned six-fault corpus and isolated, repository-pinned runner are implemented with a clean Go 1.26.6 result: three checker-backed safety kills and three separately reported conformance kills. Strict adapter-neutral semantic plans, bilateral capability preflight, projection accounting, normalized outcomes/comparisons, source-provenance verification, and a two-adapter research CLI are implemented. Immutable, CI-verified cross-adapter control and faulted-workload cases are published. A clean-provenance 21-trial bounded harness/accounting study, raw trial observations, paired cache contrast, and pinned related-work matrix are also published; real-bug effectiveness and diagnosis-time claims remain outside the measured evidence.

A seed says how to repeat one pseudorandom run in one implementation. A useful counterexample should say which semantic choices mattered, retain evidence of the violated invariant, survive irrelevant changes in random-number consumption, and be portable to another implementation.

d-raft's working research direction is therefore:

Portable, minimized, independently checkable semantic Raft counterexamples that replay across implementations and versions.

Deterministic simulation, pure Step APIs, seeded replay, and trace reduction all have substantial prior art. d-raft does not present those techniques alone as novel; RESEARCH.md defines the narrower thesis and evaluation plan.

The root module is dependency-free and uses no wall-clock sleeps or background goroutines. The isolated nested etcd/raft adapter carries its production-core dependency. Both target Go 1.26 and declare the current Go 1.26.6 toolchain.

See MUTANTS.md for the seeded-fault evaluation contract and runner trust boundary. See SEMANTIC_PLANS.md for the cross-adapter portability and comparison contract. See EVALUATION.md for the bounded study, RELATED_WORK.md for pinned positioning, and REPRODUCIBILITY.md for verification and archival steps. Release-level changes are summarized in CHANGELOG.md.

flowchart LR
    Scenario[Scenario / faults / proposals]
    Decisions[Semantic decider]
    Harness[Durable Raft harness]
    Model[Pure Raft model]
    Runtime[Virtual time + network]
    Checker[Package-separated checker]
    Artifact[Decision tape + observations]

    Scenario --> Harness
    Decisions --> Harness
    Harness <--> Model
    Harness <--> Runtime
    Harness --> Checker
    Decisions --> Artifact
    Runtime --> Artifact
    Checker --> Artifact

The persistence boundary is explicit:

Step(input)
  -> Persist(token, state)
  -> simulated durable completion
  -> Step(Persisted(token))
  -> dependent messages, timers, apply, and snapshot-install effects

A crash destroys volatile raft.Node state. Restart creates a fresh node only from the durable store. This lets tests distinguish a crash immediately before persistence from one immediately after it.

config := raftsim.DefaultConfig("a", "b", "c")
config.Seed = 42

cluster, err := raftsim.New(config)
if err != nil {
    log.Fatal(err)
}
if _, err := cluster.RunUntil(2 * time.Second); err != nil {
    log.Fatal(err)
}

leader, ok := cluster.Leader()
if !ok {
    log.Fatal("no unique leader")
}
if err := cluster.ProposeTo(leader, []byte("set x=1")); err != nil {
    log.Fatal(err)
}

Partitions, scheduled crashes, restarts, and crash-after-persist boundaries are available directly on raftsim.Cluster. Every semantic step is checked; when StopOnViolation is enabled, a run stops at the first safety violation.

Recording is separate from the observational trace. The choice tape describes election timeouts, network loss and latency, and storage completion latency by stable semantic identity.

recorder := decision.NewRecorder(decision.NewSeedDecider(42))
config.Decider = recorder
original, _ := raftsim.New(config)
_, _ = original.RunUntil(2 * time.Second)

tape := recorder.Tape()
replay, _ := decision.NewTapeDecider(tape)
config.Seed = 999 // infrastructure RNG no longer controls these choices
config.Decider = replay
replayed, _ := raftsim.New(config)
_, _ = replayed.RunUntil(2 * time.Second)
if err := replay.Finish(); err != nil {
    log.Fatal(err)
}

TapeDecider stops at the first choice ID, kind, domain, or selection mismatch. This makes replay drift explicit instead of silently producing a different run. Exact execution also requires the same scenario version, external actions, and run horizon; d-raft.run/v3 bundles those inputs with the tape.

Build the research CLI and create a self-contained run artifact:

go build -o draft ./cmd/draft
./draft run --seed 42 --duration 2s --out run.json
./draft canonical --seed 1 --out portable-faults.json portable-faults-v1
./draft inspect run.json
./draft replay run.json
./draft explore --depth 6 --max-runs 1000
./draft explore --cache=false --depth 6 --max-runs 1000 # matched baseline
./draft minimize --out minimized.json failing.json

go build -buildvcs=true -o draft-eval ./cmd/draft-eval
./draft-eval --trials 21 --out d-raft-evaluation.json
./draft-eval --verify d-raft-evaluation.json

draft replay starts from a clean cluster, consumes the stored choice tape, rejects any semantic drift, and verifies the recorded outcome status, step count, virtual end time, violation fingerprints, and versioned canonical observation digest. Artifact writes use a temporary file and atomic no-clobber publication, so an encoding or filesystem failure does not leave a plausible partial result. Artifacts remain private (0600) by default.

draft run and draft explore generate a steady all-voter scenario. The named portable-faults-v1 canonical scenario generates four portable KV proposals around a minority partition/heal and a crash/restart, followed by a 2.6-second quiet convergence tail. Its version fixes the three-node timing, 2% network loss, command bytes, and seed 1; a different semantic stream requires a new canonical version rather than silently changing the published experiment. The v3 schema and Go APIs can also execute scheduled proposals, snapshots, joint membership transitions, partitions, healing, crashes, and restarts, including the crash-after-persistence-before-acknowledgement boundary. Role changes are limited to a pre-provisioned Members universe; they do not perform dynamic discovery or process creation. Additional named fault suites and broader production adapters are later milestones. The experimental etcd/raft production-core adapter and separate draft-etcd CLI are documented in ADAPTERS.md. Canonical reference-state caching is enabled by default for draft explore and bounded with --cache-entries and --cache-bytes. See ARTIFACTS.md, MEMBERSHIP.md, SNAPSHOTS.md, EXPLORATION.md, CANONICAL_STATE.md, and MINIMIZATION.md. The opt-in cross-adapter application profile is specified in APPLICATION_ORACLE.md.

draft-eval runs the fixed balanced study only from a clean, VCS-stamped Linux build. It records all raw trials, machine and schema identity, status-separated terminal outcomes, processed event attempts, and the paired cache-on minus cache-off contrast. Its common runner-invocation ceiling does not make random full runs and DFS frontier probes equal computational work. If the workload has no cache hits or state pruning, the result characterizes cache overhead with no repeated exact cache identity; it is not evidence of pruning efficacy or real-bug detection probability. The output path is preflighted before the run and published as a durable, 0600, no-clobber artifact.

sim.JSONLRecorder emits globally ordered d-raft.trace/v1 JSON Lines. trace.Decoder enforces schema and sequence rules, supports compatible and strict validation, bounds record size, and preserves protocol payloads as json.RawMessage so 64-bit terms and indexes are never routed through float64.

The observational trace and semantic decision tape are deliberately distinct: the former explains what happened; the latter drives an execution.

For a fixed d-raft version, initial state, API-call sequence, decision tape or seed, and deterministic callbacks, d-raft reproduces virtual time, event and packet order, protocol state, durable state, and trace output. Unordered map iteration, wall-clock reads, external I/O, concurrent callbacks, and mutation after an inadequately cloned send are outside that guarantee. See COMPATIBILITY.md.

go test ./...
go vet ./...
go test -race ./...

cd adapters/etcdraft
go test ./...
go vet ./...
go test -race ./...

Contributions should include a deterministic regression test and, for protocol changes, a crash-boundary test where persistence matters. See CONTRIBUTING.md.

d-raft builds on ideas demonstrated by etcd/raft's deterministic core and TLA+ trace validation, FoundationDB simulation, DEMi, SAMC, MadSim/MadRaft, Oddity, Coyote, Turmoil, VOPR, and commercial deterministic-testing systems such as Antithesis. The research target is a portable semantic counterexample format and reduction/evaluation workflow, not another claim that seeded simulation itself is new.

If d-raft supports published work, cite the repository using CITATION.cff and record the release or commit, Go toolchain, scenario version, adapter version, and semantic tape schema.

Licensed under Apache 2.0.