this is my first time writing a writeup, so please excuse my language.
so detsim, its a library for deterministic simulation testing like the FoundationDB/TigerBeetle style of testing, mocks all I/O, runs on virtual time, seeds everything and replays any failure exactly. a couple of attempts at this exist (gosim, simtest-go) but nothings established. this is where detsim comes in. detsim on github if you want to look at it directly.
detsim, a standalone go module
three layers.
the deterministic kernel (sim.go, plus the heap itself split out into eventqueue.go once it got big enough to earn its own file). a discrete-event simulator, basically a min-heap of events ordered by (virtual_time, insertion_sequence). no real sleeps anywhere in it. Sim.After(duration, callback) schedules work on virtual time instead. same seed always produces the identical event order, so a scenario simulating an hour of network traffic runs in milliseconds and replays byte for byte identical every time. added RunUntil(t) too, for stuff like leader heartbeats that reschedule themselves forever and would otherwise never let the queue drain on its own.
network.go, fault-injectable networking. seeded message drops, random delays, network partitions where you can block specific node pairs and heal them later, all running off the same seeded rng as the kernel so its all reproducible together.
storage.go, fault-injectable storage. FaultyStorage stands in for a real disk, does torn writes, byte corruption, syncs that silently do nothing (looks committed to the caller, isnt actually), reordered syncs, and a Crash() that just discards whatever wasnt durable yet. had to rebuild this one already, more on that below.
dogfooding it on two real things
didnt want to just build the library and call it done, a tool that only proves it works against its own test suite doesnt really prove anything. so i built two real systems on top of it.
examples/raft: from scratch, single threaded, event driven raft. leader election, log replication, the fast backtrack optimization from the paper (§5.3 if you want to look it up). written straight against detsim's callback model instead of goroutines and real timers. funny side effect of that: since everything runs single threaded during a sim run, theres zero mutexes anywhere in the implementation. no locks, no races. the whole thing never needed to be threadsafe in the first place, so i never had to write a single mutex to make it one. ended up splitting it into separate files as it grew (messages, node, election, replication, handlers) instead of one giant file, easier to navigate once the state machine logic and the message handling stopped being the same thought.
the test that actually matters here is TestThousandsOfSeedsNoSplitBrain. spins up 5000 independently seeded 5 node clusters, each one goes through normal operation, gets partitioned, then healed, and the whole time it checks the one thing raft exists to guarantee: no two leaders in the same term, ever. all 5000 of those trials run in about 7 seconds. a version of this same test using real timers and real goroutines could never afford anywhere close to that many runs.
examples/kv: small write ahead log key value store sitting on FaultyStorage, with a checksum toggle so i could actually prove checksums matter instead of just saying they do like everyone does.
the bugs, this is the part worth reading
bug 1. the raft cluster test setup started all the nodes with a for loop over a go map. go deliberately randomizes map iteration order, i knew that going in and still tripped on it. that randomized start order changed which node's timer landed first whenever two election timeouts happened to fire at the exact same virtual time, which fed into the tie break the kernel uses to order simultaneous events. so i ran TestSeedIsExactlyReproducible twice with the literal same seed and got two different outcomes, run one had a leader elected in term 3, run two had no leader at all yet. that broke the one promise this whole project exists to make, and it got caught by a test whose entire job was to catch exactly that. fixed it by iterating the ordered id slice instead of the map.
bug 2. FaultyStorage itself had a byte addressing bug. the first version stored committed data as map[offset][]byte, treating every write as one opaque block keyed by its exact starting offset. thats not how a real disk works. the kv store's recovery code reads a fixed size header at some offset, then separately reads the variable length body starting right after it, and since that second offset was never itself a write target, the read just silently came back empty. first time i ran TestBasicPutSyncRecover it failed immediately, read back nothing at all. traced it down to the storage layer, not my kv code. rebuilt FaultyStorage around an actual growable byte buffer with real offset:offset+len semantics, like disks actually behave.
bug 3, smaller but worth mentioning. my first torn write test kept writing the exact same string to the exact same offset a hundred times in a row. once i fixed the byte addressing bug above, a torn (truncated) write would just leave the old identical bytes sitting untouched in the tail, so the corruption became literally invisible, the "torn" write reassembled back into the correct value purely by coincidence. fixed it by varying the payload every iteration. good reminder that a fault injection harness can pass for the completely wrong reason if the workload underneath it isnt varied enough to actually make corruption visible.
the proof
TestNoChecksumStoreCanServeCorruptData runs the no-checksum version of the kv store through up to 5000 seeded fault scenarios, 30% torn write rate, 40% corruption rate. found a seed, seed 3, where it serves back straight up corrupted data with no error whatsoever. TestChecksummedStoreNeverServesCorruptData runs the identical fault injection across 2000 seeds on the checksummed version. never once serves corrupted data. seed 3 is the actual proof, sitting right there in the test output.
numbers if you care
5000 raft cluster trials, partition plus heal each, run in about 7 seconds. 2000 kv checksum integrity trials run in about 0.2 seconds. the 5000 no-checksum corruption hunting trials found the bad seed at seed 3. zero mutexes anywhere in the raft implementation. two real bugs found and fixed through the harness itself, one in its own determinism guarantee, one in the storage primitive, plus one test design bug alongside those.
what this isnt
doesnt intercept the real go scheduler for true goroutine level determinism the way gosim does, that needs patching the actual go runtime which is a whole different project. anything you want to test against this has to be written event driven, callback style, against the kernel. you cant just drop in some existing goroutine heavy code unmodified and expect it to work.
whats left
raft snapshotting and log compaction, linearizable reads, cluster membership changes without a full restart, a wal replay idempotency test specifically for crashing during recovery itself, a disk full fault type, actual benchmarks, ci, no fuzzing wired up yet either.
update, everything above got built
oof, took me alot of time but all of it is done finally!
snapshotting/log compaction: CompactLog trims the log to a boundary, a leader whose nextIndex for some peer falls behind that boundary sends InstallSnapshot instead of AppendEntries. had to make every log index in the implementation base-relative to support this, that touched more files than i expected. tested by partitioning a node away, running the leader well past the compaction threshold so the entries that node needs are just gone from the log, healing the partition, and checking the node actually catches up via the snapshot instead of hanging forever. 300 seeds of that under packet loss.
linearizable reads via ReadIndex: leader records its current commit index, runs a quorum-confirmation round over new PingRequest/PingReply messages (tagged with a per-call sequence number so a stray ack from an earlier round cant satisfy a newer read), waits for its own apply to catch up to that recorded index, then answers. any node that steps down mid-read fails it immediately instead of leaving it hanging forever. the test i actually care about here partitions the leader away, lets the other side elect a new leader and commit writes the old one never sees, then issues a read against the isolated old leader while it still believes its the leader. verified that belief is real (not a vacuous pass) by logging that it self-reported Leader in all 300 trials, then confirmed the read never resolves successfully in any of them.
cluster membership changes: single-server add/remove, a ConfigChange log entry that takes effect the moment its appended, not just once committed, which is raft's actual safety argument for this (waiting for commit first would let you get quorum math from two different membership epochs at once). tested add (new node catches up over normal replication) and remove (removed node correctly excluded from quorum math) separately, then a 300-seed stress test of remove under packet loss. proved that one wasnt vacuous too: logged that the removed node kept campaigning for leader in every single trial (nobody tells it its been removed) and raft's own per-term single-vote rule still held split brain off regardless. theres one known gap i left alone on purpose: an uncommitted config-change entry that gets truncated away by a later conflicting leader doesnt get rolled back. documented, not silently ignored.
wal replay idempotency: recovering twice off unchanged storage now provably gives identical state, and writing more after a recovery correctly resumes from the right offset instead of clobbering anything. the 200-seed stress test here took two tries to get right. first version asserted "everything that was ever synced must always come back," which is wrong on two separate counts, once i actually thought about it: it used SkipSyncRate fault injection, which deliberately makes some synced writes non-durable on purpose (thats a different bug class being tested elsewhere, not this one). pulled that out, and the assertion was still wrong even with just torn writes, because a torn write early in the log correctly makes a sequential parser stop reading right there, so later perfectly intact cycles become legitimately unreachable. thats correct wal behavior, not a bug, so asserting against it was asserting a false thing. rewrote it around the actual invariant: once a key has been recovered, it never disappears or changes value in a later cycle. monotonicity, not completeness.
disk full: FaultProfile.MaxSize caps the virtual disk, WriteAt past that returns ErrDiskFull and writes nothing at all instead of queueing something thatll silently vanish later. kv.Store.Put now returns an ok bool and actually surfaces that failure instead of quietly losing the write.
benchmarks: an 8-simulated-second election-plus-partition-plus-heal cycle runs in about 1.3ms of real wall clock. 100 committed log entries, about 0.9ms. a snapshot compaction cycle, about 1.8ms. simulated time is basically free, which was the entire point of building this in the first place.
ci: build, vet, gofmt check, full test suite under -race, on github actions. go version pinned via go-version-file so it cant silently drift from go.mod.
fuzzing: FuzzRecoverNeverPanics feeds arbitrary bytes straight into kv.Store.Recover (through a new FaultyStorage.SeedRaw), both checksums on and off. ran it for 340k+ executions, zero panics.
also ran the whole suite under -race for real, which actually means something here since the entire architecture's claim is single-threaded-per-simulation, so running it under the race detector confirms that instead of just asserting it in a readme somewhere. and fixed a dumb one along the way: node ids used to come from rune('A' + i), so past node 26 youd get weird non-letter ascii symbols instead of clean names. still technically unique and functional past 26, just ugly, so i wouldnt call it a real functional cap like i almost wrote in the commit message before catching myself. switched to fmt.Sprintf("node%d", i) and checked a 30-node cluster actually works.
update, closing the gosim shaped gap
the "what this isnt" section above kept bugging me. detsim only worked if you wrote the system under test directly against its callback api, real go code with goroutines and channels just couldnt run on it at all. thats the actual thing gosim does that this didnt, and it was a real limitation, not something to wave away. so i built the rest of it: a deterministic green thread runtime, a source rewriter that takes normal goroutine/channel code and points it at that runtime, and a test harness on top that sweeps seeds and shrinks failures. three stages, each one a real chunk of work on its own.
stage 1, rt: the turnstile. new package, detsim/rt. every "goroutine" is still a real go goroutine underneath, but theres a turnstile forcing strict single stepping, at any instant exactly one logical goroutine holds a token and is allowed to run. each one gets a private control channel. the scheduler hands over the token by sending on it, and gets it back by receiving from a second channel the goroutine sends on right before it blocks on anything, a channel send, a mutex lock, a sleep, whatever. no runtime patch, no GOMAXPROCS=1 requirement, real parallelism just never actually happens because only one control channel is ever unblocked at a time.
the part i like most about this design, i didnt need a wake/notify mechanism anywhere. every blocking primitive is just a predicate function plus a park call, and the scheduler recomputes the entire ready set from scratch every single scheduling round instead of trying to maintain it incrementally. that one decision is what makes sync.Cond's Signal-vs-Broadcast distinction fall out for free: even if a Signal makes several waiters predicate-true in the same instant, only one of them actually gets to run that round, and it consumes the signal before the others get re-checked next round. no separate wake list needed per primitive.
deadlock detection was the fun part. instead of trying to snapshot arbitrary goroutines from outside, which means parsing runtime.Stack(buf, true) output and guessing which chunk belongs to who, each primitive just calls debug.Stack() from inside the blocking goroutine itself right before it parks. cheap and exact, since youre literally standing in that goroutine's own call stack when you capture it. so a real deadlock now comes back as a clean error naming every stuck goroutine and exactly where it was blocked, not a test that just hangs until go test's own timeout kills it.
shipped with a worker pool example and a deliberately deadlocking test (two goroutines each sending to a channel only the other one would ever receive from) asserting the error names both of them. full test suite passes under -race, which means something here for the same reason it did on the original kernel, the whole design's claim is that only one goroutine ever touches shared state at a time, so running under the race detector actually checks that instead of just asserting it.
stage 2, the rewrite pass, go f(a, b) is scarier than it looks. new package that uses go/packages with full type info plus go/ast to rewrite a target package's source, then hands the rewritten copies to the real go toolchain through go build -overlay, which is the actual mechanism go exposes for swapping file contents at build time without touching the real source tree.
the channel and sync rewrites were mechanical. chan T becomes *rt.Chan[T], ch <- v becomes ch.Send(v), var m sync.Mutex becomes m := rt.NewMutex(sched). the go statement rewrite is where i almost shipped a real bug. my first instinct was go f(a, b) becomes rt.Go(func(){ f(a, b) }). thats wrong. real go evaluates f, a, and b right there at the go statement, then spawns. wrapping it in a closure like that defers evaluating a and b until the scheduler actually gets around to running the new goroutine, which means if the calling goroutine mutates a in the next line before yielding, the spawned goroutine sees the mutated value instead of what was actually there when you wrote go f(a, b). fixed it by having the rewrite capture everything with a plain := first, then spawn a closure over those captures. go's own type inference handles the typing, i never have to print a single type name anywhere in generated code.
select was the one i called the fiddliest part going in, and it mostly was, until i found a way around the hardest bit. i originally assumed id need to reconstruct channel element types as printed go syntax to build generic RecvCase[T]/SendCase[T] calls, which meant walking go/types output back into parseable source, genuinely annoying. then i realized the case body itself can just call ch.Recv()/ch.Send() on its own, inside a plain untyped closure, and go infers the generic type parameter from the channel argument automatically. no type printing anywhere. that one insight took select from "budget the most design time here" to maybe an hour of actual coding.
two bugs came out of building select for real, both while writing its own tests, which is exactly what a test suite is for. first, my own hand written select test deadlocked, because Select's park loop wasnt registering itself as a waiter on the channel the same way the plain blocking Recv/Send do. an unbuffered channel send only succeeds once it can see a receiver is actually parked waiting, and select was checking readiness without ever incrementing that waiting counter, so a select recv case could never rendezvous with a plain blocked send. fixed by having each select case register/unregister itself against the channel while parked, same bookkeeping the plain blocking path already does.
second, and this one was worse because it would have compiled fine and silently done the wrong thing: if a select case has a return in it (a really common pattern, select { case <-done: return }), moving that case body into a closure means the return now only exits the closure, not the real function. it still compiles. it just quietly stops doing what the original code did. so the rewriter scans every case body for return/break/continue/goto before touching anything, and if it finds one, it refuses to rewrite the whole select statement. and then i found a second layer under that one: if a select gets left alone but its channels get rewritten to *rt.Chan[T] anyway because theyre used elsewhere in the file, the untouched select statement wont even compile against the new channel type. so the actual fix is a whole file scan before any mutation happens at all, if any select anywhere in the file is unsupported, skip rewriting that entire file, dont just skip the one statement. safe over broad, exactly the tradeoff the whole project keeps making on purpose.
struct fields that embed a sync type directly, type Foo struct { mu sync.Mutex }, i did not make work. a zero valued rt.Mutex has no scheduler reference and nothing safe to fall back to without bringing back a global "current scheduler" variable, which breaks the instant two tests run in parallel with different schedulers in the same process. worse, leaving that field as a real untouched sync.Mutex is actively dangerous, not neutral, because a real mutex lock blocks the actual os thread, and the turnstile scheduler is sitting there waiting to hear back from a goroutine that just went and blocked itself for real outside the whole cooperative system. that hangs the entire test binary with no clean deadlock report at all, way worse than a compile error. so instead of silently leaving that landmine in place, the rewriter now scans for it and refuses to touch the whole package if it finds one, with a message explaining exactly why.
os file redirection turned out easier than i expected once i actually looked at what FaultyStorage is, a flat byte addressable disk with zero concept of virtual time in it at all. os.Open/os.Create route to a small rt.FileSystem that hands out one FaultyStorage instance per filename, and os.File's Read/Write/Sync/Close just get implemented directly on the new rt.File type with matching signatures, so call sites dont need touching at all, only the os.Open/os.Create/*os.File spots need rewriting. proved it end to end, real corrupted bytes coming back out the other side of a rewritten, unmodified-looking os.Create/os.Open pair once you crank the corruption rate up.
net.Dial/net.Listen redirection i sat on for a while because of the two clocks problem, the existing Network fault injector runs on the original kernel's event queue clock, rt has its own separate turnstile driven virtual clock, and bridging them correctly is a real architecture decision, not a quick patch. came back to it and the actual fix was to just not bridge them at all. wrote a new type, rt.Network, living entirely on rt's own clock, seeded drop rate and delay range implemented with rt.Sleep-backed delivery goroutines instead of touching the original event queue anywhere. not literal reuse of Network's code, same fault-injection idea, different clock underneath it. proved the whole thing end to end with a real listen/serve/echo/dial roundtrip through the actual rewrite, overlay, compile, run pipeline, same output on repeated runs of the same seed.
first pass at this only gave you address level partitioning, cut off entirely or not, because net.Dial has no concept of "who is dialing" the way detsim's own NodeID-addressed api does, real go doesnt give you that either, the os picks your local address for you, so there was nothing for a pairwise check to key off of at the automatically rewritten call site. went back and closed that gap properly instead of leaving it as a known limitation: rt.Network.Partition(groupA, groupB) now mirrors the original Network's pairwise mesh semantics exactly, same symmetric blocking, rechecked again at delivery time so a message already in flight when a partition lands still gets dropped mid-transit, not just new sends. the trick was adding a second entry point, rt.Network.DialFrom(from, to), for when the SUT already knows its own address, which is the normal case for anything NodeID shaped. the automatic rewrite still only ever produces the plain Dial(addr) form since it genuinely cant infer "my own address" from an ordinary net.Dial call site, so true mesh partitioning lives on the rt side, reachable directly, not through the automatic rewrite of unmodified stdlib code. proved it with a 3 node broadcast scenario, partition A from B and confirm A-C and B-C still deliver fine while A-B genuinely doesnt, ran it 30 times clean before trusting it.
stage 3, the actual product. detsim test wraps go test and sweeps seeds against the rewritten overlay, one env var telling the harness which seed to use. found a real bug in my own sweep tool almost immediately, if you pass -keep-going so it doesnt stop at the first failure, every failing seed overwrites the same trace file, so by the time minimization runs it grabs whatever seed happened to fail last, not the first one it reported. fixed by having the tool deterministically re-run the specific failing seed it cares about right before minimizing, instead of trusting whatever's sitting on disk.
minimization is a standard delta debugging pass (ddmin) over the scheduler's own decision trace, the sequence of which goroutine won each scheduling round. needed a second, lenient replay mode for this, since ddmin deliberately feeds it mangled, incomplete traces and the strict "this must match exactly or error" replay mode used for honest reproduction would just blow up on every candidate. ran it for real against an injected bug in the worker pool example, an assertion that only fails depending on the actual schedule shape, and it took a 9 decision trace down to 1, and that single decision trace still reproduces the exact same failure completely independently, verified by running it again from cold.
seed sweeping alone doesnt tell you much about whether youre actually covering different interleavings or just running the same handful of schedules 500 times with different rng streams that happen to converge. so theres a novelty search, tracks a hashed prefix of each trial's decision trace and stops once a run of trials in a row all produce schedules already seen. built it in process first, ran it against the worker pool for real, 1005 trials, found 211 genuinely distinct schedule shapes before it went dry.
then wired it into detsim test itself, the actual subprocess-driven cli. Sched.SetDecisionLimit(n) makes the scheduler return a sentinel error after exactly n decisions instead of draining every goroutine to completion, checked once at the top of Run()'s own loop. the harness sets it from an env var before spawning anything, and when it sees that sentinel it skips its own assertions entirely, since a decision-limited run is deliberately incomplete and checking real assertions against it would just be checking garbage, it only needs to hand back the partial trace.
detsim test -novelty now runs two phases per seed, a cheap decision-limited peek first, and only if that prefix is actually new does it pay for a full run to check assertions for real. proved it genuinely skips work and doesnt just claim to, ran it against a scenario with zero scheduling diversity and got 11 peeks against exactly 1 full run, the other 10 correctly short circuited. measured the real wall clock difference instead of assuming one, on the actual worker pool example a peek trial ran about 13% faster than a full one, real but modest, because go test's own subprocess and build cache overhead dominates the total cost of a trial far more than the length of the scheduling work does for something this small. still finds real bugs, an injected schedule-shape-dependent failure got caught clean on the full-run phase at seed 1, minimized down the same way the plain sweep does.
update, widening os/net coverage past the original slice
got asked straight up whether this is "gosim level" now. answer is no, and its worth saying exactly why instead of rounding up. gosim patches the actual go runtime, a forked GOROOT with the scheduler and syscall layer replaced, so it gets every goroutine and every syscall in the process uniformly, including inside dependencies you never touched, with real preemption. this project rewrites source and runs a cooperative userspace scheduler. same two categories of coverage as gosim on paper, concurrency primitives and net/os, completely different mechanism to get there, and that mechanism gap is real: unrewritten dependencies are invisible to it, a goroutine that never touches an rt primitive can live-lock a real os thread, select cases with return/break/continue cant be safely rewritten at all. forking the go toolchain to actually close that gap is a different, much bigger project, not an extension of this one, so instead of pretending to do that, widened what the existing rewrite mechanism honestly covers.
os side turned out to have a real bug hiding in it once i actually looked hard at it. Open() and Create() both silently auto-created a file on first touch, meaning os.Open on a file that was never created just succeeded instead of failing like real os.Open does. fixed that first, Open now fails with the same io/fs.ErrNotExist-wrapped error real go returns, Create actually truncates instead of quietly reusing whatever was already there under that name. then added os.Remove/Rename/Stat/ReadFile/WriteFile. the one i like: os.IsNotExist/IsExist/IsPermission/IsTimeout needed zero rewriting at all, left them completely untouched, because they just unwrap an error via errors.Is, and since the filesystem's errors are real io/fs.PathError values wrapping real io/fs.ErrNotExist, the unmodified stdlib functions already work correctly against them. redirecting them would've been pure unnecessary surface.
net side, tried to add Listener.Addr()/Conn.LocalAddr()/RemoteAddr(), a genuinely common pattern that would've failed to compile once rewritten, since the original Addr() just returned a bare string with no .String() method. first attempt made rt.Addr a type Addr string value type, same shape i used for os.FileInfo. broke immediately, first real go vet run against the rewritten overlay caught it: an error-path return nil, err against a net.Addr-typed return, completely idiomatic go for an interface type, doesnt type-check against a concrete value type. switched Addr to a small pointer-returning struct instead, same shape as Conn/Listener already use, and nil works exactly like the real interface does. exactly the kind of thing this project keeps catching by actually running the rewritten code through the real toolchain instead of reasoning about it on paper, which is the whole discipline this thing is built around.
full suite, rt plus the rewriter plus the minimizer plus the worker pool example, all still clean under -race, go vet, and gofmt -l.
update, detsim-trace, or: reading a minimized failure without the decoder ring
the last-mile problem with the whole flow above: detsim-test -minimize could hand you a 12-decision trace that reproduces your bug exactly, and that trace was a list of bare goroutine ids. [7,3,7,3,12,3]. which one is 7? no idea. go look it up in your head against code you wrote three days ago. everything upstream of that point was automated, and then the final output was a puzzle for the human.
so traces are self-describing now. two additions to rt, and neither one changes replay at all, the replay-reproduces-the-outcome test passes untouched: rt.GoNamed("producer", fn) labels a goroutine and the label rides along in Trace() and SaveTrace automatically, and every scheduling decision records a step now, the virtual time it ran at plus what the goroutine did when its turn ended (blocked on chan send, blocked on mutex lock, finished, whatever it was). the part i like most: this needed zero new synchronization. parkCurrent already set its reason before sending on the parked channel, and the scheduler already receives that channel at the exact end of the goroutine's turn, so by the time the scheduler reads the reason back, the channel ordering has already guaranteed it's set. the feature fell out of a handshake that existed for a completely different reason. both new fields are optional in the trace json, so old trace files still load and replay fine.
the viewer is a new traceview package plus cmd/detsim-trace, three subcommands. view renders the annotated decision timeline, one line per pick, with the goroutine's name, what it blocked on, and the virtual time (#1 consumer blocked on chan recv at t=0s). stats is per-goroutine pick counts busiest-first, block-reason histograms (chan send×2, finished×1), and the biggest virtual-time jumps in the run. diff is the one i actually built the whole thing for: it takes the original trace and the minimized one and marks every original decision keep or drop, so instead of trusting that ddmin did something sensible you can see exactly which decisions it decided were load-bearing. it also verifies the minimized trace actually is a subsequence of the original, because the lenient replay mode ddmin relies on can drift, and if it did, the tool says so and prints both shapes instead of drawing a comparison that would be a lie.
detsim-test -minimize carries the labels and surviving steps into detsim_minimized_trace.json now too, so viewing the minimizer's output shows "producer blocked on chan send" instead of "g0 picked".
one honest test-design bug from building this, because there's always one: my first drift test case for diff used original [0,1,0,1,0,1,1] against minimized [0,1,1,1] expecting "not a subsequence", and the test failed, because [0,1,1,1] IS a valid subsequence of that original, i just wasnt reading carefully enough. had to pick one that genuinely isnt, [1,1,0], which needs two 1s the original doesnt have. same lesson as the torn-write test from bug 3 up top: a test that passes for the wrong reason is worse than no test, and i keep stepping on it.
verified the whole chain end to end: wrote a producer/consumer/watchdog scenario that genuinely deadlocks, ran it, ddmin'd the trace from 5 decisions down to 1, and the diff output reads "keep watchdog / drop consumer / drop producer", which is exactly the right skeleton for that failure. color is auto, on for terminals, off when piped, NO_COLOR respected, so it composes with less and ci logs without turning into escape soup.
code's public, detsim on github if you want to poke at it or find the next bug before i do.