A consensus protocol is usually argued about in its steady state. Operators meet it in every other state: a rack loses power mid-epoch, a link drops a tenth of its packets, a machine boots with its clock a minute off, a validator comes back after an hour and has to find its way into an epoch that has moved on. Formal models catch a great deal, and we keep them for Tenzro's commit rule and checkpoint voting. They do not run the real binary, the real storage engine or the real network stack. For that we break a running network on purpose.
This post describes the method and one class of bug it found, because the method is only worth something if it finds things.
The harness: seeded, written down first, replayable
The approach follows the principles of chaos engineering: state a steady-state hypothesis, vary real-world events, and look for a difference. Tests in the spirit of Jepsen add the rule that matters most for consensus: check safety from the outside, against what clients observe, not against what nodes claim about themselves.
Our soak runs against a private network of real node binaries, with software TPMs standing in for hardware keys, all inside one network namespace so nothing touches other traffic. From a seed it draws a schedule of faults and writes the whole schedule to disk before the first fault fires. A failing run is therefore a seed and a file, and can be replayed.
The faults it schedules:
- Stop and restart a validator at a random point within an epoch, so restarts land before, during and after checkpoint votes and epoch handovers.
- Degrade the link with latency, jitter and packet loss for a window, using netem on the namespace's interface.
- Partition the nodes into two groups with nftables rules, then heal.
- Skew a clock: restart a validator with its wall clock offset.
One fault it does not produce is equivocation. A release binary cannot sign two conflicting checkpoints without a test build, and a test build is not what operators run. Equivocation is covered by the formal model and by unit tests of the evidence path instead.
What the observer checks
Every five seconds the harness samples each node's head, its finalized height and the hash of its latest finalized block, and checks three properties:
- Safety. No two nodes ever finalize different blocks at the same height. One violation fails the run.
- Liveness. While at least 2f+1 validators are up and connected, a new checkpoint certificate appears within a bound.
- Rejoin. A restarted validator reaches the network's finalized height within a bound after the network is live again.
The liveness check taught us something about our own instruments before it taught us anything about the protocol. Tenzro makes a block only when there are transactions to order, and a quiet chain keeps certifying checkpoints without new heights. A soak that sends no traffic would see finalized height stand still and report a stall that is not there. So liveness is measured by the newest certificate any node reports, as an (epoch, index) pair, and heights are kept for the report only. The liveness clock also starts at genesis time rather than at launch, because no certificate can exist before genesis. Both were bugs in the observer. A chaos harness is software too, and it gets debugged alongside the system it tests.
What a stall teaches you
When a run fails, it almost never fails on safety. It fails on rejoin: a validator comes back and does not catch up, or catches up to one epoch and stops. A stall is the most useful failure there is, because the node is still running and its state can be inspected. Three habits came out of chasing them.
Make the node say why. An early stall showed a rejoining node logging that it was catching up thousands of times while its height never moved. A catch-up round could end silently in two ways: the peer sent no blocks, or the import advanced neither the tip nor the known epoch. Both now log the height and epoch the round stopped at. Half of fixing a stall is being able to see which of several quiet exits it took.
Prefer evidence over timeouts. One stall came from a validator whose consensus core had stopped advancing while the network moved on. The core waited for a garbage-collection depth that a silent core never reaches. The fix fast-forwards such a core to a checkpoint, but only when it holds a verified certificate showing the network at least four rounds ahead in its own epoch. Requiring a verified certificate ahead means a stall of the whole network can never trigger it.
Expect each fix to reveal the next stall. A validator restarted on a chain with no new transactions took a handover at the end of an epoch whose last block it had executed before the restart, and waited for that block to execute again. Handing the driver its executed tip at start moved it past that epoch, and the same test then showed it stopping one epoch later. That second stall was recorded as open, not declared fixed.
A real class of bug: progress that was not progress
The most instructive bug was about what counts as progress during catch-up.
A node far behind catches up by asking peers for ranges of blocks, each served with the checkpoint certificate that covers it. The node's consensus handle tracked one height that rose for two different reasons: when a block was executed, and when a certificate was recorded. The catch-up logic used that height to decide how far it had executed.
A node that received certificates ahead of the blocks it had executed therefore believed it had executed those heights. It ended catch-up early, treated the heights as done, and then dropped their blocks as missing from storage. It never built them. Measured by the counter, the range was progress. Measured by the store, nothing had been imported.
The fix was to keep a separate executed height, raised only when a block is executed, seeded or resumed, and to drive catch-up from that.
The same root cause showed up in mirror image. The core takes a served certificate only after the import call returns, so the epoch change it causes appeared a moment later. A range that had in fact advanced the node was logged as fruitless, and the retry logic waited a full interval before the next request. A node many epochs behind then gained one epoch per interval and could never catch a network whose epochs were shorter than that. The fix waits briefly for the epoch, tip or checkpoint to change before judging the round, and then loops straight on.
Both bugs passed unit tests, because each component was correct in isolation. They lived in the gap between two components' definitions of "done", and only a node rejoining a moving network under a fault schedule walked into it.
What this means for operators
The properties worth asking any network about are the ones above, stated so a third party can check them: no conflicting finalized blocks, certificates keep coming while a quorum is up, and a restarted machine rejoins within a bound. For an operator, the practical consequence is simpler. Restart a node by restarting the unit; never wipe its data directory. A validator rebuilds its consensus state from a journal it writes before sending anything, and catch-up is driven by what it actually executed.
Where Tenzro fits
Built: the seeded fault schedule (restarts, latency, jitter, loss, partitions, clock skew), the safety, liveness and rejoin checks, the formal models and conformance replay that sit beside them, and the catch-up fixes described here. The network is live in preview, and soak testing continues while governance decides the steps to testnet and mainnet. We publish the method, not a claim that the work is finished: some stalls are recorded as open, and that record is part of the engineering.
