Skip to content

Verify a spawned fork is actually a fork: finality, divergence, and usable bootNodes #119

Description

@mordamax

Origin: review of https://github.com/paritytech/preview-net-v1/pull/152, where @pepoviola suggested the divergence check "can be incorporated in zombie-bite directly" and noted "we should ensure that the bootnodes are the new ones".

Context

PPN forks previewnet with bite + spawn and has been running it in CI. Three failures showed up
that a successful spawn does not catch, all of the same shape: the network comes up, produces
blocks, and looks healthy while being wrong. We ended up writing the checks in PPN
(spawner/src/fork/verify.ts), but they belong in zombie-bite — every consumer needs them, and
zombie-bite is the only layer that knows the bite block.

1. monit_progress tracks best block but not finality

src/monit.rs:22 reads block_height{status="best"} and restarts a node whose best block stops
advancing. Nothing reads finalized height.

A fork that produces blocks and finalizes none passes this monitoring indefinitely. We hit exactly
that: blocks advancing, finality frozen, every node "healthy" by best-block progress.

Worth knowing if you add this: parachain finality legitimately sits still for the first minute or so
after spawn, because a parachain block is only final once the relay block carrying its candidate is
finalized by GRANDPA. Measured on our fork, all five chains took 68–120s to take their first
finality step. A naive "finalized must advance within 30s" check fails a perfectly good fork — the
gate has to wait for the pipeline to prime first.

2. Nothing checks the fork diverged from the network it was bitten from

generate_chain_spec clears bootNodes (src/doppelganger.rs:875, and :165 for custom paras), and
that does work — every spec in our bundle has bootNodes: []. But nothing verifies the outcome, and
the failure mode is silent: a fork that reaches the source network follows its chain, which looks
healthy on every metric while not being a fork at all.

It's cheap to check, since bite knows the block it captured:

fork.chain_getBlockHash(biteBlock + 5) != source.chain_getBlockHash(biteBlock + 5)

Equal hashes mean the fork is following production. Our implementation is
checkDiverged in spawner/src/fork/verify.ts.

This matters more than "we already clear bootNodes" suggests, because consumers reintroduce them:
PPN runs a patch-bootnodes process that rewrites the published specs so external nodes can dial
in, and any such path can put a production address back.

3. Cleared bootNodes are safe but not usable

Related, and the reason patch-bootnodes exists at all. bootNodes: [] is the safe default, but a
spec published from a running fork with no bootNodes is unusable to anything off the box — the
peer wiring lives only in the spawned nodes' arguments. Every consumer that wants a dialable network
has to rewrite the spec itself.

Populating bootNodes with the spawned fork's own node addresses after spawn (or exposing a hook
that does) would remove that duplicated work, and is what we understood @pepoviola to mean by
"ensure the bootnodes are the new ones."

4. Also silent: parachains sharing one chain spec

Not a zombie-bite bug — chain is mandatory alongside chain_spec_path and we simply omitted it —
but the symptom is worth a validation error. Without chain, zombienet applies one spec to every
parachain, last one wins, and every collator silently runs the same chain. The only sign is the
parachains' heights drifting together; comparing genesis hashes makes it unambiguous.

Rejecting a config that supplies chain_spec_path without chain would have turned an afternoon
into a startup error.

Suggested scope

  • Add finalized-height progress to monit_progress, with a prime-wait before it starts asserting.
  • Add a post-spawn divergence assertion against the source, keyed on the bite block.
  • Populate bootNodes with the spawned network's own nodes, or expose a hook.
  • Validate chain_spec_path without chain.

Reference implementation, MIT/Apache and yours to lift verbatim:
spawner/src/fork/verify.ts in paritytech/preview-net-v1 (checkChains, waitForChains,
checkDistinctChains, checkDiverged), with tests in spawner/tests/fork-verify.test.ts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions