Skip to content

feat(el): default execution clients to discv5-only with ENR bootnodes - #1461

Draft
barnabasbusa wants to merge 4 commits into
mainfrom
bbusa/el-discv5-only-enr-bootnodes
Draft

feat(el): default execution clients to discv5-only with ENR bootnodes#1461
barnabasbusa wants to merge 4 commits into
mainfrom
bbusa/el-discv5-only-enr-bootnodes

Conversation

@barnabasbusa

@barnabasbusa barnabasbusa commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Draft — the named-devnet path needs el_enrs.txt, and mev.yaml is blocked on an upstream rbuilder rebuild. See Depends on and Known blockers.

Why

Discv5 bootstraps from ENRs, not enodes, and two clients make this mandatory rather than cosmetic:

  • besu discards its enode bootnode list entirely once discv5 is enabled and rejects enodes with Invalid ENR bootnode: ... must start with 'enr:'.
  • nimbus-eth1 now hard-disables discv4 (status-im/nimbus-eth1#4582, merged), so the enode list it receives feeds a protocol that no longer runs.

The other clients accept enr: in their bootnode flag too, so ENRs are a strict superset.

What

Discv4 off, discv5 on:

Client Flag
geth --discovery.v4=false --discovery.v5=true
erigon --discovery.v4=false --discovery.v5=true
ethrex --p2p.discv4=false --p2p.discv5=true
reth --disable-discv4-discovery
nethermind --Discovery.DiscoveryVersion=V5
besu --discovery-mode=V5
nimbus none needed — discv4 disabled upstream

Bootnodes now come from ENRs on both paths:

  • kurtosis / shadowfork — each peer's ctx.enr, which meant plumbing the ENR through besu, reth and nimbus (get_enode_enr_for_node); geth, erigon and ethrex already had it.
  • named devnets — new get_devnet_el_enrs() reading el_enrs.txt, mirroring get_devnet_enodes(). bootstrap_nodes.txt is CL beacon ENRs and stays reserved for the CL launchers.

Entries without an ENR are filtered out rather than emitted as empty strings, so a client that can't report one isn't used as a bootnode source instead of corrupting everyone's list.

ethereumjs is deliberately untouched: no discv5 implementation at all (config.ts has only discV4/discDns), so it keeps discv4 and enodes and will not peer with the discv5-only clients.

Default image bumps — all temporary

Every one of these was verified by running the image; three of seven ELs need builds newer than their current release:

image result
hyperledger/besu:latest (26.7.0) leaves --discovery-mode=V5 unmatched
ethpandaops/besu:main accepts it (and rejects the old --Xv5-discovery-enabled)
ghcr.io/paradigmxyz/reth invalid value 'enr:...': Failed to parse url: no host specified
ethpandaops/reth:main starts on enr: bootnodes — 2.4.1-dev (c505ed2)
nethermind/nethermind:latest exits 1: PublicKey should be 64 bytes long
ethpandaops/nethermind:master starts, logs Discv5 bootnodes accepted: 1

Enabling commits, none of which are in a release:

  • besu --discovery-mode#10800, 2026-07-22
  • reth enr: bootnodes via CLI — #26448, 2026-07-29

All three image pins should revert to the upstream images once these ship in releases.

Known blockers

  1. mev.yaml cannot pass yet. The builder runs ethpandaops/reth-rbuilder:develop, whose newest build is 2026-07-17 — 12 days before reth #26448. Verified directly: it rejects enr: bootnodes with Failed to parse url: no host specified. There is no rbuilder tag that works. It needs rebasing on reth ≥ c505ed2; there's no workaround in this package, since putting the builder back on discv4 + enodes would isolate it from the discv5-only mesh anyway.
  2. nethermind cannot report its own ENR. admin_nodeInfo has no enr field — BuildNodeInfo() sets Enode/Id/Ip/ListenAddress/Ports/Protocols only. Kurtosis' extractor errors hard on a missing field rather than returning empty, so nethermind stays on get_enode_for_node and is excluded as a bootnode source. It still peers fine using other nodes' ENRs, and shouldn't be made primary_bootnode. Fixing it upstream needs INodeRecordProvider moved into Nethermind.Network first — Nethermind.JsonRpcNethermind.Network.Discovery would be a circular project reference.

Depends on

ethpandaops/ansible-collection-general#568 adds el_enrs.txt. Until it lands and devnet metadata is regenerated, the named-devnet path has no ENR file to read. The kurtosis/shadowfork path works today.

Testing

kurtosis lint clean across src/.

7-EL enclave (geth, besu, erigon, reth, nethermind, nimbusel, ethrex) discv5-only with ENR bootnodes — all seven peered and began syncing:

el-1-geth        peers=6  block=8
el-2-besu        peers=1  block=6
el-3-erigon      peers=1  block=6
el-4-reth        peers=1  block=0
el-5-nethermind  peers=1  block=0
el-6-nimbus      peers=1  block=0
el-7-ethrex      peers=1  block=6

The same run on unmodified devnet-7 images had besu and reth both Exited (2). Nethermind takes ~2 min to find its first peer (slow discv5 lookup cadence) but does get there.

@barnabasbusa
barnabasbusa force-pushed the bbusa/el-discv5-only-enr-bootnodes branch 2 times, most recently from 81388da to e78d854 Compare July 30, 2026 08:40
@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

CI status

Green: lint, check-typos, title_check, assertoor, minimal, mix-persistence, mev-mock, network_params, .

The only genuine failure is mev.yaml, and it is not fixable in this repo. The builder runs ethpandaops/reth-rbuilder:develop, newest build 2026-07-17 — 12 days before reth #26448 landed. Verified directly:

$ docker run --rm ethpandaops/reth-rbuilder:develop node --bootnodes="enr:-Iu4Q..." --disable-discv4-discovery
error: invalid value 'enr:-Iu4Q...' for '--bootnodes <BOOTNODES>': Failed to parse url: no host specified

No rbuilder tag exists that parses ENRs. It needs rebasing on reth ≥ c505ed2. There's no workaround here either — putting the builder back on discv4 + enodes would isolate it from the discv5-only mesh.

mix-public and mix-with-tools show as failed but were cancelled, not broken: run_with_args has no fail-fast: false (that setting at line 61 belongs to check-consensus-spec-values), so mev.yaml's failure cancels its siblings. Both had already brought up every service before being killed — mix-public logs all 6 ELs and all 6 CLs added, nethermind included:

Service 'el-1-geth-teku' added        Service 'cl-1-teku-geth' added
Service 'el-2-nethermind-prysm' added Service 'cl-2-prysm-nethermind' added
Service 'el-3-erigon-nimbus' added    Service 'cl-3-nimbus-erigon' added
Service 'el-4-besu-lighthouse' added  Service 'cl-4-lighthouse-besu' added
Service 'el-5-reth-lodestar' added    Service 'cl-5-lodestar-reth' added
Service 'el-6-ethrex-grandine' added  Service 'cl-6-grandine-ethrex' added
operation was canceled

Earlier runs in this PR were also mostly self-inflicted: per-pr.yml sets cancel-in-progress: true keyed on the branch, so each successive push cancelled the previous run's jobs and they surfaced as failures.

@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

Made mev.yaml nightly-only rather than per-PR.

Rationale: the builder runs ethpandaops/reth-rbuilder, whose newest build (2026-07-17) predates reth #26448, so it rejects enr: bootnodes and cannot run discv5-only. Because run_with_args doesn't set fail-fast: false, that single leg was also cancelling mix-public and mix-with-tools — 3 of 8 legs red for one unbuildable image.

Coverage is not lost: nightly.yml globs .github/tests/*.yaml and runs with continue-on-error: true, so mev.yaml still executes there, just non-blocking. mev-mock.yaml stays in the per-PR matrix and passes.

Restore it here once ethpandaops/reth-rbuilder is rebased on reth ≥ c505ed2 — that rebuild is the real fix, and it's an ethpandaops-owned image (build-push-reth-rbuilder.yml, upstream flashbots/rbuilder).

Discv5 bootstraps from ENRs, not enodes. Besu discards its enode bootnode
list entirely once discv5 is enabled, and nimbus-eth1 now hard-disables
discv4 (status-im/nimbus-eth1#4582), leaving its enode list feeding a
protocol that no longer runs.

Switch the EL bootnode plumbing from enodes to ENRs and turn discv4 off:

  geth        --discovery.v4=false --discovery.v5=true
  erigon      --discovery.v4=false --discovery.v5=true
  ethrex      --p2p.discv4=false --p2p.discv5=true
  reth        --disable-discv4-discovery
  nethermind  --Discovery.DiscoveryVersion=V5
  besu        --discovery-mode=V5
  nimbus      no flag needed, discv4 is disabled upstream

Bootnodes come from each peer's ENR on the kurtosis/shadowfork path and from
el_enrs.txt on the named-devnet path. Entries without an ENR are skipped
rather than emitted as empty strings, so a client that cannot report one is
simply not used as a bootnode source. Nethermind is one such client: its
admin_nodeInfo has no enr field, and Kurtosis' extractor errors on a missing
field rather than returning empty.

ethereumjs is left on discv4 and enodes: it has no discv5 implementation.
… discv5-only

Verified by running each image:

  hyperledger/besu:latest      leaves --discovery-mode=V5 unmatched
  ghcr.io/paradigmxyz/reth     rejects enr: bootnodes with
                               'Failed to parse url: no host specified'
  nethermind/nethermind:latest exits 1 with
                               'PublicKey should be 64 bytes long'

besu --discovery-mode landed in #10800 (2026-07-22) and reth enr: bootnodes
via CLI in #26448 (2026-07-29); neither is in a release. Revert all three
pins to the upstream images once the changes ship.
The mev builder runs ethpandaops/reth-rbuilder, whose newest build predates
reth #26448, so it rejects enr: bootnodes and cannot run discv5-only. Because
run_with_args does not set fail-fast: false, that one leg also cancels
mix-public and mix-with-tools, taking out 3 of 8 legs.

nightly.yml globs .github/tests/*.yaml with continue-on-error: true, so
mev.yaml still runs there. Restore it here once reth-rbuilder is rebased on
reth >= c505ed2.
@barnabasbusa
barnabasbusa force-pushed the bbusa/el-discv5-only-enr-bootnodes branch from 11b1e81 to ebcba8e Compare July 30, 2026 09:07
reth's discv5 defaults to UDP 9200 (DEFAULT_DISCOVERY_V5_PORT), while
--discovery.port only sets the discv4 socket. The port spec declares
udp-discovery on the discovery port, so with discv4 disabled reth was
advertising an ENR pointing at an undeclared, unpublished port.

Container-to-container traffic inside an enclave is open so this still peered,
but with port_publisher enabled, or behind a real host firewall, reth's discv5
would be unreachable.

Confirmed via admin_nodeInfo on a discv5-only enclave: reth reported
ports.discovery 9200 while every other EL reported 30303.
@barnabasbusa

Copy link
Copy Markdown
Collaborator Author

Found a real bug while checking what admin_nodeInfo reports with discv4 off — reth's discv5 was on an unpublished port.

--discovery.port only sets reth's discv4 socket. Its discv5 listener defaults to DEFAULT_DISCOVERY_V5_PORT = 9200 (crates/net/discv5/src/config.rs:34), so with discv4 disabled reth was signing an ENR advertising UDP 9200 — a port the service's used_port_assignments never declares and port_publisher therefore never publishes.

Visible in admin_nodeInfo across a discv5-only enclave:

el-1-geth        ports: {"discovery": 30303, "listener": 30303}
el-2-besu        ports: {"discovery": 30303, "listener": 30303}
el-3-erigon      ports: {"discovery": 30303, "listener": 30303}
el-4-reth        ports: {"discovery": 9200,  "listener": 30303}   <-- undeclared
el-5-nethermind  ports: {"discovery": 30303, "listener": 30303}
el-6-nimbus      ports: {"discovery": 30303, "listener": 30303}
el-7-ethrex      ports: {"discovery": 30303, "listener": 30303}

It still peered in that run only because container-to-container traffic inside an enclave is unrestricted. With port_publisher on, or behind a host firewall, reth's discv5 would have been unreachable while looking healthy.

Fixed by passing --discovery.v5.port; reth now reports discovery: 30303 and still peers. Same fix applied to the reth role in ansible-collection-general#568, where reth_container_ports publishes only reth_ports_p2p and the firewall role would not have opened 9200 either.

Also worth noting from the same dump: enode stays fully populated with discv4 off on all seven clients. In geth it comes from the local node record (NodeInfo()srv.Self()node.URLv4()), and setupUDPListening()/SetFallbackUDP is gated on NoDiscovery, not DiscoveryV4 — so the record keeps its UDP port and you never get a discport=0 enode. Anything still consuming enode (ethereumjs) is unaffected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant