Degrade Gloas envelope-by-range serving and validation gracefully instead of poisoning peer scores - #17274
Conversation
|
I don't understand this:
The only example other than an EL bug is that we no longer hold the older data, but this by definition will not be therefore a prefix of the requested range. |
|
You're right for that case — pruned older data leaves a suffix, so the prefix fallback finds nothing and the request degrades to The prefix case is a different failure we also hit in production: an EL behind its CL (rewound after a crash, or still syncing behind a checkpoint-synced beacon). There reconstruction fails for blocks newer than the EL head, so what's servable is exactly a prefix of the range. So: EL behind → serve the prefix it has; data pruned → |
…tead of poisoning peer scores Serving execution_payload_envelopes_by_range requires reconstructing full envelopes from execution-client data that may legitimately be gone: blinded envelopes are stored without transactions/withdrawals/BAL, and engine_getPayloadBodiesByHashV2 may return null blockAccessList once the EL prunes it (or null bodies entirely while the EL is syncing). Since OffchainLabs#17174 any such gap fails the whole request with a server error, which requesting peers score as a bad response; after a handful of requests every peer disconnects and the serving node is isolated - exactly when it needs peers. Observed on glamsterdam-devnet-7 prysm-geth-1: "block access list unavailable" against a healthy geth, then "missing required field 'parentHash'" against a rewound one; a Lighthouse peer disconnected us with "bad responses scorer: peer exceeded bad responses threshold: got 5". Serve side: on batch reconstruction failure, retry per payload and serve the longest contiguous prefix; if nothing is reconstructable, respond with resource-unavailable (already used for out-of-retention ranges) instead of a server error. Request side: a mid-response parent-hash discontinuity is expected when the serving peer reorgs mid-response during non-finality and is not proof of a faulty peer. Keep the contiguous prefix instead of discarding the response with ErrInvalidFetchedData and downscoring the peer. All other validations (slot range, monotonic slots, count, SSZ) remain fatal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
9ff9671 to
2dd78f3
Compare
|
Done — removed the per-payload prefix fallback. When the EL fails to reconstruct (batch error or a missing payload), the request now answers |
2 participants
What
Two hardening fixes to
ExecutionPayloadEnvelopesByRange, one on each side of the wire:streamCanonicalEnvelopes): payload reconstruction legitimately fails when the execution client no longer holds the required data (pruned block access lists, EL syncing/rewound). Today that returns a generic server error for the whole request. Now the server falls back to per-payload reconstruction and serves the longest contiguous prefix, answeringResourceUnavailableonly when nothing in the range is reconstructable.SendExecutionPayloadEnvelopesByRangeRequest): a mid-response parent-hash discontinuity (the serving peer reorged mid-stream during non-finality) is currently treated asErrInvalidFetchedData. Now the requester keeps the already-validated contiguous prefix instead of discarding the whole response.Why
Both paths punish honest peers with bad-responder downscoring, and five strikes means a goodbye/ban. In production on glamsterdam-devnet-7 this was self-reinforcing isolation at the worst possible moment: nodes serving envelope ranges from a degraded EL got banned by every requester, and syncing requesters banned their own last remaining sync sources ("bad responses scorer ... got 5, threshold 5") — a direct contributor to permanent initial-sync wedges observed on two production nodes. An envelope range a peer cannot fully serve, or a response that stops chaining because of a reorg, is a resource/availability condition, not evidence of a faulty peer; scoring it as malice removes exactly the peers a recovering node needs.
Testing
Sync-package envelope/by-range tests green, including 3 new regression subtests (server prefix-degradation, empty-range
ResourceUnavailable, requester contiguous-prefix) that fail without this fix.🤖 Generated with Claude Code
https://claude.ai/code/session_01F6buHMiS3KiNHcnpUx32FU