fix(agent): bound concurrent beacon state and bad block fetches

[Savid]

Problem

Beacon states and execution bad blocks are both read fully into memory (io.ReadAll) before being compressed and stored. In single mode one process runs an agent per node, and every agent reacts to the same block event, so nothing limited how many of those reads ran concurrently — peak memory scaled with the node count rather than with any configured bound.

On a glamsterdam devnet with ~100MB states, 47 agents sawtoothed between 2.3GB and 5.67GB against a 6GB limit and were OOM killed every 35–64s. Trimming the agent list only moved the deadline (20 agents lasted ~101s), which is what you'd expect if the peak is a function of concurrent fetches rather than of agent count directly.

Heap profile from the failing pod, 28s apart:

	t=0	t=14s	t=28s
total	354MB	558MB	2658MB
FetchRawBeaconState	—	—	1281MB (48%)
ethrpc.Provider.Do → GetBadBlocks	241MB	36MB	837MB (31%)
io.ReadAll (flat)	234MB	35MB	2118MB (80%)

Change

A process-wide semaphore for each path. The slot is held across the fetch, the compression and the upload, because both the raw and compressed copies are live for that whole window. For bad blocks it is held across the indexing pass, since the decoded slice is retained until that completes — releasing after the fetch would not bound anything.

Also drops the raw state reference once compressed, so only one copy survives the upload.

Two new config knobs, both defaulted to 10:

ethereum:
  maxConcurrentBeaconStateFetches: 10
  maxConcurrentExecutionBadBlockFetches: 10

Results

47 agents against a mock beacon node serving 100MB states, 6GB / 2 CPU, config otherwise unchanged:

	peak	vs limit	states indexed
before	5.67 GB	94%	—
limit 4	1.09 GB	18%	679
limit 10	1.44 GB	24%	752

5.2x lower peak with all 47 agents running, and the sawtooth is replaced by a narrow band with GC comfortably ahead.

Bounding concurrency also made state archival actually work — the production deployment this came from had persisted 20 states in 10 days, because the fetches were contending badly enough to die before persisting.

Notes for review

• The limits are process-wide and fixed by whichever agent starts first, since agents are constructed independently via agent.New. Threading a limiter through the single config would avoid the first-writer-wins wart at the cost of changing that signature — happy to do that instead if preferred.
• The beacon state result is measured. The bad block limiter is sound by construction but not independently measured — the mock's synthetic payload exercised the io.ReadAll and decode but produced no valid BadBlock records, so the indexing half of that loop did not run. Its limit of 10 is reasoned from the relative payload sizes (~18MB vs ~130MB per item) rather than benchmarked.
• The real fix for both is streaming rather than buffering, but FetchRawBeaconState returns []byte from ethpandaops/beacon with no streaming variant, so that is a two-repo change. This bounds the damage in the meantime.
• golang.org/x/sync was already a direct dependency; no go.mod change.

🤖 Generated with Claude Code