Implement the active-active mode in the REST validator client. by nalepae · Pull Request #17075 · OffchainLabs/prysm

nalepae · 2026-07-02T14:40:56Z

What type of PR is this?
Feature

Depends on:

saveHead: Clear the head cache when the head is updated. #17143

The active-passive connection scheme

Currently, a Prysm validator client can connect to multiple beacon nodes via REST, but only one node is actively used for SSE events and HTTP requests at any given time. This is called the active beacon node, while the others are passive.

If the active node fails or goes offline, the validator client automatically switches to one of the passive nodes, which then becomes active. This is the active-passive connection scheme.

While this approach generally works well, it has limitations in certain situations. Consider a validator client connected to BN-1, BN-2, and BN-3, with only BN-1 active. If BN-1 experiences issues (such as poor peering or slow block execution due to disk constraints) and fails to import a block within the 4-second deadline, while BN-2 and BN-3 successfully import it, the validator client faces a problem.

Because the validator client only listens to BN-1's SSE events, it will not receive the head event for the new block before the deadline. Consequently, when it requests attestation data from BN-1, it receives stale data corresponding to the previous block. The validator client signs and broadcasts this attestation through BN-1, which means the validator misses the head vote, and potentially the source and target votes if this is the first block of an epoch.

This situation could have been avoided if the validator client could listen to SSE events from all connected beacon nodes (not just the active one) and request fresh attestation data from any of them (not just the active one).

Here enters the active-active connection scheme.

The active-active connection scheme

With the active-active connection scheme, a validator client actively listens to and requests data from all connected beacon nodes. When responses differ between beacon nodes, the validator client selects the best one based on the context.

This pull request implements the active-active connection scheme.

To implement this connection scheme, we introduce four new concepts:

The multi-event stream
The validator client needs to listen to SSE events from multiple beacon nodes. The multi-event stream:

connects to (and reconnects to, if needed) each beacon node's event stream,
merges these streams into a single stream,
removes potential duplicates, and
sends the result to an output stream.

host-1 --\
host-2 ---+--> merged --> deduper --> out
host-3 --/

The multi-handler
The multi-handler is a framework that defines how HTTP requests are sent to connected beacon nodes and which response is selected. It provides the following options:

WithRace: Should requests be sent sequentially or concurrently to the beacon nodes?
WithAccept/WithSSZAccept: Should all 2xx responses from any beacon node be accepted, or should additional acceptance criteria be applied?
WithDeadline: How long should the validator client wait if no acceptable response is returned?
WithRepoll: Should the validator client retry if no acceptable response is returned?

WithRepoll takes a re-poll mode that decides what stops the re-polling:

UntilAccepted: keep re-polling until an accept-passing (fresh) response arrives, or the deadline fires.
UntilAny2xx: re-poll only while no usable response at all has arrived, stopping as soon as any node returns one.

The freshness options
Now that we've implemented the multi-handler, we need to define how to use it (which options to set) for the 4 objects that require it.

Function	Endpoint	Object requested
`attestationFreshnessOptions`	/eth/v1/validator/attestation_data	`AttestationData`
`syncCommitteeFreshnessOptions`	/eth/v1/beacon/blocks/head/root	`BlockRootResponse`
`blockFreshnessOptions`	/eth/v3/validator/blocks/{slot} (or /eth/v4/validator/blocks/{slot} for Gloas)	`BeaconBlock`
`payloadAttestationFreshnessOptions`	/eth/v1/validator/payload_attestation_data	`PayloadAttestationData`

attestationFreshnessOptions, syncCommitteeFreshnessOptions and payloadAttestationFreshnessOptions use WithRace, an accept criterion, WithDeadline, and WithRepoll(UntilAccepted).

WithRace queries all beacon nodes concurrently.
The accept criterion (WithAccept for the JSON reads, WithSSZAccept for the SSZ payload-attestation read) prioritizes any response whose root matches the expected head root. If the deadline is reached without a matching response, the validator client falls back to the first response received. (Voting for a stale block is preferable to not voting at all.)
WithDeadline works together with the accept criterion to decide when to use the fallback response. It is floored to readFreshnessBudget so that a lagging node still gets time to import the announced head.
WithRepoll(UntilAccepted) continuously re-queries all beacon nodes until either a matching response is received or the deadline is reached.

blockFreshnessOptions uses WithRace, WithSSZAccept, WithRepoll(UntilAny2xx), and WithDeadline.

WithRace queries all beacon nodes concurrently.
WithSSZAccept prioritizes the block that is built on top of the expected head, i.e. the block whose parent root matches the expected head root.
WithRepoll(UntilAny2xx) keeps re-polling until at least one node returns a block (any 2xx).
WithDeadline is the caller's context deadline (the slot deadline).

The head tracker
The freshness options above all steer requests toward the node that already imported the expected head. Something has to decide what that expected head is. This is the head tracker.

The head tracker records the latest head (block root and its slot) the validator client has learned about from the head events of all connected beacon nodes (fed by the multi-event stream). On each head event, it keeps the head with the highest slot and ignores any event for an older slot. So the "expected head" is the most advanced head announced by any single beacon node.

When the validator client is about to attest, participate in a sync committee, propose a block or attest to a payload, it attaches the tracked head (the expected head root, plus a deadline) to the request context as a freshness hint. The freshness options then read that hint to build their accept criteria.

This is what closes the loop and solves the problem described at the top: even if a node is lagging, a head event from any other node advances the head tracker, so the validator client knows which head to expect and can request fresh data from whichever node already imported it.

head events (all nodes) --> head tracker (best head) --> hint on ctx --> freshness options

To test

Use a validator client with the --enable-beacon-rest-api flag and more than one beacon node:

--beacon-rest-api-provider=http://beacon-1,http://beacon-2,http://beacon-3

Test by powering down all beacon nodes except 1 (regardless the chosen beacon node is).

Also, the interesting case is the following: If, amongst all the connected beacon node, at least one imported the block before the 4 seconds deadline, then all the votes (head, source, target) should be correct.

Other notes for review

Note

Please read commit by commit, with commit messages.
Only the REST validator client is impacted. No change on gRPC.

Important

As the --enable-beacon-rest-api flag is still in the experimental mode, we chose to replace the current active-passive connection scheme by the new proposed active-active connection scheme.

Acknowledgements

I have read CONTRIBUTING.md.
I have included a uniquely named changelog fragment file.
I have added a description with sufficient context for reviewers to understand this PR.
I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

…am shutdown. Reused across runner restarts, so closing it panicked the next StartEventStream; also return early in ProcessEvent on empty events.

No functional change

This is useful when we are actively listening to multiple beacon nodes.

A multi event stream listens to multiple beacon nodes, merges the events, dedups them if needed and outputs all deduped events in a single channel. host-1 --\ host-2 ---+--> merged --> deduper --> out host-3 --/

james-prysm · 2026-07-20T14:50:20Z

@@ -0,0 +1,6 @@
+### Changed
+
+- Implement the active-active validator client: Using the `--enable-beacon-rest-api` flag,


I think we should mention removal of active-passive in favor of active active

Addressed in 87f8019.

james-prysm · 2026-07-20T14:50:48Z

 }

+// DecodeHex32 takes a string and validates whether the string is a hex and has the correct length of 32 bytes.
+func DecodeHex32(s string) ([32]byte, error) {


surprised we didn't have something like this already

We have +/-, but each in private packages. This function in public and generic, so everybody can use it.

james-prysm · 2026-07-20T19:02:28Z

+// trim shortens a string (e.g. a hex-encoded root like "0x9927a089f167...") to
+// its first 14 characters.
+func trim(s string) string {
+	const maxLen = 14 // "0x" + 12 hex characters (6 bytes).


why is it hard coded to first 14 character? was that what it was before? maybe missed it

For log purposes, to avoid very long log lines.
This is a new function.

james-prysm · 2026-07-20T19:04:22Z

 )

+// sinceSlotStartTime returns the elapsed time between the start of the provided slot and now.
+func (v *validator) sinceSlotStartTime(slot primitives.Slot) (time.Duration, error) {


why not make this function something in the slots package?

wait i think we have one already, slots.SinceSlotStart

Not really:
This function takes only one parameter: slot. (genesisTime is a field of validator.)
slots.SinceSlotStart needs also genesis.

However, it's true that this function can use slots.SinceSlotStart instead of recomputing

sinceSlotStartTime := prysmTime.Now().Sub(startTime)

Done in 3922f43.

james-prysm · 2026-07-20T19:56:54Z

 		newBalance := float64(balAfterEpoch) / gweiPerEth
-		prevBalance := float64(balBeforeEpoch) / gweiPerEth
-		startBalance := float64(v.startBalances[pubKeyBytes]) / gweiPerEth
-		percentNet := (newBalance - prevBalance) / prevBalance


why the changes here for summary?

For readability purposes.
Now we add the slot field, this line started to be very long.

Now:

[2026-07-29 09:13:47.76] INFO client: Previous epoch voting summary balanceEth=32.003817448 correctlyVotedHead=true correctlyVotedSource=true correctlyVotedTarget=true diffGwei=8484 epoch=464822 pubkey=0x... slot=14874324

james-prysm · 2026-07-22T19:14:11Z

-
+// ConnectionCounter always returns 0: the active-active handler queries every
+// configured host rather than switching between them.
 func (p *restConnectionProvider) ConnectionCounter() uint64 {


is this still useful?

Addressed in 66695c0.

james-prysm · 2026-07-22T19:14:39Z

-	// switch, distinguishing a host0 → host1 → host0 bounce from no change.
+	// switch. This always stays 0: it exists to satisfy the ValidatorClient
+	// connection-tracking API.
 	ConnectionCounter() uint64


why not remove this?

Addressed in 66695c0.

syjn99

Some random questions and comments:

1. We won't use the concept of fallback for REST VC - so methods like EnsureReady can be safely deleted after we deprecate gRPC VC, right?
1. I'd suggest to advertise this feature as loud as we can (e.g., use our marketing channels, provide comprehensive guides in our docs). What's the rollout plan, and when are we gonna feel "safe" to recommend this flag to our users? IMO this feature should work for happy paths but I'd like to see some chaotic cases.
- Maybe we can enable this in glamsterdam-devnet-*, I think it's going to be a good stress test.
1. Not a scope of this PR, but may we add a custom endpoint in VC that registers/deletes new REST providers so that NOs don't have to restart VC process? I didn't think of security issues so far, but this looks good for UX as I remember some NOs complain about their reluctance of restarting the process.

syjn99 · 2026-07-23T04:33:33Z

+	h.mu.Lock()
+	defer h.mu.Unlock()
+
+	if h.set && slot < h.slot {


What if reorg happens and this update rejects a new head event that is lower than the pinned head slot?

It's safe thanks to the fallback.
I added a godoc in f78f556 for the future reader.

syjn99 · 2026-07-23T04:55:14Z

+		return vals[0].body, vals[0].header, nil
+	}
+
+	return nil, nil, errors.Join(errs...)


This can return nil, nil, nil when errs is empty. This might happen when ctx is cancelled fast during broadcastWriteAll. Regression test:

t.Run("returns the context error when canceled before any node accepts", func(t *testing.T) { release := make(chan struct{}) srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { <-release })) t.Cleanup(srv.Close) t.Cleanup(func() { close(release) }) ctx, cancel := context.WithCancel(context.Background()) cancel() mh := multi(t, srv.URL, srv.URL) _, _, err := mh.PostSSZ(ctx, "/publish", nil, bytes.NewBufferString(body)) assert.Equal(t, true, errors.Is(err, context.Canceled)) })

So can we check ctx.Err() as well before this?

Good catch. Addressed in 6585681.

syjn99 · 2026-07-23T05:12:37Z

+
+		// If r.val satisfies accept, return it immediately.
+		if accept(r.val) {
+			return r.val, true, true, errs


Let's examine this loop for blockFreshnessOptions. I'm trying to think of edge cases with Codex:

accept callback = checking whether the new block's parent root is matched with the current head root.

VC will keep polling UntilAny2xx.

Suppose we have BN-1 and BN-2. BN-1 returns a block (B1) but it isn't accepted, so in this loop B1 is saved as a fallback response. (BN-1 can be lagged for example.)

BN-2 hangs, so it is not responding. Note that health check (IsReady) passes because BN-1 directly responds with 200.

As the line r := <-results is blocked, this loop would be finished at the deadline.

As a result, although VC could have proposed B1, it fails to propose because of BN-2. B1 might be orphaned if proposed, but anyway VC should propose it (shound't it?) as B1 is the best-effort choice for the VC.

Is this feasible? I assume active-active design should not be affected bt one faulty BN.

Nice catch. Addressed in b9ecd80.

syjn99 · 2026-07-23T05:21:23Z

+const readFreshnessBudget = 500 * time.Millisecond // Floor for a read's deadline.
+
+var (
+	attestationRootExtractor = rootExtractor("beacon_block_root")


Only checking beacon_block_root can make post-Gloas attestation non-determinsitic as index now signals FULL or EMPTY payload status. I'm not sure which attestation data is preferrable here - maybe we can leverage the fact that head_v2 event has payload_status field?

Nice catch. Addressed in 4f7215d.

syjn99 · 2026-07-23T06:21:19Z

+		return nil, nil
+	}
+
+	// non-2XX codes are a failure.


How do we handle 2XX response that deliberately indicates to do nothing like GET /eth/v1/validator/payload_attestation_data? It returns 204 No Content when no block has been seen so that VC doesn't have to do anything. My guess is this can be filtered with hint, is it right?

A 204 is a 2xx so it's a usable response, but it fails the accept criterion (no root to extract), so it only ever becomes the fallback. A node that actually has data wins the race.

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

…ead. A validator asking for attestation data right after the beacon node imported the block of the current slot could be served data produced earlier in the same slot, from the previous head, and vote for the parent of the head - losing its head vote. #17143 attacked this from the writer side, by evicting the cache when the head changes (`go c.Clear()` in `saveHead`). That turned a guaranteed stale vote into a rare one, but it cannot be made complete: 1. The eviction is not ordered against the notifications that wake validators. Both leave the same `saveHead` call, and the eviction has to run in a goroutine: clearing inline takes the attestation cache lock under the forkchoice lock. So a validator can be told about the new head and still be served the old entry. 2. Even perfectly ordered, an eviction cannot help. Because the cache write lock is held across the whole computation, and `TargetRootForEpoch` inside it waits on the forkchoice lock that `ReceiveBlock` holds for the entire import, a request that read the previous head writes its entry AFTER the head changed - after any eviction point. The cache is re-staled by a writer that no invalidation can reach. Check freshness when reading instead. An entry records the head it was produced from (`HeadRoot`, plus `HeadFull`), and is served only if that matches the current head, read from `HeadRootAndFull`, which takes only the head lock, so the fast path cannot block behind a block import. A mismatch is treated as a miss and the data is recomputed.

nalepae force-pushed the active-active branch 6 times, most recently from ca78cf8 to 9375332 Compare July 7, 2026 14:25

nalepae force-pushed the active-active branch 9 times, most recently from 9fbdce1 to b5ac4fc Compare July 16, 2026 14:22

nalepae changed the title ~~Active active~~ Implement the "active-active" mode in the REST validator client. Jul 16, 2026

nalepae force-pushed the active-active branch from 7f45ee2 to 1916a4d Compare July 16, 2026 16:27

nalepae and others added 4 commits July 17, 2026 09:44

ProcessEvent: Don't close the owned by caller event channel on stre…

a2acdf3

…am shutdown. Reused across runner restarts, so closing it panicked the next StartEventStream; also return early in ProcessEvent on empty events.

Event.EventType ==> Event.Type.

639f6f5

No functional change

Move some tests.

10f228e

Improve the Received head event log.

a936ee5

nalepae force-pushed the active-active branch from 1916a4d to 7a92c76 Compare July 17, 2026 07:45

nalepae added 2 commits July 17, 2026 09:48

Listening to Beacon API events log: Add beacon node URL.

0b161f1

This is useful when we are actively listening to multiple beacon nodes.

Implement multi event stream.

e38b827

A multi event stream listens to multiple beacon nodes, merges the events, dedups them if needed and outputs all deduped events in a single channel. host-1 --\ host-2 ---+--> merged --> deduper --> out host-3 --/

nalepae force-pushed the active-active branch from 7a92c76 to fc7f072 Compare July 17, 2026 09:30

nalepae changed the title ~~Implement the "active-active" mode in the REST validator client.~~ Implement the active-active mode in the REST validator client. Jul 17, 2026

nalepae force-pushed the active-active branch 4 times, most recently from 25aa36d to 79f8112 Compare July 17, 2026 21:19

james-prysm reviewed Jul 20, 2026

View reviewed changes

syjn99 reviewed Jul 21, 2026

View reviewed changes

Comment thread validator/client/validator.go Outdated

syjn99 self-requested a review July 22, 2026 13:42

james-prysm reviewed Jul 22, 2026

View reviewed changes

syjn99 reviewed Jul 23, 2026

View reviewed changes

nalepae and others added 15 commits July 27, 2026 23:07

Merge branch 'develop' into active-active

9d9a846

Address @james-prysm's comment

87f8019

Address @james-prysm's comment.

3922f43

Address @syjn99's comment

c361f55

Address @james-prysm's comment.

66695c0

Update api/rest/multi_handler.go

c68cc97

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

Update validator/client/beacon-api/freshness_test.go

d43856a

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

Address @syjn99's comment.

c4dd668

Update validator/client/payload_availability.go

bfa2de6

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

Address @syjn99's comment.

42a7785

Update api/client/event/event_stream.go

69d4a3f

Co-authored-by: Jun Song <87601811+syjn99@users.noreply.github.com>

Address @syjn99's comment.

0ddb028

Address @syjn99's comment.

f78f556

Address @syjn99's comment.

6585681

Address @syjn99's comment.

b9ecd80

syjn99 self-requested a review July 30, 2026 05:56

Address @syjn99's comment.

4f7215d

nalepae force-pushed the active-active branch from b4fff37 to 4f7215d Compare July 30, 2026 10:30

nalepae added 2 commits July 30, 2026 16:46

Previous epoch voting summary log: Align everything.

de0ed3a

		@@ -0,0 +1,6 @@
		### Changed

		- Implement the active-active validator client: Using the `--enable-beacon-rest-api` flag,

Uh oh!

Conversation

nalepae commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The active-passive connection scheme

The active-active connection scheme

To test

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nalepae Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syjn99 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nalepae Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nalepae Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

nalepae commented Jul 2, 2026 •

edited

Loading

nalepae Jul 29, 2026 •

edited

Loading

nalepae Jul 29, 2026 •

edited

Loading

nalepae Jul 30, 2026 •

edited

Loading