Skip to content

chore: stabilize fastConfirmation perf bench - #9494

Merged
nazarhussain merged 3 commits into
unstablefrom
nh/fcr-benchmark-fix
Jun 10, 2026
Merged

chore: stabilize fastConfirmation perf bench#9494
nazarhussain merged 3 commits into
unstablefrom
nh/fcr-benchmark-fix

Conversation

@nazarhussain

@nazarhussain nazarhussain commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Motivation

fastConfirmation.test.ts perf bench was flaky on CI.

Summary

The per-iteration everyoneVotes(...) (100K–1M validator loop) and updateHead() in beforeEach created GC pressure inside the µs-scale measurement window. The "flip votes" logic was also a silent no-op — addLatestMessage rejects same-epoch votes. Since runFastConfirmationRules doesn't mutate store, beforeEach is now a pass-through.

Locally: ~80K–160K samples/config (was 25–424), ~0.3–5s wall time/config (was 2–17s).

Follow-ups (separate PR)

  • Rename to runFastConfirmationRules.test.ts to match the bench id.
  • Add a state-backed bench that exercises findLatestConfirmedDescendant end-to-end using generatePerfTestCachedStateElectra from @lodestar/state-transition/test-utils.

Test plan

  • pnpm benchmark:files 'packages/fork-choice/test/perf/forkChoice/fastConfirmation.test.ts' — stable across 3 consecutive runs.
  • check-types + lint clean.
  • CI bench job.

🤖 AI-assisted with Claude Code (Opus 4.7).

The per-iteration `everyoneVotes(...)` (100K–1M validator loop) and
`updateHead()` in `beforeEach` were generating GC pressure right inside
the measurement window, which on CI inflated the µs-scale fn timings
enough to trip the regression threshold. The "flip votes" logic also
never actually flipped — same-epoch votes are rejected by
`addLatestMessage`, so the second `everyoneVotes` call was a no-op.

Replace `beforeEach` with a pass-through. `runFastConfirmationRules`
doesn't mutate `store`, so each iteration replays the same work and no
per-iteration reset is required.

Bench is now stable across runs: ~80K–160K samples per config (was
25–424), wall time per config ~0.3–5s (was 2–17s), and the vc/eq
variations no longer move the number — which is the correct behavior,
since neither parameter enters the executed code path; their old
apparent effect was the GC pressure from the removed beforeEach work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nazarhussain
nazarhussain requested a review from a team as a code owner June 9, 2026 15:54

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request simplifies the Fast Confirmation Rules (FCR) benchmark by removing the per-iteration vote-flipping and slot-advancing logic in the beforeEach hook, as the rule runner does not mutate the store. The review feedback identifies a correctness bug in the everyoneVotes helper function where an Epoch is incorrectly passed instead of a Slot to addLatestMessage, causing votes to be registered in epoch 0 and treated as expired. Additionally, the pass-through beforeEach hook is redundant and can be safely removed because the benchmark runner defaults to passing the data from before to fn when beforeEach is omitted.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

if (!prevBlock) throw Error("no prevBlock");

// Vote everyone for head
// Vote everyone for head so the FCR vote-map paths see a populated voteNextIndices.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is a correctness bug in the helper function everyoneVotes (defined at the bottom of this file) that is called here.\n\nIn everyoneVotes, nextEpoch is passed as the second argument to forkChoice["addLatestMessage"]:\ntypescript\nfunction everyoneVotes(vote: ProtoBlock, forkChoice: ForkChoice): void {\n const nextEpoch = computeEpochAtSlot(vote.slot);\n const nextRoot = vote.blockRoot;\n for (let i = 0; i < forkChoice["balances"].length; i++) {\n forkChoice["addLatestMessage"](i, nextEpoch, nextRoot, PayloadStatus.FULL);\n }\n}\n\n\nHowever, addLatestMessage expects a Slot as its second parameter, not an Epoch:\ntypescript\n private addLatestMessage(\n validatorIndex: ValidatorIndex,\n nextSlot: Slot,\n nextRoot: RootHex,\n nextPayloadStatus: PayloadStatus\n ): void\n\n\nBecause an epoch number (e.g., 2) is passed instead of a slot number (e.g., 64), addLatestMessage internally computes the epoch of the vote as computeEpochAtSlot(2), which evaluates to 0.\n\nThis causes all validator votes to be registered as belonging to epoch 0 (instead of epoch 2). Since the benchmark advances the slot to epoch 3, these votes are treated as expired/inactive by the Fast Confirmation Rules (FCR), meaning the benchmark is not actually executing the realistic code paths where active votes are processed and counted.\n\n### Suggested Fix\nUpdate everyoneVotes to pass vote.slot instead of nextEpoch:\ntypescript\nfunction everyoneVotes(vote: ProtoBlock, forkChoice: ForkChoice): void {\n const nextSlot = vote.slot;\n const nextRoot = vote.blockRoot;\n for (let i = 0; i < forkChoice["balances"].length; i++) {\n forkChoice["addLatestMessage"](i, nextSlot, nextRoot, PayloadStatus.FULL);\n }\n}\n

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirming Gemini's analysis: addLatestMessage signature is (validatorIndex, nextSlot, ...) and internally calls computeEpochAtSlot(nextSlot) to determine vote epoch. With vc:600000 bc:96, the head's vote.slot=96nextEpoch=3 is passed as nextSlot=3, so voteNextSlots[i]=3 and the internal epoch resolves to computeEpochAtSlot(3)=0. After updateTime(97) (epoch 3), all votes look like they belong to epoch 0 — i.e. stale by 3 epochs. The realistic path through the FCR vote-map paths is not being exercised.

Fix is one line as Gemini suggested:

forkChoice["addLatestMessage"](i, vote.slot, nextRoot, PayloadStatus.FULL);

cc @gemini-code-assist — good catch. @nazarhussain — the GC-pressure removal still delivers the flakiness fix on its own, but this means the benchmarked code path isn't faithfully representing real validator state. Worth folding into this PR before merge so the bench numbers actually reflect FCR-active votes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 99872b5. vote.slot is now passed directly; the now-unused computeEpochAtSlot import was dropped. Verified via local bench run (still stable, ~80K–135K samples/config).

Note: this is independent of the GC-pressure flakiness fix in the first commit — fn() at currentSlot=97 short-circuits at Rule 4 (confirmedEpoch+1 < currentEpoch after Rule 2 resets to genesis) so it never reaches the vote-map path either way. But fixing the bug now means the future state-backed bench (planned follow-up) will start from a correct setup.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Confirmed — 99872b50ca matches what Gemini and I flagged: vote.slot is passed straight to addLatestMessage and the now-unused computeEpochAtSlot import is gone. Thanks for closing it out.

Comment on lines 62 to +66
},
// Pass-through: the bench harness invokes fn(beforeEach(inputAll)), so without a
// beforeEach fn would receive undefined. store is not mutated by the rule runner so
// no per-iteration reset is required.
beforeEach: (data) => data,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The beforeEach pass-through and its accompanying comment are redundant.\n\nIn @chainsafe/benchmark, if beforeEach is omitted, the benchmark runner automatically defaults to passing the data returned by before directly to fn. We can safely remove this block to simplify the benchmark setup.

Suggested change
},
// Pass-through: the bench harness invokes fn(beforeEach(inputAll)), so without a
// beforeEach fn would receive undefined. store is not mutated by the rule runner so
// no per-iteration reset is required.
beforeEach: (data) => data,
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing back on this one — checked @chainsafe/benchmark's runBenchmarkFn.js:

const input = beforeEach ? await beforeEach(inputAll, runIdx) : undefined;

When beforeEach is omitted, fn receives undefined — not the data from before. With the pass-through removed, fn: ({ctx, store}) => ... would destructure undefined and throw on every iteration.

The comment is there precisely because the pass-through looks redundant at a glance but isn't. Suggestion to remove it would break the benchmark. cc @gemini-code-assist.

@nazarhussain — leaving this one as-is, comment is accurate.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the clarification, @lodekeeper. You are absolutely correct; my previous assessment regarding the redundancy of the beforeEach pass-through was based on an incorrect assumption about the benchmark runner's default behavior when beforeEach is omitted. I appreciate you pointing out that fn would receive undefined in that case, which would indeed cause the benchmark to fail. The current implementation is necessary for the benchmark to function correctly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @lodekeeper — keeping the pass-through. Confirmed in @chainsafe/benchmark's runBenchmarkFn.js (const input = beforeEach ? await beforeEach(inputAll, runIdx) : undefined): without beforeEach, fn would receive undefined and the destructure on {ctx, store} would throw on every iteration. The comment block above the function is there precisely to flag this for the next reader.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for confirming. Agreed — the beforeEach pass-through is load-bearing: without it @chainsafe/benchmark hands fn undefined and the {ctx, store} destructure throws. Good to keep.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: b41507a Previous: b00d95d Ratio
Array.fill - length 1000000 7.6802 ms/op 2.3832 ms/op 3.22
Full benchmark results
Benchmark suite Current: b41507a Previous: b00d95d Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 986.73 us/op 991.33 us/op 1.00
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 40.964 us/op 39.086 us/op 1.05
BLS verify - blst 732.39 us/op 718.67 us/op 1.02
BLS verifyMultipleSignatures 3 - blst 1.3854 ms/op 1.2966 ms/op 1.07
BLS verifyMultipleSignatures 8 - blst 2.2562 ms/op 2.0583 ms/op 1.10
BLS verifyMultipleSignatures 32 - blst 7.0173 ms/op 6.5330 ms/op 1.07
BLS verifyMultipleSignatures 64 - blst 13.496 ms/op 12.728 ms/op 1.06
BLS verifyMultipleSignatures 128 - blst 25.948 ms/op 24.933 ms/op 1.04
BLS deserializing 10000 signatures 630.22 ms/op 617.27 ms/op 1.02
BLS deserializing 100000 signatures 6.4115 s/op 6.2171 s/op 1.03
BLS verifyMultipleSignatures - same message - 3 - blst 711.69 us/op 763.76 us/op 0.93
BLS verifyMultipleSignatures - same message - 8 - blst 841.27 us/op 871.80 us/op 0.96
BLS verifyMultipleSignatures - same message - 32 - blst 1.5787 ms/op 1.5161 ms/op 1.04
BLS verifyMultipleSignatures - same message - 64 - blst 2.4218 ms/op 2.3563 ms/op 1.03
BLS verifyMultipleSignatures - same message - 128 - blst 4.0499 ms/op 3.9695 ms/op 1.02
BLS aggregatePubkeys 32 - blst 17.626 us/op 17.318 us/op 1.02
BLS aggregatePubkeys 128 - blst 63.235 us/op 61.503 us/op 1.03
getSlashingsAndExits - default max 50.526 us/op 54.141 us/op 0.93
getSlashingsAndExits - 2k 345.70 us/op 410.85 us/op 0.84
proposeBlockBody type=full, size=empty 638.61 us/op 1.6291 ms/op 0.39
isKnown best case - 1 super set check 171.00 ns/op 164.00 ns/op 1.04
isKnown normal case - 2 super set checks 167.00 ns/op 160.00 ns/op 1.04
isKnown worse case - 16 super set checks 166.00 ns/op 160.00 ns/op 1.04
validate api signedAggregateAndProof - struct 1.5685 ms/op 1.4784 ms/op 1.06
validate gossip signedAggregateAndProof - struct 1.5582 ms/op 1.4630 ms/op 1.07
batch validate gossip attestation - vc 640000 - chunk 32 110.71 us/op 109.15 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 64 98.124 us/op 94.521 us/op 1.04
batch validate gossip attestation - vc 640000 - chunk 128 88.996 us/op 88.802 us/op 1.00
batch validate gossip attestation - vc 640000 - chunk 256 88.308 us/op 87.446 us/op 1.01
bytes32 toHexString 296.00 ns/op 280.00 ns/op 1.06
bytes32 Buffer.toString(hex) 163.00 ns/op 172.00 ns/op 0.95
bytes32 Buffer.toString(hex) from Uint8Array 237.00 ns/op 230.00 ns/op 1.03
bytes32 Buffer.toString(hex) + 0x 165.00 ns/op 169.00 ns/op 0.98
Return object 10000 times 0.21870 ns/op 0.20940 ns/op 1.04
Throw Error 10000 times 3.3656 us/op 3.2741 us/op 1.03
toHex 90.051 ns/op 87.330 ns/op 1.03
Buffer.from 81.319 ns/op 78.893 ns/op 1.03
shared Buffer 55.228 ns/op 51.680 ns/op 1.07
fastMsgIdFn sha256 / 200 bytes 1.4900 us/op 1.4510 us/op 1.03
fastMsgIdFn h32 xxhash / 200 bytes 154.00 ns/op 152.00 ns/op 1.01
fastMsgIdFn h64 xxhash / 200 bytes 204.00 ns/op 205.00 ns/op 1.00
fastMsgIdFn sha256 / 1000 bytes 4.8350 us/op 4.6410 us/op 1.04
fastMsgIdFn h32 xxhash / 1000 bytes 251.00 ns/op 243.00 ns/op 1.03
fastMsgIdFn h64 xxhash / 1000 bytes 257.00 ns/op 247.00 ns/op 1.04
fastMsgIdFn sha256 / 10000 bytes 42.676 us/op 41.387 us/op 1.03
fastMsgIdFn h32 xxhash / 10000 bytes 1.2930 us/op 1.2730 us/op 1.02
fastMsgIdFn h64 xxhash / 10000 bytes 822.00 ns/op 815.00 ns/op 1.01
enrSubnets - fastDeserialize 64 bits 736.00 ns/op 729.00 ns/op 1.01
enrSubnets - ssz BitVector 64 bits 276.00 ns/op 261.00 ns/op 1.06
enrSubnets - fastDeserialize 4 bits 104.00 ns/op 103.00 ns/op 1.01
enrSubnets - ssz BitVector 4 bits 275.00 ns/op 260.00 ns/op 1.06
prioritizePeers score -10:0 att 32-0.1 sync 2-0 222.39 us/op 200.52 us/op 1.11
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 265.07 us/op 236.67 us/op 1.12
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 353.29 us/op 350.49 us/op 1.01
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 643.65 us/op 597.09 us/op 1.08
prioritizePeers score 0:0 att 64-1 sync 4-1 720.23 us/op 690.15 us/op 1.04
array of 16000 items push then shift 1.2991 us/op 1.2340 us/op 1.05
LinkedList of 16000 items push then shift 7.5900 ns/op 7.5300 ns/op 1.01
array of 16000 items push then pop 71.070 ns/op 67.508 ns/op 1.05
LinkedList of 16000 items push then pop 6.7150 ns/op 5.9020 ns/op 1.14
array of 24000 items push then shift 1.9284 us/op 1.8409 us/op 1.05
LinkedList of 24000 items push then shift 7.6690 ns/op 7.3250 ns/op 1.05
array of 24000 items push then pop 103.90 ns/op 93.709 ns/op 1.11
LinkedList of 24000 items push then pop 6.3900 ns/op 6.0140 ns/op 1.06
intersect bitArray bitLen 8 5.1550 ns/op 4.6740 ns/op 1.10
intersect array and set length 8 30.442 ns/op 28.898 ns/op 1.05
intersect bitArray bitLen 128 24.871 ns/op 23.685 ns/op 1.05
intersect array and set length 128 519.61 ns/op 494.10 ns/op 1.05
bitArray.getTrueBitIndexes() bitLen 128 1.0670 us/op 1.0240 us/op 1.04
bitArray.getTrueBitIndexes() bitLen 248 1.8070 us/op 1.7990 us/op 1.00
bitArray.getTrueBitIndexes() bitLen 512 3.7690 us/op 3.6810 us/op 1.02
Full columns - reconstruct all 6 blobs 127.08 us/op 122.97 us/op 1.03
Full columns - reconstruct half of the blobs out of 6 72.622 us/op 72.095 us/op 1.01
Full columns - reconstruct single blob out of 6 36.451 us/op 33.913 us/op 1.07
Half columns - reconstruct all 6 blobs 410.20 ms/op 384.20 ms/op 1.07
Half columns - reconstruct half of the blobs out of 6 206.29 ms/op 192.78 ms/op 1.07
Half columns - reconstruct single blob out of 6 73.625 ms/op 68.649 ms/op 1.07
Full columns - reconstruct all 10 blobs 221.09 us/op 221.10 us/op 1.00
Full columns - reconstruct half of the blobs out of 10 105.22 us/op 155.64 us/op 0.68
Full columns - reconstruct single blob out of 10 30.181 us/op 31.734 us/op 0.95
Half columns - reconstruct all 10 blobs 672.65 ms/op 620.38 ms/op 1.08
Half columns - reconstruct half of the blobs out of 10 335.06 ms/op 315.38 ms/op 1.06
Half columns - reconstruct single blob out of 10 72.540 ms/op 68.014 ms/op 1.07
Full columns - reconstruct all 20 blobs 1.5058 ms/op 1.3090 ms/op 1.15
Full columns - reconstruct half of the blobs out of 20 183.55 us/op 265.59 us/op 0.69
Full columns - reconstruct single blob out of 20 31.030 us/op 30.209 us/op 1.03
Half columns - reconstruct all 20 blobs 1.3160 s/op 1.2662 s/op 1.04
Half columns - reconstruct half of the blobs out of 20 667.20 ms/op 625.85 ms/op 1.07
Half columns - reconstruct single blob out of 20 71.822 ms/op 67.361 ms/op 1.07
Set add up to 64 items then delete first 2.6200 us/op 2.0634 us/op 1.27
OrderedSet add up to 64 items then delete first 3.4820 us/op 3.2745 us/op 1.06
Set add up to 64 items then delete last 2.4653 us/op 2.0492 us/op 1.20
OrderedSet add up to 64 items then delete last 3.3463 us/op 3.2300 us/op 1.04
Set add up to 64 items then delete middle 2.1486 us/op 2.0607 us/op 1.04
OrderedSet add up to 64 items then delete middle 4.8723 us/op 4.7182 us/op 1.03
Set add up to 128 items then delete first 4.2914 us/op 5.8737 us/op 0.73
OrderedSet add up to 128 items then delete first 6.7678 us/op 6.3611 us/op 1.06
Set add up to 128 items then delete last 4.1752 us/op 5.6867 us/op 0.73
OrderedSet add up to 128 items then delete last 5.9543 us/op 5.7545 us/op 1.03
Set add up to 128 items then delete middle 3.9232 us/op 5.7045 us/op 0.69
OrderedSet add up to 128 items then delete middle 12.073 us/op 11.538 us/op 1.05
Set add up to 256 items then delete first 7.9812 us/op 12.355 us/op 0.65
OrderedSet add up to 256 items then delete first 12.464 us/op 12.344 us/op 1.01
Set add up to 256 items then delete last 7.7284 us/op 11.489 us/op 0.67
OrderedSet add up to 256 items then delete last 11.725 us/op 11.506 us/op 1.02
Set add up to 256 items then delete middle 7.6967 us/op 11.358 us/op 0.68
OrderedSet add up to 256 items then delete middle 35.863 us/op 36.032 us/op 1.00
runFastConfirmationRules vc:100000 bc:96 eq:0 2.2390 us/op 3.2240 us/op 0.69
runFastConfirmationRules vc:600000 bc:96 eq:0 2.3710 us/op 11.029 us/op 0.21
runFastConfirmationRules vc:1000000 bc:96 eq:0 2.2970 us/op 10.653 us/op 0.22
runFastConfirmationRules vc:600000 bc:320 eq:0 5.4680 us/op 12.950 us/op 0.42
runFastConfirmationRules vc:600000 bc:1200 eq:0 19.291 us/op 21.117 us/op 0.91
runFastConfirmationRules vc:600000 bc:96 eq:1000 2.0500 us/op 4.6900 us/op 0.44
runFastConfirmationRules vc:600000 bc:96 eq:10000 2.3660 us/op 16.200 us/op 0.15
runFastConfirmationRules vc:600000 bc:96 eq:300000 2.3770 us/op 21.254 us/op 0.11
pass gossip attestations to forkchoice per slot 2.4951 ms/op 2.4957 ms/op 1.00
forkChoice updateHead vc 100000 bc 64 eq 0 499.22 us/op 390.71 us/op 1.28
forkChoice updateHead vc 600000 bc 64 eq 0 2.6490 ms/op 2.3807 ms/op 1.11
forkChoice updateHead vc 1000000 bc 64 eq 0 4.8949 ms/op 3.9193 ms/op 1.25
forkChoice updateHead vc 600000 bc 320 eq 0 2.9530 ms/op 2.3979 ms/op 1.23
forkChoice updateHead vc 600000 bc 1200 eq 0 2.9892 ms/op 2.5837 ms/op 1.16
forkChoice updateHead vc 600000 bc 7200 eq 0 3.3478 ms/op 3.1510 ms/op 1.06
forkChoice updateHead vc 600000 bc 64 eq 1000 4.8329 ms/op 2.3902 ms/op 2.02
forkChoice updateHead vc 600000 bc 64 eq 10000 3.1260 ms/op 2.5109 ms/op 1.25
forkChoice updateHead vc 600000 bc 64 eq 300000 6.6791 ms/op 6.6832 ms/op 1.00
computeDeltas 1400000 validators 0% inactive 12.285 ms/op 12.337 ms/op 1.00
computeDeltas 1400000 validators 10% inactive 11.510 ms/op 11.393 ms/op 1.01
computeDeltas 1400000 validators 20% inactive 10.093 ms/op 10.589 ms/op 0.95
computeDeltas 1400000 validators 50% inactive 7.8829 ms/op 8.2496 ms/op 0.96
computeDeltas 2100000 validators 0% inactive 17.913 ms/op 18.780 ms/op 0.95
computeDeltas 2100000 validators 10% inactive 16.818 ms/op 17.384 ms/op 0.97
computeDeltas 2100000 validators 20% inactive 15.260 ms/op 15.855 ms/op 0.96
computeDeltas 2100000 validators 50% inactive 8.8564 ms/op 9.3154 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 2.5222 ms/op 2.3047 ms/op 1.09
altair processAttestation - 250000 vs - 7PWei worstcase 2.8196 ms/op 3.1455 ms/op 0.90
altair processAttestation - setStatus - 1/6 committees join 106.96 us/op 98.543 us/op 1.09
altair processAttestation - setStatus - 1/3 committees join 215.22 us/op 198.62 us/op 1.08
altair processAttestation - setStatus - 1/2 committees join 297.94 us/op 284.71 us/op 1.05
altair processAttestation - setStatus - 2/3 committees join 391.00 us/op 374.49 us/op 1.04
altair processAttestation - setStatus - 4/5 committees join 522.63 us/op 827.45 us/op 0.63
altair processAttestation - setStatus - 100% committees join 618.80 us/op 615.06 us/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase 3.7958 ms/op 4.6754 ms/op 0.81
altair processBlock - 250000 vs - 7PWei normalcase hashState 14.042 ms/op 14.480 ms/op 0.97
altair processBlock - 250000 vs - 7PWei worstcase 21.270 ms/op 24.062 ms/op 0.88
altair processBlock - 250000 vs - 7PWei worstcase hashState 42.995 ms/op 49.052 ms/op 0.88
phase0 processBlock - 250000 vs - 7PWei normalcase 1.4097 ms/op 1.4936 ms/op 0.94
phase0 processBlock - 250000 vs - 7PWei worstcase 19.199 ms/op 16.467 ms/op 1.17
altair processEth1Data - 250000 vs - 7PWei normalcase 288.44 us/op 286.24 us/op 1.01
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 3.4990 us/op 4.1260 us/op 0.85
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 20.406 us/op 22.548 us/op 0.91
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 5.7980 us/op 8.4470 us/op 0.69
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 4.7130 us/op 5.0570 us/op 0.93
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 95.594 us/op 103.46 us/op 0.92
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 1.4729 ms/op 1.4849 ms/op 0.99
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.9134 ms/op 1.8140 ms/op 1.05
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.9378 ms/op 1.7497 ms/op 1.11
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 5.2311 ms/op 3.5424 ms/op 1.48
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.2117 ms/op 2.0522 ms/op 1.08
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 4.1307 ms/op 4.1913 ms/op 0.99
Tree 40 250000 create 358.23 ms/op 343.25 ms/op 1.04
Tree 40 250000 get(125000) 93.557 ns/op 92.156 ns/op 1.02
Tree 40 250000 set(125000) 1.0545 us/op 966.11 ns/op 1.09
Tree 40 250000 toArray() 16.925 ms/op 14.899 ms/op 1.14
Tree 40 250000 iterate all - toArray() + loop 16.721 ms/op 15.257 ms/op 1.10
Tree 40 250000 iterate all - get(i) 44.859 ms/op 40.287 ms/op 1.11
Array 250000 create 2.3879 ms/op 2.1522 ms/op 1.11
Array 250000 clone - spread 681.25 us/op 671.27 us/op 1.01
Array 250000 get(125000) 0.28700 ns/op 0.29400 ns/op 0.98
Array 250000 set(125000) 0.29500 ns/op 0.29400 ns/op 1.00
Array 250000 iterate all - loop 57.263 us/op 56.916 us/op 1.01
phase0 afterProcessEpoch - 250000 vs - 7PWei 38.722 ms/op 39.136 ms/op 0.99
Array.fill - length 1000000 7.6802 ms/op 2.3832 ms/op 3.22
Array push - length 1000000 7.5818 ms/op 9.4883 ms/op 0.80
Array.get 0.20699 ns/op 0.19947 ns/op 1.04
Uint8Array.get 0.24595 ns/op 0.22782 ns/op 1.08
phase0 beforeProcessEpoch - 250000 vs - 7PWei 18.140 ms/op 15.719 ms/op 1.15
altair processEpoch - mainnet_e81889 340.10 ms/op 301.69 ms/op 1.13
mainnet_e81889 - altair beforeProcessEpoch 44.369 ms/op 38.035 ms/op 1.17
mainnet_e81889 - altair processJustificationAndFinalization 8.4860 us/op 6.9870 us/op 1.21
mainnet_e81889 - altair processInactivityUpdates 3.9216 ms/op 6.4626 ms/op 0.61
mainnet_e81889 - altair processRewardsAndPenalties 21.341 ms/op 19.574 ms/op 1.09
mainnet_e81889 - altair processRegistryUpdates 555.00 ns/op 529.00 ns/op 1.05
mainnet_e81889 - altair processSlashings 128.00 ns/op 139.00 ns/op 0.92
mainnet_e81889 - altair processEth1DataReset 129.00 ns/op 126.00 ns/op 1.02
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.6930 ms/op 6.2076 ms/op 0.27
mainnet_e81889 - altair processSlashingsReset 709.00 ns/op 669.00 ns/op 1.06
mainnet_e81889 - altair processRandaoMixesReset 1.5390 us/op 1.4200 us/op 1.08
mainnet_e81889 - altair processHistoricalRootsUpdate 128.00 ns/op 134.00 ns/op 0.96
mainnet_e81889 - altair processParticipationFlagUpdates 459.00 ns/op 448.00 ns/op 1.02
mainnet_e81889 - altair processSyncCommitteeUpdates 102.00 ns/op 106.00 ns/op 0.96
mainnet_e81889 - altair afterProcessEpoch 42.486 ms/op 40.367 ms/op 1.05
capella processEpoch - mainnet_e217614 1.0870 s/op 828.08 ms/op 1.31
mainnet_e217614 - capella beforeProcessEpoch 64.484 ms/op 55.504 ms/op 1.16
mainnet_e217614 - capella processJustificationAndFinalization 8.9480 us/op 6.3620 us/op 1.41
mainnet_e217614 - capella processInactivityUpdates 17.822 ms/op 15.115 ms/op 1.18
mainnet_e217614 - capella processRewardsAndPenalties 101.24 ms/op 90.777 ms/op 1.12
mainnet_e217614 - capella processRegistryUpdates 4.6170 us/op 4.3120 us/op 1.07
mainnet_e217614 - capella processSlashings 132.00 ns/op 126.00 ns/op 1.05
mainnet_e217614 - capella processEth1DataReset 126.00 ns/op 125.00 ns/op 1.01
mainnet_e217614 - capella processEffectiveBalanceUpdates 18.123 ms/op 16.862 ms/op 1.07
mainnet_e217614 - capella processSlashingsReset 692.00 ns/op 660.00 ns/op 1.05
mainnet_e217614 - capella processRandaoMixesReset 1.6570 us/op 1.2900 us/op 1.28
mainnet_e217614 - capella processHistoricalRootsUpdate 131.00 ns/op 124.00 ns/op 1.06
mainnet_e217614 - capella processParticipationFlagUpdates 495.00 ns/op 417.00 ns/op 1.19
mainnet_e217614 - capella afterProcessEpoch 109.70 ms/op 107.00 ms/op 1.03
phase0 processEpoch - mainnet_e58758 436.59 ms/op 326.98 ms/op 1.34
mainnet_e58758 - phase0 beforeProcessEpoch 82.713 ms/op 70.634 ms/op 1.17
mainnet_e58758 - phase0 processJustificationAndFinalization 7.9070 us/op 6.3660 us/op 1.24
mainnet_e58758 - phase0 processRewardsAndPenalties 17.926 ms/op 15.682 ms/op 1.14
mainnet_e58758 - phase0 processRegistryUpdates 2.2990 us/op 2.1860 us/op 1.05
mainnet_e58758 - phase0 processSlashings 130.00 ns/op 126.00 ns/op 1.03
mainnet_e58758 - phase0 processEth1DataReset 127.00 ns/op 124.00 ns/op 1.02
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 841.37 us/op 850.63 us/op 0.99
mainnet_e58758 - phase0 processSlashingsReset 1.0220 us/op 856.00 ns/op 1.19
mainnet_e58758 - phase0 processRandaoMixesReset 1.5350 us/op 1.2180 us/op 1.26
mainnet_e58758 - phase0 processHistoricalRootsUpdate 133.00 ns/op 129.00 ns/op 1.03
mainnet_e58758 - phase0 processParticipationRecordUpdates 1.3590 us/op 1.2140 us/op 1.12
mainnet_e58758 - phase0 afterProcessEpoch 34.559 ms/op 32.358 ms/op 1.07
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.1141 ms/op 950.62 us/op 1.17
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.7536 ms/op 1.5048 ms/op 1.17
altair processInactivityUpdates - 250000 normalcase 12.354 ms/op 12.079 ms/op 1.02
altair processInactivityUpdates - 250000 worstcase 12.099 ms/op 12.019 ms/op 1.01
phase0 processRegistryUpdates - 250000 normalcase 2.9280 us/op 2.0470 us/op 1.43
phase0 processRegistryUpdates - 250000 badcase_full_deposits 145.04 us/op 134.22 us/op 1.08
phase0 processRegistryUpdates - 250000 worstcase 0.5 77.390 ms/op 67.706 ms/op 1.14
altair processRewardsAndPenalties - 250000 normalcase 17.859 ms/op 16.522 ms/op 1.08
altair processRewardsAndPenalties - 250000 worstcase 17.137 ms/op 15.444 ms/op 1.11
phase0 getAttestationDeltas - 250000 normalcase 5.4628 ms/op 5.3498 ms/op 1.02
phase0 getAttestationDeltas - 250000 worstcase 5.4515 ms/op 5.4497 ms/op 1.00
phase0 processSlashings - 250000 worstcase 64.619 us/op 61.963 us/op 1.04
altair processSyncCommitteeUpdates - 250000 13.807 ms/op 12.389 ms/op 1.11
BeaconState.hashTreeRoot - No change 167.00 ns/op 162.00 ns/op 1.03
BeaconState.hashTreeRoot - 1 full validator 91.527 us/op 87.518 us/op 1.05
BeaconState.hashTreeRoot - 32 full validator 1.7932 ms/op 854.28 us/op 2.10
BeaconState.hashTreeRoot - 512 full validator 15.885 ms/op 9.4566 ms/op 1.68
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 121.40 us/op 106.83 us/op 1.14
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.4993 ms/op 1.5574 ms/op 1.60
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 28.937 ms/op 21.127 ms/op 1.37
BeaconState.hashTreeRoot - 1 balances 105.11 us/op 82.518 us/op 1.27
BeaconState.hashTreeRoot - 32 balances 1.5077 ms/op 758.08 us/op 1.99
BeaconState.hashTreeRoot - 512 balances 10.967 ms/op 8.6228 ms/op 1.27
BeaconState.hashTreeRoot - 250000 balances 197.95 ms/op 151.00 ms/op 1.31
aggregationBits - 2048 els - zipIndexesInBitList 19.819 us/op 21.631 us/op 0.92
regular array get 100000 times 23.473 us/op 23.204 us/op 1.01
wrappedArray get 100000 times 23.589 us/op 23.188 us/op 1.02
arrayWithProxy get 100000 times 9.7925 ms/op 17.273 ms/op 0.57
ssz.Root.equals 21.840 ns/op 76.051 ns/op 0.29
byteArrayEquals 21.504 ns/op 22.729 ns/op 0.95
Buffer.compare 8.8780 ns/op 9.6110 ns/op 0.92
processSlot - 1 slots 11.236 us/op 12.009 us/op 0.94
processSlot - 32 slots 2.3766 ms/op 2.1689 ms/op 1.10
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 3.9708 ms/op 5.4772 ms/op 0.72
getCommitteeAssignments - req 1 vs - 250000 vc 1.6980 ms/op 1.6515 ms/op 1.03
getCommitteeAssignments - req 100 vs - 250000 vc 3.4197 ms/op 3.3508 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 3.6769 ms/op 3.6222 ms/op 1.02
findModifiedValidators - 10000 modified validators 905.59 ms/op 775.91 ms/op 1.17
findModifiedValidators - 1000 modified validators 503.19 ms/op 460.31 ms/op 1.09
findModifiedValidators - 100 modified validators 333.08 ms/op 298.57 ms/op 1.12
findModifiedValidators - 10 modified validators 246.09 ms/op 236.47 ms/op 1.04
findModifiedValidators - 1 modified validators 195.41 ms/op 164.84 ms/op 1.19
findModifiedValidators - no difference 153.44 ms/op 195.14 ms/op 0.79
migrate state 1500000 validators, 3400 modified, 2000 new 3.6493 s/op 3.4140 s/op 1.07
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 3.6600 ns/op 3.6500 ns/op 1.00
state getBlockRootAtSlot - 250000 vs - 7PWei 390.17 ns/op 402.63 ns/op 0.97
computeProposerIndex 100000 validators 1.3599 ms/op 1.3639 ms/op 1.00
getNextSyncCommitteeIndices 1000 validators 2.9352 ms/op 2.8767 ms/op 1.02
getNextSyncCommitteeIndices 10000 validators 25.924 ms/op 25.611 ms/op 1.01
getNextSyncCommitteeIndices 100000 validators 88.358 ms/op 91.556 ms/op 0.97
computeProposers - vc 250000 655.03 us/op 554.37 us/op 1.18
computeEpochShuffling - vc 250000 39.681 ms/op 39.615 ms/op 1.00
getNextSyncCommittee - vc 250000 9.4916 ms/op 10.744 ms/op 0.88
nodejs block root to RootHex using toHex 99.019 ns/op 94.457 ns/op 1.05
nodejs block root to RootHex using toRootHex 59.327 ns/op 53.106 ns/op 1.12
nodejs fromHex(blob) 840.32 us/op 918.76 us/op 0.91
nodejs fromHexInto(blob) 662.20 us/op 649.99 us/op 1.02
nodejs block root to RootHex using the deprecated toHexString 490.49 ns/op 522.61 ns/op 0.94
nodejs byteArrayEquals 32 bytes (block root) 25.521 ns/op 26.142 ns/op 0.98
nodejs byteArrayEquals 48 bytes (pubkey) 36.907 ns/op 38.124 ns/op 0.97
nodejs byteArrayEquals 96 bytes (signature) 40.322 ns/op 34.435 ns/op 1.17
nodejs byteArrayEquals 1024 bytes 43.883 ns/op 41.547 ns/op 1.06
nodejs byteArrayEquals 131072 bytes (blob) 1.7660 us/op 1.7793 us/op 0.99
browser block root to RootHex using toHex 145.40 ns/op 146.10 ns/op 1.00
browser block root to RootHex using toRootHex 130.34 ns/op 132.50 ns/op 0.98
browser fromHex(blob) 1.6215 ms/op 1.5964 ms/op 1.02
browser fromHexInto(blob) 601.28 us/op 618.55 us/op 0.97
browser block root to RootHex using the deprecated toHexString 316.01 ns/op 342.53 ns/op 0.92
browser byteArrayEquals 32 bytes (block root) 27.331 ns/op 27.324 ns/op 1.00
browser byteArrayEquals 48 bytes (pubkey) 38.185 ns/op 38.543 ns/op 0.99
browser byteArrayEquals 96 bytes (signature) 72.845 ns/op 71.361 ns/op 1.02
browser byteArrayEquals 1024 bytes 739.13 ns/op 727.06 ns/op 1.02
browser byteArrayEquals 131072 bytes (blob) 91.848 us/op 92.597 us/op 0.99

by benchmarkbot/action

@nflaig nflaig changed the title fix: stabilize fastConfirmation perf bench chore: stabilize fastConfirmation perf bench Jun 9, 2026
@nflaig

nflaig commented Jun 9, 2026

Copy link
Copy Markdown
Member

benchmarks failed

@lodekeeper

Copy link
Copy Markdown
Contributor

Opened #9496 against this branch with the everyoneVotes epoch→slot fix Gemini flagged at discussion_r3382054138. With that landed, runFastConfirmationRules will actually exercise the populated vote-map paths instead of measuring the empty-vote path.

Note: the noise/sendData benchmark failure on bench job 27218604384 (StreamResetError: The stream has been reset in 8 send data - 1000 NB messages cases) is pre-existing and reproduces on unstable head — same stack on bench runs 27225662417 (b00d95d0, 18:00 UTC) and 27203232811 (8d5a6a4f, 11:32 UTC). Not from this PR.

#9496

lodekeeper and others added 2 commits June 10, 2026 19:41
…tes (#9496)

## Summary

Applies the `everyoneVotes` fix Gemini flagged at
`discussion_r3382054138` (confirmed in `discussion_r3382950784`):
`addLatestMessage`'s second arg is `nextSlot: Slot`, not an epoch. The
helper passed `computeEpochAtSlot(vote.slot)` (= `3` for the `bc:96`
snapshot), so `voteNextSlots[i] = 3` got stored. On the next
`updateTime(97)` tick, `computeEpochAtSlot(3) = 0` then made every vote
look 3 epochs stale, and `runFastConfirmationRules` measured the
empty-vote path instead of the intended fully-voted path.

Pass `vote.slot` directly so `computeEpochAtSlot(nextSlot)` inside
`addLatestMessage` lands on the right epoch (= 3 for the snapshot).
`computeEpochAtSlot` was the only reason for the
`@lodestar/state-transition` import in this file; drop it too.

## Why it matters

The original PR (`#9494`) already fixed the GC-pressure flakiness, but
the benchmark numbers weren't measuring what the bench id
(`runFastConfirmationRules vc:600000 bc:96 ...`) claims they're
measuring. With this fix, the bench actually exercises the FCR vote-map
paths with a populated `voteNextIndices`.

## Test plan

- [ ] CI: `Benchmarks` job runs cleanly on the merged commit, with
`runFastConfirmationRules` cases reflecting populated-vote-map cost.
- [ ] CI: `Type Checks` passes (the import drop doesn't break anything
else).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
@nazarhussain

Copy link
Copy Markdown
Contributor Author

@lodekeeper your PR #9496 is merged. Could you now re-review the PR and see if it's looks good to merge?

@lodekeeper lodekeeper left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, ready to merge once CI finishes.

State review:

  • 3f3d194d (your original fix): removes the per-iteration GC pressure (move everyoneVotes + updateHead out of beforeEach, defer cache+snapshot+rules to fn, pass-through beforeEach: data => data so vitest-benchmark routes data correctly).
  • 3c5c63e5 (#9496 merge): fixes the everyoneVotes epoch-vs-slot trap so the bench actually measures the populated-vote-map path instead of the empty-vote shortcut.
  • c300a911 (unstable merge): green diff against current unstable.

Gemini's two earlier inlines are settled — comment 1 (discussion_r3382054138) was the real epoch/slot bug, fixed in #9496; comment 2 (discussion_r3382054140) was incorrect about @chainsafe/benchmark defaulting fn to receive before's output when beforeEach is omitted, Gemini conceded in discussion_r3382952400.

CI on current head c300a911: 12 success / 7 in progress / 0 failure. No blockers.

The one perf-alert on the PR (issuecomment-4661689785, phase0 getAttestationDeltas - 250000 normalcase 3.49x) is from older commit 4b96e5a6 and on an unrelated benchmark — getAttestationDeltas is in packages/state-transition, this PR only touches packages/fork-choice/test/perf/forkChoice/fastConfirmation.test.ts. Same CI-noise pattern as the proposeBlockBody / blob recon trips on #9481/#9484/#9489 yesterday.

@lodekeeper

Copy link
Copy Markdown
Contributor

Approved — pullrequestreview-4468839687. With #9496 in, the bench actually measures the populated-vote-map path. CI green on c300a911 (12 success / 7 pending), no failures. The earlier getAttestationDeltas 3.49x perf alert is on an unrelated benchmark and from a pre-#9496 commit; same CI-noise pattern as the other PRs that tripped yesterday.

@nazarhussain

Copy link
Copy Markdown
Contributor Author

@lodekeeper The benchmark failed again on this PR, seems not related to FCR. Could you investigate that and if it's unrelated then fix it in separate PR.

@lodekeeper

Copy link
Copy Markdown
Contributor

Investigated — unrelated. The flagged bench arrayWithProxy get 100000 times lives at packages/state-transition/test/perf/misc/proxy.test.ts (added in #9440 as a misc V8 Proxy-overhead reference). It's a 100K-iteration hot loop with no per-iteration work besides the property read, so timings are dominated by V8 inline-cache + engine-version effects and very high run-to-run variance on the shared CI runner.

Three new commits between the alert's baseline b00d95d and current head b41507a:

None of them touch state-transition or anything that could plausibly change V8 Proxy IC behavior. Confirmed no src/ code uses new Proxy(...) either, so the bench isn't tracking a real regression surface — it's just been generating noise (e.g. proposeBlockBody type=full, size=empty and Full columns - reconstruct half of the blobs out of 20 fired similarly on #9481/#9484/#9489 with the same unrelated-code-path pattern).

Opened #9502 against unstable to switch the three Proxy-cost benches to bench.skip(...). Source stays in place + a docstring on the describe block explains why, so it can be flipped back to bench(...) locally for ad-hoc measurements.

#9502

@nazarhussain
nazarhussain merged commit 87884a8 into unstable Jun 10, 2026
18 of 20 checks passed
@nazarhussain
nazarhussain deleted the nh/fcr-benchmark-fix branch June 10, 2026 14:57
@wemeetagain

Copy link
Copy Markdown
Member

🎉 This PR is included in v1.44.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants