chore: stabilize fastConfirmation perf bench - #9494
Conversation
The per-iteration `everyoneVotes(...)` (100K–1M validator loop) and `updateHead()` in `beforeEach` were generating GC pressure right inside the measurement window, which on CI inflated the µs-scale fn timings enough to trip the regression threshold. The "flip votes" logic also never actually flipped — same-epoch votes are rejected by `addLatestMessage`, so the second `everyoneVotes` call was a no-op. Replace `beforeEach` with a pass-through. `runFastConfirmationRules` doesn't mutate `store`, so each iteration replays the same work and no per-iteration reset is required. Bench is now stable across runs: ~80K–160K samples per config (was 25–424), wall time per config ~0.3–5s (was 2–17s), and the vc/eq variations no longer move the number — which is the correct behavior, since neither parameter enters the executed code path; their old apparent effect was the GC pressure from the removed beforeEach work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request simplifies the Fast Confirmation Rules (FCR) benchmark by removing the per-iteration vote-flipping and slot-advancing logic in the beforeEach hook, as the rule runner does not mutate the store. The review feedback identifies a correctness bug in the everyoneVotes helper function where an Epoch is incorrectly passed instead of a Slot to addLatestMessage, causing votes to be registered in epoch 0 and treated as expired. Additionally, the pass-through beforeEach hook is redundant and can be safely removed because the benchmark runner defaults to passing the data from before to fn when beforeEach is omitted.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if (!prevBlock) throw Error("no prevBlock"); | ||
|
|
||
| // Vote everyone for head | ||
| // Vote everyone for head so the FCR vote-map paths see a populated voteNextIndices. |
There was a problem hiding this comment.
There is a correctness bug in the helper function everyoneVotes (defined at the bottom of this file) that is called here.\n\nIn everyoneVotes, nextEpoch is passed as the second argument to forkChoice["addLatestMessage"]:\ntypescript\nfunction everyoneVotes(vote: ProtoBlock, forkChoice: ForkChoice): void {\n const nextEpoch = computeEpochAtSlot(vote.slot);\n const nextRoot = vote.blockRoot;\n for (let i = 0; i < forkChoice["balances"].length; i++) {\n forkChoice["addLatestMessage"](i, nextEpoch, nextRoot, PayloadStatus.FULL);\n }\n}\n\n\nHowever, addLatestMessage expects a Slot as its second parameter, not an Epoch:\ntypescript\n private addLatestMessage(\n validatorIndex: ValidatorIndex,\n nextSlot: Slot,\n nextRoot: RootHex,\n nextPayloadStatus: PayloadStatus\n ): void\n\n\nBecause an epoch number (e.g., 2) is passed instead of a slot number (e.g., 64), addLatestMessage internally computes the epoch of the vote as computeEpochAtSlot(2), which evaluates to 0.\n\nThis causes all validator votes to be registered as belonging to epoch 0 (instead of epoch 2). Since the benchmark advances the slot to epoch 3, these votes are treated as expired/inactive by the Fast Confirmation Rules (FCR), meaning the benchmark is not actually executing the realistic code paths where active votes are processed and counted.\n\n### Suggested Fix\nUpdate everyoneVotes to pass vote.slot instead of nextEpoch:\ntypescript\nfunction everyoneVotes(vote: ProtoBlock, forkChoice: ForkChoice): void {\n const nextSlot = vote.slot;\n const nextRoot = vote.blockRoot;\n for (let i = 0; i < forkChoice["balances"].length; i++) {\n forkChoice["addLatestMessage"](i, nextSlot, nextRoot, PayloadStatus.FULL);\n }\n}\n
There was a problem hiding this comment.
Confirming Gemini's analysis: addLatestMessage signature is (validatorIndex, nextSlot, ...) and internally calls computeEpochAtSlot(nextSlot) to determine vote epoch. With vc:600000 bc:96, the head's vote.slot=96 → nextEpoch=3 is passed as nextSlot=3, so voteNextSlots[i]=3 and the internal epoch resolves to computeEpochAtSlot(3)=0. After updateTime(97) (epoch 3), all votes look like they belong to epoch 0 — i.e. stale by 3 epochs. The realistic path through the FCR vote-map paths is not being exercised.
Fix is one line as Gemini suggested:
forkChoice["addLatestMessage"](i, vote.slot, nextRoot, PayloadStatus.FULL);cc @gemini-code-assist — good catch. @nazarhussain — the GC-pressure removal still delivers the flakiness fix on its own, but this means the benchmarked code path isn't faithfully representing real validator state. Worth folding into this PR before merge so the bench numbers actually reflect FCR-active votes.
There was a problem hiding this comment.
Good catch — fixed in 99872b5. vote.slot is now passed directly; the now-unused computeEpochAtSlot import was dropped. Verified via local bench run (still stable, ~80K–135K samples/config).
Note: this is independent of the GC-pressure flakiness fix in the first commit — fn() at currentSlot=97 short-circuits at Rule 4 (confirmedEpoch+1 < currentEpoch after Rule 2 resets to genesis) so it never reaches the vote-map path either way. But fixing the bug now means the future state-backed bench (planned follow-up) will start from a correct setup.
There was a problem hiding this comment.
👍 Confirmed — 99872b50ca matches what Gemini and I flagged: vote.slot is passed straight to addLatestMessage and the now-unused computeEpochAtSlot import is gone. Thanks for closing it out.
| }, | ||
| // Pass-through: the bench harness invokes fn(beforeEach(inputAll)), so without a | ||
| // beforeEach fn would receive undefined. store is not mutated by the rule runner so | ||
| // no per-iteration reset is required. | ||
| beforeEach: (data) => data, |
There was a problem hiding this comment.
The beforeEach pass-through and its accompanying comment are redundant.\n\nIn @chainsafe/benchmark, if beforeEach is omitted, the benchmark runner automatically defaults to passing the data returned by before directly to fn. We can safely remove this block to simplify the benchmark setup.
| }, | |
| // Pass-through: the bench harness invokes fn(beforeEach(inputAll)), so without a | |
| // beforeEach fn would receive undefined. store is not mutated by the rule runner so | |
| // no per-iteration reset is required. | |
| beforeEach: (data) => data, | |
| }, |
There was a problem hiding this comment.
Pushing back on this one — checked @chainsafe/benchmark's runBenchmarkFn.js:
const input = beforeEach ? await beforeEach(inputAll, runIdx) : undefined;When beforeEach is omitted, fn receives undefined — not the data from before. With the pass-through removed, fn: ({ctx, store}) => ... would destructure undefined and throw on every iteration.
The comment is there precisely because the pass-through looks redundant at a glance but isn't. Suggestion to remove it would break the benchmark. cc @gemini-code-assist.
@nazarhussain — leaving this one as-is, comment is accurate.
There was a problem hiding this comment.
Thank you for the clarification, @lodekeeper. You are absolutely correct; my previous assessment regarding the redundancy of the beforeEach pass-through was based on an incorrect assumption about the benchmark runner's default behavior when beforeEach is omitted. I appreciate you pointing out that fn would receive undefined in that case, which would indeed cause the benchmark to fail. The current implementation is necessary for the benchmark to function correctly.
There was a problem hiding this comment.
Agree with @lodekeeper — keeping the pass-through. Confirmed in @chainsafe/benchmark's runBenchmarkFn.js (const input = beforeEach ? await beforeEach(inputAll, runIdx) : undefined): without beforeEach, fn would receive undefined and the destructure on {ctx, store} would throw on every iteration. The comment block above the function is there precisely to flag this for the next reader.
There was a problem hiding this comment.
👍 Thanks for confirming. Agreed — the beforeEach pass-through is load-bearing: without it @chainsafe/benchmark hands fn undefined and the {ctx, store} destructure throws. Good to keep.
|
| Benchmark suite | Current: b41507a | Previous: b00d95d | Ratio |
|---|---|---|---|
| Array.fill - length 1000000 | 7.6802 ms/op | 2.3832 ms/op | 3.22 |
Full benchmark results
| Benchmark suite | Current: b41507a | Previous: b00d95d | Ratio |
|---|---|---|---|
| getPubkeys - index2pubkey - req 1000 vs - 250000 vc | 986.73 us/op | 991.33 us/op | 1.00 |
| getPubkeys - validatorsArr - req 1000 vs - 250000 vc | 40.964 us/op | 39.086 us/op | 1.05 |
| BLS verify - blst | 732.39 us/op | 718.67 us/op | 1.02 |
| BLS verifyMultipleSignatures 3 - blst | 1.3854 ms/op | 1.2966 ms/op | 1.07 |
| BLS verifyMultipleSignatures 8 - blst | 2.2562 ms/op | 2.0583 ms/op | 1.10 |
| BLS verifyMultipleSignatures 32 - blst | 7.0173 ms/op | 6.5330 ms/op | 1.07 |
| BLS verifyMultipleSignatures 64 - blst | 13.496 ms/op | 12.728 ms/op | 1.06 |
| BLS verifyMultipleSignatures 128 - blst | 25.948 ms/op | 24.933 ms/op | 1.04 |
| BLS deserializing 10000 signatures | 630.22 ms/op | 617.27 ms/op | 1.02 |
| BLS deserializing 100000 signatures | 6.4115 s/op | 6.2171 s/op | 1.03 |
| BLS verifyMultipleSignatures - same message - 3 - blst | 711.69 us/op | 763.76 us/op | 0.93 |
| BLS verifyMultipleSignatures - same message - 8 - blst | 841.27 us/op | 871.80 us/op | 0.96 |
| BLS verifyMultipleSignatures - same message - 32 - blst | 1.5787 ms/op | 1.5161 ms/op | 1.04 |
| BLS verifyMultipleSignatures - same message - 64 - blst | 2.4218 ms/op | 2.3563 ms/op | 1.03 |
| BLS verifyMultipleSignatures - same message - 128 - blst | 4.0499 ms/op | 3.9695 ms/op | 1.02 |
| BLS aggregatePubkeys 32 - blst | 17.626 us/op | 17.318 us/op | 1.02 |
| BLS aggregatePubkeys 128 - blst | 63.235 us/op | 61.503 us/op | 1.03 |
| getSlashingsAndExits - default max | 50.526 us/op | 54.141 us/op | 0.93 |
| getSlashingsAndExits - 2k | 345.70 us/op | 410.85 us/op | 0.84 |
| proposeBlockBody type=full, size=empty | 638.61 us/op | 1.6291 ms/op | 0.39 |
| isKnown best case - 1 super set check | 171.00 ns/op | 164.00 ns/op | 1.04 |
| isKnown normal case - 2 super set checks | 167.00 ns/op | 160.00 ns/op | 1.04 |
| isKnown worse case - 16 super set checks | 166.00 ns/op | 160.00 ns/op | 1.04 |
| validate api signedAggregateAndProof - struct | 1.5685 ms/op | 1.4784 ms/op | 1.06 |
| validate gossip signedAggregateAndProof - struct | 1.5582 ms/op | 1.4630 ms/op | 1.07 |
| batch validate gossip attestation - vc 640000 - chunk 32 | 110.71 us/op | 109.15 us/op | 1.01 |
| batch validate gossip attestation - vc 640000 - chunk 64 | 98.124 us/op | 94.521 us/op | 1.04 |
| batch validate gossip attestation - vc 640000 - chunk 128 | 88.996 us/op | 88.802 us/op | 1.00 |
| batch validate gossip attestation - vc 640000 - chunk 256 | 88.308 us/op | 87.446 us/op | 1.01 |
| bytes32 toHexString | 296.00 ns/op | 280.00 ns/op | 1.06 |
| bytes32 Buffer.toString(hex) | 163.00 ns/op | 172.00 ns/op | 0.95 |
| bytes32 Buffer.toString(hex) from Uint8Array | 237.00 ns/op | 230.00 ns/op | 1.03 |
| bytes32 Buffer.toString(hex) + 0x | 165.00 ns/op | 169.00 ns/op | 0.98 |
| Return object 10000 times | 0.21870 ns/op | 0.20940 ns/op | 1.04 |
| Throw Error 10000 times | 3.3656 us/op | 3.2741 us/op | 1.03 |
| toHex | 90.051 ns/op | 87.330 ns/op | 1.03 |
| Buffer.from | 81.319 ns/op | 78.893 ns/op | 1.03 |
| shared Buffer | 55.228 ns/op | 51.680 ns/op | 1.07 |
| fastMsgIdFn sha256 / 200 bytes | 1.4900 us/op | 1.4510 us/op | 1.03 |
| fastMsgIdFn h32 xxhash / 200 bytes | 154.00 ns/op | 152.00 ns/op | 1.01 |
| fastMsgIdFn h64 xxhash / 200 bytes | 204.00 ns/op | 205.00 ns/op | 1.00 |
| fastMsgIdFn sha256 / 1000 bytes | 4.8350 us/op | 4.6410 us/op | 1.04 |
| fastMsgIdFn h32 xxhash / 1000 bytes | 251.00 ns/op | 243.00 ns/op | 1.03 |
| fastMsgIdFn h64 xxhash / 1000 bytes | 257.00 ns/op | 247.00 ns/op | 1.04 |
| fastMsgIdFn sha256 / 10000 bytes | 42.676 us/op | 41.387 us/op | 1.03 |
| fastMsgIdFn h32 xxhash / 10000 bytes | 1.2930 us/op | 1.2730 us/op | 1.02 |
| fastMsgIdFn h64 xxhash / 10000 bytes | 822.00 ns/op | 815.00 ns/op | 1.01 |
| enrSubnets - fastDeserialize 64 bits | 736.00 ns/op | 729.00 ns/op | 1.01 |
| enrSubnets - ssz BitVector 64 bits | 276.00 ns/op | 261.00 ns/op | 1.06 |
| enrSubnets - fastDeserialize 4 bits | 104.00 ns/op | 103.00 ns/op | 1.01 |
| enrSubnets - ssz BitVector 4 bits | 275.00 ns/op | 260.00 ns/op | 1.06 |
| prioritizePeers score -10:0 att 32-0.1 sync 2-0 | 222.39 us/op | 200.52 us/op | 1.11 |
| prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 | 265.07 us/op | 236.67 us/op | 1.12 |
| prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 | 353.29 us/op | 350.49 us/op | 1.01 |
| prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 | 643.65 us/op | 597.09 us/op | 1.08 |
| prioritizePeers score 0:0 att 64-1 sync 4-1 | 720.23 us/op | 690.15 us/op | 1.04 |
| array of 16000 items push then shift | 1.2991 us/op | 1.2340 us/op | 1.05 |
| LinkedList of 16000 items push then shift | 7.5900 ns/op | 7.5300 ns/op | 1.01 |
| array of 16000 items push then pop | 71.070 ns/op | 67.508 ns/op | 1.05 |
| LinkedList of 16000 items push then pop | 6.7150 ns/op | 5.9020 ns/op | 1.14 |
| array of 24000 items push then shift | 1.9284 us/op | 1.8409 us/op | 1.05 |
| LinkedList of 24000 items push then shift | 7.6690 ns/op | 7.3250 ns/op | 1.05 |
| array of 24000 items push then pop | 103.90 ns/op | 93.709 ns/op | 1.11 |
| LinkedList of 24000 items push then pop | 6.3900 ns/op | 6.0140 ns/op | 1.06 |
| intersect bitArray bitLen 8 | 5.1550 ns/op | 4.6740 ns/op | 1.10 |
| intersect array and set length 8 | 30.442 ns/op | 28.898 ns/op | 1.05 |
| intersect bitArray bitLen 128 | 24.871 ns/op | 23.685 ns/op | 1.05 |
| intersect array and set length 128 | 519.61 ns/op | 494.10 ns/op | 1.05 |
| bitArray.getTrueBitIndexes() bitLen 128 | 1.0670 us/op | 1.0240 us/op | 1.04 |
| bitArray.getTrueBitIndexes() bitLen 248 | 1.8070 us/op | 1.7990 us/op | 1.00 |
| bitArray.getTrueBitIndexes() bitLen 512 | 3.7690 us/op | 3.6810 us/op | 1.02 |
| Full columns - reconstruct all 6 blobs | 127.08 us/op | 122.97 us/op | 1.03 |
| Full columns - reconstruct half of the blobs out of 6 | 72.622 us/op | 72.095 us/op | 1.01 |
| Full columns - reconstruct single blob out of 6 | 36.451 us/op | 33.913 us/op | 1.07 |
| Half columns - reconstruct all 6 blobs | 410.20 ms/op | 384.20 ms/op | 1.07 |
| Half columns - reconstruct half of the blobs out of 6 | 206.29 ms/op | 192.78 ms/op | 1.07 |
| Half columns - reconstruct single blob out of 6 | 73.625 ms/op | 68.649 ms/op | 1.07 |
| Full columns - reconstruct all 10 blobs | 221.09 us/op | 221.10 us/op | 1.00 |
| Full columns - reconstruct half of the blobs out of 10 | 105.22 us/op | 155.64 us/op | 0.68 |
| Full columns - reconstruct single blob out of 10 | 30.181 us/op | 31.734 us/op | 0.95 |
| Half columns - reconstruct all 10 blobs | 672.65 ms/op | 620.38 ms/op | 1.08 |
| Half columns - reconstruct half of the blobs out of 10 | 335.06 ms/op | 315.38 ms/op | 1.06 |
| Half columns - reconstruct single blob out of 10 | 72.540 ms/op | 68.014 ms/op | 1.07 |
| Full columns - reconstruct all 20 blobs | 1.5058 ms/op | 1.3090 ms/op | 1.15 |
| Full columns - reconstruct half of the blobs out of 20 | 183.55 us/op | 265.59 us/op | 0.69 |
| Full columns - reconstruct single blob out of 20 | 31.030 us/op | 30.209 us/op | 1.03 |
| Half columns - reconstruct all 20 blobs | 1.3160 s/op | 1.2662 s/op | 1.04 |
| Half columns - reconstruct half of the blobs out of 20 | 667.20 ms/op | 625.85 ms/op | 1.07 |
| Half columns - reconstruct single blob out of 20 | 71.822 ms/op | 67.361 ms/op | 1.07 |
| Set add up to 64 items then delete first | 2.6200 us/op | 2.0634 us/op | 1.27 |
| OrderedSet add up to 64 items then delete first | 3.4820 us/op | 3.2745 us/op | 1.06 |
| Set add up to 64 items then delete last | 2.4653 us/op | 2.0492 us/op | 1.20 |
| OrderedSet add up to 64 items then delete last | 3.3463 us/op | 3.2300 us/op | 1.04 |
| Set add up to 64 items then delete middle | 2.1486 us/op | 2.0607 us/op | 1.04 |
| OrderedSet add up to 64 items then delete middle | 4.8723 us/op | 4.7182 us/op | 1.03 |
| Set add up to 128 items then delete first | 4.2914 us/op | 5.8737 us/op | 0.73 |
| OrderedSet add up to 128 items then delete first | 6.7678 us/op | 6.3611 us/op | 1.06 |
| Set add up to 128 items then delete last | 4.1752 us/op | 5.6867 us/op | 0.73 |
| OrderedSet add up to 128 items then delete last | 5.9543 us/op | 5.7545 us/op | 1.03 |
| Set add up to 128 items then delete middle | 3.9232 us/op | 5.7045 us/op | 0.69 |
| OrderedSet add up to 128 items then delete middle | 12.073 us/op | 11.538 us/op | 1.05 |
| Set add up to 256 items then delete first | 7.9812 us/op | 12.355 us/op | 0.65 |
| OrderedSet add up to 256 items then delete first | 12.464 us/op | 12.344 us/op | 1.01 |
| Set add up to 256 items then delete last | 7.7284 us/op | 11.489 us/op | 0.67 |
| OrderedSet add up to 256 items then delete last | 11.725 us/op | 11.506 us/op | 1.02 |
| Set add up to 256 items then delete middle | 7.6967 us/op | 11.358 us/op | 0.68 |
| OrderedSet add up to 256 items then delete middle | 35.863 us/op | 36.032 us/op | 1.00 |
| runFastConfirmationRules vc:100000 bc:96 eq:0 | 2.2390 us/op | 3.2240 us/op | 0.69 |
| runFastConfirmationRules vc:600000 bc:96 eq:0 | 2.3710 us/op | 11.029 us/op | 0.21 |
| runFastConfirmationRules vc:1000000 bc:96 eq:0 | 2.2970 us/op | 10.653 us/op | 0.22 |
| runFastConfirmationRules vc:600000 bc:320 eq:0 | 5.4680 us/op | 12.950 us/op | 0.42 |
| runFastConfirmationRules vc:600000 bc:1200 eq:0 | 19.291 us/op | 21.117 us/op | 0.91 |
| runFastConfirmationRules vc:600000 bc:96 eq:1000 | 2.0500 us/op | 4.6900 us/op | 0.44 |
| runFastConfirmationRules vc:600000 bc:96 eq:10000 | 2.3660 us/op | 16.200 us/op | 0.15 |
| runFastConfirmationRules vc:600000 bc:96 eq:300000 | 2.3770 us/op | 21.254 us/op | 0.11 |
| pass gossip attestations to forkchoice per slot | 2.4951 ms/op | 2.4957 ms/op | 1.00 |
| forkChoice updateHead vc 100000 bc 64 eq 0 | 499.22 us/op | 390.71 us/op | 1.28 |
| forkChoice updateHead vc 600000 bc 64 eq 0 | 2.6490 ms/op | 2.3807 ms/op | 1.11 |
| forkChoice updateHead vc 1000000 bc 64 eq 0 | 4.8949 ms/op | 3.9193 ms/op | 1.25 |
| forkChoice updateHead vc 600000 bc 320 eq 0 | 2.9530 ms/op | 2.3979 ms/op | 1.23 |
| forkChoice updateHead vc 600000 bc 1200 eq 0 | 2.9892 ms/op | 2.5837 ms/op | 1.16 |
| forkChoice updateHead vc 600000 bc 7200 eq 0 | 3.3478 ms/op | 3.1510 ms/op | 1.06 |
| forkChoice updateHead vc 600000 bc 64 eq 1000 | 4.8329 ms/op | 2.3902 ms/op | 2.02 |
| forkChoice updateHead vc 600000 bc 64 eq 10000 | 3.1260 ms/op | 2.5109 ms/op | 1.25 |
| forkChoice updateHead vc 600000 bc 64 eq 300000 | 6.6791 ms/op | 6.6832 ms/op | 1.00 |
| computeDeltas 1400000 validators 0% inactive | 12.285 ms/op | 12.337 ms/op | 1.00 |
| computeDeltas 1400000 validators 10% inactive | 11.510 ms/op | 11.393 ms/op | 1.01 |
| computeDeltas 1400000 validators 20% inactive | 10.093 ms/op | 10.589 ms/op | 0.95 |
| computeDeltas 1400000 validators 50% inactive | 7.8829 ms/op | 8.2496 ms/op | 0.96 |
| computeDeltas 2100000 validators 0% inactive | 17.913 ms/op | 18.780 ms/op | 0.95 |
| computeDeltas 2100000 validators 10% inactive | 16.818 ms/op | 17.384 ms/op | 0.97 |
| computeDeltas 2100000 validators 20% inactive | 15.260 ms/op | 15.855 ms/op | 0.96 |
| computeDeltas 2100000 validators 50% inactive | 8.8564 ms/op | 9.3154 ms/op | 0.95 |
| altair processAttestation - 250000 vs - 7PWei normalcase | 2.5222 ms/op | 2.3047 ms/op | 1.09 |
| altair processAttestation - 250000 vs - 7PWei worstcase | 2.8196 ms/op | 3.1455 ms/op | 0.90 |
| altair processAttestation - setStatus - 1/6 committees join | 106.96 us/op | 98.543 us/op | 1.09 |
| altair processAttestation - setStatus - 1/3 committees join | 215.22 us/op | 198.62 us/op | 1.08 |
| altair processAttestation - setStatus - 1/2 committees join | 297.94 us/op | 284.71 us/op | 1.05 |
| altair processAttestation - setStatus - 2/3 committees join | 391.00 us/op | 374.49 us/op | 1.04 |
| altair processAttestation - setStatus - 4/5 committees join | 522.63 us/op | 827.45 us/op | 0.63 |
| altair processAttestation - setStatus - 100% committees join | 618.80 us/op | 615.06 us/op | 1.01 |
| altair processBlock - 250000 vs - 7PWei normalcase | 3.7958 ms/op | 4.6754 ms/op | 0.81 |
| altair processBlock - 250000 vs - 7PWei normalcase hashState | 14.042 ms/op | 14.480 ms/op | 0.97 |
| altair processBlock - 250000 vs - 7PWei worstcase | 21.270 ms/op | 24.062 ms/op | 0.88 |
| altair processBlock - 250000 vs - 7PWei worstcase hashState | 42.995 ms/op | 49.052 ms/op | 0.88 |
| phase0 processBlock - 250000 vs - 7PWei normalcase | 1.4097 ms/op | 1.4936 ms/op | 0.94 |
| phase0 processBlock - 250000 vs - 7PWei worstcase | 19.199 ms/op | 16.467 ms/op | 1.17 |
| altair processEth1Data - 250000 vs - 7PWei normalcase | 288.44 us/op | 286.24 us/op | 1.01 |
| getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 | 3.4990 us/op | 4.1260 us/op | 0.85 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 | 20.406 us/op | 22.548 us/op | 0.91 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 | 5.7980 us/op | 8.4470 us/op | 0.69 |
| getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 | 4.7130 us/op | 5.0570 us/op | 0.93 |
| getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 | 95.594 us/op | 103.46 us/op | 0.92 |
| getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 | 1.4729 ms/op | 1.4849 ms/op | 0.99 |
| getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 | 1.9134 ms/op | 1.8140 ms/op | 1.05 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 | 1.9378 ms/op | 1.7497 ms/op | 1.11 |
| getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 | 5.2311 ms/op | 3.5424 ms/op | 1.48 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 | 2.2117 ms/op | 2.0522 ms/op | 1.08 |
| getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 | 4.1307 ms/op | 4.1913 ms/op | 0.99 |
| Tree 40 250000 create | 358.23 ms/op | 343.25 ms/op | 1.04 |
| Tree 40 250000 get(125000) | 93.557 ns/op | 92.156 ns/op | 1.02 |
| Tree 40 250000 set(125000) | 1.0545 us/op | 966.11 ns/op | 1.09 |
| Tree 40 250000 toArray() | 16.925 ms/op | 14.899 ms/op | 1.14 |
| Tree 40 250000 iterate all - toArray() + loop | 16.721 ms/op | 15.257 ms/op | 1.10 |
| Tree 40 250000 iterate all - get(i) | 44.859 ms/op | 40.287 ms/op | 1.11 |
| Array 250000 create | 2.3879 ms/op | 2.1522 ms/op | 1.11 |
| Array 250000 clone - spread | 681.25 us/op | 671.27 us/op | 1.01 |
| Array 250000 get(125000) | 0.28700 ns/op | 0.29400 ns/op | 0.98 |
| Array 250000 set(125000) | 0.29500 ns/op | 0.29400 ns/op | 1.00 |
| Array 250000 iterate all - loop | 57.263 us/op | 56.916 us/op | 1.01 |
| phase0 afterProcessEpoch - 250000 vs - 7PWei | 38.722 ms/op | 39.136 ms/op | 0.99 |
| Array.fill - length 1000000 | 7.6802 ms/op | 2.3832 ms/op | 3.22 |
| Array push - length 1000000 | 7.5818 ms/op | 9.4883 ms/op | 0.80 |
| Array.get | 0.20699 ns/op | 0.19947 ns/op | 1.04 |
| Uint8Array.get | 0.24595 ns/op | 0.22782 ns/op | 1.08 |
| phase0 beforeProcessEpoch - 250000 vs - 7PWei | 18.140 ms/op | 15.719 ms/op | 1.15 |
| altair processEpoch - mainnet_e81889 | 340.10 ms/op | 301.69 ms/op | 1.13 |
| mainnet_e81889 - altair beforeProcessEpoch | 44.369 ms/op | 38.035 ms/op | 1.17 |
| mainnet_e81889 - altair processJustificationAndFinalization | 8.4860 us/op | 6.9870 us/op | 1.21 |
| mainnet_e81889 - altair processInactivityUpdates | 3.9216 ms/op | 6.4626 ms/op | 0.61 |
| mainnet_e81889 - altair processRewardsAndPenalties | 21.341 ms/op | 19.574 ms/op | 1.09 |
| mainnet_e81889 - altair processRegistryUpdates | 555.00 ns/op | 529.00 ns/op | 1.05 |
| mainnet_e81889 - altair processSlashings | 128.00 ns/op | 139.00 ns/op | 0.92 |
| mainnet_e81889 - altair processEth1DataReset | 129.00 ns/op | 126.00 ns/op | 1.02 |
| mainnet_e81889 - altair processEffectiveBalanceUpdates | 1.6930 ms/op | 6.2076 ms/op | 0.27 |
| mainnet_e81889 - altair processSlashingsReset | 709.00 ns/op | 669.00 ns/op | 1.06 |
| mainnet_e81889 - altair processRandaoMixesReset | 1.5390 us/op | 1.4200 us/op | 1.08 |
| mainnet_e81889 - altair processHistoricalRootsUpdate | 128.00 ns/op | 134.00 ns/op | 0.96 |
| mainnet_e81889 - altair processParticipationFlagUpdates | 459.00 ns/op | 448.00 ns/op | 1.02 |
| mainnet_e81889 - altair processSyncCommitteeUpdates | 102.00 ns/op | 106.00 ns/op | 0.96 |
| mainnet_e81889 - altair afterProcessEpoch | 42.486 ms/op | 40.367 ms/op | 1.05 |
| capella processEpoch - mainnet_e217614 | 1.0870 s/op | 828.08 ms/op | 1.31 |
| mainnet_e217614 - capella beforeProcessEpoch | 64.484 ms/op | 55.504 ms/op | 1.16 |
| mainnet_e217614 - capella processJustificationAndFinalization | 8.9480 us/op | 6.3620 us/op | 1.41 |
| mainnet_e217614 - capella processInactivityUpdates | 17.822 ms/op | 15.115 ms/op | 1.18 |
| mainnet_e217614 - capella processRewardsAndPenalties | 101.24 ms/op | 90.777 ms/op | 1.12 |
| mainnet_e217614 - capella processRegistryUpdates | 4.6170 us/op | 4.3120 us/op | 1.07 |
| mainnet_e217614 - capella processSlashings | 132.00 ns/op | 126.00 ns/op | 1.05 |
| mainnet_e217614 - capella processEth1DataReset | 126.00 ns/op | 125.00 ns/op | 1.01 |
| mainnet_e217614 - capella processEffectiveBalanceUpdates | 18.123 ms/op | 16.862 ms/op | 1.07 |
| mainnet_e217614 - capella processSlashingsReset | 692.00 ns/op | 660.00 ns/op | 1.05 |
| mainnet_e217614 - capella processRandaoMixesReset | 1.6570 us/op | 1.2900 us/op | 1.28 |
| mainnet_e217614 - capella processHistoricalRootsUpdate | 131.00 ns/op | 124.00 ns/op | 1.06 |
| mainnet_e217614 - capella processParticipationFlagUpdates | 495.00 ns/op | 417.00 ns/op | 1.19 |
| mainnet_e217614 - capella afterProcessEpoch | 109.70 ms/op | 107.00 ms/op | 1.03 |
| phase0 processEpoch - mainnet_e58758 | 436.59 ms/op | 326.98 ms/op | 1.34 |
| mainnet_e58758 - phase0 beforeProcessEpoch | 82.713 ms/op | 70.634 ms/op | 1.17 |
| mainnet_e58758 - phase0 processJustificationAndFinalization | 7.9070 us/op | 6.3660 us/op | 1.24 |
| mainnet_e58758 - phase0 processRewardsAndPenalties | 17.926 ms/op | 15.682 ms/op | 1.14 |
| mainnet_e58758 - phase0 processRegistryUpdates | 2.2990 us/op | 2.1860 us/op | 1.05 |
| mainnet_e58758 - phase0 processSlashings | 130.00 ns/op | 126.00 ns/op | 1.03 |
| mainnet_e58758 - phase0 processEth1DataReset | 127.00 ns/op | 124.00 ns/op | 1.02 |
| mainnet_e58758 - phase0 processEffectiveBalanceUpdates | 841.37 us/op | 850.63 us/op | 0.99 |
| mainnet_e58758 - phase0 processSlashingsReset | 1.0220 us/op | 856.00 ns/op | 1.19 |
| mainnet_e58758 - phase0 processRandaoMixesReset | 1.5350 us/op | 1.2180 us/op | 1.26 |
| mainnet_e58758 - phase0 processHistoricalRootsUpdate | 133.00 ns/op | 129.00 ns/op | 1.03 |
| mainnet_e58758 - phase0 processParticipationRecordUpdates | 1.3590 us/op | 1.2140 us/op | 1.12 |
| mainnet_e58758 - phase0 afterProcessEpoch | 34.559 ms/op | 32.358 ms/op | 1.07 |
| phase0 processEffectiveBalanceUpdates - 250000 normalcase | 1.1141 ms/op | 950.62 us/op | 1.17 |
| phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 | 1.7536 ms/op | 1.5048 ms/op | 1.17 |
| altair processInactivityUpdates - 250000 normalcase | 12.354 ms/op | 12.079 ms/op | 1.02 |
| altair processInactivityUpdates - 250000 worstcase | 12.099 ms/op | 12.019 ms/op | 1.01 |
| phase0 processRegistryUpdates - 250000 normalcase | 2.9280 us/op | 2.0470 us/op | 1.43 |
| phase0 processRegistryUpdates - 250000 badcase_full_deposits | 145.04 us/op | 134.22 us/op | 1.08 |
| phase0 processRegistryUpdates - 250000 worstcase 0.5 | 77.390 ms/op | 67.706 ms/op | 1.14 |
| altair processRewardsAndPenalties - 250000 normalcase | 17.859 ms/op | 16.522 ms/op | 1.08 |
| altair processRewardsAndPenalties - 250000 worstcase | 17.137 ms/op | 15.444 ms/op | 1.11 |
| phase0 getAttestationDeltas - 250000 normalcase | 5.4628 ms/op | 5.3498 ms/op | 1.02 |
| phase0 getAttestationDeltas - 250000 worstcase | 5.4515 ms/op | 5.4497 ms/op | 1.00 |
| phase0 processSlashings - 250000 worstcase | 64.619 us/op | 61.963 us/op | 1.04 |
| altair processSyncCommitteeUpdates - 250000 | 13.807 ms/op | 12.389 ms/op | 1.11 |
| BeaconState.hashTreeRoot - No change | 167.00 ns/op | 162.00 ns/op | 1.03 |
| BeaconState.hashTreeRoot - 1 full validator | 91.527 us/op | 87.518 us/op | 1.05 |
| BeaconState.hashTreeRoot - 32 full validator | 1.7932 ms/op | 854.28 us/op | 2.10 |
| BeaconState.hashTreeRoot - 512 full validator | 15.885 ms/op | 9.4566 ms/op | 1.68 |
| BeaconState.hashTreeRoot - 1 validator.effectiveBalance | 121.40 us/op | 106.83 us/op | 1.14 |
| BeaconState.hashTreeRoot - 32 validator.effectiveBalance | 2.4993 ms/op | 1.5574 ms/op | 1.60 |
| BeaconState.hashTreeRoot - 512 validator.effectiveBalance | 28.937 ms/op | 21.127 ms/op | 1.37 |
| BeaconState.hashTreeRoot - 1 balances | 105.11 us/op | 82.518 us/op | 1.27 |
| BeaconState.hashTreeRoot - 32 balances | 1.5077 ms/op | 758.08 us/op | 1.99 |
| BeaconState.hashTreeRoot - 512 balances | 10.967 ms/op | 8.6228 ms/op | 1.27 |
| BeaconState.hashTreeRoot - 250000 balances | 197.95 ms/op | 151.00 ms/op | 1.31 |
| aggregationBits - 2048 els - zipIndexesInBitList | 19.819 us/op | 21.631 us/op | 0.92 |
| regular array get 100000 times | 23.473 us/op | 23.204 us/op | 1.01 |
| wrappedArray get 100000 times | 23.589 us/op | 23.188 us/op | 1.02 |
| arrayWithProxy get 100000 times | 9.7925 ms/op | 17.273 ms/op | 0.57 |
| ssz.Root.equals | 21.840 ns/op | 76.051 ns/op | 0.29 |
| byteArrayEquals | 21.504 ns/op | 22.729 ns/op | 0.95 |
| Buffer.compare | 8.8780 ns/op | 9.6110 ns/op | 0.92 |
| processSlot - 1 slots | 11.236 us/op | 12.009 us/op | 0.94 |
| processSlot - 32 slots | 2.3766 ms/op | 2.1689 ms/op | 1.10 |
| getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei | 3.9708 ms/op | 5.4772 ms/op | 0.72 |
| getCommitteeAssignments - req 1 vs - 250000 vc | 1.6980 ms/op | 1.6515 ms/op | 1.03 |
| getCommitteeAssignments - req 100 vs - 250000 vc | 3.4197 ms/op | 3.3508 ms/op | 1.02 |
| getCommitteeAssignments - req 1000 vs - 250000 vc | 3.6769 ms/op | 3.6222 ms/op | 1.02 |
| findModifiedValidators - 10000 modified validators | 905.59 ms/op | 775.91 ms/op | 1.17 |
| findModifiedValidators - 1000 modified validators | 503.19 ms/op | 460.31 ms/op | 1.09 |
| findModifiedValidators - 100 modified validators | 333.08 ms/op | 298.57 ms/op | 1.12 |
| findModifiedValidators - 10 modified validators | 246.09 ms/op | 236.47 ms/op | 1.04 |
| findModifiedValidators - 1 modified validators | 195.41 ms/op | 164.84 ms/op | 1.19 |
| findModifiedValidators - no difference | 153.44 ms/op | 195.14 ms/op | 0.79 |
| migrate state 1500000 validators, 3400 modified, 2000 new | 3.6493 s/op | 3.4140 s/op | 1.07 |
| RootCache.getBlockRootAtSlot - 250000 vs - 7PWei | 3.6600 ns/op | 3.6500 ns/op | 1.00 |
| state getBlockRootAtSlot - 250000 vs - 7PWei | 390.17 ns/op | 402.63 ns/op | 0.97 |
| computeProposerIndex 100000 validators | 1.3599 ms/op | 1.3639 ms/op | 1.00 |
| getNextSyncCommitteeIndices 1000 validators | 2.9352 ms/op | 2.8767 ms/op | 1.02 |
| getNextSyncCommitteeIndices 10000 validators | 25.924 ms/op | 25.611 ms/op | 1.01 |
| getNextSyncCommitteeIndices 100000 validators | 88.358 ms/op | 91.556 ms/op | 0.97 |
| computeProposers - vc 250000 | 655.03 us/op | 554.37 us/op | 1.18 |
| computeEpochShuffling - vc 250000 | 39.681 ms/op | 39.615 ms/op | 1.00 |
| getNextSyncCommittee - vc 250000 | 9.4916 ms/op | 10.744 ms/op | 0.88 |
| nodejs block root to RootHex using toHex | 99.019 ns/op | 94.457 ns/op | 1.05 |
| nodejs block root to RootHex using toRootHex | 59.327 ns/op | 53.106 ns/op | 1.12 |
| nodejs fromHex(blob) | 840.32 us/op | 918.76 us/op | 0.91 |
| nodejs fromHexInto(blob) | 662.20 us/op | 649.99 us/op | 1.02 |
| nodejs block root to RootHex using the deprecated toHexString | 490.49 ns/op | 522.61 ns/op | 0.94 |
| nodejs byteArrayEquals 32 bytes (block root) | 25.521 ns/op | 26.142 ns/op | 0.98 |
| nodejs byteArrayEquals 48 bytes (pubkey) | 36.907 ns/op | 38.124 ns/op | 0.97 |
| nodejs byteArrayEquals 96 bytes (signature) | 40.322 ns/op | 34.435 ns/op | 1.17 |
| nodejs byteArrayEquals 1024 bytes | 43.883 ns/op | 41.547 ns/op | 1.06 |
| nodejs byteArrayEquals 131072 bytes (blob) | 1.7660 us/op | 1.7793 us/op | 0.99 |
| browser block root to RootHex using toHex | 145.40 ns/op | 146.10 ns/op | 1.00 |
| browser block root to RootHex using toRootHex | 130.34 ns/op | 132.50 ns/op | 0.98 |
| browser fromHex(blob) | 1.6215 ms/op | 1.5964 ms/op | 1.02 |
| browser fromHexInto(blob) | 601.28 us/op | 618.55 us/op | 0.97 |
| browser block root to RootHex using the deprecated toHexString | 316.01 ns/op | 342.53 ns/op | 0.92 |
| browser byteArrayEquals 32 bytes (block root) | 27.331 ns/op | 27.324 ns/op | 1.00 |
| browser byteArrayEquals 48 bytes (pubkey) | 38.185 ns/op | 38.543 ns/op | 0.99 |
| browser byteArrayEquals 96 bytes (signature) | 72.845 ns/op | 71.361 ns/op | 1.02 |
| browser byteArrayEquals 1024 bytes | 739.13 ns/op | 727.06 ns/op | 1.02 |
| browser byteArrayEquals 131072 bytes (blob) | 91.848 us/op | 92.597 us/op | 0.99 |
by benchmarkbot/action
|
benchmarks failed |
|
Opened #9496 against this branch with the Note: the noise/sendData benchmark failure on bench job |
…tes (#9496) ## Summary Applies the `everyoneVotes` fix Gemini flagged at `discussion_r3382054138` (confirmed in `discussion_r3382950784`): `addLatestMessage`'s second arg is `nextSlot: Slot`, not an epoch. The helper passed `computeEpochAtSlot(vote.slot)` (= `3` for the `bc:96` snapshot), so `voteNextSlots[i] = 3` got stored. On the next `updateTime(97)` tick, `computeEpochAtSlot(3) = 0` then made every vote look 3 epochs stale, and `runFastConfirmationRules` measured the empty-vote path instead of the intended fully-voted path. Pass `vote.slot` directly so `computeEpochAtSlot(nextSlot)` inside `addLatestMessage` lands on the right epoch (= 3 for the snapshot). `computeEpochAtSlot` was the only reason for the `@lodestar/state-transition` import in this file; drop it too. ## Why it matters The original PR (`#9494`) already fixed the GC-pressure flakiness, but the benchmark numbers weren't measuring what the bench id (`runFastConfirmationRules vc:600000 bc:96 ...`) claims they're measuring. With this fix, the bench actually exercises the FCR vote-map paths with a populated `voteNextIndices`. ## Test plan - [ ] CI: `Benchmarks` job runs cleanly on the merged commit, with `runFastConfirmationRules` cases reflecting populated-vote-map cost. - [ ] CI: `Type Checks` passes (the import drop doesn't break anything else). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
|
@lodekeeper your PR #9496 is merged. Could you now re-review the PR and see if it's looks good to merge? |
lodekeeper
left a comment
There was a problem hiding this comment.
LGTM, ready to merge once CI finishes.
State review:
3f3d194d(your original fix): removes the per-iteration GC pressure (moveeveryoneVotes+updateHeadout ofbeforeEach, defer cache+snapshot+rules tofn, pass-throughbeforeEach: data => dataso vitest-benchmark routes data correctly).3c5c63e5(#9496 merge): fixes theeveryoneVotesepoch-vs-slot trap so the bench actually measures the populated-vote-map path instead of the empty-vote shortcut.c300a911(unstable merge): green diff against current unstable.
Gemini's two earlier inlines are settled — comment 1 (discussion_r3382054138) was the real epoch/slot bug, fixed in #9496; comment 2 (discussion_r3382054140) was incorrect about @chainsafe/benchmark defaulting fn to receive before's output when beforeEach is omitted, Gemini conceded in discussion_r3382952400.
CI on current head c300a911: 12 success / 7 in progress / 0 failure. No blockers.
The one perf-alert on the PR (issuecomment-4661689785, phase0 getAttestationDeltas - 250000 normalcase 3.49x) is from older commit 4b96e5a6 and on an unrelated benchmark — getAttestationDeltas is in packages/state-transition, this PR only touches packages/fork-choice/test/perf/forkChoice/fastConfirmation.test.ts. Same CI-noise pattern as the proposeBlockBody / blob recon trips on #9481/#9484/#9489 yesterday.
|
Approved — |
|
@lodekeeper The benchmark failed again on this PR, seems not related to FCR. Could you investigate that and if it's unrelated then fix it in separate PR. |
|
Investigated — unrelated. The flagged bench Three new commits between the alert's baseline
None of them touch Opened #9502 against unstable to switch the three Proxy-cost benches to |
|
🎉 This PR is included in v1.44.0 🎉 |
Motivation
fastConfirmation.test.tsperf bench was flaky on CI.Summary
The per-iteration
everyoneVotes(...)(100K–1M validator loop) andupdateHead()inbeforeEachcreated GC pressure inside the µs-scale measurement window. The "flip votes" logic was also a silent no-op —addLatestMessagerejects same-epoch votes. SincerunFastConfirmationRulesdoesn't mutatestore,beforeEachis now a pass-through.Locally: ~80K–160K samples/config (was 25–424), ~0.3–5s wall time/config (was 2–17s).
Follow-ups (separate PR)
runFastConfirmationRules.test.tsto match the benchid.findLatestConfirmedDescendantend-to-end usinggeneratePerfTestCachedStateElectrafrom@lodestar/state-transition/test-utils.Test plan
pnpm benchmark:files 'packages/fork-choice/test/perf/forkChoice/fastConfirmation.test.ts'— stable across 3 consecutive runs.check-types+lintclean.🤖 AI-assisted with Claude Code (Opus 4.7).