chore: v1.45.0 release#9674
Open
matthewkeil wants to merge 48 commits into
Open
Conversation
Relevant spec ethereum/consensus-specs#5306 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Motivation
`PeerDiscovery` reads `this.transports` inside `handleDiscoveredPeer()`
(to check transport compatibility), but the constructor assigns
`this.transports` only at the **end** — after it registers the discovery
listeners and, when `network.connectToDiscv5Bootnodes` is enabled,
synchronously processes every bootENR via `onDiscoveredENR()` →
`handleDiscoveredPeer()`.
So with `--network.connectToDiscv5Bootnodes` set, each bootENR is
handled while `this.transports` is still `undefined`, throwing:
```
error: Error onDiscovered - Cannot read properties of undefined (reading 'includes')
at PeerDiscovery.onDiscoveredENR (.../network/peers/discover.js)
```
(from `this.transports.includes("tcp")`). The error is caught and
returns `DiscoveredPeerStatus.error`, so it is **not fatal** — but it
means **none of the bootnodes are dialed at startup**, which hurts peer
bootstrapping exactly when a freshly (re)started node needs peers most.
### Observed in the wild
On ethpandaops `glamsterdam-devnet-6`, restarted Lodestar supernodes
(run with `--network.connectToDiscv5Bootnodes`) repeatedly failed to
re-establish peers after a restart (stuck at 0–2 peers), and the log
spammed this TypeError on every startup.
## Description
Move the `this.transports` initialization above the listener
registration and the bootENR loop, with a comment documenting the
ordering requirement to avoid regression. Pure reordering — no other
behavior change.
## Testing
`discover.test.ts` has no constructor harness today (a regression test
would need to mock `libp2p.services.components.transportManager`, the
discv5 worker, etc.) — happy to add one if preferred. Verified the fix
removes the startup TypeError and restores bootnode dialing on the
affected devnet nodes.
`ProtoArray.onBlock` can potentially reject the block to be imported. So we should update our fork choice store only after the import is complete. See #9565 (review) for more context Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…9476) **Motivation** - as found in #9475 we did not add PayloadEnvelopeInput to the seen cache when we receive "unknown parent" block - this caused some lodestar nodes to stay on "EMPTY" branch of the common ancestor block 23956 forever **Description** issue on `lodestar-erigon-1` `lodestar-nethermind-2` - add it to the cache right after we have it - do the same thing for range sync issue on `lodestar-nethermind-1` - don't remove EMPTY PayloadEnvelopeInput because we may need it to download FULL variant later part of #9475 **AI Assistance Disclosure** created with the help of Claude --------- Co-authored-by: twoeths <twoeths@users.noreply.github.com>
The bid commits to blob KZG commitments even when its payload is orphaned, whose columns are pruned and unservable, so the batch must not be truncated and re-requested (which stalls range sync). Skip such blocks instead; data availability is enforced on envelope processing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…spin loop (#9501) ### Motivation While testing `glamsterdam-devnet-5` via Kurtosis, `lodestar` reliably crashed with a JS heap OOM (exit 134) ~60-100s after startup, whenever the CL was following the chain while its EL was still syncing: ``` <--- Last few GCs ---> [7:0xffffac790000] 65680 ms: Scavenge (interleaved) 8124.2 (8168.0) -> 8101.1 (8172.3) MB, pooled: 0 MB, 14.92 / 0.00 ms (average mu = 0.254, current mu = 0.204) allocation failure; [7:0xffffac790000] 66835 ms: Mark-Compact (reduce) 8114.3 (8173.3) -> 8103.7 (8116.8) MB, pooled: 0 MB, 45.22 / 0.09 ms (+ 1044.7 ms in 209 steps since start of marking, biggest step 6.3 ms, walltime since start of marking 1140 ms) (average mu = 0.257 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace ----- 1: 0x73f17c node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node] 2: 0xbaa8dc [node] 3: 0xbaa9b4 [node] 4: 0xe26c4c [node] 5: 0xe26c7c [node] 6: 0xe26fd4 [node] 7: 0xe36784 [node] 8: 0xe3a3e4 [node] 9: 0x18226c4 [node] ``` Debug logs captured the cause: an infinite spin loop in `BlockInputSync` payload sync, re-processing the same payload root at ~25 iterations **per millisecond** until the 8GB heap (`--max-old-space-size=8192`) was exhausted, while starving the event loop (no status logs, dead REST API for the final ~90s): ``` BlockInputSync.downloadPayload() slot=unknown, root=0x2869…afd0, pendingPayloads=2 BlockInputSync.fetchPayloadInput: successful download … hasPayload=true, hasAllData=true Processed payload from unknown sync slot=43213, root=0x2869…afd0 Added new payload rootHex to BlockInputSync.pendingPayloads root=0x2869…afd0 No unknown block, process ancestor downloaded blocks pendingBlocks=1, ancestorBlocks=1, processedBlocks=0 (repeats ~25x per ms, cycling through every connected peer, until OOM) ``` The loop: 1. A pending block's missing dependency is its parent payload; the payload is not in fork choice yet because its real import job is still in flight (e.g. blocked on regen/EL while the EL syncs). 2. `advancePendingBlock()` re-adds the payload root to `pendingPayloads`. 3. The scheduler "downloads" it instantly from `seenPayloadEnvelopeInputCache`. 4. `processPayload()` → `PayloadEnvelopeProcessor.processPayloadEnvelopeJob()` sees the input already has an `importStatus` and **returns immediately — silent success** — while the actual import has not completed. 5. `BlockInputSync` logs "Processed payload", deletes the `pendingPayloads` entry, re-triggers the search → back to step 1. No backoff, no progress check. ### Description Replace the `importStatus` enum `WeakMap` in `PayloadEnvelopeProcessor` with a `WeakMap` of in-flight import promises. Duplicate callers now await the real import outcome instead of resolving early: - While the import is in flight, deduped callers wait — `BlockInputSync` no longer sees a fake success, so the spin loop cannot form. - On import failure, the rejection propagates to all callers — `BlockInputSync.processPayload` already has correct bounded handling for `PayloadError`s (`BLOCK_NOT_IN_FORK_CHOICE` → re-enqueue block, engine errors → retry on next scheduler pass), it just never saw the errors before. The failed entry is dropped from the map so a later attempt can retry. - On success, subsequent calls still resolve immediately without re-running the import (unchanged behavior). Added unit tests for the processor; the first two fail against the previous implementation. Note: the bug is currently only reachable on gloas networks (devnets), but the code is live on `unstable`. ### AI disclosure This PR was diagnosed and authored with AI assistance (Claude Code), reviewed by me.
As per ethereum/beacon-APIs#608 > Validator clients SHOULD begin submitting signed proposer preferences one epoch before the Gloas fork activates, so the proposer preference caches kept by both beacon nodes and builders are warm at the fork boundary, letting builders prepare valid bids for the first Gloas slots. Also related to ethereum/consensus-specs#4947 but we already do this since we subscribe to new gossip topics 2 epochs before the fork (`SUBSCRIPTIONS_LOOKAHEAD_EPOCHS = 2`).
## Motivation
Lodestar's `--rcConfig` file currently requires nested options to be
written as **dotted keys**:
```yaml
rest.address: "0.0.0.0"
rest.port: 9596
```
This surprised a user who expected the idiomatic nested YAML form
(`rest:` → `address:`). The nested form is nicer, but we don't want to
break existing dotted configs — so this PR supports **both**.
## Description
### Nested maps → dotted keys
Nested maps in the config file are flattened to dotted keys before being
handed to yargs, so both of these produce identical results:
```yaml
# dotted (still works)
rest.address: "0.0.0.0"
rest.port: 9596
# nested (now works too)
rest:
address: "0.0.0.0"
port: 9596
```
**Why flattening:** `cli.ts` sets `parserConfiguration({"dot-notation":
false})` (needed so `.strict()` keeps working with dotted option names),
so options are registered as literal dotted keys (`"rest.address"`) and
yargs `.config()` only matches literal dotted keys from the file —
nested maps are silently ignored today. Flattening the parsed file in
the `rcConfig` read callback bridges the gap without touching parser
config or the strictness guarantees.
- **Arrays are preserved** as values (not flattened by index) so array
options (`rest.namespace`, `bootnodes`, …) keep working.
- **Already-dotted keys pass through unchanged** → fully backward
compatible.
- **Prototype-pollution safe:** `__proto__` is skipped and non-plain
input returns `{}`.
- If an option is given in both nested and dotted form, the value
appearing last in the file wins.
- Applies to every command that shares the global `--rcConfig` option
(beacon / validator / bootnode).
### `.enabled` → bare on/off flag
On/off options are registered as a bare boolean flag (e.g. `--metrics`),
which reads awkwardly as a nested map. After flattening,
`{prefix}.enabled` is rewritten to the bare `{prefix}` flag so the
natural nested form works:
```yaml
metrics:
enabled: true # → the --metrics flag
port: 8008
```
`metrics.enabled: true` → `metrics: true`. If both `{prefix}` and
`{prefix}.enabled` are present, `.enabled` is left as-is so the conflict
surfaces at yargs' strict check instead of being silently swallowed.
## Testing
- **`flattenObject`** — 13 unit tests: nested → dotted, dotted
pass-through, mixed nested+dotted, array preservation,
arrays-of-objects, primitive/null values, deep nesting, top-level
scalars, empty-map elision, nested/dotted collision precedence,
`__proto__` guard, non-plain input, and non-plain object leaves
(`Date`).
- **`translateEnabledKeys`** — 7 unit tests: `.enabled` → bare flag, the
nested `metrics: {enabled, port}` form, multiple prefixes, the
already-set conflict case, no-op when absent, a top-level `enabled` is
not translated, and deeply-prefixed keys.
- `check-types`, `biome lint`, full build, and the existing cli `util` /
`options` / `beacon` unit suites pass — no regressions.
## Docs
Adds a compact example to the `--rcConfig` option showing both forms (a
nested map and a dotted key), so it renders in the generated CLI
reference
([beacon-cli#--rcconfig](https://chainsafe.github.io/lodestar/run/beacon-management/beacon-cli#--rcconfig)).
Because `--rcConfig` is a **global** option, its example renders in
every command's docs, so it uses options common to all of them
(`network` / `logLevel` / `metrics`) rather than a beacon-only option.
Rendering an option `example` that's a config snippet (no shell command)
needed a small docsgen change: `renderOption` renders the `example`
description directly, and `CliOptionDefinition.example` becomes
description-only (`Omit<CliExample, "title" | "command">`). The
`example` is docs-only (yargs ignores it, so `--help` is unchanged) and
the generated `*-cli.md` files are gitignored.
This supersedes the dedicated docs page in #9574 (now closed) — a whole
page was overkill; an inline example is enough.
---
🤖 Generated with AI assistance
---------
Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
) **Motivation** `PersistentCheckpointStateCache.add()` can orphan a persisted checkpoint-state file on disk. **Description** `add()` only carried `persistedKey` forward when the existing cache item was *persisted*. But an in-memory item can also hold a `persistedKey` — `getStateOrReload()` sets `{type: inMemory, state, persistedKey}` when it faults a persisted state back into memory. When `add()` was then called for that same checkpoint, it fell into the `else` branch and dropped the `persistedKey`. Since persisted files are deleted by the *stored* key only (`isPersistedCacheItem(cacheItem) ? cacheItem.value : cacheItem.persistedKey`), dropping it leaves the on-disk copy untracked — so a later prune/finalize never removes it and it stays orphaned on disk until the next restart (`init()` wipes all on-disk keys). Trigger sequence: 1. A checkpoint state is persisted → `{persisted, value: key}`, file written. 2. `getStateOrReload()` reloads it → `{inMemory, state, persistedKey}`. 3. `add()` is called for the same checkpoint → `else` branch → `{inMemory, state}` (key dropped). 4. The epoch is pruned/finalized → cleanup finds no `persistedKey` → the file is left behind. The fix preserves an existing in-memory item's `persistedKey` too: ```ts const persistedKey = cacheItem && (isPersistedCacheItem(cacheItem) ? cacheItem.value : cacheItem.persistedKey); ``` **Testing** Added a regression test in `persistentCheckpointsCache.test.ts` (`persist → reload → re-add → finalize` asserts the on-disk file is removed, not orphaned). Confirmed it **fails on the pre-fix code** (`expected true to be false` — file orphaned) and **passes with the fix**. Full `PersistentCheckpointStateCache` suite (24 tests) green; `biome check` and `tsgo` type-check clean. **AI Assistance Disclosure** - [x] I have read the [contributor guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice) and disclosed my usage of AI below. The root-cause analysis, the fix, and the regression test were developed with AI assistance (Claude Code), then reviewed and verified by me — red→green confirmed, full suite green, lint + type-check clean.
### Motivation Follow-up to #9571. The `submitted` map in `ProposerPreferencesService` tracks which preferences were sent per epoch but was never pruned, so it grows by one entry each epoch for the lifetime of the process. Raised in review by @twoeths. ### Description Drop tracking for past epochs at the start of each task run; only `currentEpoch` and `currentEpoch + 1` are ever processed.
- Update `on_execution_payload` anchor to `on_execution_payload_envelope` (renamed in consensus-specs#5249). - Replace the dangling `is_supporting_vote` reference with `get_supported_node`
**Motivation** Remove `SeenProposerPreferences` cache as `ProposerPreferencesPool` can be used instead. **Description** Removes `SeenProposerPreferences` and uses `ProposerPreferencesPool` instead. Inside `validateGossipProposerPreferences`, preference addition to `SeenProposerPreferences` is completely removed, as if we replaced it with `ProposerPreferencesPool` addition that would make a duplicate. Part of #9379
**Motivation** We want produced blocks to carry Lodestar / EL client info when validators leave enough room in graffiti, without taking over user supplied graffiti. This keeps the default useful for client diversity/debugging while still allowing node operators to opt out. **Description** When a validator supplies graffiti, the beacon node appends the richest client info suffix that fits, always separated by a space: - full EL+CL watermark, e.g. ` BU9b0eLS80c2` - EL+CL client codes, e.g. ` BULS` - CL client code, e.g. ` LS` If none of these fit, the user graffiti is kept unchanged. The feature is enabled by default, use `--graffitiAppend false` to disable it. `--private` suppresses client info as before. --------- Co-authored-by: Lodekeeper <258435968+lodekeeper@users.noreply.github.com> Co-authored-by: matthewkeil <me@matthewkeil.com>
**Motivation** There is a possibility of a race condition occurring between 2 preferences submissions. This can be fixed by adding another check of preference presence for a validator in question after the bls signature verification. **Description** This PR introduces a second preference presence check to prevent the race condition. Also, it moves the preference addition to the ProposerPreferencePool inside the `validateGossipProposerPreferences` - this tie seems cleaner, and it is also a pattern already applied in `validateGossipAggregateAndProof`. Closes #9612
…vote (#9597) - closes ethereum-bounty/lodestar#4 --------- Co-authored-by: Nazar Hussain <nazarhussain@gmail.com>
## Summary The fast confirmation rule (FCR) recomputed the **total active balance** on every `is_one_confirmed` evaluation. That value is invariant per balance source within a single FCR run, but it was read ~3× per evaluation (proposer score, committee weight, adversarial weight) across every block of the epoch-boundary chain walk. For a state-backed balance source each call also **allocates and scans the full validator set** (`getEffectiveBalanceIncrementsZeroInactive`). This memoizes it on the per-run cache, keyed by the balance source's state/balances reference, so the full-set work runs **once per source per run** instead of once per evaluation. ## Impact / how it was found Observed on a mainnet node: at an epoch boundary the FCR blocked the Node.js **event loop for ~11s** (zero gossip/attestation/block processing during the window), seen in logs as ~56 serial `Fast confirmation one-confirmed evaluation` lines ~150–300ms apart. The FCR runs on the chain thread (`onClockSlot → forkChoice.updateTime → runFastConfirmationRules`), so this delayed block import and attestation processing. Root cause: at an epoch boundary the FCR evaluates dozens of blocks (the `isConfirmedChainSafe` chain walk **plus** the descendant search), and each evaluation recomputed the full-validator-set total active balance several times. ## Perf analysis **Deterministic call-count (the regression guard)** — number of full-set `getEffectiveBalanceIncrementsZeroInactive` computations for a 32-block chain walk: | | computations | |---|---| | before | **96** (≈3 × per evaluation × 32 blocks) | | after | **1** (memoized per balance source per run) | A 96× reduction for a single 32-block walk; an epoch-boundary run (~56 evaluations) drops from ~168 full-set computations to 1 per source. **Benchmark** (`runFastConfirmationRules`, epoch-boundary path; `test/perf/forkChoice/fastConfirmation.test.ts`): | case | ms/op | |---|---| | vc 100k, bc 96 | 4.6 ms | | vc 600k, bc 96 | 32 ms | | vc 1M, bc 96 | 54 ms | | vc 600k, bc 320 | 34 ms | | vc 100k, bc 96, eq 1000 | 1.27 s (equivocation-path coverage) | (The harness uses a flat-balances stub, so its absolute cost is far below production — where each `getTotalActiveBalance` call also allocates + tree-scans the validator set — but it captures the per-evaluation redundancy the fix removes.) ## Changes - `fastConfirmation/utils.ts`: `getTotalActiveBalance` reads/writes a per-run memo; threaded `cache` through `estimateCommitteeWeightBetweenSlots` / `computeProposerScore`; routed the FFG-justification helpers (`computeHonestFfgSupportForCurrentTarget`, `willNoConflictingCheckpointBeJustified`, `willCurrentTargetBeJustified`) through it too. - `fastConfirmation/types.ts` + `data.ts`: `totalActiveBalanceByKey` cache field + init. - `test/unit/.../fastConfirmation.test.ts`: two deterministic memoization regression tests (chain walk + current-target justification). - `test/perf/.../fastConfirmation.test.ts` + `util.ts`: benchmark exercises the epoch-boundary path; lightweight stub state (reusing the unit-test `makeState`) so the FFG path is reachable. ## Testing - `check-types` ✅, biome lint ✅ - fork-choice unit suite ✅ 184/184 (incl. the 2 new memoization tests) - perf benchmark ✅ 5/5 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
**Motivation**
Implement `POST /eth/v1/beacon/states/{state_id}/builders` added to the
beacon-api spec in ethereum/beacon-APIs#614.
**Description**
- add `getStateBuilders` route, filterable by builder ids (hex encoded
public key or builder index) and statuses (`pending`, `active`,
`exited`)
- add `getBuildersLength()` to the beacon state view to enumerate the
builder registry
- returns 400 if the requested state is prior to gloas, ids that do not
match a known builder are skipped without error as per spec
- builder pubkey lookups are a linear scan over the registry, can
consider a builder pubkey cache if this becomes a performance issue
---------
Co-authored-by: Lodekeeper <258435968+lodekeeper@users.noreply.github.com>
Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
**Motivation** There has been some recent updates to the era spec in e2store-format-specs , This PR make sure the lodestar era implementation adheres to the recent changes. Pls see eth-clients/e2store-format-specs@247cb54 for more details. **AI Assistance Disclosure** - [x] External Contributors: I have read the [contributor guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice) and disclosed my usage of AI below. <!-- Insert any AI assistance disclosure here --> I have not used any AI for the commit , as the commit is fairly straight forward
…oas (#9592) **Motivation** Implements the spec change from ethereum/beacon-APIs#621. Gloas blocks no longer have block number info and there are edge cases where serving the parent block number becomes unnecessarily complex. ELs can get the parent block number from the parent block hash. **Description** - remove `parentBlockNumber` from the gloas `SSEPayloadAttributes` type - only set `parent_block_number` on the `payload_attributes` event pre-gloas, the event is unchanged for those forks - removes the fork choice lookup on the gloas path which could throw if the parent block was not found The `payload_attributes` example in the oapi spec test data is kept pre-gloas until we bump the pinned spec version to a release that includes the updated example.
…ock (#9589) As per ethereum/beacon-APIs#612 (comment) > I am beginning to feel more strongly about using 204 here, on glamsterdam-devnet-6, there are a lot of missed blocks right now and lodestar is emitting this as warnings (our standard request handler), I don't wanna special case this api, and this is quite a frequent error/warning being emitted. Today, this was flagged incorrectly by AI as a potential CL issue to me but it's perfectly normal behavior, so let's use a status code that indicates that, 404 does not. the spec has not settled yet but to avoid noise in the logs we should use 204 for now. Can always revert back to 404 if that's what we decide to go with on the spec side.
Aligns our implementation with ethereum/beacon-APIs#608 - endpoint moved `/eth/v1/beacon/pool/proposer_preferences` --> `/eth/v1/validator/proposer_preferences` - drop the `GET /eth/v1/beacon/pool/proposer_preferences` endpoint, we can reconsider adding a `/lodestar` endpoint for this if it would be useful for debugging or propose it to the spec, but for now I don't think it's that useful to have - list capped at `(MIN_SEED_LOOKAHEAD + 1) * SLOTS_PER_EPOCH` per spec - derive `Eth-Consensus-Version` header and event `version` from proposal slot
…oad (#9588) We currently only cast payload attestations for canonical blocks at slot https://github.com/ChainSafe/lodestar/blob/51a1c44b27069d392acdbd2c09a838da9b336eba/packages/beacon-node/src/api/impl/validator/index.ts#L1113-L1114 other approaches like #9453 do not work currently as outlined > this approach is incorrect, we cannot vote for any block (on any branch), it has to match the head views shuffling/dependent root, otherwise the PTC attestation we produce is invalid as we are not part of the committee on that other branch but our current code has a flaw https://github.com/ChainSafe/lodestar/blob/51a1c44b27069d392acdbd2c09a838da9b336eba/packages/validator/src/services/ptc.ts#L62-L65 `waitForExecutionPayloadAvailableSlot` resolves for **any** payload we receive for `slot` even if it's not on our canonical branch, this would cause us to call `producePayloadAttestationData` too early and cast `payload_present/blob_data_available = false` for the canonical block before that block's own payload had arrived. This PR makes sure that we **only** cast payload attestations early if the `execution_payload_available` is emitted for the payload matching our canonical head root.
**Motivation** - client hardening **Description** - Add `getAllowedTopics` function which enumerates all valid topics ever accepted by this node. This includes all known forks, all attestation subnets, and all data column subnets. - Configure gossipsub `allowedTopics` in gossipsub construction **AI Assistance Disclosure** - claude assistance
Add client code for Caplin and Nimbus-EL per ethereum/execution-apis#844 and ethereum/execution-apis#848
…p validation (#9624) ## Problem `validateExecutionPayloadBid` (Gloas execution payload bid gossip validation) does not implement the spec's `[REJECT] bid.builder_index < len(state.builders)` bounds check. It looks up the builder inside a `try/catch` meant to turn an out-of-range index into a `GossipReject`: ```ts let builder: gloas.Builder; try { builder = state.getBuilder(bid.builderIndex); } catch { throw new ExecutionPayloadBidError(GossipAction.REJECT, {code: BUILDER_NOT_ELIGIBLE, ...}); } if (!isActiveBuilder(builder, state.finalizedCheckpoint.epoch)) { ... } ``` But `state.getBuilder(i)` returns a **lazy** SSZ view (`builders.getReadonly(i)`) that is not bounds-checked eagerly. An out-of-range `builder_index` therefore does **not** throw at `getBuilder` — it throws `LeafNode has no right node` later, on deferred field access inside `isActiveBuilder` (`builder.depositEpoch`), which is outside the `try/catch`. The net effect is an **uncaught error** on the gossip validation path instead of a clean `REJECT`. ## Fix Add an explicit `bid.builder_index < len(state.builders)` bounds check up front using the existing `state.getBuildersLength()`, and drop the now-ineffective `try/catch`. ## Testing Found by running the consensus-specs [#5294](ethereum/consensus-specs#5294) Gloas networking reference tests (spec `v1.7.0-alpha.12`). The `gossip_execution_payload_bid__reject_builder_index_out_of_range` case now returns `REJECT` (previously an uncaught throw), with no regression across the rest of the `gossip_execution_payload_bid` suite (minimal + mainnet presets). > Note: these Gloas gossip validators currently have no unit-test coverage on `unstable`; the reftest suite (wired up in #9372) is their canonical coverage. 🤖 Generated with AI assistance Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com>
…ation (#9627) ## Problem The Gloas execution payload bid gossip validation checked the slot with an exact match and **no** `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance: ```ts const currentSlot = chain.clock.currentSlot; if (bid.slot !== currentSlot && bid.slot !== currentSlot + 1) { throw new ExecutionPayloadBidError(GossipAction.IGNORE, {code: INVALID_SLOT, ...}); } ``` Every other gossip slot check in Lodestar applies the disparity allowance (see `block.ts`, `blobSidecar.ts`, `attestation.ts`). Without it, a bid that arrives within `MAXIMUM_GOSSIP_CLOCK_DISPARITY` of a slot boundary is wrongly `IGNORE`d. ## Fix Implement the spec's `is_within_slot_range(state, bid.slot, 1, current_time_ms)` semantics: the current time must fall within `[start(bid.slot - 1), start(bid.slot + 1)]`, extended by `± MAXIMUM_GOSSIP_CLOCK_DISPARITY` on both ends — i.e. the clock is in slot `bid.slot - 1` (bid.slot is the *next* slot) or `bid.slot` (the *current* slot). Implemented with `chain.clock.msFromSlot(...)` because the disparity boundary is **sub-slot**: the passing/failing reftests are exactly **1 ms** apart at the edge (e.g. `95500` valid vs `95499` ignore; `108500` valid vs `108501` ignore). The slot-granular helpers (`currentSlotWithGossipDisparity` / `slotWithFutureTolerance`) floor to integer slots and produce identical values on either side of that 1 ms boundary, so they cannot express this check — a millisecond-precise comparison is required. ## Testing Found by running the consensus-specs [#5294](ethereum/consensus-specs#5294) Gloas networking reference tests (spec `v1.7.0-alpha.12`). All five `gossip_execution_payload_bid` slot cases now pass (minimal + mainnet): - `valid_slot_at_lower_disparity`, `valid_slot_at_upper_disparity` — now `valid` (were `ignore`) - `ignore_slot_outside_lower_disparity`, `ignore_slot_outside_upper_disparity`, `ignore_slot_too_far_future` — still `ignore` (no regression on the 1 ms-outside boundary) > Note: these Gloas gossip validators currently have no unit-test coverage on `unstable`; the reftest suite (wired up in #9372) is their canonical coverage. 🤖 Generated with AI assistance --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
**Motivation** The [spec](https://github.com/ethereum/consensus-specs/blob/v1.7.0-alpha.10/specs/phase0/p2p-interface.md?plain=1#L486-L498) requires the gossipsub `message-id` to be 20 bytes and clients must reject other sizes. In our previous testing, we found that lodestar already rejected `msgId` longer than 20 bytes, which is good according to the spec. However, IDs shorter than 20 bytes were accepted. Since `Buffer.set()` only overwrites the prefix, the remaining bytes were reused from the previous conversion. This allowed malformed control-message IDs to alias to stale message IDs. For example: - peer A sends or triggers conversion of a valid 20-byte message ID - peer B sends an empty `IWANT` message ID - Lodestar converts the empty ID to the stale previous ID and returns the cached gossip message I run a PoC and confirmed this in the kurtosis testnet. I further investigated its impact, but it seems that there is no security impact since the gossip data is public. So this is just a correctness bug. This PR adds length check in `msgIdToStrFn()` following similar style of [`toRootHex`](https://github.com/ChainSafe/lodestar/blob/4ac6f45830a47b5e895e81b50753a5f3a417f5d3/packages/utils/src/bytes/nodejs.ts#L17-L20). **Description** - Reject non-20 bytes `msgId`. - Add unit test for `msgId` `=`, `>`, `<` 20 bytes . After the code changes, I run the tests and all passed: - pnpm build - pnpm lint - pnpm check-types - pnpm test:unit **AI Assistance Disclosure** - [x] External Contributors: I have read the [contributor guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice) and disclosed my usage of AI below. I asked claude-opus-4-8 to generate the test and I reviewed the added test, I think it suffice. My first lodestar PR, would love some feedback :) Thanks for your attention! Co-authored-by: Nico Flaig <nflaig@protonmail.com>
## Motivation The peer manager performs the network's most safety-critical housekeeping — scoring, pruning, discovery, subnet balancing — but much of it was poorly observable: the heaviest heartbeat phases were hidden inside a single `heartbeat_duration` timer, and the algorithm's *decisions* (score-state changes, prune reasons, relevance rejections) weren't recorded at all. This PR adds metrics and a dashboard to monitor and debug the peer manager in production. The metrics fall into two dimensions: - **Performance** — where heartbeat time goes, normalized per peer. - **Behavioral / health** — the scoring state machine, prune reasons, and the invariants the algorithm is supposed to maintain. (These also make it possible to compare the peer manager against the in-progress native [Zig port](ChainSafe/lodestar-z#308), but the metrics stand on their own as long-lived observability.) ## Metrics added **Performance** | Metric | Notes | |---|---| | `lodestar_peer_manager_prioritize_peers_seconds` | Isolates the core selection/pruning phase (was buried in `heartbeat_duration`) | | `lodestar_peer_score_update_seconds` | Score decay + prune over all peers | | `lodestar_peer_manager_peers_evaluated_count` | Denominator for time-per-peer on `prioritize_peers_seconds` | **Behavioral / health** | Metric | Notes | |---|---| | `lodestar_peer_score_state_transitions_total` `{from,to}` | Healthy/Disconnected/Banned crossings — score-engine churn | | `lodestar_peer_relevance_check_total` `{result}` | `assertPeerRelevance` outcomes by reason (previously only logged) | | `lodestar_peer_manager_peers_pruned_total` `{reason}` | Completes prune accounting — includes top-of-heartbeat bad-score disconnects that `peersRequestedToDisconnect` missed | | `lodestar_peer_manager_peers_per_active_subnet` `{type}` | Min-peers-per-subnet invariant (p10 reveals under-subscription) | | `lodestar_peer_manager_outbound_peers_ratio` | Outbound floor invariant (~10%, #2215) | | `lodestar_peer_manager_score_map_size` | Score store entry count (retention check) | | `lodestar_peer_manager_connected_peers_map_size` | Connected-peers entry count — leak signal vs `libp2p_peers` | ## Dashboard Adds a **Peer manager internals** row to the networking dashboard (9 panels): - **Duration panels** show **p95 per instance** with a table legend (mean/max/last) — per-node latency is the point. - **Breakdown/health panels** are **aggregated across the fleet** by their meaningful label (transition, reason, result, subnet type); drill into a single node via the `Filters` variable. - Every panel carries help text (`ⓘ`) with the metric name and how to read it. Purely additive — no existing panels were modified. <img width="1182" height="758" alt="image" src="https://github.com/user-attachments/assets/ee762f96-0715-4bf2-b517-465f2c9bbaf1" /> ## Notes / scope decisions - **Entries, not bytes** for the map-size gauges: V8 `Map` overhead isn't a meaningful byte comparison; total memory is left to process-level instrumentation. - **Dial success ratio** is a dashboard/PromQL expression over existing `dial_time{status}` / `dial_attempts` — no new series. - All metrics are guarded with optional chaining — no-op when metrics are disabled. ## Testing - `pnpm lint` — clean - `pnpm check-types` (beacon-node) — 0 errors - `pnpm vitest run --project unit test/unit/network/peers/` — 51/51 passing ## AI disclosure This PR was authored with AI assistance (Claude Code). All changes were reviewed by the author. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
**Motivation**
Hello developers,
In our testing, we found that a signature-valid ENR carrying a malformed
`ip` field reaches `enr.getLocationMultiaddr()`, which throws
`InvalidMultiaddrError`. On the discv5 request-timeout / partial-NODES
path this throw is not handled, so it escapes as an uncaught exception
inside the worker thread.
```
attacker answers lodestar FINDNODE with NODES(total=2, enrs=[badENR]) and withholds packet 2
→ discv5 buffers the partial response, request timer fires (rpcFailure)
→ "discovered" event emitted for the buffered ENR
→ onDiscovered(enr) worker.ts:74 (no try/catch)
→ enrRelevance(enr, config, clock) utils.ts:15-21 (no try/catch)
→ enr.getLocationMultiaddr(ENRKey.tcp) throws InvalidMultiaddrError
```
This is the logs of lodestar from local kurtosis testnet. Luckily, the
worker runtime (@chainsafe/threads) catches it, so it is not fatal and
chain health continues.
```
[network] UncaughtException: IPv4 address was incorrect length
InvalidMultiaddrError: IPv4 address was incorrect length
at Object.ip4ToString [as bytesToValue] (file:///usr/app/node_modules/.pnpm/@multiformats+multiaddr@13.0.1/node_modules/@multiformats/multiaddr/dist/src/utils.js:138:15)
at ENR.getLocationMultiaddr (file:///usr/app/node_modules/.pnpm/@ChainSafe+enr@6.0.1/node_modules/@chainsafe/enr/lib/enr.js:283:92)
at ENR.getLocationMultiaddr (file:///usr/app/node_modules/.pnpm/@ChainSafe+enr@6.0.1/node_modules/@chainsafe/enr/lib/enr.js:264:25)
at enrRelevance (file:///usr/app/packages/beacon-node/lib/network/discv5/utils.js:14:30)
at Discv5.onDiscovered (file:///usr/app/packages/beacon-node/lib/network/discv5/worker.js:62:20)
at Discv5.emit (node:events:509:28)
at Discv5.discovered (file:///usr/app/node_modules/.pnpm/@ChainSafe+discv5@12.0.1/node_modules/@chainsafe/discv5/lib/service/service.js:582:18)
at SessionService.rpcFailure (file:///usr/app/node_modules/.pnpm/@ChainSafe+discv5@12.0.1/node_modules/@chainsafe/discv5/lib/service/service.js:885:26)
at SessionService.emit (node:events:509:28)
at SessionService.failRequest (file:///usr/app/node_modules/.pnpm/@ChainSafe+discv5@12.0.1/node_modules/@chainsafe/discv5/lib/session/service.js:640:14) - IPv4 address was incorrect length
```
**Description**
- Wrap the `getLocationMultiaddr` calls in a try/catch so a
malformed/unconvertible ENR location returns `ENRRelevance.no_transport`
- Added a unit test to confirm the fix works
Notes
- I think use `no_transport` to explain can not be converted to a
multiaddr is enough
- Other call site (discover.ts onDiscoveredENR) of
`getLocationMultiaddr` is guarded.
**AI Assistance Disclosure**
- [x] External Contributors: I have read the [contributor
guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice)
and disclosed my usage of AI below.
I used claude-opus-4-7 to add the unit test and I checked it.
…9629) **Motivation** Aligns `producePayloadAttestationData` with [beacon-APIs#626](ethereum/beacon-APIs#626) (merged): `slot` moves from a path parameter to a query parameter, consistent with the other "produce data to sign for a slot" validator endpoints — `produceAttestationData` and `produceSyncCommitteeContribution` — which all take `slot` as a required query param. **Description** - `GET /eth/v1/validator/payload_attestation_data/{slot}` → `GET /eth/v1/validator/payload_attestation_data?slot={slot}` - Only the request wire encoding changes (path → query). The endpoint args, the impl handler (`{slot}`), and the response are unchanged. - The `204`-on-no-block response already landed in #9589. Closes #9628 **oapiSpec conformance** The pinned beacon-APIs spec (`v5.0.0-alpha.2`) predates beacon-APIs#626 and still defines `slot` as a path param, so `producePayloadAttestationData` is temporarily added to `ignoredOperations` in `oapiSpec.test.ts` to avoid a false mismatch. **Remove that ignore once the pin is bumped to a release that includes #626.** 🤖 Generated with AI assistance --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…9633) **Motivation** The Gloas `proposer_preferences` gossip validation `[IGNORE] preferences.proposal_slot has not already passed` compares `proposal_slot` against the exact `current_slot`. A preference whose `proposal_slot` falls within `MAXIMUM_GOSSIP_CLOCK_DISPARITY` of the next-slot boundary is accepted as `valid` when it should be `ignore`d. The boundary is sub-slot (a 1 ms shift flips the outcome), so the integer `current_slot` comparison cannot express it. This is the same class of gap fixed for the `execution_payload_bid` slot check in #9627. **Description** Use `slotWithFutureTolerance(MAXIMUM_GOSSIP_CLOCK_DISPARITY / 1000)` as the `current_slot` reference, matching the existing convention in `verifyPropagationSlotRange` (`attestation.ts`). `MAXIMUM_GOSSIP_CLOCK_DISPARITY` is a global gossip allowance that applies even where the [spec](https://github.com/ethereum/consensus-specs/blob/master/specs/gloas/p2p-interface.md) states the check as a plain `preferences.proposal_slot > current_slot` (as confirmed for the analogous bid check). **Verification** Ran the Gloas networking gossip reference tests from [consensus-specs PR #5294](ethereum/consensus-specs#5294): - `gossip_proposer_preferences__ignore_slot_outside_disparity` → now `ignore` (was `valid`) - `gossip_proposer_preferences__valid_slot_at_disparity_edge` → still `valid` - `gossip_proposer_preferences__valid_at_lookahead_upper_edge` → still `valid` Full `gossip_proposer_preferences`: **13/13 minimal + 13/13 mainnet**, no regressions in the rest of the Gloas networking suite. 🤖 Generated with AI assistance --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Small, low-risk cleanup of the fork choice / fast confirmation
dashboard. No new
metrics or code changes — dashboard JSON only.
## Changes
- **Removed node-specific dead config**: the "updateHead() errors" panel
had a
color override matcher hardcoded to `{group="beta",
instance="contabo-13", …}`.
Reduced to the plain metric name so it is no longer tied to a specific
node.
- **Legend rename**: `Fast Confirmation Duration` → `Duration` on the
Fast
Confirmation Duration avg panel.
## Testing
- Dashboard JSON validates (`python -m json.tool`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
we should follow the spec, thanks Sam for reporting
## Summary Updates `@chainsafe/libp2p-quic` from 2.0.1 to 2.1.1, including the platform-native package lock entries. This consumes the automated 2.1.1 release published after the npm provenance publishing regression was corrected upstream. ## Validation - `pnpm lint` - `pnpm test:unit` - `pnpm vitest run --project unit packages/beacon-node/test/unit/network/libp2p/createNodeJsLibp2p.test.ts` - `pnpm check-types` (fails on existing validator errors outside this diff: `proposerPreferences.ts` and `ptc.test.ts`) > This PR was written with Codex assistance.
Previously in #9565 we added dependent root check, but didn't remove the proposer index check.
## Summary - update `@chainsafe/ssz` from `^1.4.0` to `^1.6.2` across workspace dependencies and the `spec-test-util` peer dependency - align `@chainsafe/persistent-merkle-tree` from `^1.2.5` to `^1.3.0` - align `@chainsafe/as-sha256` from `^1.2.0` to `^1.2.4` - refresh the pnpm lockfile and remove the duplicate older transitive versions ## Impact Lodestar now uses the latest published SSZ release with a single version of its core merkle-tree and SHA-256 dependencies. This avoids duplicate package instances and their associated bundle-size, memory, and cross-version object compatibility risks. This is a dependency-only update with no application code changes. ## Validation - `pnpm lint` - `pnpm check-types` - `pnpm test:unit` (308 files passed, 1 skipped; 3,228 tests passed) > This PR was written primarily by Codex.
**Motivation** - discussion in discord **Description** - Add a line to AGENTS.md with some very basic communication style rules to avoid verbosity.
## Problem A far-behind node (e.g. resumed from a stale DB, hundreds of epochs behind) that is polled for a beacon state via REST can trigger a `regen` that needs to walk back further than `SLOTS_PER_HISTORICAL_ROOT` (8192) slots. Regen can't get a block root more than 8192 slots in the past, so the node **wedges**: the request grinds on the main thread, 0 blocks get imported, and the only recovery is detaching the poller or checkpoint-syncing fresh. ## Fix Guard the shared `getStateResponseWithRegen` entry point with the existing `notWhileSyncing` check, so REST state requests return **503 `NodeIsSyncing`** while the node is behind — before regen is ever reached. This matches how the validator duties endpoints already behave (and other clients). - Extract `notWhileSyncing` + `SYNC_TOLERANCE_EPOCHS` from the validator API into shared `api/impl/utils.ts` and reuse them (no behavior change for validator endpoints). - Thread `sync` into `getStateResponseWithRegen` and its callers (beacon `state`, `proof`, `debug`, `lodestar`, validator duties). It's the single choke point for all REST state-serving endpoints, so guarding it there covers all of them. - `head`/`finalized`/`justified`/`genesis` are exempt from the guard — they resolve to already-available states (no regen), so they keep serving during sync (observability, dashboards, VC checks). - Added unit tests: `getStateResponseWithRegen` throws `NodeIsSyncing` for a regen-capable slot while syncing, and still serves the four always-available state ids. ## Relation to #9634 This is the `notWhileSyncing` alternative to #9634, per @nflaig's review. It's simpler (drops the custom `RegenError`/`SLOT_TOO_FAR_FROM_BLOCK` machinery) and covers all REST state-serving endpoints, not just the far-slot regen case. If this is preferred, #9634 can be closed. ## Behavior note While the node is more than `SYNC_TOLERANCE_EPOCHS` behind, this returns 503 for regen-capable state lookups (arbitrary slots / state roots). `head`/`finalized`/`justified`/`genesis` keep serving (they never trigger regen). This is broader than the far-slot-only guard in #9634 but consistent with the duties endpoints and other clients. 🤖 Generated with AI assistance --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
**Motivation** The pre-gloas builder circuit breaker (`updateBuilderStatus`) counts missed slots via fork choice and disables the external builder flow when the chain is unhealthy. This mechanism does not apply to gloas: - `produceBlockV4` always prefers a builder bid from the bid pool without any chain health check - post-gloas the beacon block is produced by the proposer regardless of bid source, so missed blocks are no longer a useful builder health signal. The failure mode to protect against is unrevealed payloads - the missed slot counting via `getSlotsPresent` also breaks post-gloas since each gloas block inserts multiple protoArray nodes (PENDING/EMPTY/FULL variants) **Description** - add `BuilderCircuitBreaker` which counts blocks with unrevealed payloads (no FULL variant in fork choice) within the fault inspection window and ignores builder bids in `produceBlockV4` while the breaker is active - the fault budget is scaled by blocks present in the window so sparse windows still trigger on high non-reveal rates - the current slot is excluded from the window as its payload reveal may still be in flight - blocks on all branches are counted, this keeps the result independent of which branch is head when the breaker is evaluated and errs toward local building when forks are frequent - the local block is always built, meaning fallback to the self-built block requires no extra work - the breaker will apply uniformly once bids are also accepted via the builder api (#9594), bid value comparison vs. the local payload is handled in #9595 - update breaker status during `prepareNextSlot` and lazily at block production time, the pre-gloas builder status update is now gated to pre-gloas forks - the existing `--builder.faultInspectionWindow` and `--builder.allowedFaults` flags configure the breaker on every fork, defaults are randomized per boot as before - add metrics - `beacon_builder_circuit_breaker_active`: whether the breaker is currently active (1) causing builder bids to be ignored - `beacon_builder_circuit_breaker_faults`: count of blocks with unrevealed payloads in the fault inspection window - `beacon_builder_circuit_breaker_blocks_present`: count of blocks present in the fault inspection window - `beacon_builder_circuit_breaker_payloads_revealed`: count of blocks with revealed payloads in the fault inspection window
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4d9d78d33
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+686
to
+690
| if (node.slot < fromSlot) { | ||
| if (isGloasBlock(node) && node.payloadStatus === PayloadStatus.FULL) { | ||
| continue; | ||
| } | ||
| break; |
There was a problem hiding this comment.
When an old Gloas block is imported after newer blocks, its PENDING node is appended at the end of nodes, so this break exits before examining earlier in-window nodes. This occurs when resolving a current fork whose ancestors are imported after the existing canonical chain. The resulting incomplete counts can incorrectly activate or deactivate the builder circuit breaker, causing block production to select the wrong bid source. Skip below-window nodes instead of assuming PENDING nodes are ordered by slot.
Useful? React with 👍 / 👎.
Member
There was a problem hiding this comment.
this comment might be valid, but it's not relevant for the release and related to gloas circuit breaker, but also there, doesn't seem that urgent to address, will look into it separately
Member
There was a problem hiding this comment.
Contributor
Performance Report✔️ no performance regression detected Full benchmark results
|
## Summary Fixes #9672. `GET /eth/v1/beacon/states/{state_id}/validators` (and `.../validator_balances`) returns `400 id must be array` when a validator client sends more than 20 ids as a comma-separated list (`?id=a,b,c,...`). This regressed in v1.44.0 and breaks interop with clients that use comma-separated `id` encoding, notably the Nimbus validator client. ## Root cause The REST query string is parsed with `qs` using `comma: true` but no explicit `arrayLimit`, so it defaulted to **20**. v1.44.0 bumped `qs` `6.14.1 -> 6.15.2` (#9399), which enforces `arrayLimit` on comma-parsed values: once a comma-separated (or repeated) array exceeds the limit, `qs` converts it into an object (`{0: ..., 1: ...}`), which then fails Ajv `type: "array"` validation and surfaces as `id must be array`. The default cap also applies to the repeated form (`?id=1&id=2&...`). ## Fix Raise `arrayLimit` to `NUMBER_OF_COLUMNS` (128), the largest array any beacon-API query param can carry — a full data-column custody set on `getDebugDataColumnSidecars` (`indices`); the validator `id` lists are capped at 64 by the spec. The route schemas set no per-request `maxItems`, so the qs `arrayLimit` is the only cap on query-array length. This is not a DoS boundary (`parseArrays: false` already disables index-based sparse arrays, and the URL/header size limits bound total input), so a single generous global cap is safe. The `@lodestar/api` test-server helper (`getTestServer`) mirrors this `arrayLimit` so it stays in parity with the real server. 🤖 Generated with AI assistance --------- Co-authored-by: lodekeeper <lodekeeper@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v1.45.0