fix: slot-ticker rename, eventsync log accuracy, ProbeAll error masking - #2896
fix: slot-ticker rename, eventsync log accuracy, ProbeAll error masking#2896Roy-blox wants to merge 3 commits into
Conversation
Closes #2784. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The fatal log was capturing lastProcessedBlock from the historical sync, which on a long-running node points the debugger at the wrong block. Capture the block SyncOngoing was actually started from instead. Closes #2878. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When one component fails, ProbeAll cancels siblings via its internal cancel(). Those siblings return context.Canceled from the retry-wait select, which was being joined into the final error. This caused errors.Is(err, context.Canceled) to be true even when a real probe failure was present, masking it for callers. Only join errors that are not context.Canceled. Closes #2880. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Greptile SummaryThree independent targeted fixes:
Confidence Score: 4/5Safe to merge; all three fixes are narrow, well-scoped, and leave no behavioral regression in the changed paths. The rename is mechanical and exhaustive, the eventsync log fix correctly captures the value before goroutine spawn, and the ProbeAll change is consistent with the pre-existing context.Canceled handling already present in probeComponent. The only gap is two doc-comment lines in the SlotTicker interface that still say Next after the rename. operator/slotticker/slotticker.go — two interface doc-comment lines still reference the old method name Next. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
PA["ProbeAll(ctx)"] --> CC["context.WithCancel(ctx)"]
CC --> PG["Launch goroutine per component"]
PG --> PC["probeComponent(ctx, n)"]
PC -->|success| WD["wg.Done() — silent"]
PC -->|context.Canceled| NEW["NEW: skip errsCh send\nwg.Done() — silent"]
PC -->|real error| ES["errsCh ← fmt.Errorf(...)"]
ES --> CL["cancel() — cancels siblings"]
CL --> NEW
WD --> WG["wg.Wait() → close(errsCh)"]
NEW --> WG
WG --> RL["range errsCh → errors.Join"]
RL --> RET["return joined real errors only"]
style NEW fill:#d4edda,stroke:#28a745
style CL fill:#fff3cd,stroke:#ffc107
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
PA["ProbeAll(ctx)"] --> CC["context.WithCancel(ctx)"]
CC --> PG["Launch goroutine per component"]
PG --> PC["probeComponent(ctx, n)"]
PC -->|success| WD["wg.Done() — silent"]
PC -->|context.Canceled| NEW["NEW: skip errsCh send\nwg.Done() — silent"]
PC -->|real error| ES["errsCh ← fmt.Errorf(...)"]
ES --> CL["cancel() — cancels siblings"]
CL --> NEW
WD --> WG["wg.Wait() → close(errsCh)"]
NEW --> WG
WG --> RL["range errsCh → errors.Join"]
RL --> RET["return joined real errors only"]
style NEW fill:#d4edda,stroke:#28a745
style CL fill:#fff3cd,stroke:#ffc107
|
QA — Deploy Test Results ✅
1. Commit Verification ✅All 4 nodes confirmed running the PR head commit: 2. Duty Execution ✅1000 duty-start events on node-97 in a 5-minute window — the slot ticker fires on every slot. Duties completing successfully end-to-end: 3. QBFT Consensus ✅Round 1 decisions observed across all duty types — no round changes, no timeouts: 4. Errors ✅No application errors or panics. Only benign P2P gossip noise: ("validation ignored" = peer rejected a late/duplicate message — expected background noise.) 5. Metrics ✅
Node-99 shows slightly elevated consensus latency (~12s vs ~8.5s for peers). This is within normal staging variance and predates this PR — no consensus logic was changed. PR-Specific Checks
|
|
Follow-up on consensus duration figures above The COMMITTEE consensus avg figures in the metrics table (8–12s) were startup noise — queried at Re-queried at
No consensus regression. Values are comparable to the stage branch control group. QA conclusion stands: PASS. |
| @@ -18,11 +18,11 @@ type Provider func() SlotTicker | |||
| // Note, the caller is RESPONSIBLE for calling Next method periodically in order for | |||
There was a problem hiding this comment.
Might be worth updating this — the doc still says calling Next method periodically, but the method is now Advance. Since the whole point of the rename was making the name honest, the comment should follow.
| // comes. | ||
| Next() <-chan time.Time | ||
| Advance() <-chan time.Time | ||
| // Slot returns the slot number that corresponds to Next. |
There was a problem hiding this comment.
Same here: corresponds to Next should now read Advance.
| err := p.probeComponent(ctx, n) | ||
| if err != nil { | ||
| // Relay the error and quit early. | ||
| if err != nil && !errors.Is(err, context.Canceled) { |
There was a problem hiding this comment.
Worth considering a regression test for the masking path this guard fixes?
All current cases in prober_test.go use retryDelay: 0, so the retry-wait select is never where a sibling cancel() lands — the exact scenario this addresses isn't exercised.
A two-component test (one failing fast, one with a non-zero retryDelay) would lock the fix in.
Summary
Three independent small fixes bundled together:
SlotTicker.Next()→Advance()across interface, implementation, generated mock, and all callers (18 files). The old name was misleading — the method advances the ticker as a side-effect;Advancemakes that explicit.eventsync: the fatal log onSyncOngoingfailure was emittinglast_processed_blockcaptured from the historical sync end, not the blockSyncOngoingactually started from. On a long-running node these differ by millions of blocks. Now capturesfrom_blockat goroutine-spawn time.hprobe.ProbeAll: when one component fails it callscancel(), causing sibling goroutines to returncontext.Canceledfrom their retry-wait select. These were beingerrors.Join'd into the result, soerrors.Is(err, context.Canceled)was true even when a real probe failure was present — masking it for callers. Now silently dropscontext.Canceledfrom siblings.Test plan
go test ./operator/slotticker/... ./operator/duties/... ./operator/fee_recipient/... ./doppelganger/... ./ibft/storage/... ./hprobe/... ./cli/operator/...— 167 passmake lint— 0 issues🤖 Generated with Claude Code