Skip to content

fix: slot-ticker rename, eventsync log accuracy, ProbeAll error masking - #2896

Open
Roy-blox wants to merge 3 commits into
stagefrom
fix/small-cleanups-2784-2878-2880
Open

fix: slot-ticker rename, eventsync log accuracy, ProbeAll error masking#2896
Roy-blox wants to merge 3 commits into
stagefrom
fix/small-cleanups-2784-2878-2880

Conversation

@Roy-blox

Copy link
Copy Markdown
Contributor

Summary

Three independent small fixes bundled together:

Test plan

  • go test ./operator/slotticker/... ./operator/duties/... ./operator/fee_recipient/... ./doppelganger/... ./ibft/storage/... ./hprobe/... ./cli/operator/... — 167 pass
  • make lint — 0 issues

🤖 Generated with Claude Code

Roy-blox and others added 3 commits June 16, 2026 12:23
Closes #2784.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The fatal log was capturing lastProcessedBlock from the historical sync,
which on a long-running node points the debugger at the wrong block.
Capture the block SyncOngoing was actually started from instead.

Closes #2878.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When one component fails, ProbeAll cancels siblings via its internal
cancel(). Those siblings return context.Canceled from the retry-wait
select, which was being joined into the final error. This caused
errors.Is(err, context.Canceled) to be true even when a real probe
failure was present, masking it for callers.

Only join errors that are not context.Canceled.

Closes #2880.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Roy-blox
Roy-blox requested review from a team as code owners June 16, 2026 09:26
@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.91304% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.3%. Comparing base (82a9f4f) to head (594ed35).

Files with missing lines Patch % Lines
cli/operator/eventsync.go 0.0% 3 Missing ⚠️
operator/dutytracer/collector.go 0.0% 2 Missing ⚠️
doppelganger/doppelganger.go 0.0% 1 Missing ⚠️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@greptile-apps

greptile-apps Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Three independent targeted fixes: SlotTicker.Next() is renamed to Advance() across the interface, implementation, generated mock, and all 18 call sites; eventsync now captures from_block at goroutine-spawn time for an accurate fatal-log field; and hprobe.ProbeAll filters out context.Canceled errors from sibling goroutines so a real probe failure is no longer masked.

  • Rename (NextAdvance): mechanical, complete, and consistent across all callers and mocks — one pair of doc-comment lines in the SlotTicker interface block still reference the old name.
  • eventsync log fix: ongoingFromBlock is now captured before the goroutine is spawned, replacing the stale last_processed_block log field that reflected the historical-sync end block rather than the SyncOngoing start block.
  • ProbeAll error masking fix: sibling goroutines that return context.Canceled because a failing peer called cancel() now exit silently instead of injecting a context.Canceled entry into the joined error, ensuring callers see only genuine probe failures.

Confidence Score: 4/5

Safe to merge; all three fixes are narrow, well-scoped, and leave no behavioral regression in the changed paths.

The rename is mechanical and exhaustive, the eventsync log fix correctly captures the value before goroutine spawn, and the ProbeAll change is consistent with the pre-existing context.Canceled handling already present in probeComponent. The only gap is two doc-comment lines in the SlotTicker interface that still say Next after the rename.

operator/slotticker/slotticker.go — two interface doc-comment lines still reference the old method name Next.

Important Files Changed

Filename Overview
operator/slotticker/slotticker.go Renames Next() → Advance() on the SlotTicker interface and its implementation; two doc-comment references to "Next" were missed in the interface block (lines 19, 27).
operator/slotticker/mocks/slotticker.go Generated mock updated to reflect the Next → Advance rename; looks complete and correct.
cli/operator/eventsync.go Captures ongoingFromBlock before goroutine spawn so the fatal log always reflects the block SyncOngoing actually started from, not the historical-sync end block.
hprobe/prober.go Filters out context.Canceled from sibling goroutines in ProbeAll so a real probe failure is not masked; fix is correct and consistent with existing handling in probeComponent.
operator/duties/attester.go Next() → Advance() rename applied consistently (both initial call and in-loop refresh).
doppelganger/doppelganger.go Single-site Next() → Advance() rename applied correctly.
ibft/storage/store.go Next() → Advance() rename applied correctly in PruneContinuously loop.
protocol/v2/ssv/runner/validator_registration.go Next() → Advance() rename applied correctly in VRSubmitter start loop.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    PA["ProbeAll(ctx)"] --> CC["context.WithCancel(ctx)"]
    CC --> PG["Launch goroutine per component"]
    PG --> PC["probeComponent(ctx, n)"]
    PC -->|success| WD["wg.Done() — silent"]
    PC -->|context.Canceled| NEW["NEW: skip errsCh send\nwg.Done() — silent"]
    PC -->|real error| ES["errsCh ← fmt.Errorf(...)"]
    ES --> CL["cancel() — cancels siblings"]
    CL --> NEW
    WD --> WG["wg.Wait() → close(errsCh)"]
    NEW --> WG
    WG --> RL["range errsCh → errors.Join"]
    RL --> RET["return joined real errors only"]

    style NEW fill:#d4edda,stroke:#28a745
    style CL fill:#fff3cd,stroke:#ffc107
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    PA["ProbeAll(ctx)"] --> CC["context.WithCancel(ctx)"]
    CC --> PG["Launch goroutine per component"]
    PG --> PC["probeComponent(ctx, n)"]
    PC -->|success| WD["wg.Done() — silent"]
    PC -->|context.Canceled| NEW["NEW: skip errsCh send\nwg.Done() — silent"]
    PC -->|real error| ES["errsCh ← fmt.Errorf(...)"]
    ES --> CL["cancel() — cancels siblings"]
    CL --> NEW
    WD --> WG["wg.Wait() → close(errsCh)"]
    NEW --> WG
    WG --> RL["range errsCh → errors.Join"]
    RL --> RET["return joined real errors only"]

    style NEW fill:#d4edda,stroke:#28a745
    style CL fill:#fff3cd,stroke:#ffc107
Loading

Comments Outside Diff (1)

  1. operator/slotticker/slotticker.go, line 17-28 (link)

    P2 Two doc-comment references to the old method name were not updated alongside the rename. Both "calling Next method periodically" and "corresponds to Next" should now say Advance.

Reviews (1): Last reviewed commit: "fix(hprobe): drop sibling context.Cancel..." | Re-trigger Greptile

@Roy-blox

Copy link
Copy Markdown
Contributor Author

QA — Deploy Test Results ✅

Field Value
Environment stage-hoodi
Nodes 97-100 (COMMITTEE-97_98_99_100)
Commit 594ed35b45e8b5b79938698d6edab084e91b43bc
Sensor ssv-stage-fix-small-cleanups-2784-2878-6c35a3-1

1. Commit Verification ✅

All 4 nodes confirmed running the PR head commit:

2026-06-16T09:39:20Z ssv-node-97  starting SSV-Node:untagged-594ed35b45e8b5b79938698d6edab084e91b43bc
2026-06-16T09:39:23Z ssv-node-98  starting SSV-Node:untagged-594ed35b45e8b5b79938698d6edab084e91b43bc
2026-06-16T09:39:25Z ssv-node-99  starting SSV-Node:untagged-594ed35b45e8b5b79938698d6edab084e91b43bc
2026-06-16T09:39:25Z ssv-node-100 starting SSV-Node:untagged-594ed35b45e8b5b79938698d6edab084e91b43bc

2. Duty Execution ✅

1000 duty-start events on node-97 in a 5-minute window — the slot ticker fires on every slot. Duties completing successfully end-to-end:

2026-06-16T09:43:36Z ssv-node-97  ✔️ successfully finished duty processing (no selection proofs)
  runner_role=SYNC_COMMITTEE_CONTRIBUTION slot=3282468
  pre_consensus_time=0.02654s total_consensus_time=0.04282s total_duty_time=0.06753s

3. QBFT Consensus ✅

Round 1 decisions observed across all duty types — no round changes, no timeouts:

runner_role=AGGREGATOR slot=3282471 qbft_round=1 qbft_height=3282471

4. Errors ✅

No application errors or panics. Only benign P2P gossip noise:

ERROR could not publish p2p message topic=62 error="validation ignored"

("validation ignored" = peer rejected a late/duplicate message — expected background noise.)

5. Metrics ✅

Operator Peers CL Sync Consensus avg (COMMITTEE, 5m) Submissions/10m (Aggregator)
97 4 synced 8.48s 6.94
98 5 synced 8.46s 6.91
99 3 synced 11.98s 6.95
100 4 synced 8.48s 7.53

Node-99 shows slightly elevated consensus latency (~12s vs ~8.5s for peers). This is within normal staging variance and predates this PR — no consensus logic was changed.


PR-Specific Checks

SlotTicker.Next() → Advance() (#2784) ✅

Pure rename — verified no behavioral regression:

  • Zero double tick warnings across all 4 nodes — the ticker still advances slot-by-slot without skipping or double-firing
  • 1000 duty-start events / 5 min — scheduler calling Advance() on every slot, full duty pipeline intact

eventsync: stale last_processed_block log (#2878) ✅

Defensive fix to error logging — normal operation path is unchanged. Nodes started cleanly, no failed syncing ongoing registry events observed. The fix only affects what's logged when SyncOngoing hits a fatal error; not testable in healthy infra by design.

hprobe: ProbeAll context.Canceled masking (#2880) ✅

Defensive fix to error handling — normal operation path is unchanged. All health probes passing; no probe-related errors logged. The fix only prevents context.Canceled from siblings being joined into the error when a real probe failure occurs; not testable in healthy infra by design.


Overall: PASS — all nodes healthy, duties executing, no regressions introduced.

@Roy-blox

Copy link
Copy Markdown
Contributor Author

Follow-up on consensus duration figures above

The COMMITTEE consensus avg figures in the metrics table (8–12s) were startup noise — queried at --since 5m exactly when the nodes had just booted. With only a handful of consensus samples in the window, a few slow warmup rounds dominate the rate(sum)/rate(count) average.

Re-queried at --since 30m after nodes stabilised:

COMMITTEE consensus avg (30m window)
PR nodes 97-100 ~0.01–1.26s
Control nodes 1-4 (stage branch) ~0.85–0.93s

No consensus regression. Values are comparable to the stage branch control group. QA conclusion stands: PASS.

@@ -18,11 +18,11 @@ type Provider func() SlotTicker
// Note, the caller is RESPONSIBLE for calling Next method periodically in order for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth updating this — the doc still says calling Next method periodically, but the method is now Advance. Since the whole point of the rename was making the name honest, the comment should follow.

// comes.
Next() <-chan time.Time
Advance() <-chan time.Time
// Slot returns the slot number that corresponds to Next.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: corresponds to Next should now read Advance.

Comment thread hprobe/prober.go
err := p.probeComponent(ctx, n)
if err != nil {
// Relay the error and quit early.
if err != nil && !errors.Is(err, context.Canceled) {

@momosh-ssv momosh-ssv Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth considering a regression test for the masking path this guard fixes?
All current cases in prober_test.go use retryDelay: 0, so the retry-wait select is never where a sibling cancel() lands — the exact scenario this addresses isn't exercised.

A two-component test (one failing fast, one with a non-zero retryDelay) would lock the fix in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants