Skip to content

runner: clarify queue state management (with respect to slot) - #2778

Draft
iurii-ssv wants to merge 15 commits into
stagefrom
runner-clarify-queue-state-management-slot
Draft

runner: clarify queue state management (with respect to slot)#2778
iurii-ssv wants to merge 15 commits into
stagefrom
runner-clarify-queue-state-management-slot

Conversation

@iurii-ssv

@iurii-ssv iurii-ssv commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Based on #2777 (the actual change is in the latest commit).

While working on #2777 it became apparent that Validator/Committee queues (the Queues that hold all duty-related messages, Runners process those messages sequentially) are not using rState.Slot to prioritize messages (specifically this affects partial-sig messages, when older "irrelevant" message can unnecessarily delay the processing of an up-to-date "relevant" one).

This PR should fix that.

Related: F-ssv-186 F-ssv-054 https://github.com/ssvlabs/ssv-node-board/issues/764

@iurii-ssv
iurii-ssv requested review from a team as code owners April 6, 2026 16:44
@iurii-ssv
iurii-ssv marked this pull request as draft April 6, 2026 16:44
@codecov

codecov Bot commented Apr 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.3%. Comparing base (c2fb7be) to head (624971a).
⚠️ Report is 90 commits behind head on stage.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@iurii-ssv iurii-ssv changed the title runner: clarify queue state management slot runner: clarify queue state management (with respect to slot) Apr 6, 2026
@greptile-apps

greptile-apps Bot commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes message prioritization in Validator and Committee queues by ensuring rState.Slot reflects the runner's current duty slot when comparing partial-signature messages. Previously the queue.State passed to the prioritizer had Slot either uninitialized (validator path) or fixed at queue-creation time (committee path), meaning older "irrelevant" partial-sig messages could be prioritized equally with current-slot ones — potentially delaying the processing of up-to-date messages.

Key changes:

  • queue_validator.go: adds rState.Slot = r.GetCurrentDutySlot() inside the consumer loop so it tracks the runner's active slot across slot transitions
  • committee_queue.go: initializes rState.Slot = r.GetCurrentDutySlot() before the loop (correct for committee runners, which handle exactly one slot per runner lifetime); rState.Height, rState.Round and rState.HasRunningInstance are still refreshed each iteration from the live runner state
  • runner.go: removes sync.RWMutex from BaseRunner — concurrent safety is now enforced entirely by the sequential queue consumer goroutine, as documented in the new comment; HasRunningDuty() moves into the Getters interface and GetCurrentDutySlot() is added to support slot-aware prioritization
  • The queueContainer wrapper struct is removed; queues are stored directly as queue.Queue
  • OnTimeoutQBFT gains guards for both "no running duty" and "wrong slot/height", safely dropping stale timeout events from old duties
  • timer.go: moves the HasRunningDuty guard out of the timer goroutine and into OnTimeoutQBFT (queue consumer goroutine), eliminating a previously-racy cross-goroutine state read

Confidence Score: 4/5

Safe to merge; the slot fix is targeted and correct, with one minor nil-safety concern in HasRunningQBFTInstance.

The queue-state slot fix is correct and well-scoped. The mutex removal is justified by the single-goroutine queue consumer model. The minor concern is HasRunningQBFTInstance() directly accessing RunningInstance.State.Decided without guarding against nil, unlike the old IsDecided() method. This relies on a documented by-construction invariant and is unlikely to cause issues in practice.

protocol/v2/ssv/runner/runner.go (HasRunningQBFTInstance nil-safety)

Important Files Changed

Filename Overview
protocol/v2/ssv/runner/runner.go Removes sync.RWMutex, moves HasRunningDuty to Getters, adds GetCurrentDutySlot()/HasStartedQBFTInstance(), inlines baseSetupForNewDuty, adds slot and duty guards to OnTimeoutQBFT
protocol/v2/ssv/validator/committee_queue.go Removes queueContainer wrapper; ConsumeQueue takes queue.Queue and CommitteeRunner directly; rState.Slot initialized once from runner before loop
protocol/v2/ssv/validator/queue_validator.go Core fix: rState.Slot = r.GetCurrentDutySlot() added inside consumer loop to correctly prioritize partial sig messages by current duty slot
protocol/v2/ssv/validator/committee.go Changes Queues map from map[Slot]queueContainer to map[Slot]queue.Queue; gracefully returns nil (instead of error) for timeout events targeting pruned runners
protocol/v2/ssv/runner/runner_state.go Removes nil guard on CurrentDuty in MarshalJSON, enforcing by-construction invariant; now returns error instead of silently producing malformed JSON when CurrentDuty is nil
protocol/v2/ssv/validator/timer.go Removes HasRunningDuty check from timer goroutine (moved to OnTimeoutQBFT on queue consumer goroutine); upgrades log levels for failures from Warn/Debug to Error
protocol/v2/ssv/queue/message_prioritizer.go Documentation-only: clarifies State struct purpose as runner state used in message priority comparison
protocol/v2/ssv/runner/aggregator.go Uses r.beacon and r.network directly instead of GetBeaconNode()/GetNetwork() accessor methods; minor whitespace addition between methods
protocol/v2/ssv/runner/committee.go Reorders GetOperatorSigner and GetDoppelgangerHandler methods for consistency; no behavioral change
protocol/v2/ssv/runner/runner_validations.go Uses HasStartedQBFTInstance() helper instead of direct nil check on State.RunningInstance
protocol/v2/ssv/validator/committee_queue_test.go Major test refactor: removes queueContainer usage, adds newCommitteeRunnerForTest/newCommitteeQueueStateForTest helpers, adds QBFTController to test runner stubs
protocol/v2/ssv/spectest/util.go Replaces manual State/RunningInstance nil checks with HasStartedQBFTInstance() helper

Sequence Diagram

sequenceDiagram
    participant Timer as Round Timer Goroutine
    participant Q as queue.Queue
    participant Consumer as Queue Consumer Goroutine
    participant R as Runner (Committee / Validator)
    participant QBFT as QBFTController

    Note over Consumer: Init: rState.Slot = r.GetCurrentDutySlot()
    loop Each message
        Consumer->>R: HasRunningQBFTInstance()
        R-->>Consumer: bool
        Consumer->>R: GetLastHeight() / GetLastRound()
        R-->>Consumer: height, round
        Consumer->>R: GetCurrentDutySlot() [validator: each iteration]
        R-->>Consumer: slot
        Consumer->>Q: Pop(ctx, prioritizer(rState), filter)
        Q-->>Consumer: SSVMessage
        Consumer->>R: ProcessMessage(msg)
        R-->>Consumer: error / nil
    end

    Timer->>Q: TryPush(timeout event)
    Note over Consumer: Dequeues timeout event
    Consumer->>R: OnTimeoutQBFT(timeoutData)
    Note over R: Guard 1: hasDutyRunning() check
    Note over R: Guard 2: timeoutData.Height == GetCurrentDutySlot()
    R->>QBFT: OnTimeout(ctx, timeoutData)
Loading

Reviews (1): Last reviewed commit: "runner: clarify queue state management (..." | Re-trigger Greptile

Comment on lines 115 to +117
func (b *BaseRunner) HasRunningQBFTInstance() bool {
var runningInstance *instance.Instance
if b.HasRunningDuty() {
runningInstance = b.State.RunningInstance
if runningInstance != nil {
decided, _ := runningInstance.IsDecided()
return !decided
}
}
return false
// Note: RunningInstance.State cannot be nil for existing RunningInstance by construction.
return b.hasDutyRunning() && b.HasStartedQBFTInstance() && !b.State.RunningInstance.State.Decided

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Potential nil panic: RunningInstance.State no longer guarded

The previous implementation delegated to runningInstance.IsDecided(), which guards against a nil inner State:

func (i *Instance) IsDecided() (bool, []byte) {
    if state := i.State; state != nil {
        return state.Decided, state.DecidedValue
    }
    return false, nil
}

The new code accesses b.State.RunningInstance.State.Decided directly. If RunningInstance.State is ever nil (e.g., after deserializing a malformed or legacy snapshot), this will panic instead of returning false gracefully. The existing comment documents the by-construction invariant, but it may be worth keeping the nil guard defensively:

Suggested change
func (b *BaseRunner) HasRunningQBFTInstance() bool {
var runningInstance *instance.Instance
if b.HasRunningDuty() {
runningInstance = b.State.RunningInstance
if runningInstance != nil {
decided, _ := runningInstance.IsDecided()
return !decided
}
}
return false
// Note: RunningInstance.State cannot be nil for existing RunningInstance by construction.
return b.hasDutyRunning() && b.HasStartedQBFTInstance() && !b.State.RunningInstance.State.Decided
func (b *BaseRunner) HasRunningQBFTInstance() bool {
// Note: RunningInstance.State cannot be nil for existing RunningInstance by construction.
return b.hasDutyRunning() && b.HasStartedQBFTInstance() &&
b.State.RunningInstance.State != nil && !b.State.RunningInstance.State.Decided
}

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 30 days if there are no updates. Please comment if you would like to keep it open.

@github-actions github-actions Bot added stale and removed stale labels Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant