observability: restart-resilient counters + proposal-parent labels

[iurii-ssv]

Summary

Phase 1 of ssvlabs/ssv-node-board#1069.

• Restart-resilient sparse counters: new observability/metrics.RegisterSparseCounter + EmitBaselines so PromQL increase()/rate() return correct values after process restart. Applied to 26 sparse counters across 10 packages (the proposal_parent and attestation_data sets in beacon/goclient plus 16 vulnerable counters from a broader audit). Single EmitBaselines call in node.go startup.
• Per-attribute-set baselines via RegisterLabeledBaseline: lets packages pre-emit baselines for the specific label combinations they'll use at runtime, so per-label PromQL queries (e.g. rate({beacon_client="..."})) also work after restart. Applied to the four proposal_parent counters for each configured beacon address.
• operator_id as OTel resource attribute: split observability.Initialize into InitializeLogger (early) and Initialize (late). New WithResourceAttributes option carries late-known facts like operator_id from operatorDataStore. Every metric and trace is now automatically labeled with ssv.operator_id (when the operator is on-chain).
• proposal_parent per-call labels: verifyProposalParent now receives bestClient from getProposalParallel and labels all four counter sites with ssv.beacon.client. Plus two quick wins: slot == 0 underflow guard and Warn → Info for the mismatch log (which was alert noise during normal fork resolution — see Phase-0 evidence below).

Phase-0 evidence behind the design

From 30 days of stage data analyzed in #1069 — 28,975 verifications, 16 mismatches across 16 unique operators (uniform distribution, not concentrated).

The metric is a noisy proxy, not a true-staleness signal. Detailed analysis of the 2026-05-24 fork event showed that the operator whose BN was actually on the stale fork did not fire the mismatch warning — its cache agreed with its (stale) BN. The three operators on the winning fork fired the warning, because their caches still held the earlier head root while their BNs had already updated. So the metric systematically flags operators with the fresher BN view, not the staler one. Acting on this signal directly (tier-1/2/3 refetch logic, SIP-83-style leader selection) would harm fork resolution rather than help it.

Other observations from the same window:

• 100% of mismatches were slot-1 (in-flight build race during fork, not deeply-stale beacons)
• Mismatch rate is 0.055% — even this noisy proxy fires rarely; true proposal-staleness is likely far lower

This PR ships only the observability + cache-improvement subset; no behavior changes to the duty hot path.

Limitations / known follow-ups

• beacon_client label not yet applied to the 8 attestation_data counters — the simpleAttestationData path uses gc.multiClient which doesn't expose which underlying client returned the data. Needs separate design.
• ssvsigner sparse counters (3 of them) not covered — ssvsigner is a separate Go module and would need a small duplicate of the baseline mechanism.
• Per-attribute-set baselines covered only for proposal_parent — other counters labeled at runtime (e.g. by validator role) still need each package to register its own labeled baseline via RegisterLabeledBaseline. The infrastructure is in place; just hasn't been wired up for those counters yet.

Incidental cleanups

• observability/metrics and observability/traces package loggers default to zap.NewNop() so they don't panic on a nil pointer if InitLogger hasn't been called yet (latent bug — instruments are created during package init, before main()).
• operator/validator validators.removed description corrected from "total number of validator errors" copy-paste.

Test plan

• go build ./... — clean
• go vet ./... — clean
• go test on all packages touched (observability, beacon/goclient, network/streams, network/peers/connections, network/discovery, message/validation, eth/executionclient, eth/eventhandler, operator/validator, protocol/v2/ssv/runner, protocol/v2/qbft/instance, cli/operator) — all green
• Broader go test ./operator/... ./protocol/... ./api/... ./exporter/... — all green
• Staging deploy: confirm ssv_cl_proposal_parent_*_total series include ssv.operator_id and ssv.beacon.client labels
• Staging deploy: restart a node, confirm increase(ssv_cl_proposal_parent_verify_total{cluster="stage"}[30d]) returns non-zero (proves unlabeled baseline emission works)
• Staging deploy: restart a node, confirm increase(ssv_cl_proposal_parent_verify_total{beacon_client="..."}[30d]) returns non-zero for each configured beacon (proves per-label baseline emission works)
• Staging deploy: confirm the mismatch log is at info level, not warn
• Staging deploy: confirm metrics emitted before operatorDataStore is ready are silently dropped (no errors in startup logs)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.8%. Comparing base (0916f7c) to head (cc34c77).
⚠️ Report is 1 commits behind head on stage.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

This PR introduces three coordinated observability improvements: restart-resilient sparse counters via a new EmitBaselines registry, operator_id as an OTel resource attribute (available after operatorDataStore is ready), and ssv.beacon.client per-call labels on the four proposal_parent_* counters in verifyProposalParent. A bonus fix guards against uint64 underflow at slot 0.

• Startup split: observability.InitializeLogger now runs early (logger immediately available), while observability.Initialize is deferred until after operatorDataStore is set up, baking operator_id into every emitted metric and trace. Metrics emitted before Initialize are silently dropped (uses the OTel no-op proxy until the real provider is installed), which the code explicitly documents as an acceptable trade-off.
• Sparse-counter baseline mechanism: 26 counters across 10 packages are registered via metrics.RegisterSparseCounter; a single metrics.EmitBaselines(ctx) call after Initialize writes Add(ctx, 0) for each, giving Prometheus a reset anchor so increase()/rate() return correct values after process restarts. Labeled counters (e.g. proposal_parent_* keyed by ssv.beacon.client) receive only the unlabeled baseline — per-attribute-set baselining is called out as future work.

Confidence Score: 4/5

Safe to merge; no changes to the duty hot path, all observability-only.

The core logic is sound: bestClient is always set alongside bestProposal in getProposalParallel, the slot-0 underflow guard is correct, and the OTel proxy pattern means instruments created at package init time properly delegate to the real provider after otel.SetMeterProvider is called. Two counters (messageValidationsIgnored/Rejected, connected/disconnected) are registered as sparse when they likely fire frequently in production — functionally harmless but worth a follow-up on the next audit pass.

message/validation/observability.go and network/peers/connections/observability.go carry counters that may not be truly sparse; cli/operator/node.go has a longer startup window before metrics land (acceptable trade-off, but worth watching on the staging deploy checklist).

Important Files Changed

Filename	Overview
observability/metrics/baseline.go	New file: global sparse-counter registry with RegisterSparseCounter/EmitBaselines; thread-safe with mutex, correctly returns counter unchanged for inline use.
observability/configurator.go	Splits Initialize into InitializeLogger (early) + Initialize (late); cleanly removes logger config from Config struct; localLogger now derived from zap.L() after InitializeLogger sets it up.
cli/operator/node.go	InitializeLogger called early; Initialize deferred until after operatorDataStore is ready; EmitBaselines called after Initialize; observabilityShutdown nil-guarded in defer for fatal-before-Initialize paths.
beacon/goclient/proposer.go	getProposalParallel now returns (proposal, bestClient, error); verifyProposalParent takes beaconClient; slot==0 underflow guard added; mismatch log demoted Warn to Info.
beacon/goclient/observability.go	10 counters wrapped with RegisterSparseCounter; proposal_parent counters will use per-call ssv.beacon.client attribute via verifyProposalParent.
observability/attributes.go	Adds OperatorIDAttribute (resource attr) and BeaconClientAttribute (per-call attr); correctly namespaced per team style.
message/validation/observability.go	messageValidationsIgnoredCounter and messageValidationsRejectedCounter registered as sparse; audit comment acknowledges uncertainty about actual fire rate.
network/peers/connections/observability.go	connectedCounter, disconnectedCounter, filteredCounter registered as sparse; connection events can be frequent in production but functionally harmless.
observability/resources.go	buildResources accepts extraAttrs; base attrs built first, then extras appended; correct OTel merge pattern maintained.
operator/validator/observability.go	validatorsRemovedCounter description corrected; validatorErrorsCounter and routerDroppedMessagesCounter wrapped as sparse.

Sequence Diagram

sequenceDiagram
    participant N as node.go (StartNodeCmd)
    participant OL as observability.InitializeLogger
    participant ODS as operatorDataStore
    participant OI as observability.Initialize
    participant OTel as OTel MeterProvider
    participant EB as metrics.EmitBaselines

    N->>OL: InitializeLogger(level, format, ...)
    OL-->>N: global zap logger set, metrics/traces loggers propagated

    Note over N: Setup: network, beacon client, storage, signing config...
    Note over N: (Metrics emitted here to no-op provider, silently dropped)

    N->>ODS: setupOperatorDataStore(logger, nodeStorage, pubKey)
    ODS-->>N: operatorDataStore (with OperatorID if registered on-chain)

    alt OperatorIDReady()
        N->>N: append WithResourceAttributes(OperatorIDAttribute(id))
    end

    N->>OI: Initialize(ctx, appName, appVersion, options...)
    OI->>OTel: SetMeterProvider(prometheusProvider)
    OI-->>N: shutdownFn

    N->>EB: EmitBaselines(ctx)
    EB->>OTel: counter.Add(ctx, 0) x26 sparse counters
    EB-->>N: baseline zeros written, Prometheus has reset anchor

    Note over N: Node continues startup (network, duties, API...)

Loading

Comments Outside Diff (2)

1. message/validation/observability.go, line 437-452 (link)

   Ignored/rejected counters may not be truly sparse

   messageValidationsIgnoredCounter tracks messages that are valid but not destined for this operator's validators — on a busy network with many operators these can fire thousands of times per slot. messageValidationsRejectedCounter also fires on every rejected message from misbehaving/lagging peers. The "UNKNOWN from audit" comment flags this uncertainty. Registering non-sparse counters here is functionally harmless (a zero baseline at startup is idempotent), but it inflates the sparse-counter list with entries that gain nothing from baselining and adds noise to EmitBaselines output. Worth confirming actual fire rates from staging data before the next baseline audit pass.

2. network/peers/connections/observability.go, line 484-507 (link)

   Connection counters are not sparse on a live node

   connectedCounter and disconnectedCounter fire on every libp2p connection lifecycle event. A production SSV node with dozens of active peers will generate many connection/disconnection events during normal operation (reconnects, peer rotation, etc.). These counters already have dense enough sample coverage that increase()/rate() works without a startup baseline. Registering them is harmless but inconsistent with the intent of the sparse-counter mechanism (which targets counters that would have no samples in a window). Same caveat: no correctness impact, just a classification question for the next audit pass.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "observability: restart-resilient counter..." | Re-trigger Greptile}

[momosh-ssv]

Might be worth a closer look here: EmitBaselines emits Add(ctx, 0) with no attributes, but it seems most of the counters registered as sparse are only ever incremented with a per-call attribute set — connected/disconnected/filtered carry direction, events_failed the event name, rounds_changed role/round/reason, submissions.failed the role, router.messages.dropped the reason, etc.

For those, the zero baseline lands on a phantom {} series that production never touches, so the real labeled series still start cold after a restart — the exact increase()/rate() gap this is meant to close. The proposal_parent counters handle this correctly via RegisterLabeledBaseline.

Maybe we either extend that pattern to the labeled counters, or explicitly de-scope them and tighten the docs so the mechanism's current reach is clear? The genuinely-unlabeled ones (multi_client.method_errors, validator errors, attestation_data.*) do work as-is.

[momosh-ssv]

Worth considering a short note guarding this trade-off: the dropped-metrics window looks sound today (the only counter I found emitting before Initialize is beaconNodeStatusGauge, a re-emitted gauge, so the first dropped sample is harmless).

But it rests on an unasserted invariant — that nothing one-shot fires before Initialize. A one-liner here (or near goclient.New) warning future refactors not to emit one-shot counters before observability is initialized would keep this from silently breaking.

[momosh-ssv]

Might be worth require.NoError on the marshal/unmarshal here.

The deep-clone nicely fixes the shared-singleton mutation, but if UnmarshalSSZ ever errored we'd silently get a zero-valued block and a confusing downstream failure — a guard would localize any future fixture breakage.