Skip to content

chore: add more logs and metrics - #2871

Draft
alok-ssv wants to merge 1 commit into
attack-sim-observability-cleanfrom
attack-sim-observability-clean.2
Draft

chore: add more logs and metrics#2871
alok-ssv wants to merge 1 commit into
attack-sim-observability-cleanfrom
attack-sim-observability-clean.2

Conversation

@alok-ssv

@alok-ssv alok-ssv commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Expand observability for pubsub validation, highlighted-peer tracing, connection handshakes, and stream operations so failures and latency can be attributed to the right stage, topic, and peer.

What Changed

  • Added validation-stage tagging to SSV validation errors so failures can be traced to the exact phase that failed.
  • Extended SSV validation observer events with:
    • validation stage
    • pubsub topic
    • payload size
  • Added pubsub validation metrics for:
    • messages received
    • messages validated by result
    • validation duration
    • handler errors after validation succeeds
  • Added highlighted-peer pubsub observability for:
    • reject events
    • drop events
  • Updated the pubsub tracer so batched DROP_RPC events emit one drop datapoint per real topic instead of collapsing into a synthetic multiple bucket.
  • Added p2p connection handshake observability with:
    • handshake outcome
    • handshake reason
    • handshake duration
  • Added stream error metrics with operation and reason labels.
  • Extended tests across validation, connections, streams, topics, and highlighted-peer tracing to cover the new metrics and event fields.

Why

The previous observability was too coarse to explain where failures happened or which traffic was affected. This change makes dashboards, alerts, and incident triage more accurate by distinguishing:

  • validation stage failures versus successful validation followed by handler failure
  • topic-specific pubsub behavior
  • real connection handshake outcomes versus ignored or duplicate connection paths
  • actual drop events versus normal RPC delivery paths
  • per-topic behavior for batched pubsub drops

Validation

  • go test ./network/topics -run TestPsTracerLogDropRPCRecordsEachTopic -count=1
  • go test ./network/topics

@alok-ssv
alok-ssv requested a review from julienharbulot June 3, 2026 09:40
@codecov

codecov Bot commented Jun 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 62.85714% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.1%. Comparing base (18f6006) to head (11b9f12).

Files with missing lines Patch % Lines
network/topics/tracer.go 18.1% 16 Missing and 2 partials ⚠️
message/validation/errors.go 52.1% 9 Missing and 2 partials ⚠️
network/peers/connections/conn_handler.go 80.0% 7 Missing ⚠️
network/streams/controller.go 56.2% 7 Missing ⚠️
network/topics/controller.go 44.4% 5 Missing ⚠️
message/validation/partial_validation.go 42.8% 4 Missing ⚠️
message/validation/validation.go 71.4% 4 Missing ⚠️
network/peers/peertrace/observer.go 84.0% 2 Missing and 2 partials ⚠️
message/validation/consensus_validation.go 66.6% 3 Missing ⚠️
message/validation/stage.go 86.6% 1 Missing and 1 partial ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant