Skip to content

Add exponential backoff jitter to gRPC channel readiness #85

Description

@dwkang

Background

C++ agent gRPC reconnect behavior currently relies on fixed channel readiness waits. When collectors are down or restarting, many agents can retry on similar intervals and create a thundering herd risk.

The Java agent mitigates this with ExponentialBackoffReconnectJob using:

  • initialInterval: 3000ms
  • multiplier: 1.2
  • randomizationFactor: 0.3
  • maxInterval: 30000ms

Scope

  • Add a reusable exponential backoff helper with jitter.
  • Apply jittered backoff only in the shared gRPC channel readiness path.
  • Use WaitForStateChange during the backoff window instead of sleeping, with 1s slices.
  • Call readyChannel() before metadata unary RPCs.
  • Reset backoff after the channel becomes ready.
  • Add unit tests for delay growth, cap, reset, jitter bounds, and metadata channel-not-ready behavior.

Validation

  • cmake --build --preset debug-cached
  • ctest --preset debug-cached

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions