diff --git a/docs/plan.md b/docs/plan.md new file mode 100644 index 00000000..2dcf831f --- /dev/null +++ b/docs/plan.md @@ -0,0 +1,207 @@ +# Plan for #882: Support and send Client Reports + +## Summary +Implement best-effort client reports in `sentry-rust` so dropped data is visible in Sentry outcomes, aligned with develop docs: +- [Client Reports](https://develop.sentry.dev/sdk/telemetry/client-reports/) +- [Dealing with network failures](https://develop.sentry.dev/sdk/expected-features/#dealing-with-network-failures) +- [Rate limiting](https://develop.sentry.dev/sdk/expected-features/rate-limiting/) +- [Logs (client reports for dropped logs)](https://develop.sentry.dev/sdk/telemetry/logs/#client-reports) + +This should be implemented incrementally because it touches `sentry-types`, `sentry-core`, and all HTTP transports. + +## Relevant requirements from develop docs + +1. SDKs should send `client_report` envelope items with aggregated `discarded_events` (`reason`, `category`, `quantity`). +2. Reports are best-effort; exactness is not required. +3. SDKs should not send one report per drop; aggregate and flush periodically / piggyback on normal envelopes. +4. HTTP handling: + - `2xx`: success + - `4xx/5xx` (except `429`): discard and record `send_error` + - `413`: discard and record `send_error` + - `429`: discard but **do not** record client report (to avoid double counting) +5. Logs: + - track dropped log count as `log_item` + - track dropped log size as `log_byte` (approximate size is acceptable) +6. Logs buffering spec also requires a hard cap of 1000 queued logs. + - **Scope note for #882:** keep implementation scope narrow to client reports; track the 1000-cap as a separate follow-up issue unless we discover a low-risk one-line fix while implementing. +7. Self-hosted minimum Sentry version for client reports: `21.9.0`. + - Documentation should explicitly recommend setting `send_client_reports = false` on older self-hosted deployments. + +## Current gaps in this repo + +- No protocol model or envelope item support for `client_report`. +- No `send_client_reports`/`sendClientReports` option in `ClientOptions`. +- No central drop accounting. +- Drops currently happen but are unreported at many points: + - event processor / `before_send` / sampling drops + - transaction unsampled drops + - transport queue overflow + - rate-limit filtering + - HTTP/network failures + - `before_send_log` drops +- No `log_byte` accounting. +- Logs queue does not currently enforce the 1000-item hard cap from develop logs docs (tracked as a follow-up to keep #882 focused). + +## Implementation approach + +### 1) Protocol + envelope support (foundation) + +Add `client_report` item support in `sentry-types`: +- Introduce protocol structs for client reports and discarded entries. +- Extend envelope parsing/serialization: + - item header `{"type":"client_report"}` + - payload JSON per spec. +- Add unit tests for roundtrip serialization and parsing. + +### 2) Client report aggregation primitive + +Add a thread-safe aggregation component (e.g. `ClientReports`) with: +- `record(reason, category, quantity)` +- `take_report(now)` to drain aggregated counters into one report payload. +- keying by `(reason, category)`. + +Design target: usable from both `sentry-core` and transport internals. + +### 3) Public option and defaults + +Add to `ClientOptions`: +- `send_client_reports: bool` (default `true`). + +Behavior: +- when `false`, do no bookkeeping and never attach/send client report items. + +### 4) Instrument drop points in `sentry-core` + +Record reasons where data is dropped before transport send: +- `event_processor` +- `before_send` +- `sample_rate` (events + transactions) +- `before_send_log` + +For transactions: include special-case `span` outcome when a transaction is dropped (per client reports spec). + +### 5) Transport-level instrumentation (`sentry` crate) + +Record drop reasons in transport paths: +- `queue_overflow`: when transport channel is full (`try_send` failure) +- `ratelimit_backoff`: when envelope/items are dropped by active rate limits +- `network_error`: request failed and envelope is discarded +- `send_error`: non-2xx HTTP responses except 429; include 413 +- no client report for `429` + +Apply consistently across `reqwest`, `curl`, `ureq`, and `embedded_svc_http`. + +### 6) Attach client reports to outgoing envelopes + +Before sending an envelope, attach pending `client_report` item when available: +- piggyback on normal envelopes to avoid extra requests +- never create recursion (don’t report drops of client report items themselves) +- avoid generating empty report items. + +### 7) Log-specific quantity rules + +For dropped logs, emit both: +- `log_item`: dropped log count +- `log_byte`: dropped bytes (estimated, e.g. approximate serialized size) + +Implement a reusable log-size estimation helper and use it in all log drop paths. + +### 8) Tests + +Add coverage for: +- protocol/envelope serialization of `client_report` +- aggregation semantics and drain behavior +- reason/category recording at core drop points +- transport queue overflow and rate limit filtering reports +- HTTP status mapping (`2xx`, `413`, `429`, other `4xx/5xx`) +- logs: `log_item` + `log_byte` +- `send_client_reports = false` disables reports + +## Proposed sub-issues + +Splitting is recommended. This is a cross-cutting change touching core APIs, transport internals, and protocol. + +### Sub-issue A: Add protocol and envelope support for `client_report` +**Description** +- Add client report payload structs and `EnvelopeItem` variant support in `sentry-types`. +- Add serializer/parser tests. + +**Dependencies** +- None (foundational). + +--- + +### Sub-issue B: Add shared client report aggregator and `send_client_reports` option +**Description** +- Implement thread-safe aggregation primitive. +- Add `ClientOptions::send_client_reports` defaulting to `true`. +- Wire through client initialization. + +**Dependencies** +- Depends on A. + +--- + +### Sub-issue C: Record core-side drops (event/log/transaction pipeline) +**Description** +- Add reporting for `event_processor`, `before_send`, `sample_rate`, `before_send_log`. +- Handle transaction/span special-case accounting. + +**Dependencies** +- Depends on B. + +--- + +### Sub-issue D: Record transport-side drops + attach client reports to envelopes +**Description** +- Add drop recording in transport queue overflow, rate-limit filtering, and HTTP send outcomes. +- Add piggyback attachment of pending client reports to outgoing envelopes. +- Ensure behavior parity across reqwest/curl/ureq/embedded transports. + +**Dependencies** +- Depends on B. +- Can run in parallel with C after B. + +--- + +### Sub-issue E: Log byte/count outcomes and docs update +**Description** +- Ensure every dropped log path records both `log_item` and `log_byte`. +- Update README/docs for `send_client_reports`, including: + - minimum self-hosted version (`21.9.0`) + - explicit recommendation: set `send_client_reports = false` on older self-hosted versions +- Add integration tests covering logs client outcomes. + +**Dependencies** +- Depends on C and D. + +--- + +### Sub-issue F (follow-up): Logs hard queue cap parity +**Description** +- Enforce the hard cap of 1000 queued logs (develop logs spec). +- Record client outcomes for logs dropped due to this cap. + +**Dependencies** +- Can be implemented after E; intentionally out of scope for #882 unless trivial/low-risk. + +--- + +### Sub-issue G (follow-up): Additional category/precision parity +**Description** +- Extend client-report accounting beyond initial #882 scope (e.g. monitor/check-in and attachment-byte precision) as needed for full parity. + +**Dependencies** +- Depends on base client-report infrastructure from A-D. + +## Scope decisions (resolved) + +1. Keep #882 narrow to client-report infrastructure and primary drop paths. +2. Do **not** expand #882 to broader category/precision parity; handle that in follow-ups. +3. In docs, explicitly recommend `send_client_reports = false` for self-hosted versions older than `21.9.0`. + +## Suggested delivery order + +A → B → (C and D in parallel) → E + +Follow-ups: F, then G. diff --git a/pi-session-2026-02-16T16-21-39-591Z_b8b1f2f3-5399-49fa-9c36-a00791f330e8.html b/pi-session-2026-02-16T16-21-39-591Z_b8b1f2f3-5399-49fa-9c36-a00791f330e8.html new file mode 100644 index 00000000..58348259 --- /dev/null +++ b/pi-session-2026-02-16T16-21-39-591Z_b8b1f2f3-5399-49fa-9c36-a00791f330e8.html @@ -0,0 +1,3882 @@ + + + + + + Session Export + + + + + +
+ +
+
+
+
+
+ +
+
+ + + + + + + + + + + + +