Skip to content

fix: stop counting expected dependency failures as APM errors - #798

Draft
shyim wants to merge 1 commit into
mainfrom
cursor/otel-expected-dependency-errors-f98b
Draft

fix: stop counting expected dependency failures as APM errors#798
shyim wants to merge 1 commit into
mainfrom
cursor/otel-expected-dependency-errors-f98b

Conversation

@shyim

@shyim shyim commented Aug 12, 2026

Copy link
Copy Markdown
Member

Why

Production APM error rate is dominated by expected/transient dependency noise (~130k status:error spans/hour), not shopmon bugs:

Source Signal Volume (approx)
Shopware Store API error.type:429 on otelhttp client spans ~100k+/hour
Tenant shop Admin API 401/403 auth failures hundreds/day
Sitespeed upstream 503 / connection refused (job retries) hundreds/day
SES / SMTP soft 451 (and similar) on gomailer.send intermittent

otelhttp marks every client HTTP status ≥400 as span Error. That makes Datadog error rate useless for finding real regressions. Outcome counters from #793 (shopmon.*.outcome) already cover cheap alerting for rate limits / auth / mail — this PR aligns span status with the same taxonomy.

Taxonomy

Stable attribute: error.expected=true

Kind Span status Attribute Examples
Expected / degraded dependency Ok (not Error) error.expected=true; keep http.response.status_code / error.type / SMTP code 429, 401/403, 502–504, SMTP 421/450/451/452, retryable net errors
Hard failure Error no error.expected unexpected 5xx, panics, non-retryable bugs, final hard job failures

Datadog query for real errors:

env:production status:error -@error.expected:true

Find expected degradations (forensics, not alerts):

env:production @error.expected:true

Prefer metrics for volume alerts (from #793): shopmon.store_sync.outcome{outcome:rate_limited}, shopmon.scrape.outcome{outcome:auth_error}, shopmon.sitespeed.outcome, shopmon.mail.send.

What

  • New api/internal/otelx helpers + unit tests (status/error classifiers, RecordExpected / RecordHard / RecordDependency)
  • HTTP client transport: after otelhttp sets Error for expected statuses, downgrade to Ok + error.expected=true (span still open until body close)
  • Scrape auth + catalog sync rate-limit parent spans use RecordDependency
  • Sitespeed scrape span treats 503 / connection-refused as expected (job retries)
  • Mail: classifying gomailer.send tracer (replaces otelmw) so soft SMTP codes do not Error; shopmon.mail.send outcomes unchanged

Out of scope

  • Changing job retry policy / go-queue middleware
  • Alert monitor edits in Datadog (query above is the contract)

Test plan

  • Unit tests for classifiers + HTTP transport status downgrade
  • mise run lint (API golangci-lint + frontend oxlint/format/tsc/vitest)
  • mise run test (go test ./internal/...)
Open in Web Open in Cursor 

Classify Store API 429s, tenant shop 401/403, Sitespeed 503s, and soft
SMTP failures so otelhttp/gomailer spans keep status codes as attributes
with error.expected=true instead of status=Error. Real bugs and final
hard failures still Error. Coordinates with shopmon.*.outcome metrics.

Co-authored-by: Soner <github@shyim.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants