fix: stop counting expected dependency failures as APM errors - #798
Draft
shyim wants to merge 1 commit into
Draft
Conversation
Classify Store API 429s, tenant shop 401/403, Sitespeed 503s, and soft SMTP failures so otelhttp/gomailer spans keep status codes as attributes with error.expected=true instead of status=Error. Real bugs and final hard failures still Error. Coordinates with shopmon.*.outcome metrics. Co-authored-by: Soner <github@shyim.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Production APM error rate is dominated by expected/transient dependency noise (~130k
status:errorspans/hour), not shopmon bugs:error.type:429onotelhttpclient spansgomailer.sendotelhttp marks every client HTTP status ≥400 as span
Error. That makes Datadog error rate useless for finding real regressions. Outcome counters from #793 (shopmon.*.outcome) already cover cheap alerting for rate limits / auth / mail — this PR aligns span status with the same taxonomy.Taxonomy
Stable attribute:
error.expected=trueOk(not Error)error.expected=true; keephttp.response.status_code/error.type/ SMTP codeErrorerror.expectedDatadog query for real errors:
Find expected degradations (forensics, not alerts):
Prefer metrics for volume alerts (from #793):
shopmon.store_sync.outcome{outcome:rate_limited},shopmon.scrape.outcome{outcome:auth_error},shopmon.sitespeed.outcome,shopmon.mail.send.What
api/internal/otelxhelpers + unit tests (status/error classifiers,RecordExpected/RecordHard/RecordDependency)error.expected=true(span still open until body close)RecordDependencygomailer.sendtracer (replaces otelmw) so soft SMTP codes do not Error;shopmon.mail.sendoutcomes unchangedOut of scope
Test plan
mise run lint(API golangci-lint + frontend oxlint/format/tsc/vitest)mise run test(go test ./internal/...)