feat(jobs): facet Consumer spans by environment.id in Datadog - #795
Conversation
Annotate go-queue otel Consumer spans for EnvironmentScrape and SitespeedScrape with low-cardinality environment.id (and outcome) so Datadog can facet process spans without opening child Internal spans. Co-authored-by: Soner <github@shyim.de>
Greptile SummaryThe PR adds environment and outcome facets to Consumer spans for environment-scoped jobs. The implementation preserves handler errors and avoids placing high-cardinality payload fields on entry spans, but returned failures should also be recorded on the active span.
Confidence Score: 4/5The PR appears safe to merge, with a non-blocking observability issue around recording returned job errors on the active span. The handler behavior and returned errors are preserved, but the new helper only assigns an outcome attribute rather than recording failures with the repository-required OTel error metadata. Files Needing Attention: api/internal/jobs/consumer_span.go
|
| Filename | Overview |
|---|---|
| api/internal/jobs/consumer_span.go | Adds shared Consumer-span annotations, but returned job errors are not explicitly recorded on the span. |
| api/internal/jobs/handlers.go | Routes environment and sitespeed scrape handlers through the annotation helper while preserving their return values. |
| api/internal/jobs/consumer_span_test.go | Verifies success/error attributes and confirms annotations remain on the parent Consumer span. |
Sequence Diagram
sequenceDiagram
participant Q as go-queue Consumer span
participant H as runEnvironmentJob
participant S as Scrape service
Q->>H: Invoke handler(ctx, environmentID)
H->>Q: Set environment.id
H->>S: Scrape(ctx, environmentID)
S-->>H: error or nil
H->>Q: Set outcome
H-->>Q: Return error or nil
Reviews (1): Last reviewed commit: "feat(jobs): set environment.id on Consum..." | Re-trigger Greptile
| err := run(ctx) | ||
| annotateConsumerOutcome(ctx, err) | ||
| return err |
There was a problem hiding this comment.
When an environment or sitespeed scraper returns an error, runEnvironmentJob sets only outcome="error" before returning it. The background-job observability convention also requires recording the error and setting the span status, otherwise the Consumer span lacks the expected diagnostic error event and status.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Summary
Copy
environment.id(and a tinyoutcomeenum) onto go-queue Consumer entry spans for environment-scoped jobs so Datadog can filter/facetEnvironmentScrape process/SitespeedScrape processwithout opening child Internal spans.Before / after
messaging.message.*onlyenvironment.id,outcome(ok/error)environment.scrapeEnvironmentScrape process, etc.)Approach
go-queue’s otel middleware has no message→attribute mapping. Handlers already receive the Consumer span in
ctx(trace.SpanFromContext). We annotate before scrape services start their Internal child spans:EnvironmentScrape→environment.id+outcomeSitespeedScrape→ same*Cleanup,ShopwareChangelogSync) or high-cardinality payloads (StoreExtensionSync.Names) — left untouchedHelper:
runEnvironmentJobinapi/internal/jobs/consumer_span.go.Test plan
runEnvironmentJob(ok/error outcome; attributes stay on parent Consumer span)mise run lint(local) + CI Lint