feat: OTLP metrics export for low-cardinality business outcomes

[shyim]

Why

Datadog monitors that scan millions of APM spans for SES 451s, Store API 429s, scrape failures, or Sitespeed errors are expensive and noisy. Cheap OTLP outcome counters with tiny enum labels let us alert on business results without high-cardinality span queries.

What

• Extend telemetry setup with an optional OTLP HTTP MeterProvider (OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, falling back to OTEL_EXPORTER_OTLP_ENDPOINT like traces/logs). Empty endpoint keeps metrics disabled (no-op meters).
• Add api/internal/metrics helpers that register instruments after the MeterProvider is installed, and coerce unexpected attribute values to unknown so cardinality stays bounded.
• Instrument natural success/failure points (existing spans unchanged):

Metric	Labels	Outcomes
shopmon.scrape.outcome	outcome	ok, error, auth_error, data_fetch_error
shopmon.store_sync.outcome	outcome	ok, rate_limited, error (no-op freshness skips not counted)
shopmon.sitespeed.outcome	outcome	ok, error, skipped
shopmon.mail.send	outcome	ok, error (no recipient addresses)
shopmon.notify.delivery	outcome, channel	ok/error × email/in_app

Config

# Generic fallback for all signals:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Optional per-signal overrides (same pattern as traces/logs):
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=...

Out of scope

HTTP span renaming (separate PR).

Test plan

• Unit tests for attribute normalization + ManualReader emission
• mise run lint (API + frontend)
• mise run test

  

[greptile-apps]

Greptile Summary

This PR adds optional OTLP HTTP metric export and bounded business-outcome counters for scrapes, store synchronization, sitespeed checks, mail, and notification delivery.

• Adds metrics endpoint configuration with generic OTLP endpoint fallback.
• Installs and shuts down an OTLP MeterProvider alongside tracing and logging providers.
• Introduces normalized low-cardinality metric instruments and tests their emitted attributes.
• Records outcomes at existing success, failure, rate-limit, authentication, data-fetch, and skip branches.

Confidence Score: 5/5

The PR appears safe to merge; no concrete blocking or independently actionable non-blocking defects were identified.

Metrics-only configuration reaches provider registration correctly, disabled telemetry remains a safe no-op, outcome branches avoid duplicate counts, and attribute normalization keeps the new series bounded.

Important Files Changed

Filename	Overview
api/internal/telemetry/telemetry.go	Adds optional OTLP metrics exporter setup, global provider registration, and coordinated shutdown.
api/internal/metrics/metrics.go	Defines thread-safe outcome counters with strict enum normalization to bound cardinality.
api/internal/monitoring/scrape/service.go	Records one scrape outcome for lookup failures, normal completion, authentication failures, data-fetch failures, and returned errors.
api/internal/catalog/sync/service.go	Records store-sync outcomes while intentionally excluding freshness no-ops and distinguishing rate limits.
api/internal/monitoring/sitespeed/service.go	Records sitespeed success, error, and configuration-skip outcomes across current return paths.
api/internal/notify/dispatcher.go	Records per-channel notification delivery outcomes without double-counting failed sends.
api/internal/mail/mail.go	Records SMTP send outcomes without recipient-derived attributes.
api/internal/config/config.go	Adds metrics-specific OTLP endpoint configuration with generic endpoint fallback.
api/telemetry.go	Propagates the configured metrics endpoint into telemetry setup.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Config["OTLP environment configuration"] --> Setup["telemetry.Setup"]
  Setup --> Provider["OTLP MeterProvider"]
  Provider --> Register["metrics.Register"]
  Register --> Counters["Bounded outcome counters"]
  Scrape["Environment scrape"] --> Counters
  StoreSync["Store catalog sync"] --> Counters
  Sitespeed["Sitespeed scrape"] --> Counters
  Mail["Mail send"] --> Counters
  Notify["Notification delivery"] --> Counters
  Counters --> Collector["OTLP collector"]

Loading

_{Reviews (1): Last reviewed commit: "feat: export OTLP metrics for low-cardin..." | Re-trigger Greptile}