Opt-in application-level metrics for the STAC path (/metrics + scraping + dashboard panels)

[lhoupert]

Goal

Add native, application-level Prometheus metrics for the STAC path, as a generic opt-in capability (everything off by default):

• /metrics on stac-fastapi-pgstac and stac-auth-proxy (upstream in those repos)
• opt-in scraping config and RED (rate / error / latency) dashboard panels in this chart

I'm proposing this from the EOEPCA project (See discussion here), where eoAPI is deployed behind a platform monitoring stack, but the work is intended to be generic (no EOEPCA-specific code anywhere).

Current gap

The chart's current monitoring is infrastructure-level. The platform can see that a request was slow, but not which STAC operation (search vs get-item vs list-collections) was slow, and error rates can't be attributed to an operation either.

Claude's analysis of the gap:

This was looked at before: EOEPCA/data-access#41 deliberately chose the ingress-exporter route to avoid forking the upstream apps, and EOEPCA/data-access#134 scoped out possible pathways without landing anything. Two things have changed since:

• stac-fastapi now takes a middlewares parameter, so instrumentation is a plain opt-in middleware — no fork needed.
• There's a merged precedent in the STAC ecosystem: Add metrics instrumentation and optional dependency for prometheus stac-utils/stac-fastapi-elasticsearch-opensearch#622 added prometheus-fastapi-instrumentator as an optional metrics extra with route-template grouping. Add observability to STAC, Raster APIs eoAPI#193 discusses the same need and the cardinality argument for doing it in the app rather than at the ingress.

Propose work

1. stac-utils/stac-fastapi-pgstac: optional metrics extra (prometheus-fastapi-instrumentator, gated by an ENABLE_METRICS env var following the existing settings pattern. Request counters + duration histograms labelled by route template, method and status class. (Raised on OpenTelemetry instrumentation stac-utils/stac-fastapi-pgstac#286, which tracks OpenTelemetry instrumentation, this Prometheus /metrics endpoint would be complementary to OTel tracing, not a replacement.)
2. developmentseed/stac-auth-proxy: same pattern, making sure /metrics is served by the proxy itself and not forwarded upstream.
3. This chart (eoapi-k8s): opt-in values to set ENABLE_METRICS on the stac container, a named service port, scraping enablement, and RED panels for the STAC service in the Grafana dashboard + unit tests coverage and docs changes

Out of scope for this first slice: raster/vector/multidim (can follow the same pattern later), tracing, and per-collection analytics

Are you happy with this proposal @vincentsarago @alukach @hrodmn @gadomski @ciaransweet @pantierra

Questions about the new eoapi-k8s helm chart

A. Prometheus scraping

The current version of the chart bundles scraping through prometheus.io/scrape anotation. EOPECA run the Prometheus operator, in this case we should deploy a servicemonitor object. Should we ship both mechanisms default to off? So a user with the bundled Prometheus flips on annotations an a user with the operator (like EOEPCA) flips on the ServiceMonitor.

B. Values naming

Where does the on/off switch live in values.yaml? per-service toggle (stac.metrics.enabled, like autoscaling) or under the global monitoring block? I'll follow the recent #262 conventions either way.

[gadomski]

Does this need to touch stac-fastapi-pgstac? Can't you just add instrumentation to your app instance in whatever deploy stack you have (e.g. in https://github.com/developmentseed/eoapi-devseed/blob/main/runtimes/eoapi/stac/eoapi/stac/app.py)?

[pantierra]

We would like to avoid having to create and maintain runtimes for adding prometheus. This would be quite a lot of overhead. When prometheus seems to be a common interface anyway. But this is exactly the point we need to get an agreement (or not) on.

[gadomski]

This would be quite a lot of overhead.

The overhead has to live somewhere, no? My instinct is to keep the stac-fastapi-* packages as lean as possible, and allow instrumentation via configuration/hooks; other folks that use stac-fastapi-pgstac might want different instrumentation. But happy to be convinced otherwise.

[lhoupert]

The idea would be to have it as an opt-in option via an env variable e.g. ENABLE_METRICS. Would it be more acceptable? or the main issue is the extra dependency for prometheus?

[pantierra]

Yes, the overhead has to live somewhere, totally fair point. It's just that upstream integration is really lightweight: one additional dependency (prometheus-fastapi-instrumentator) plus a handful of lines of code, whereas custom runtimes bring a lot more surface area to manage (container builds, etc.).

I completely get the motivation to keep the upstream code lean, and I know it's a trade-off either way. I'd love for there to be a cleaner option here, but Prometheus has become something of a quasi-standard, so I do think it's worth adding.

Just to be transparent about our thinking: if you (upstream maintainers) would rather not take this on, that's okay; we'll go ahead and build the custom runtimes for our services instead. Just wanted to suggest that as the alternative path that would make things much easier for us.

[gadomski]

Just to be transparent about our thinking: if you (upstream maintainers) would rather not take this on, that's okay; we'll go ahead and build the custom runtimes for our services instead. Just wanted to suggest that as the alternative path that would make things much easier for us.

I think an issue on https://github.com/stac-utils/stac-fastapi-pgstac/issues would be a good place to discuss those tradeoffs, and would give other non-devseed folks a chance to chime in.

[pantierra]

Thanks, yes, you are right, let's talk over there with all the maintainers. Instead of creating an issue, I did it as a draft PR, so we have an example of what code changes this would imply: stac-utils/stac-fastapi-pgstac#398

[pantierra]

Also created one for developmentseed/stac-auth-proxy#186

[alukach]

I'm generally fine with a /metrics endpoint on stac-auth-proxy as an extras opt-in feature. I do agree that adding it to stac-fastapi-pgstac is a bit of a stickier subject

[pantierra]

I think this is a duplicate. Let's continue conversation in the initial issues over here: developmentseed/eoAPI#193