diff --git a/proposals/00089-query-cost.md b/proposals/00089-query-cost.md new file mode 100644 index 0000000..d79b751 --- /dev/null +++ b/proposals/00089-query-cost.md @@ -0,0 +1,133 @@ +## Query cost estimation and limits + +* **Owners:** + * Julien Pivotto [@roidelapluie](https://github.com/roidelapluie) + +* **Implementation Status:** Not Implemented. + +* **Related Issues and PRs:** + * `` + +* **Other docs or links:** + +> TL;DR: A single expensive query can hurt a whole Prometheus. We have knobs to cap it (`--query.max-samples`, `--query.timeout`), but no way to tell a user *before* they run a query how expensive it is, and no per-query, reloadable ceilings. This proposal adds a cheap cost *estimate* (series touched, samples scanned) exposed through `/api/v1/query_cost`, reloadable cost *limits* enforced during execution, and an estimated-vs-actual `cost` object on the query response. All behind a `query-cost` feature flag. + +## Why + +Prometheus already protects itself from runaway queries, but the tools are blunt: + +* `--query.max-samples` caps peak samples in memory, not the total scanned. +* `--query.timeout` and `--query.max-concurrency` are process-wide flags, not reloadable and not per-query. +* Nothing tells a user, an autocomplete UI, or an alerting rule author how heavy a query is *before* it runs. + +Operators want ceilings they can tune without a restart. Users and tools (Grafana, dashboards, recording rules) want a cheap way to gauge cost up front so they can refuse or rewrite a query before it lands on the server. + +### Pitfalls of the current solution + +* The existing limits are set at startup. Changing them means a restart. +* They are global. A single tenant or dashboard cannot be given a tighter budget. +* There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. +* `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". + +## Goals + +* Give a cheap, index-based cost *estimate* (series touched, samples scanned) without executing the query. +* Expose the estimate through a new API so clients can gauge cost before running a query. +* Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. +* Let a client *lower* those ceilings per query, never raise them. +* Surface estimated-vs-actual cost on the normal query response, so the estimate can be validated against reality. +* Keep it all opt-in behind a feature flag until the model is proven. + +### Audience + +Operators running shared Prometheus servers, and UI/tooling authors (Grafana, recording rules) that build queries on a user's behalf. + +## Non-Goals + +* Not replacing `--query.max-samples`, `--query.timeout`, or `--query.max-concurrency`. +* Not a billing or chargeback system. The numbers are upper bounds, not exact accounting. +* Not a slow-query log. +* Not per-tenant configuration, as Prometheus is not multi-tenant. Limits are global, with per-query lowering only. +* Not exact cost prediction. The estimate is intentionally cheap and approximate. + +## How + +Three pieces, all gated by `--enable-feature=query-cost`. + +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most `histogramSampleLimit` (50) points per selector to size histograms. + +**2. API.** Two new endpoints estimate cost without executing: + +``` +GET|POST /api/v1/query_cost +GET|POST /api/v1/query_range_cost +``` + +They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and return: + +```json +{ + "estimate": { + "seriesTouched": 42, + "samplesScanned": 5040 + } +} +``` + +The instant and range endpoints also gain a `cost` parameter. When set, the response `data` carries an estimated-vs-actual comparison: + +```json +"cost": { + "estimated": { "seriesTouched": 42, "samplesScanned": 5040 }, + "actual": { "seriesTouched": 40, "samplesScanned": 4980, "peakSamples": 320 } +} +``` + +Note: `cost=1` adds a second index lookup on top of executing the query. + +**3. Limits.** Three reloadable knobs under `global:`: + +```yaml +global: + query_max_series: 0 # 0 = no limit + query_max_samples_scanned: 0 + query_max_duration: 0s +``` + +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`; these can only tighten, never loosen, the operator-set value. The estimate is never used to reject a query — enforcement is always on the real cost. + +### Testing and verification + +* Unit tests for limit enforcement (reject paths) in `promql`. +* Estimation-accuracy tests against known fixtures, plus the `cost` object which lets us compare estimated and actual on every executed query. +* API tests for the new endpoints and the `cost` parameter. +* OpenAPI golden files updated for the new paths and schemas. + +### Migration + +Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. Nothing to migrate. + +### Known unknowns + +* **Estimate accuracy.** `SeriesTouched` over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the scrape interval and that sampled series are representative. Is an upper bound the right contract, or do we want something tighter? +* **Scrape interval.** The estimator uses the global scrape interval; per-target intervals are not modelled. +* **Subqueries.** Only one level of nesting is modelled exactly. +* **Lookback delta.** The storage-only estimator uses the package default, not the engine's configured value. +* **Agent mode.** Estimation is unavailable (no queryable index). +* **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? + +## Alternatives + +1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. +2. **Reject queries based on the estimate.** Rejected: the estimate is an upper bound and can be wrong in both directions. Rejecting on an estimate would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. +3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. +4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. + +## Action Plan + +* [ ] `promql.EstimateCost` and the sample-unit cost model +* [ ] `/api/v1/query_cost` and `/api/v1/query_range_cost` endpoints +* [ ] `cost` parameter on instant/range queries (estimated vs actual) +* [ ] Reloadable `query_max_series` / `query_max_samples_scanned` / `query_max_duration` under `global:` +* [ ] Per-query lowering via `max_series` / `max_samples_scanned` / `max_query_duration` +* [ ] `query-cost` feature flag, docs, OpenAPI spec, UI surfacing