Skip to content

add detail about interaction with zero injection #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ywwg merged 2 commits into from
Jun 30, 2026
Merged

add detail about interaction with zero injection #87

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f7d0c1a
add detail about interaction with zero injection
ywwg Jun 29, 2026
48648d7
address note
ywwg Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion proposals/0060-st-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
* [PROM-48 (Delta type)](https://github.com/prometheus/proposals/pull/48), [Delta WG](https://docs.google.com/document/d/1G0d_cLHkgrnWhXYG9oXEmjy2qp6GLSX2kxYiurLYUSQ/edit)

> TL;DR: We propose to extend Prometheus TSDB storage sample definition to include an extra int64 that will represent the start timestamp (ST) (previously called created timestamp (CT)) for the cumulative types as well for the future delta temporality ([PROM-48](https://github.com/prometheus/proposals/pull/48)).
> We propose introducing persisting ST logic behind a single flag `st-storage`. Once implemented, we could propose eventual removal of the `created-timestamps-zero-injection` experimental feature.
> We propose introducing persisting ST logic behind a single flag `st-storage`.

## Why

Expand Down Expand Up @@ -164,6 +164,7 @@ ST[2], T[2]

* [MUST] Prometheus can reliably store, query, ingest and export cumulative start timestamp (ST) information (long term plan for [PROM-29](https://github.com/prometheus/proposals/blob/main/proposals/0029-created-timestamp.md#:~:text=For%20those%20reasons%2C%20created%20timestamps%20will%20also%20be%20stored%20as%20metadata%20per%20series%2C%20following%20the%20similar%20logic%20used%20for%20the%20zero%2Dinjection.))
* [MUST] Prometheus can reliably store, query, ingest and export delta start time information. This unblocks [PROM-48 delta proposal](https://github.com/prometheus/proposals/pull/48). Notably adding delta feature later on should ideally not require another complex storage design or implementation.
* [MUST] Co-exist peacefully with pre-existing `created-timestamps-zero-injection` feature.
* [SHOULD] Overhead of the solution should be minimal--initial overhead target set to maximum of 10% CPU, 10% of memory and 15% of disk space.
* [SHOULD] Improve complexity/tech-debt of TSDB on the way if possible.
* [SHOULD] Complexity of consuming STs should be minimal (e.g. low amount of knowledge needed to use it).
Expand Down Expand Up @@ -203,6 +204,16 @@ Notably, given persistence of this feature, similar to example storage, if users

This feature could be considered to be switched to opt-out, only after it's finished (this proposal is fully implemented) stable, provably adopted and when the previous LTS Prometheus version is compatible with this feature.

#### Interaction with `created-timestamps-zero-injection`

The existing zero-injection feature is useful and has been around long enough that some large users are relying on it. The new ST support will be in addition to or an alternative to zero-injection. In order to better differentiate zero-injection from st-per-sample, we propose to rename `created-timestamps-zero-injection` to `start-time-injection-at-scrape`.

@bwplotka bwplotka Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to rename feature flag.

@ywwg ywwg Jun 30, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, someone in the meeting mentioned renaming it so that's why I wrote it down


We also propose that the st setting be be tri-state on a per-series basis. Start Time per Sample support can be either `off`, `scrape`, or `per-sample`. Large installations may have a mix of implementations and there is no compatibility concern with having series have differing settings.

@bwplotka bwplotka Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure we want per series? Can we configure it per series?

I'd say per scrape job is perhaps enough.

either off, scrape, or per-sample

Do we mean scrape job is planned to be either off, stZeroInjection, stPerSample?

@ywwg ywwg Jun 30, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping to add just enough text to indicate the intent, not the implementation, so we could merge the proposal


In the case where a series has both scrape-injected zeroes and st-per-sample, the scrape-injected zeros take precedence.
Comment on lines +211 to +213

@bwplotka bwplotka Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't above already blocking this (tri state)?

Also you can't detect that after a fact and this statement assumes we will?

@ywwg ywwg Jun 30, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refers to when tsdb blocks being queried have both, for some reason. Ideally we shouldn't end up with data like that.


We also intend to graduate zero-injection from an experimental feature, which could involve replacing the existing flag with, for example, a `stZeroInjection` term to the scrape config. The exact implementation is out of scope of this proposal.

@bwplotka bwplotka Jun 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So why we explained before exactly how we planned it? (:

@ywwg ywwg Jun 30, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

People asked for per-series configuration, so that's what I wrote down, but I didn't want to specify exactly where that configuration would go. You mention scrape job, other people seemed to agree with "per series"... that's why I didn't want to continue to slow down this PR with that detail and leave it for implementation


### Proposed ST semantics and validation

See the [official ST semantics in the ecosystem](#background-official-st-semantics). In this section we propose what semantics we recommend (or enforce).
Expand Down
Loading