Skip to content

proposal: TSDB Support for Start Timestamp (ST) - #60

Merged
bwplotka merged 20 commits into
mainfrom
ctstorage
Jul 8, 2026
Merged

proposal: TSDB Support for Start Timestamp (ST)#60
bwplotka merged 20 commits into
mainfrom
ctstorage

Conversation

@bwplotka

@bwplotka bwplotka commented Sep 9, 2025

Copy link
Copy Markdown
Member

As discussed in various places (e.g. prometheus/prometheus#17036 (comment) and delta WG) we decided to create a formal proposal on how CT/ST native Prometheus storage could look like and how to make it useful (unblock) delta temporality.

@bwplotka bwplotka changed the title proposal[PROM-60]: Prometheus CT Storage proposal: Prometheus CT Storage Sep 9, 2025
@bwplotka
bwplotka force-pushed the ctstorage branch 4 times, most recently from 4f9eb07 to 03c37e3 Compare September 10, 2025 14:04
@bwplotka bwplotka changed the title proposal: Prometheus CT Storage proposal: Native TSDB Support for Cumulative CT (and Delta (ST) on the way) Sep 10, 2025
@bwplotka bwplotka changed the title proposal: Native TSDB Support for Cumulative CT (and Delta (ST) on the way) proposal: Native TSDB Support for Cumulative CT (and Delta ST on the way) Sep 10, 2025
@bwplotka bwplotka changed the title proposal: Native TSDB Support for Cumulative CT (and Delta ST on the way) proposal: TSDB Support for Cumulative CT (and Delta ST on the way) Sep 10, 2025
@bwplotka
bwplotka force-pushed the ctstorage branch 4 times, most recently from 033d077 to 704dee5 Compare September 11, 2025 13:23
@bwplotka
bwplotka marked this pull request as ready for review September 11, 2025 13:23
Comment thread proposals/0060-ct-storage.md Outdated
@bwplotka

bwplotka commented Sep 12, 2025

Copy link
Copy Markdown
Member Author

FYI: We met for 1h with the delta WG (@ArthurSens @carrieedwards @fionaliao @ywwg) for an initial discussion around this proposal decisions. Thanks for this productive time!

Here are some notes:

  • Bartek introducing proposal details.
  • Fiona adding more context on reset hints proposal: TSDB Support for Start Timestamp (ST) #60 (comment)
  • Fiona sharing suggestions for delta being a "mini-cumulative" story.
  • General alignment on technical decisions.
  • Fiona suggested we double check if start/create : end time is inclusive (left TODO, Otel is inclusive on end time only).
  • Fiona asked around gauge vs counter CT storages differences, especially given gauges in some systems can have CT.
    • Bartek (now): There are none for now, mentioned that in general decisions.
  • On TSDB read (programmatic) interfaces:
    • Owen: Should we discuss here failover algorithms, what can you do? Don't take too much invariants/assumptions, leave room for flexibility e.g lack of CTs
    • Fiona: Is it worth adding any assumptions around CT semantics (e.g. on append)
    • Owen: We should document our assumption, and evolve with reality
    • Arthur: Fiona has a lot of details proposal.
    • Bartek: So far we didn't put ANY requirements on CT on write
    • Bartek (now): I added related section # Proposed CT semantics and validation -- @ywwg could you help me explore how those restriction could look like? And what if we do SHOULD or MUST on those?
  • Artur noticed delta feature is a "SHOULD" goal for CT proposal, he aligned expectations around Grafana interest to deliver delta support.
    • Bartek: Ack. Happy to move to MUST if it helps. I left should to be open minded for extreme cases when solution to cumulative CT and delta ST are better to be entirely different, it would silly to push in single inefficient direction in this proposal. I don't see this being a case now though.
  • Outlining pros & cons for CT -> ST renaming alternative:
    • Fiona/Ar/B: Just stick with one naming
    • Bartek/Fiona: No strong opinion at this point
    • Arthur: I'd vote for CT
    • Owen: There are more future users than previous users, I'd vote for changing to ST.
    • Bartek (now): I added one more argument to keep CT -- CT or ST naming is equally correct/incorrect in this context - Prometheus is cumulative-first system so choosing CT might be fair.

Also updated proposal today with some learnings. Finally proposed a single feature flag for this work (ct-storage).

Still lots of TODOs and anyone is welcome to help!

Signed-off-by: bwplotka <bwplotka@gmail.com>

@ywwg ywwg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this proposal! I added a bunch of comments, some of which are answered by the paragraph right after the comment 😅 . I think my main concern is nailing down the Goals section. This is not at all to question whether we should do the work, just that I think our statement of intent needs to be unequivocable.

Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
@dashpole

Copy link
Copy Markdown
Contributor

This makes a lot of sense to me. I think performance/benchmarks are probably the biggest potential blocker.

Comment thread proposals/0060-ct-storage.md Outdated
@bwplotka

bwplotka commented Oct 6, 2025

Copy link
Copy Markdown
Member Author

Back from some PTO/leave, will try to address comments and finalize interface and TSDB piece soon

@ArthurSens ArthurSens left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm adding just a few stylish/correction comments. I'm midway through the document and still haven't reviewed the proposed interfaces.

Regarding the CT vs ST discussion, I see the point that we'll always have more new users than old ones, but I feel like the CT terminology is so ingrained in the ecosystem that even if we change it now, people will continue to call it CT. Of course this is based on "voices in my head" and there's no real confirmation that this is gonna happen in the future 😅

Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated
Comment thread proposals/0060-ct-storage.md Outdated

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I addressed first pass, thanks for reviews!

bwplotka added 3 commits May 14, 2026 11:56
Incorporate feedback from reviewers: update delta support goal to MUST,
add section explaining unknown start-time resets mapping to 0 at
ingestion, clarify storage flexibility across metric types, fix broken
markdown links, and correct multiple minor typos.

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
@bwplotka

Copy link
Copy Markdown
Member Author

Addressed all comments, plus it's already implemented 🎉

Waiting for formal approvals as discussed in Slack

@ywwg ywwg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Co-authored-by: David Ashpole <dashpole@google.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

@krajorama krajorama left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, I think the unknown start time is the sticking point.

Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
@ywwg

ywwg commented Jun 1, 2026

Copy link
Copy Markdown
Member

Per our meeting today, we will be updating this proposal with some clarifications and additions:

  • We will not be deprecating start time injection
  • But we will rename it and/or change how it is configured to work with st-per-sample
  • We will add detail to the proposal about how those features will interact (e.g., zero-injected st takes precedence)

ywwg added 3 commits June 29, 2026 10:22
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
add detail about interaction with zero injection
@ywwg

ywwg commented Jun 30, 2026

Copy link
Copy Markdown
Member

ok I merged in the last changes talking about zero injection, are we good to merge?

@bwplotka

Copy link
Copy Markdown
Member Author

I found some nits on https://github.com/prometheus/proposals/pull/87/changes#r3501661834 also curious about #60 (comment) - even if we don't plan to decide, let's state so (I might have lost context, need to check what's the latest 🙈 )

@krajorama krajorama left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposing some changes.

Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
Comment thread proposals/0060-st-storage.md Outdated
bwplotka and others added 3 commits July 3, 2026 14:29
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>

@krajorama krajorama left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

bwplotka added 2 commits July 3, 2026 16:59
Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
@ywwg

ywwg commented Jul 7, 2026

Copy link
Copy Markdown
Member

thanks for clarifying the zero-injection updates

@bwplotka
bwplotka merged commit 9cce1e5 into main Jul 8, 2026
2 checks passed
@bwplotka
bwplotka deleted the ctstorage branch July 8, 2026 09:21
@github-project-automation github-project-automation Bot moved this from In progress to Done in Delta Temporality Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants