[feat] PIP-471: Metadata-Driven Transactions for Scalable Topics#25693
Open
merlimat wants to merge 1 commit intoapache:masterfrom
Open
[feat] PIP-471: Metadata-Driven Transactions for Scalable Topics#25693merlimat wants to merge 1 commit intoapache:masterfrom
merlimat wants to merge 1 commit intoapache:masterfrom
Conversation
Sub-PIP of PIP-460 (Scalable Topics). Defines transactional support for scalable topics by replacing the in-stream commit/abort marker model with metadata-store-backed state. Adds parallel implementations of TransactionBuffer, PendingAckStore, and Transaction Coordinator that write nothing to any data stream, so sealed segments (split/merge) no longer strand in-flight transactions. Reuses dispatcher, client API, and TC wire commands; legacy in-stream-marker components remain unchanged for persistent:// topics.
merlimat
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sub-PIP of PIP-460: Scalable Topics.
Proposes a transaction model for scalable topics that replaces the in-stream commit/abort marker mechanism with metadata-store-backed state, so transactions compose correctly with the segment lifecycle (splits, merges, sealed segments).
The shape of the change:
MetadataTransactionBuffer— newTransactionBufferimplementation forsegment://topics. Writes nothing to the topic's data stream; transactional state lives as records and secondary indexes in the metadata store.MetadataPendingAckStore— newPendingAckStoreimplementation forsegment://subscriptions. No sibling pending-ack topic.MetadataStoreextensions — surface partition-key co-location, sequential keys, and secondary indexes with range-watch. Default backend (Oxia) maps to native primitives; ZooKeeper and others can implement them sub-optimally without affecting correctness.Existing
persistent://topic behavior is unchanged. v4 transactions keep their current implementation byte-for-byte.The PIP is self-contained — full data model, end-to-end flows (publish, end-txn, dispatch, recovery, GC), backward-compat, and rejected alternatives are in the document.
Test plan