Skip to content

add transaction support for publishing #38

@piccione99

Description

@piccione99

if we want to re-use the functionality for topic splitting in wrpkafka, then we need to add the ability to use transactions to wrpkafka too (separate story on wrpkafka) This would involve passing the franz-go client into the publisher as a session, along with a block of wrpkafka records, rather than using the configured publisher in wrpkafka and one record at a time.

This would allow for all or nothing commits, rather than potentially committing records we really want to keep trying.

We want to probably support both sync and async transactions (Produce and ProduceSync under the covers)

The caller, this repository, would then decide based on all the errors it gets back whether to mark the records in this transaction as committed or not. If the errors in the transaction are due to buffer full, network timeout etc... in other words, the network to the kafka end user topic is down, the records in the transaction would not be committed. Otherwise they would.

Note that this could still result in duplicate records because the consumer would potentially be reprocessing records that were already published. However, what we really want to accomplish with this feature is to not mark any records for commit if the network is down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions