-
Notifications
You must be signed in to change notification settings - Fork 0
Description
if we want to re-use the functionality for topic splitting in wrpkafka, then we need to add the ability to use transactions to wrpkafka too (separate story on wrpkafka) This would involve passing the franz-go client into the publisher as a session, along with a block of wrpkafka records, rather than using the configured publisher in wrpkafka and one record at a time.
This would allow for all or nothing commits, rather than potentially committing records we really want to keep trying.
We want to probably support both sync and async transactions (Produce and ProduceSync under the covers)
The caller, this repository, would then decide based on all the errors it gets back whether to mark the records in this transaction as committed or not. If the errors in the transaction are due to buffer full, network timeout etc... in other words, the network to the kafka end user topic is down, the records in the transaction would not be committed. Otherwise they would.
Note that this could still result in duplicate records because the consumer would potentially be reprocessing records that were already published. However, what we really want to accomplish with this feature is to not mark any records for commit if the network is down.