What would you like to happen?
Add an "max idle time" option. The system would read messages from a topic and if none are received for a period, say 500ms, stop the consumer altogether, ending the Kafka reading pipeline.
We could also do a maximum number of messages read in a similar fashion.
It would support the usecase of consuming all messages on the topic (for example) once per hour for batch processing by another process.
Issue Priority
Priority: 3
Issue Component
Component: Transforms
What would you like to happen?
Add an "max idle time" option. The system would read messages from a topic and if none are received for a period, say 500ms, stop the consumer altogether, ending the Kafka reading pipeline.
We could also do a maximum number of messages read in a similar fashion.
It would support the usecase of consuming all messages on the topic (for example) once per hour for batch processing by another process.
Issue Priority
Priority: 3
Issue Component
Component: Transforms