Incremental cooperative rebalancing

[a-a-f]

Hello.

I created this (hopefully) small PR not to get it merged but to receive comments about my use case. I hope I don't write in the wrong place. I have an issue and I am exploring three hypotheses to explain it:

• I'm using cppkafka incorrectly.
• Something is not ok with librdkafka.
• Our Kafka broker coordinator is misconfigured and I should check with the staff who administers it.

At my job, as part of a project, we are tasked with adding support for incremental cooperative rebalancing in cppkafka. I built cppkafka from this PR against librdkafka v1.6.1 (yes it's an old version but it's a constraint at my job). I performed the following 2 tests:

• In the 1st test, I had 1 Kafka topic with 18 partitions and 1 consumer in a consumer group. Then another consumer joined this group. In the end, nothing happened, i.e. the 1st consumer went on receiving messages from 18 partitions while the 2nd consumer got no messages (bad).
• In the 2nd test, 2 Kafka topics were set up, each with 18 partitions. I had 2 consumers in a consumer group. After taking one consumer offline, the remaining consumer ended up receiving messages from 36 partitions (good).

There is no such odd behavior when switching to the roundrobin rebalancing strategy.

My question is: am I doing anything wrong with cppkafka?

I hope I clearly and concisely stated the problem I have been encountering and am thankful for any help.

Note: in order to make cppkafka build, I had to copy a couple of librdkafka headers manually, namely rdtypes.h and rdkafka_error.h. Indeed the external header generated by librdkafka after performing ./configure && make && make install does not export enough information.

[mfontanini]

Is this right? The documentation says:

The application must pass the partition list passed to the callback (or a copy of it), even if the list is empty.

[mfontanini]

Sorry, it took me quite a bit to reply!