fix: exit cleanly on Kafka commit ILLEGAL_GENERATION

[anaselmhamdi]

Summary

• When commit fails with ILLEGAL_GENERATION or UNKNOWN_MEMBER_ID, the consumer has been evicted from the group and its partitions are now owned by someone else
• Previously these errors were silently swallowed (commit() logged and returned), so the evicted consumer kept processing messages and duplicating writes against the new partition owner until the next rebalance
• Now the consumer is closed gracefully and the exception is re-raised, so the runner returns ERROR, the pod exits, and Kubernetes restarts it with a fresh consumer that rejoins the group cleanly
• Other Kafka commit errors (network, broker hiccups) keep the existing log-and-continue behavior

Context

Seen repeatedly in production under v0.3.10:

ERROR Kafka exception occurred during commit: KafkaError{code=ILLEGAL_GENERATION,val=22,str="Commit failed: Broker: Specified group generation id is not valid"}
INFO  Gracefully exiting without committing offsets due to Kafka exception

Test plan

• uv run pytest tests/connectors/sources/kafka/ — all 27 tests pass
• uv run ruff check && ruff format --check
• Deploy to one pipeline, trigger a rebalance, confirm: pod exits, k8s restarts, new consumer joins group cleanly

🤖 Generated with Claude Code