Skip to content

fix(ci): wait for the kafka broker before creating topics#13709

Open
AlinsRan wants to merge 1 commit into
masterfrom
test/kafka-logger-partition-flaky
Open

fix(ci): wait for the kafka broker before creating topics#13709
AlinsRan wants to merge 1 commit into
masterfrom
test/kafka-logger-partition-flaky

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

t/plugin/kafka-logger.t TEST 18 fails intermittently in CI, e.g. https://github.com/apache/apisix/actions/runs/29567552409/job/88251295446:

#   Failed test 't/plugin/kafka-logger.t TEST 18: report log to kafka by different partitions - pattern "(?^:partition_id: 1)" should match a line in error.log (req 0)'
#   Failed test 't/plugin/kafka-logger.t TEST 18: report log to kafka by different partitions - pattern "(?^:partition_id: 2)" should match a line in error.log (req 0)'

The cause is in the CI setup, not in the test. Earlier in the same job log:

Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
Created topic test4.

ci/init-plugin-test-service.sh creates the topics right after make ci-env-up returns. When the broker has not registered itself in ZooKeeper yet, creating test2 and test3 fails. The failure is ignored, and since KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true, test3 is then auto-created with the broker default of one partition on first produce.

TEST 18 expects test3 to have 3 partitions and asserts that partition ids 0, 1 and 2 all show up in the error log. With a single-partition topic every message goes to partition 0, so partition_id: 1 and partition_id: 2 never appear — exactly the observed failure. Whether the run is affected depends purely on broker startup timing, which is why it looks flaky.

Fix: retry topic creation until the broker accepts it, and fail the setup loudly instead of starting a test run against a wrong partition layout. ci/init-last-test-service.sh had the same pattern and gets the same treatment.

Verified locally by reproducing the race (broker deliberately started 20s late):

Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
Created topic test3.
real	0m25.180s

and the resulting topic has the expected layout:

Topic: test3	PartitionCount: 3	ReplicationFactor: 1

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jul 20, 2026
Topic creation ran as soon as the containers were up, so when the broker had
not registered in ZooKeeper yet the command failed with "Replication factor: 1
larger than available brokers: 0". The error was ignored, and because
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE is on, the topic was then auto-created with
the default single partition on first produce.

That is what makes t/plugin/kafka-logger.t TEST 18 flaky: it expects test3 to
have 3 partitions and asserts partition ids 0, 1 and 2 are all logged, but every
message goes to partition 0 when the topic silently degrades to one partition.

Retry topic creation until the broker accepts it, and fail the setup instead of
starting a test run against a wrong partition layout.
@AlinsRan
AlinsRan force-pushed the test/kafka-logger-partition-flaky branch from c8e53b1 to 66a43ef Compare July 20, 2026 05:17
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jul 20, 2026
@AlinsRan AlinsRan changed the title test(kafka-logger): fix flaky partition distribution assertion fix(ci): wait for the kafka broker before creating topics Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant