Skip to content

consumer: improve performance of EventsGroup - #5941

Merged
ti-chi-bot[bot] merged 1 commit into
pingcap:masterfrom
wk989898:consumer-0810
Aug 11, 2026
Merged

consumer: improve performance of EventsGroup #5941
ti-chi-bot[bot] merged 1 commit into
pingcap:masterfrom
wk989898:consumer-0810

Conversation

@wk989898

@wk989898 wk989898 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #5937

What is changed and how it works?

  • AppendMessage records whether the message is out of order or not, with the overhead of comparing neighboring CommitTs just once.
  • In normal order, ResolveInto uses sort. Search to find message boundaries with CommitTs <= resolveTs directly, instead of traversing all the search, instead of traversing all messages at once and allocating temporary resolved slices.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Scenario, 16K messages Before After Improvement
    No messages resolved 139.9 µs/op 14.6 ns/op roughly 9,500×
    Half resolved 161.4 µs/op 7.16 µs/op roughly 22.5×
    All resolved 175.6 µs/op 8.06 µs/op roughly 21.8×

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Summary by CodeRabbit

  • Performance

    • Improved event resolution when messages arrive out of timestamp order.
    • Optimized processing of buffered events, including partial and complete resolution scenarios.
  • Tests

    • Added benchmark coverage for ordered, out-of-order, partial, and full event resolution across large message sets.

Signed-off-by: wk989898 <nhsmwk@gmail.com>
@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 10, 2026
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

EventsGroup now tracks out-of-order message insertion, sorts buffered messages when needed, and resolves messages through binary search and in-place removal. A benchmark measures ordered and out-of-order resolution across partial and complete workloads.

Changes

Event resolution

Layer / File(s) Summary
Out-of-order tracking and prefix resolution
cmd/util/event_group.go
EventsGroup detects timestamp disorder. ResolveInto sorts when required, finds the resolvable prefix with binary search, logs the count, resets ordering state, and removes messages in place.
Resolution benchmark coverage
cmd/util/event_group_test.go
A benchmark measures ordered and out-of-order resolution for no-op, partial, and full cases across 16K messages while reusing buffers and suppressing logging.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit sorting timestamps bright,
Finding the prefix just right.
Buffers stay swift,
Resolved rows lift,
And benchmarks measure the flight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive The changes improve EventsGroup resolution, but the provided context does not show that they restore CREATE DATABASE synchronization for Kafka changefeeds [#5937]. Add a regression test or other evidence that confirms CREATE DATABASE events synchronize correctly under the issue's Kafka configuration.
✅ Passed checks (4 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The code and benchmark changes concern EventsGroup resolution and performance, with no clearly unrelated modifications.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly identifies the main change: improving EventsGroup performance in the consumer.
Description check ✅ Passed The description includes the issue, implementation summary, test coverage, and benchmark results, but leaves the questions and release note sections incomplete.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@wk989898

Copy link
Copy Markdown
Collaborator Author

/test all

@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 10, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 11, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-08-10 10:27:34.588932255 +0000 UTC m=+3042240.625027311: ☑️ agreed by 3AceShowHand.
  • 2026-08-11 05:14:43.901564927 +0000 UTC m=+3109869.937659973: ☑️ agreed by asddongmen.

@ti-chi-bot
ti-chi-bot Bot merged commit 3215238 into pingcap:master Aug 11, 2026
40 checks passed
@wk989898 wk989898 added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Aug 11, 2026
@ti-chi-bot

Copy link
Copy Markdown
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #5948.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v8.5.8][New Arch][kafka]Under TPCC workloads, the upstream create db is not synchronized to the downstream.

4 participants