cdc(ddl): ensure strict ordering for multi-table DDLs after split (#12450)

[ti-chi-bot]

This is an automated cherry-pick of #12450

What problem does this PR solve?

Issue Number: close #12449

What is changed and how it works?

This PR addresses an issue where split DDLs from a multi-table RENAME statement could be executed out of order downstream because they share the same CommitTs and the order of ranging map is non-deterministic.

Check List

Tests

• Unit test
• Integration test
• Manual test

Questions

Will it cause performance regression or break compatibility?

None

Do you need to update user documentation, design documentation or monitoring documentation?

None

Release note

Fix the incorrect execution order of split DDLs generated from a multi-table DDL statement (e.g., RENAME TABLE).

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign july2993 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be approved by the approvers firstly.
2. AFTER it has been approved by approvers, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[gemini-code-assist]

Summary of Changes

Hello @ti-chi-bot, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a crucial fix to ensure strict ordering for Data Definition Language (DDL) operations, particularly those involving multiple tables like RENAME TABLE. Previously, DDL events originating from such multi-table statements could be processed out of sequence downstream due to sharing the same commit timestamp and the non-deterministic nature of map iteration. By incorporating a new sequence number into DDL events and updating the DDL management logic to respect this sequence, the change guarantees that complex DDL operations are applied in their intended order, preventing potential data inconsistencies.

Highlights

• DDL Ordering Fix: Resolves an issue where multi-table DDLs (like RENAME TABLE) could execute out of order downstream due to shared commit timestamps and non-deterministic map iteration.
• Sequence Number Introduction: A new 'Seq' field is added to 'DDLEvent' to explicitly define the order of DDLs with identical commit timestamps.
• Serialization Updates: msgp serialization/deserialization logic for 'DDLEvent' and 'RedoDDLEvent' is updated to support the new 'Seq' field.
• DDL Manager Logic: The 'ddlManager' now uses the 'Seq' field to ensure strict ordering when selecting the next DDL to execute, especially for events with the same 'CommitTs'.
• Integration Test Coverage: New integration tests are added to specifically validate the correct ordering of RENAME TABLE DDL operations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly addresses a potential out-of-order execution issue for DDLs generated from a multi-table statement by introducing a Seq field to DDLEvent. The implementation looks solid, with the new field being properly handled during DDL event creation, serialization, and selection for execution. The refactoring in cdc/model/sink_gen.go to delegate serialization logic from RedoDDLEvent to DDLEvent is a great improvement for maintainability. The addition of an integration test for the multi-table RENAME scenario is also a valuable contribution to ensure the fix is effective. I have one minor suggestion regarding a typo in a comment.

[gemini-code-assist]

There's a small typo here. "splited" should be "split".

Suggested change

	// Only used in the splited DDLEvent generated by a multi-table DDL,
	// Only used in the split DDLEvent generated by a multi-table DDL,

[codecov]

Codecov Report

❌ Patch coverage is 43.18182% with 25 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-8.1@05534b3). Learn more about missing BASE report.

❌ Your patch check has failed because the patch coverage (43.1818%) is below the target coverage (60.0000%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Components	Coverage Δ
cdc	61.5778% <0.0000%> (?)
dm	51.0408% <0.0000%> (?)
engine	63.3950% <0.0000%> (?)

Flag	Coverage Δ
unit	57.3179% <43.1818%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             release-8.1     #12456   +/-   ##
================================================
  Coverage               ?   57.3179%           
================================================
  Files                  ?        854           
  Lines                  ?     126620           
  Branches               ?          0           
================================================
  Hits                   ?      72576           
  Misses                 ?      48590           
  Partials               ?       5454           

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.