Skip to content

cdc(ddl): ensure strict ordering for multi-table DDLs after split (#12450) #12457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ti-chi-bot merged 6 commits into from
Dec 11, 2025
Merged

cdc(ddl): ensure strict ordering for multi-table DDLs after split (#12450) #12457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
27b0f5a
cdc(ddl): ensure multi-table ddl execution order after split
wlwilliamx Dec 7, 2025
3f6d012
ddl event generate
wlwilliamx Dec 7, 2025
46691be
add rename tables ddl integration test case
wlwilliamx Dec 7, 2025
696f185
insert some values for integration test
wlwilliamx Dec 8, 2025
3f71132
add some check in integration test
wlwilliamx Dec 9, 2025
bd2b769
fix integration test
wlwilliamx Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cdc/entry/schema_storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,7 @@ func (s *schemaStorage) buildRenameEvents(
tableInfo := model.WrapTableInfo(info.NewSchemaID, newSchemaName,
job.BinlogInfo.FinishedTS, tableInfo)
event.FromJobWithArgs(job, preTableInfo, tableInfo, oldSchemaName, newSchemaName)
event.Seq = uint64(i)
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Assigning event.Seq = uint64(i) within the buildRenameEvents function correctly sets the order for individual DDL events originating from a multi-table RENAME statement. This ensures that the original DDL order is preserved.

ddlEvents = append(ddlEvents, event)
}
return ddlEvents, nil
Expand Down
4 changes: 4 additions & 0 deletions cdc/model/sink.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1056,6 +1056,10 @@ type DDLEvent struct {
// the DDL is executed by the primary cluster.
BDRRole string `msg:"-"`
SQLMode mysql.SQLMode `msg:"-"`
// Seq is used to order the DDLs with the same commit ts
// Only used in the splited DDLEvent generated by a multi-table DDL,
// we need to keep the order of the original multi-table DDL
Seq uint64 `msg:"seq"`
Comment on lines +1059 to +1062
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The addition of the Seq field to DDLEvent is a clear and effective way to ensure deterministic ordering for split DDLs with the same CommitTs. The comments clearly explain its purpose.

}

// FromJob fills the values with DDLEvent from DDL job
Expand Down
162 changes: 43 additions & 119 deletions cdc/model/sink_gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion cdc/owner/ddl_manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@ func (m *ddlManager) getNextDDL() *model.DDLEvent {
delete(m.pendingDDLs, tb)
continue
}
if res == nil || res.CommitTs > ddls[0].CommitTs {
if res == nil || res.CommitTs > ddls[0].CommitTs || (res.CommitTs == ddls[0].CommitTs && res.Seq > ddls[0].Seq) {
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The modification to include (res.CommitTs == ddls[0].CommitTs && res.Seq > ddls[0].Seq) in the DDL selection logic is crucial. This ensures that when multiple DDLs share the same CommitTs, they are processed in the strict order defined by their Seq field, preventing out-of-order execution for multi-table DDLs.

res = ddls[0]
}
}
Expand Down
30 changes: 30 additions & 0 deletions tests/integration_tests/common_1/data/test.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,3 +175,33 @@ VALUES (1),
UPDATE `column_is_null`
SET t = NULL
WHERE id = 1;

-- rename tables DDL should keep the order of events

CREATE TABLE `rename_t3` (
`id` INT PRIMARY KEY,
`val` INT
);

CREATE TABLE `rename_t1` (
`id` INT PRIMARY KEY,
`val` INT
);

CREATE TABLE `rename_t2` (
`id` INT PRIMARY KEY,
`val` INT
);

INSERT INTO `rename_t1` VALUES (1, 1);
INSERT INTO `rename_t2` VALUES (1, 2);
INSERT INTO `rename_t3` VALUES (1, 3);

RENAME TABLE
`rename_t1` TO `rename_t4`,
`rename_t2` TO `rename_t1`,
`rename_t3` TO `rename_t2`;

INSERT INTO `rename_t4` VALUES (2, 4);
INSERT INTO `rename_t1` VALUES (2, 1);
INSERT INTO `rename_t2` VALUES (2, 2);
Comment on lines +179 to +207
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The added integration test case for RENAME TABLE statements is valuable. It directly verifies that the fix correctly handles multi-table DDLs and maintains the expected order of events downstream.

3 changes: 3 additions & 0 deletions tests/integration_tests/common_1/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,9 @@ EOF
# sync_diff can't check non-exist table, so we check expected tables are created in downstream first
check_table_exists common_1.v1 ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
check_table_exists common_1.recover_and_insert ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
check_table_exists common_1.rename_t4 ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
check_table_exists common_1.rename_t1 ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
check_table_exists common_1.rename_t2 ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
Comment on lines +89 to +91
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Adding check_table_exists for the newly renamed tables (rename_t4, rename_t1, rename_t2) in the run.sh script is a necessary step to ensure the integration test correctly validates the DDL execution. This confirms that the tables are created and renamed as expected in the downstream.

check_table_exists common_1.finish_mark ${DOWN_TIDB_HOST} ${DOWN_TIDB_PORT}
check_sync_diff $WORK_DIR $CUR/conf/diff_config.toml

Expand Down
Loading