sink(ticdc): waiting the asynchronous ddl to synchronize

[wk989898]

What problem does this PR solve?

Issue Number: close #12114 close #12128

What is changed and how it works?

When a DDL operation takes a long time, the connection may be lost, even though the DDL is actually executed. Instead of throwing a failure state, we can query the downstream and wait for the DDL to complete.

Notice:

• The add index DDL operates the same as before and does not require waiting.
• Waiting DDL mechanism only works when downstream is TiDB and the user is at the root.

Check List

Tests

• Unit test
• Integration test
• manual test

mysql> select count(*) from information_schema.ddl_jobs where CREATE_TIME >= now() - interval 10 MINUTE;
+----------+
| count(*) |
+----------+
|     9000 |
+----------+
1 row in set (0.90 sec)
mysql> SELECT JOB_ID, JOB_TYPE, SCHEMA_STATE, SCHEMA_ID, TABLE_ID, STATE, QUERY FROM information_schema.ddl_jobs where CREATE_TIME >= now()- interval 10 MINUTE;
...
...
...
9000 rows in set (0.91 sec)

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Wait for the DDL to synchronize when meeting MySQL Invalid connection error

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[wk989898]

/test pull-cdc-integration-mysql-test

[hongyunyan]

LGTM

[asddongmen]

Please complete the "Check List" part of this PR description. Specifically, remove the items that you didn't do and the redundant items (No code).

[asddongmen]

Please conduct a manual test to ensure there are no performance issues when there are thousands of tables. I'm concerned about the execution time of the query: SELECT JOB_ID, JOB_TYPE, SCHEMA_STATE, SCHEMA_ID, TABLE_ID, STATE, QUERY FROM information_schema.ddl_jobs.

[wk989898]

Please conduct a manual test to ensure there are no performance issues when there are thousands of tables. I'm concerned about the execution time of the query: SELECT JOB_ID, JOB_TYPE, SCHEMA_STATE, SCHEMA_ID, TABLE_ID, STATE, QUERY FROM information_schema.ddl_jobs.

I test it with v9.0.0-beta.1.pre-nightly version, and you could find more information in pingcap/tidb#55748

[wk989898]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, hongyunyan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [asddongmen,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-04-11 12:04:08.077966242 +0000 UTC m=+2430741.762202325: ☑️ agreed by hongyunyan.
• 2025-04-21 07:24:31.983891754 +0000 UTC m=+254015.795682135: ☑️ agreed by asddongmen.

[wk989898]

/test pull-engine-integration-test

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #12466.
But this PR has conflicts, please resolve them!