dm: add safe-mode upsert DML strategy#12519
Conversation
Add syncer.safe-mode-dml-strategy (replace|upsert). In upsert mode, safe-mode INSERT uses INSERT ON DUPLICATE KEY UPDATE and UPDATE stays UPDATE, and we keep foreign_key_checks enabled to preserve FK cascade semantics. Also add an integration test case for ON UPDATE CASCADE under safe mode.
ti-chi-bot
bot
added
do-not-merge/needs-linked-issue
release-note
|
[FORMAT CHECKER NOTIFICATION] Notice: To remove the 📖 For more info, you can check the "Contribute Code" section in the development guide. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
area/dm
Summary of ChangesHello @GMHDBJD, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the data migration (DM) tool by providing a more flexible and robust DML strategy for safe mode operations. The new 'upsert' strategy addresses limitations of the existing 'replace' strategy, particularly concerning foreign key constraints and cascade actions. By allowing Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new upsert strategy for safe-mode DMLs, which is a valuable addition for handling foreign key constraints with cascading operations. The implementation appears correct and is well-supported by a new integration test. I've included a few suggestions to enhance code readability and maintainability, particularly in the configuration validation and DML generation logic.
| if c.SyncerConfig.SafeModeDMLStrategy != "" && c.SyncerConfig.SafeModeDMLStrategy != "replace" && c.SyncerConfig.SafeModeDMLStrategy != "upsert" { | ||
| log.L().Warn("unknown safe-mode-dml-strategy, fallback to default", | ||
| zap.String("strategy", c.SyncerConfig.SafeModeDMLStrategy), | ||
| zap.String("default", "replace")) | ||
| c.SyncerConfig.SafeModeDMLStrategy = "" | ||
| } |
There was a problem hiding this comment.
For better readability and maintainability, you could use a switch statement to validate SafeModeDMLStrategy. This makes it easier to add more strategies in the future.
switch c.SyncerConfig.SafeModeDMLStrategy {
case "", "replace", "upsert":
// valid
default:
log.L().Warn("unknown safe-mode-dml-strategy, fallback to default",
zap.String("strategy", c.SyncerConfig.SafeModeDMLStrategy),
zap.String("default", "replace"))
c.SyncerConfig.SafeModeDMLStrategy = ""
}| if inst.Syncer.SafeModeDMLStrategy != "" && inst.Syncer.SafeModeDMLStrategy != "replace" && inst.Syncer.SafeModeDMLStrategy != "upsert" { | ||
| log.L().Warn("unknown safe-mode-dml-strategy, fallback to default", | ||
| zap.String("strategy", inst.Syncer.SafeModeDMLStrategy), | ||
| zap.String("default", "replace")) | ||
| inst.Syncer.SafeModeDMLStrategy = "" | ||
| } |
There was a problem hiding this comment.
This validation logic is similar to the one in subtask.go. Using a switch statement here would also improve readability and make the code more extensible.
switch inst.Syncer.SafeModeDMLStrategy {
case "", "replace", "upsert":
// valid
default:
log.L().Warn("unknown safe-mode-dml-strategy, fallback to default",
zap.String("strategy", inst.Syncer.SafeModeDMLStrategy),
zap.String("default", "replace"))
inst.Syncer.SafeModeDMLStrategy = ""
}| if j.safeMode { | ||
| if j.dml.IsPrimaryOrUniqueKeyUpdated() { | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLDelete) | ||
| appendQueryAndArg() | ||
| if w.safeModeDMLStrategy == "upsert" { | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | ||
| } else { | ||
| if j.dml.IsPrimaryOrUniqueKeyUpdated() { | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLDelete) | ||
| appendQueryAndArg() | ||
| } | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLReplace) | ||
| } | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLReplace) | ||
| } else { | ||
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | ||
| } |
There was a problem hiding this comment.
The logic for handling RowChangeUpdate in safe mode can be simplified by using a switch statement on w.safeModeDMLStrategy. This would make the code cleaner and easier to understand.
| if j.safeMode { | |
| if j.dml.IsPrimaryOrUniqueKeyUpdated() { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLDelete) | |
| appendQueryAndArg() | |
| if w.safeModeDMLStrategy == "upsert" { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | |
| } else { | |
| if j.dml.IsPrimaryOrUniqueKeyUpdated() { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLDelete) | |
| appendQueryAndArg() | |
| } | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLReplace) | |
| } | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLReplace) | |
| } else { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | |
| } | |
| if j.safeMode { | |
| switch w.safeModeDMLStrategy { | |
| case "upsert": | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | |
| default: // "replace" or "" | |
| if j.dml.IsPrimaryOrUniqueKeyUpdated() { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLDelete) | |
| appendQueryAndArg() | |
| } | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLReplace) | |
| } | |
| } else { | |
| query, arg = j.dml.GenSQL(sqlmodel.DMLUpdate) | |
| } |
|
@GMHDBJD: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Add syncer.safe-mode-dml-strategy (replace|upsert). In upsert mode, safe-mode INSERT uses INSERT ON DUPLICATE KEY UPDATE and UPDATE stays UPDATE, and we keep foreign_key_checks enabled to preserve FK cascade semantics.
Also add an integration test case for ON UPDATE CASCADE under safe mode.
What problem does this PR solve?
Issue Number: close #xxx
What is changed and how it works?
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note