Skip to content

dxf, importinto: compare subtask task IDs as strings (#69884)#69912

Merged
ti-chi-bot[bot] merged 2 commits into
pingcap:release-nextgen-202603from
ti-chi-bot:cherry-pick-69884-to-release-nextgen-202603
Jul 20, 2026
Merged

dxf, importinto: compare subtask task IDs as strings (#69884)#69912
ti-chi-bot[bot] merged 2 commits into
pingcap:release-nextgen-202603from
ti-chi-bot:cherry-pick-69884-to-release-nextgen-202603

Conversation

@ti-chi-bot

@ti-chi-bot ti-chi-bot commented Jul 17, 2026

Copy link
Copy Markdown
Member

This is an automated cherry-pick of #69884

What problem does this PR solve?

Issue Number: ref #61702, ref #69788

Problem Summary:

Subtask task_key columns store decimal task IDs as strings, but several callers bind integer values. TiDB can coerce these comparisons to DOUBLE, so adjacent IDs above 2^53 may compare equal and equality predicates cannot use the task_key index as intended.
passing string also make sure the table index is used for querying

What changed and how does it work?

  • Add a shared canonical task-ID-to-key conversion and use it for DXF and Import Into subtask predicates.
  • Update test and test utility queries to compare string keys without numeric coercion.
  • Add regression coverage for adjacent task IDs above 2^53.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Test commands:

./tools/check/failpoint-go-test.sh pkg/dxf/framework/storage -run '^TestGetActiveSubtasks$' -count=1
./tools/check/failpoint-go-test.sh pkg/dxf/importinto/jobhistory -run '^TestGetFromHistory$' -count=1
./tools/check/failpoint-go-test.sh pkg/dxf/importinto -run '^TestGetJobLastUpdateTime$' -count=1
./tools/check/failpoint-go-test.sh tests/realtikvtest/addindextest1 -run '^TestAddIndexScheduleAway$' -count=1 -args -tikv-path 'tikv://127.0.0.1:12379?disableGC=true'
./tools/check/failpoint-go-test.sh tests/realtikvtest/addindextest3 -run '^TestMergeTempIndexBasic$' -count=1 -args -tikv-path 'tikv://127.0.0.1:12379?disableGC=true'
./tools/check/failpoint-go-test.sh tests/realtikvtest/importintotest4 -run '^TestImportInto/TestGlobalSortSummary$' -count=1 -args -tikv-path 'tikv://127.0.0.1:12379?disableGC=true'
make lint

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fix incorrect DXF subtask matching when task IDs exceed the exact integer range of floating-point comparisons.

Summary by CodeRabbit

  • Bug Fixes
    • Improved correctness of task/subtask queries by consistently treating task identifiers as string-based task_key values across active and history tables.
    • Fixed job last-update-time and history aggregation behavior, especially for large or closely numbered task IDs.
    • Improved reliability of recovery, pause/cancel/fail, and concurrency-related subtask updates.
  • Tests
    • Added coverage for GetJobLastUpdateTime and enhanced history/active filtering regression checks near 2^53.
  • Chores
    • Adjusted the importinto test sharding configuration.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603 labels Jul 17, 2026
@ti-chi-bot

Copy link
Copy Markdown
Member Author

@D3Hunter This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

@ti-chi-bot

ti-chi-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a3a4964f-36b1-4557-93b1-c8e7d88ddef0

📥 Commits

Reviewing files that changed from the base of the PR and between c5d55a3 and f16f9a2.

📒 Files selected for processing (2)
  • pkg/dxf/framework/storage/table_test.go
  • pkg/dxf/importinto/BUILD.bazel
💤 Files with no reviewable changes (1)
  • pkg/dxf/framework/storage/table_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/dxf/importinto/BUILD.bazel

📝 Walkthrough

Walkthrough

The change adds canonical string conversion for DXF task_key bindings, applies it across storage and ImportInto history queries, and expands regression coverage for large adjacent task IDs and active/history aggregation.

Changes

DXF task-key handling

Layer / File(s) Summary
Canonical task-key bindings
pkg/dxf/framework/storage/..., pkg/dxf/importinto/...
Adds TaskIDToKey and uses it across subtask state updates, history transfer, task queries, and ImportInto history aggregation.
Storage and SQL regression coverage
pkg/dxf/framework/storage/table_test.go, pkg/dxf/framework/testutil/..., pkg/dxf/framework/integrationtests/..., tests/realtikvtest/...
Updates SQL task-key handling to use strings and adds adjacent-large-ID filtering coverage.
ImportInto coverage and test configuration
pkg/dxf/importinto/job_testkit_test.go, pkg/dxf/importinto/jobhistory/history_test.go, pkg/dxf/importinto/BUILD.bazel, .agents/skills/...
Adds active/history timestamp and aggregation tests, increases test sharding, and updates the DXF test case map.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

  • pingcap/tidb#69181: Touches the overlapping DXF pausing and resume storage paths.
  • pingcap/tidb#69811: Overlaps in DXF history transfer and adjacent large task ID handling.
  • pingcap/tidb#69884: Overlaps in canonical string task_key bindings and ImportInto regression coverage.

Suggested labels: component/import, approved, lgtm

Suggested reviewers: joechenrh, wjhuang2016, gmhdbjd

Poem

A rabbit found task keys in rows,
And taught them strings where each one goes.
Through history’s trail and active sight,
Adjacent IDs stay distinct and right.
Tests bloom softly, hop by hop.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 47.62% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: comparing DXF and Import Into subtask IDs as strings.
Description check ✅ Passed The description follows the template and includes issue refs, problem summary, changes, tests, side effects, documentation, and release note.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
pkg/dxf/framework/storage/converter.go (1)

94-96: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Fix typographical errors in the documentation comment.

💡 Proposed fix
-// due to history reason, the task_key column inside subtask tables are defined
-// as varchar, but it actually stores the ID of the task which is integer. to
-// make sure TiDB query can correct handle this, we need to convert it to string.
+// due to historical reasons, the task_key column inside subtask tables is defined
+// as varchar, but it actually stores the ID of the task which is an integer. To
+// make sure TiDB queries can correctly handle this, we need to convert it to a string.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/dxf/framework/storage/converter.go` around lines 94 - 96, Correct the
typographical errors in the documentation comment describing the subtask table
task_key conversion, including grammar and wording, without changing the
converter implementation or its meaning.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/dxf/framework/storage/table_test.go`:
- Around line 303-354: Remove the unresolved Git conflict markers surrounding
the “prepare transition persists fields and exposes zero-row CAS” test in the
table test file, preserving the incoming test body and ensuring the file remains
valid Go syntax.

In `@pkg/dxf/importinto/BUILD.bazel`:
- Around line 113-117: Remove the unresolved Git conflict markers in the
shard_count assignment and retain the correct single value for shard_count
according to the intended branch change, leaving the BUILD target syntactically
valid.

---

Nitpick comments:
In `@pkg/dxf/framework/storage/converter.go`:
- Around line 94-96: Correct the typographical errors in the documentation
comment describing the subtask table task_key conversion, including grammar and
wording, without changing the converter implementation or its meaning.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: ead2db09-6074-4d19-8f81-3f8dda73001b

📥 Commits

Reviewing files that changed from the base of the PR and between 86c5dc7 and c5d55a3.

📒 Files selected for processing (18)
  • .agents/skills/tidb-test-guidelines/references/dxf-case-map.md
  • pkg/dxf/framework/integrationtests/framework_err_handling_test.go
  • pkg/dxf/framework/integrationtests/modify_test.go
  • pkg/dxf/framework/storage/converter.go
  • pkg/dxf/framework/storage/history.go
  • pkg/dxf/framework/storage/subtask_state.go
  • pkg/dxf/framework/storage/table_test.go
  • pkg/dxf/framework/storage/task_state.go
  • pkg/dxf/framework/storage/task_table.go
  • pkg/dxf/framework/testutil/table_util.go
  • pkg/dxf/importinto/BUILD.bazel
  • pkg/dxf/importinto/job.go
  • pkg/dxf/importinto/job_testkit_test.go
  • pkg/dxf/importinto/jobhistory/history.go
  • pkg/dxf/importinto/jobhistory/history_test.go
  • tests/realtikvtest/addindextest1/disttask_test.go
  • tests/realtikvtest/addindextest3/temp_index_test.go
  • tests/realtikvtest/importintotest4/import_summary_test.go

Comment thread pkg/dxf/framework/storage/table_test.go Outdated
Comment thread pkg/dxf/importinto/BUILD.bazel Outdated
Comment on lines +113 to +117
<<<<<<< HEAD
shard_count = 27,
=======
shard_count = 30,
>>>>>>> 66be7a615be (dxf, importinto: compare subtask task IDs as strings (#69884))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Resolve merge conflict markers.

The file contains unresolved Git merge conflict markers (<<<<<<< HEAD, =======, >>>>>>>) which will cause a build failure.

🐛 Proposed fix to resolve the conflict
-<<<<<<< HEAD
-    shard_count = 27,
-=======
-    shard_count = 30,
->>>>>>> 66be7a615be (dxf, importinto: compare subtask task IDs as strings (`#69884`))
+    shard_count = 30,
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
<<<<<<< HEAD
shard_count = 27,
=======
shard_count = 30,
>>>>>>> 66be7a615be (dxf, importinto: compare subtask task IDs as strings (#69884))
shard_count = 30,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/dxf/importinto/BUILD.bazel` around lines 113 - 117, Remove the unresolved
Git conflict markers in the shard_count assignment and retain the correct single
value for shard_count according to the intended branch change, leaving the BUILD
target syntactically valid.

Signed-off-by: D3Hunter <jujj603@gmail.com>
@ti-chi-bot

Copy link
Copy Markdown
Member Author

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 17, 2026
@ti-chi-bot ti-chi-bot Bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 17, 2026
@D3Hunter

Copy link
Copy Markdown
Contributor

/retest

1 similar comment
@D3Hunter

Copy link
Copy Markdown
Contributor

/retest

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-nextgen-202603@86c5dc7). Learn more about missing BASE report.

Additional details and impacted files
@@                     Coverage Diff                     @@
##             release-nextgen-202603     #69912   +/-   ##
===========================================================
  Coverage                          ?   76.3548%           
===========================================================
  Files                             ?       1937           
  Lines                             ?     542707           
  Branches                          ?          0           
===========================================================
  Hits                              ?     414383           
  Misses                            ?     128324           
  Partials                          ?          0           
Flag Coverage Δ
unit 76.3548% <93.5483%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 61.5065% <0.0000%> (?)
parser ∅ <0.0000%> (?)
br 48.7548% <0.0000%> (?)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, GMHDBJD

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 20, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-07-17 11:15:38.301862933 +0000 UTC m=+971524.337957990: ☑️ agreed by D3Hunter.
  • 2026-07-20 07:17:20.630779399 +0000 UTC m=+1216426.666874455: ☑️ agreed by GMHDBJD.

@D3Hunter

Copy link
Copy Markdown
Contributor

/retest

@ti-chi-bot
ti-chi-bot Bot merged commit 73b2477 into pingcap:release-nextgen-202603 Jul 20, 2026
18 checks passed
@ti-chi-bot
ti-chi-bot Bot deleted the cherry-pick-69884-to-release-nextgen-202603 branch July 20, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants