docs: add physical storage UUID RFC by bb7133 · Pull Request #69924 · pingcap/tidb

bb7133 · 2026-07-19T04:26:50Z

What problem does this PR solve?

Issue Number: ref #69923

Problem Summary:

This PR adds an RFC for introducing a globally restorable physical storage identity for TiDB tables and partitions. The goal is to enable supported BR restore and TiDB-X tenant migration flows to preserve physical data key prefixes instead of rewriting source physical table IDs to target-local physical table IDs.

What changed and how does it work?

This PR adds a new design document under docs/design/:

docs/design/2026-07-19-physical-storage-uuid.md

The RFC describes:

why logical UUID metadata alone cannot remove BR table ID key rewrite;
the proposed PhysicalStorageUUID metadata for tables and partitions;
UUIDv7 generation semantics for new physical data spaces;
a possible UUID-based tablecodec v2 key prefix;
DDL, BR restore, and TiDB-X tenant migration semantics;
rollout, testing, compatibility, and open questions.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2026-07-19T04:26:53Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

coderabbitai · 2026-07-19T04:26:57Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fd6fa7d1-5e42-4e80-8100-79cd8ae40936

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

ti-chi-bot · 2026-07-19T04:27:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xiongjiwei for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pingcap-cla-assistant · 2026-07-19T04:27:03Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

CodexM5 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

ti-chi-bot · 2026-07-19T05:42:56Z

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-tests-checked label, please finished the tests then check the finished items in description.

For example:

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code

‼️ Must keep the HTML comments 

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}

YangKeao · 2026-07-20T06:48:36Z

+Generation timing:
+
+* `CREATE TABLE`: generate one physical storage UUID for a non-partitioned table.
+* `CREATE TABLE` with partitions: generate one physical storage UUID per physical partition.


Maybe we also need a storage UUID for the global index 🤔 .

YangKeao · 2026-07-20T06:53:10Z

+```
+
+The new layout may keep the `t` table-key family for locality and compatibility with table-key scanning, but the byte after `t` or an equivalent field must identify the key format before decoding the physical identity.
+This is required because existing tablecodec logic assumes `t + 8-byte int64` in many places, including fixed offsets for record/index separators and table range boundaries.


I don't understand how the format_marker can make it better.

A decoder still cannot tell whether it's an old key or a new key, because the prefix of int64_physical_table_id can also be a format_marker.

joechenrh · 2026-07-20T07:29:29Z

+* `ADD PARTITION`: generate one physical storage UUID for each new partition.
+* `REORGANIZE PARTITION`: generate UUIDs for newly created physical partitions.
+* `TRUNCATE TABLE`: generate a new physical storage UUID for the new physical data space.
+* `TRUNCATE PARTITION`: generate a new physical storage UUID for the truncated partition.


Do we need to rewrite the whole table for MODIFY COLUMN if it's still using the old format?

joechenrh · 2026-07-20T07:39:16Z

+
+    // PhysicalStorageUUID identifies this physical partition data space
+    // when the partition uses the UUID-based physical key format.
+    PhysicalStorageUUID []byte


Should we store PhysicalStorageKeyFormat for each partition?

D3Hunter · 2026-07-20T09:52:18Z

see #69923 (comment), before we proceed with writing an RFC, maybe we should eval the gain/cost first

expxiaoli · 2026-07-20T10:00:18Z

+
+Even though random collision is practically negligible, metadata mutation must still enforce uniqueness within a keyspace.
+If a newly generated UUID conflicts, DDL can retry generation.
+If restore or migration sees an existing target object with the same preserved UUID, it must report a conflict unless the requested mode explicitly drops or replaces that object.


Do not reuse UUIDs until asynchronous delete-range cleanup completes; it may delete newly restored data.

expxiaoli · 2026-07-20T10:03:40Z

+* For new-format tables, BR preserves source physical storage UUIDs and can ingest KV data without rewriting the physical table key prefix.
+* If the target cluster already has the same physical storage UUID, restore reports a conflict unless the user requested a mode that drops/replaces the target object.
+* Restore must advance or reconcile metadata ID allocators as needed, but this does not require rewriting UUID-based data key prefixes.
+* Log restore should use physical storage UUID as the stable physical object identity for UUID-format tables.


Existing UUIDs owned by the same BR checkpoint should be resumable, not conflicting.

expxiaoli · 2026-07-20T10:08:27Z

+2. **Experimental key format**
+   * Add UUID-based tablecodec helpers.
+   * Enable the new format only behind an experimental switch for newly created tables.
+   * Keep old-format table behavior unchanged.


Require cluster-wide UUID-key support during upgrades, otherwise older TiDB nodes may read or write incompatible keys.

expxiaoli · 2026-07-20T10:11:58Z

+1. **Metadata preparation**
+   * Add physical storage UUID metadata fields.
+   * Generate UUIDs for new tables and partitions but keep old key format as default.
+   * Add uniqueness checks and diagnostic display.


Line 289 generates UUIDs for legacy-format tables, contradicting the empty-UUID invariant at lines 125–126.

ti-chi-bot Bot added the do-not-merge/needs-tests-checked label Jul 19, 2026

ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jul 19, 2026

ti-chi-bot Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 19, 2026

bb7133 force-pushed the bb7133/rfc-physical-storage-uuid branch from c2b35a4 to 95b35fd Compare July 19, 2026 04:28

bb7133 mentioned this pull request Jul 19, 2026

Introduce globally restorable physical table identity to avoid BR table ID rewrite #69923

Open

docs: add physical storage UUID RFC

129ca13

bb7133 force-pushed the bb7133/rfc-physical-storage-uuid branch from 95b35fd to 129ca13 Compare July 19, 2026 05:42

YangKeao reviewed Jul 20, 2026

View reviewed changes

joechenrh reviewed Jul 20, 2026

View reviewed changes

expxiaoli reviewed Jul 20, 2026

View reviewed changes

Uh oh!

Conversation

bb7133 commented Jul 19, 2026

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot Bot commented Jul 19, 2026

Uh oh!

coderabbitai Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

ti-chi-bot Bot commented Jul 19, 2026

Uh oh!

pingcap-cla-assistant Bot commented Jul 19, 2026

Uh oh!

ti-chi-bot Bot commented Jul 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

D3Hunter commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

coderabbitai Bot commented Jul 19, 2026 •

edited

Loading

D3Hunter commented Jul 20, 2026 •

edited

Loading