add API V3 keyspace identity protos by disksing · Pull Request #1458 · pingcap/kvproto

disksing · 2026-04-30T16:25:31Z

Issue: #1506

Summary

Add public API V3 KeyspaceIdentity schema and wire it through KV, PD, TSO, keyspace, GC, and related tenant-scoped protobuf APIs.
Preserve legacy keyspace_id fields for V1/V2 compatibility while documenting V3 identity semantics.
Update generated bindings and proto metadata for downstream API V3 consumers.

Validation

make proto-fmt
go build ./pkg/...
go test ./pkg/...
make check reached the expected proto.lock change prompt
cargo check is still blocked locally by the known libz/Xcode header issue

Signed-off-by: disksing <i@disksing.com>

# Conflicts: # pkg/kvrpcpb/kvrpcpb.pb.go # pkg/pdpb/pdpb.pb.go Signed-off-by: disksing <i@disksing.com>

Signed-off-by: disksing <i@disksing.com>

# Conflicts: # pkg/keyspacepb/keyspacepb.pb.go # proto/keyspacepb.proto # scripts/proto.lock

rleungx · 2026-07-09T05:55:27Z

-  // The keyspace that the request is sent to.
-  // NOTE: This field is only meaningful while the api_version is V2.
-  uint32 keyspace_id = 32;
+  oneof keyspace {


Changing this existing scalar field into a oneof is source-incompatible for generated Go users even though the wire tag is preserved. Context.KeyspaceId disappears, and downstream PD/TiDB/client-go code constructs or reads this field directly, so upgrading kvproto would break V1/V2 callers that do not use V3. The legacy scalar should stay as a normal field and keyspace_identity should be added alongside it, as was done for AutoIDRequest and KeyspaceMeta.

Thanks, fair point. I originally kept keyspace_id as a normal field and added keyspace_identity alongside it to minimize downstream Go source changes. Later I changed it to oneof based on review feedback, since semantically it is clearer that only one keyspace identifier should be set.

I think both directions are reasonable: parallel fields are more source-compatible, while oneof is clearer semantically. @sunxiaoguang what do you think? If we prefer minimizing downstream changes for kvrpcpb.Context, I can change this back to the parallel-field style.

Thanks @rleungx @disksing for the insightful discussion.
I lean towards the oneof approach here. Here is my perspective:
Confined Blast Radius: While changing to oneof is indeed source-incompatible, the generated protobuf code is primarily encapsulated within our rich clients (client-go, client-rust, client-c). The blast radius of this incompatible change is limited and well-contained within these repositories, rather than leaking indefinitely downstream.
Compiler-Guaranteed Safety: For strongly-typed languages, breaking the source code during compilation is actually a major benefit in this context. It forces the compiler to help us pinpoint every single callsite that accesses the relevant fields. This completely eliminates the risk of missing any field migrations, which could otherwise lead to silent, hard-to-debug runtime issues.
Semantics vs. Refactoring Cost: Maintaining parallel fields compromises semantic clarity and risks inconsistent states (e.g., both fields being set). The value of absolute correctness and explicit semantics brought by oneof far outweighs the cost of limited code adjustments in the client libraries.
Therefore, I'd prefer to go with oneof to ensure long-term robustness.

Signed-off-by: disksing <i@disksing.com>

overvenus · 2026-07-20T03:44:13Z

+  //
+  // `V3` uses user-key wire semantics for normal KV RPCs. Servers encode an
+  // 8-byte physical prefix at the serving boundary:
+  // mode(1) + namespace_id(4 bytes, big endian) + keyspace_id(3 bytes, big endian).


What's the semantic difference between namespace id and keyspace id?
What are the considerations of encoding namespace id first? Can it be mode(1) + keyspace_id(3 bytes, big endian) + namespace_id(4 bytes, big endian)?

namespace_id represents the higher-level namespace that groups multiple keyspaces. We intentionally encode it before keyspace_id because storage keys are ordered lexicographically. Using mode + namespace_id + keyspace_id ensures that all keyspaces belonging to the same namespace form one contiguous key range after sorting. If keyspace_id were encoded first, keyspaces from the same namespace would be scattered across different ranges. Therefore, we prefer to keep the namespace ID before the keyspace ID.

coocood · 2026-07-20T04:12:56Z

@disksing
Do we have dev branch of tidb & tikv that compiles with API V3?

Signed-off-by: disksing <i@disksing.com>

rleungx

The rest LGTM.

Signed-off-by: disksing <i@disksing.com>

ti-chi-bot · 2026-07-23T03:00:46Z

[LGTM Timeline notifier]

Timeline:

2026-07-21 05:50:58.855688624 +0000 UTC m=+1297644.891783670: ☑️ agreed by coocood.
2026-07-23 03:00:45.856590652 +0000 UTC m=+1460231.892685708: ☑️ agreed by rleungx.

windtalker

lgtm

Signed-off-by: disksing <i@disksing.com>

bb7133

LGTM

zhangjinpeng87

LGTM

ti-chi-bot · 2026-07-24T02:58:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, cfzjywxk, coocood, rleungx, windtalker, zhangjinpeng87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [bb7133,cfzjywxk,coocood,rleungx,windtalker]
~~proto/OWNERS~~ [bb7133,cfzjywxk,rleungx,windtalker,zhangjinpeng87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

niubell · 2026-07-24T03:11:29Z

@@ -16,7 +17,12 @@ option (rustproto.lite_runtime_all) = true;

 // Keyspace provides services to manage keyspaces.
 service Keyspace {


Namespace contains keyspace, but we put Namespace in the Keyspace service. Maybe adding a new Namespace service is better?

Namespace and keyspace are tightly coupled in API V3, and namespace_id is part of KeyspaceIdentity. Therefore, keeping their management APIs in the same service seems reasonable. We can revisit a separate Namespace service if it gains a more independent lifecycle later.

yudongusa · 2026-07-24T03:32:22Z

Why didn't any reviewer point out the missing github related issue and design spec?

Sync the final protocol changes from pingcap/kvproto#1458 at 7022f46937bc35125fb6d65aad5a0eedd59e7cfd onto release-8.5-keyspace. Preserve the downstream release schema and regenerate Go bindings and proto.lock with the downstream toolchain. Signed-off-by: disksing <i@disksing.com>

api: add API V3 keyspace identity protos

b17ca6f

Signed-off-by: disksing <i@disksing.com>

ti-chi-bot Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 30, 2026

ti-chi-bot Bot requested a review from henrybw April 30, 2026 16:25

ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 30, 2026

disksing changed the title ~~[codex] add API V3 keyspace identity protos~~ add API V3 keyspace identity protos Apr 30, 2026

disksing added 5 commits May 1, 2026 09:54

api: update proto lock

9484519

Signed-off-by: disksing <i@disksing.com>

api: keep TSO group response focused

5fa3c39

Signed-off-by: disksing <i@disksing.com>

api-v3: normalize keyspace identity fields

e279ac6

Signed-off-by: disksing <i@disksing.com>

api: remove namespace from keyspace lookup

d885a32

Signed-off-by: disksing <i@disksing.com>

Merge remote-tracking branch 'origin/master' into apiv3

9340947

# Conflicts: # pkg/kvrpcpb/kvrpcpb.pb.go # pkg/pdpb/pdpb.pb.go Signed-off-by: disksing <i@disksing.com>

sunxiaoguang reviewed Jun 10, 2026

View reviewed changes

Comment thread proto/keyspacepb.proto Outdated

disksing and others added 10 commits June 11, 2026 09:19

Use oneof for keyspace identity fields

58ce560

Signed-off-by: disksing <i@disksing.com>

Clarify keyspace pagination comment

52f4995

Signed-off-by: disksing <i@disksing.com>

Adjust DFS stat keyspace scope fields

214a58f

Signed-off-by: disksing <i@disksing.com>

Remove GC safe point response keyspace identity

ac39bd6

Signed-off-by: disksing <i@disksing.com>

Remove service safe point response keyspace identity

91be314

Signed-off-by: disksing <i@disksing.com>

Merge keyspace group lookup by identity

e58f3b1

Signed-off-by: disksing <i@disksing.com>

Add namespace metadata APIs

3566665

Signed-off-by: disksing <i@disksing.com>

Merge remote-tracking branch 'origin/master' into apiv3

c57bf39

# Conflicts: # pkg/keyspacepb/keyspacepb.pb.go # proto/keyspacepb.proto # scripts/proto.lock

Make keyspace meta identity backward compatible

57eabd0

Keep AutoIDRequest keyspace fields compatible

c3fb31a

disksing marked this pull request as ready for review July 8, 2026 04:27

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 8, 2026

ti-chi-bot Bot requested a review from 0xPoe July 8, 2026 04:27

rleungx reviewed Jul 9, 2026

View reviewed changes

Use oneof for remaining keyspace identities

fd32127

Signed-off-by: disksing <i@disksing.com>

overvenus reviewed Jul 20, 2026

View reviewed changes

disksing mentioned this pull request Jul 21, 2026

Validate kvproto API V3 definitions against TiKV master tikv/tikv#19856

Open

rleungx reviewed Jul 22, 2026

View reviewed changes

Comment thread proto/autoid.proto

disksing added 2 commits July 22, 2026 11:36

externalworkloadpb: support V3 keyspace identity

ec90f51

Signed-off-by: disksing <i@disksing.com>

autoid: identify keyspace in rebase requests

e30d0e9

Signed-off-by: disksing <i@disksing.com>

rleungx reviewed Jul 22, 2026

View reviewed changes

Comment thread proto/resource_manager.proto Outdated

rleungx reviewed Jul 22, 2026

View reviewed changes

resource-manager: support V3 keyspace identity

57fb9c0

Signed-off-by: disksing <i@disksing.com>

rleungx approved these changes Jul 23, 2026

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 23, 2026

windtalker approved these changes Jul 23, 2026

View reviewed changes

externalworkloadpb: use oneof for keyspace identity

64fac48

Signed-off-by: disksing <i@disksing.com>

cfzjywxk approved these changes Jul 23, 2026

View reviewed changes

pdpb: remove unused TSO keyspace identity

64bc02a

Signed-off-by: disksing <i@disksing.com>

bb7133 approved these changes Jul 24, 2026

View reviewed changes

zhangjinpeng87 approved these changes Jul 24, 2026

View reviewed changes

ti-chi-bot Bot added the approved label Jul 24, 2026

sunxiaoguang reviewed Jul 24, 2026

View reviewed changes

Comment thread proto/pdpb.proto

Apply suggestion from @sunxiaoguang

0de434e

ti-chi-bot Bot merged commit 7022f46 into master Jul 24, 2026
5 of 6 checks passed

ti-chi-bot Bot deleted the apiv3 branch July 24, 2026 03:11

niubell reviewed Jul 24, 2026

View reviewed changes

This was referenced Jul 24, 2026

keyspacepb: split namespace service #1505

Merged

Introduce namespace-scoped keyspace identity for API V3 #1506

Open

disksing mentioned this pull request Jul 27, 2026

api: sync API V3 keyspace identity protos tidbcloud/kvproto#47

Open

disksing mentioned this pull request Aug 5, 2026

*: update kvproto to apiv3 tikv/client-go#2024

Merged

wfxr mentioned this pull request Aug 13, 2026

*: support shared lock upgrade pingcap/tidb#69559

Open

14 tasks

		@@ -16,7 +17,12 @@ option (rustproto.lite_runtime_all) = true;

		// Keyspace provides services to manage keyspaces.
		service Keyspace {

Uh oh!

Conversation

disksing commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rleungx Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

disksing Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

sunxiaoguang Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

overvenus Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

disksing Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

coocood commented Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

rleungx left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Jul 23, 2026

[LGTM Timeline notifier]

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

bb7133 left a comment

Choose a reason for hiding this comment

Uh oh!

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

niubell Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

disksing Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

yudongusa commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

disksing commented Apr 30, 2026 •

edited

Loading