You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Feature] ShardCoordinator state is rebuilt from region claims — the in-code comment already names the production upgrade: journal-backed coordinator state #1188
The ShardCoordinator's own doc comment names its production upgrade — and nothing tracks it:
"State is reconstructed from Register messages: each region reports the shards it currently hosts, and the coordinator merges that with any new allocation requests. This is deliberately lightweight — a production upgrade would snapshot state to a journal so the coordinator can recover across restarts without re-allocating every shard from scratch."
Today's model: the coordinator runs on every node, leader-gated; on a leader change the new coordinator rebuilds shardHome from whatever regions happen to re-register, optionally seeded from a plain snapshot store (coordinatorStateStore). The rebuild-from-claims model is the root enabler of the filed ownership bugs: a stale region's claims overwrite the live owner's (#948) because claims are the state; a demotion racing the snapshot load repopulates cleared maps (#952); and the snapshot store, when configured, can serve a stale view filtered only against current up-members.
The tracked decision this issue asks for: give coordinator state a real durability/authority story. Candidates, in ascending order of change:
Journal-backed coordinator state (the path the comment itself names): allocation decisions are events; a new leader recovers the allocation log instead of trusting re-register claims; region claims become reconciliation input, never authority.
Replicated state via DistributedData: shardHome as a replicated map, so every candidate leader already holds a recent view and the register-claim path loses its authority role.
src/cluster/sharding/ShardCoordinator.ts:87-98 — the doc comment quoted above (v0.15.0 tree). CoordinatorState.ts — the optional plain-snapshot store that exists today.
Verification status
CONFIRMED-BY-READ — quoted from the shipped source. From the independent production-readiness pass (2026-08-14), second batch (design decisions). Concrete bugs of the current model, filed separately: #948, #952, #953, #1026; remember-entities durability rides the same store family (#649, #954).
Problem
The
ShardCoordinator's own doc comment names its production upgrade — and nothing tracks it:Today's model: the coordinator runs on every node, leader-gated; on a leader change the new coordinator rebuilds
shardHomefrom whatever regions happen to re-register, optionally seeded from a plain snapshot store (coordinatorStateStore). The rebuild-from-claims model is the root enabler of the filed ownership bugs: a stale region's claims overwrite the live owner's (#948) because claims are the state; a demotion racing the snapshot load repopulates cleared maps (#952); and the snapshot store, when configured, can serve a stale view filtered only against current up-members.The tracked decision this issue asks for: give coordinator state a real durability/authority story. Candidates, in ascending order of change:
DistributedData:shardHomeas a replicated map, so every candidate leader already holds a recent view and the register-claim path loses its authority role.ShardCoordinator.onRegisteroverwritesshardHomefor every claimed shard with no conflict check, so a region re-registering with stalelocalShardstakes ownership back from the live owner and both nodes run the same entities #948's overwrite without new persistence).Evidence
src/cluster/sharding/ShardCoordinator.ts:87-98— the doc comment quoted above (v0.15.0 tree).CoordinatorState.ts— the optional plain-snapshot store that exists today.Verification status
CONFIRMED-BY-READ — quoted from the shipped source. From the independent production-readiness pass (2026-08-14), second batch (design decisions). Concrete bugs of the current model, filed separately: #948, #952, #953, #1026; remember-entities durability rides the same store family (#649, #954).