Skip to content

[Feature] ShardCoordinator state is rebuilt from region claims — the in-code comment already names the production upgrade: journal-backed coordinator state #1188

Description

@pathosDev

Problem

The ShardCoordinator's own doc comment names its production upgrade — and nothing tracks it:

"State is reconstructed from Register messages: each region reports the shards it currently hosts, and the coordinator merges that with any new allocation requests. This is deliberately lightweight — a production upgrade would snapshot state to a journal so the coordinator can recover across restarts without re-allocating every shard from scratch."

Today's model: the coordinator runs on every node, leader-gated; on a leader change the new coordinator rebuilds shardHome from whatever regions happen to re-register, optionally seeded from a plain snapshot store (coordinatorStateStore). The rebuild-from-claims model is the root enabler of the filed ownership bugs: a stale region's claims overwrite the live owner's (#948) because claims are the state; a demotion racing the snapshot load repopulates cleared maps (#952); and the snapshot store, when configured, can serve a stale view filtered only against current up-members.

The tracked decision this issue asks for: give coordinator state a real durability/authority story. Candidates, in ascending order of change:

  1. Journal-backed coordinator state (the path the comment itself names): allocation decisions are events; a new leader recovers the allocation log instead of trusting re-register claims; region claims become reconciliation input, never authority.
  2. Replicated state via DistributedData: shardHome as a replicated map, so every candidate leader already holds a recent view and the register-claim path loses its authority role.
  3. Keep rebuild-from-claims, add authority rules: claims carry the coordinator epoch that granted them; a new coordinator rejects claims from older epochs (cheapest; closes [Bug] ShardCoordinator.onRegister overwrites shardHome for every claimed shard with no conflict check, so a region re-registering with stale localShards takes ownership back from the live owner and both nodes run the same entities #948's overwrite without new persistence).

Evidence

src/cluster/sharding/ShardCoordinator.ts:87-98 — the doc comment quoted above (v0.15.0 tree). CoordinatorState.ts — the optional plain-snapshot store that exists today.

Verification status

CONFIRMED-BY-READ — quoted from the shipped source. From the independent production-readiness pass (2026-08-14), second batch (design decisions). Concrete bugs of the current model, filed separately: #948, #952, #953, #1026; remember-entities durability rides the same store family (#649, #954).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority: mediumUseful, not urgentproduction-goalBlocks or defines the path to production readiness

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions