Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#43

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-3e9b
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#43
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-3e9b

Conversation

@cursor

@cursor cursor Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

Critical data-loss bug in disaster recovery and gossip paths.

restore_from_file() rebuilt MemoryNode objects without the stored hlc field, assigning a fresh local HLC instead. After snapshot restore, causal writes from the original cluster (gossip replay, bulk_write) were rejected with TimestampRegression — silent data loss in production failover paths.

GossipProtocol.receive() also omitted hlc, allowing stale remote events to overwrite fresher local state.

Concrete trigger

  1. Node stores memory with hlc=(5000, 10, "nodeA")
  2. Snapshot taken and restored to failover node
  3. Restored node assigns fresh HLC (wall, 0, uuid) instead of snapshot HLC
  4. Original cluster sends causal successor hlc=(5000, 11, "nodeA") via gossip
  5. Write rejected: (5000, 11) < (wall, 0)data lost

Root cause

HLC was added in v0.6.0 for causal ordering, but restore_from_file() and gossip.receive() were not updated to preserve/propagate HLC timestamps.

Fix

  • Add _parse_hlc() for snapshot/wire normalisation with legacy fallback
  • Restore hlc and advance global _hlc during snapshot restore
  • Pass hlc through bulk_write; lock restore and gc_expired
  • Gossip: propagate hlc, reject stale updates, skip missing-hlc overwrites

Validation

  • Reproduced bug before fix (HLC not preserved, causal successor rejected)
  • 37 targeted tests pass (backup, gossip, rust_brain, concurrency)
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

restore_from_file() was rebuilding MemoryNode without the stored hlc field,
assigning a fresh local HLC instead. After disaster recovery, causal writes
from the original cluster (gossip replay, bulk_write) were rejected with
TimestampRegression — silent data loss in production failover paths.

Gossip receive() also omitted hlc, allowing stale remote events to overwrite
fresher local state when monotonic enforcement compared against a fresh HLC.

- Add _parse_hlc() for snapshot/wire normalisation with legacy fallback
- Restore hlc and advance global _hlc during snapshot restore
- Pass hlc through bulk_write; lock restore and gc_expired
- Gossip: propagate hlc, reject stale updates, skip missing-hlc overwrites
- Tests for restore round-trip and gossip HLC semantics

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant