Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip replay#44

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-f055
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip replay#44
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-f055

Conversation

@cursor

@cursor cursor Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() and gossip.receive() did not preserve or apply Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After a snapshot restore, restored nodes received fresh wall-clock HLCs instead of their stored causal timestamps. Legitimate post-restore writes and gossip replays with successor HLCs were rejected as TimestampRegression, causing silent data loss in disaster-recovery and multi-node deployments.

Concrete trigger: Write key with hlc=(5000, 10, "nodeA") → snapshot → restore → attempt causal successor hlc=(5001, 11, "nodeA")TimestampRegression because restored node has fresh wall-clock HLC ~1.7e18.

Root cause

  • restore_from_file() constructed MemoryNode without reading hlc from snapshot JSON (defaulted to _hlc.now())
  • gossip.receive() called remember() without passing hlc/ts_ns
  • bulk_write() also omitted HLC passthrough

Fix

  • Add _parse_hlc() for snapshot/wire HLC normalisation (with legacy fallback from ts_ns)
  • Restore HLC on snapshot load and advance global clock via _hlc.update()
  • Pass HLC/ts_ns through gossip.receive() with stale-update rejection and skip missing-HLC updates on existing keys
  • Pass HLC through bulk_write(); lock gc_expired() during iteration

Validation

  • Reproduced bug before fix (successor write raised TimestampRegression)
  • Added regression tests: test_restore_preserves_hlc_for_replication, gossip HLC ordering tests
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

restore_from_file() dropped HLC timestamps from snapshots, assigning fresh
wall-clock HLCs instead. Post-restore causal writes and gossip replays with
legitimate successor timestamps were rejected as TimestampRegression,
causing silent data loss in disaster-recovery and multi-node deployments.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC and advance global clock on snapshot load
- Pass HLC/ts_ns through gossip.receive() with stale-update rejection
- Pass HLC through bulk_write; lock gc_expired during iteration
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant