Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#47

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-cd53
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#47
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-cd53

Conversation

@cursor

@cursor cursor Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file() and gossip.receive() did not preserve or apply Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After disaster-recovery snapshot restore, nodes received fresh wall-clock HLCs instead of their stored causal timestamps. Legitimate post-restore writes and gossip updates with correct successor HLCs were rejected as TimestampRegression, causing silent data loss in backup/restore and multi-node gossip scenarios.

Concrete trigger: Snapshot a node with hlc=(5000, 10, "nodeA"), restore, then apply a causal successor hlc=(5000, 11, "nodeA") — rejected because restore assigned a fresh wall-clock HLC.

Root cause

  • restore_from_file() rebuilt MemoryNode without reading the hlc field from the snapshot and never advanced the global _hlc.
  • gossip.receive() omitted hlc when calling remember().
  • bulk_write() also dropped HLC from row payloads.

Fix

  • Add _parse_hlc() to normalise HLC from snapshot/wire form (with legacy ts_ns fallback).
  • Restore stored HLC on snapshot load and call _hlc.update() for each restored node.
  • Pass HLC through bulk_write and gossip.receive; reject stale gossip overwrites; skip HLC-less gossip updates on existing keys.
  • Hold lock during restore_from_file mutation and gc_expired.

Validation

  • Reproduced pre-fix failure scenario.
  • Added regression tests in test_enterprise_backup.py and test_gossip.py.
  • Full suite: 192 passed, 8 skipped.
Open in Web View Automation 

After snapshot restore, nodes were assigned fresh HLC timestamps instead
of the stored values from the snapshot. Post-restore causal writes with
legitimate successor HLCs were rejected as TimestampRegression, causing
silent data loss in disaster-recovery and multi-node gossip scenarios.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC on snapshot load and advance global _hlc
- Pass HLC through bulk_write and gossip receive
- Reject stale gossip updates; skip HLC-less updates on existing keys
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant