Quorum: add a from-scratch Kademlia DHT lab (XOR-metric routing)#668
Merged
Conversation
The counterpoint to Chord and the DHT that actually runs the internet (BitTorrent Mainline, IPFS, Ethereum discv5, Storj). Built the full Maymounkov-Mazieres (2002) design on the shared kernel: - XOR metric + k-buckets with the prefer-old-live-contacts LRU rule (a full bucket pings its LRU head and evicts only on a failed ping). - Iterative, alpha-parallel lookup driven by the initiator (contrast Chord's recursive forward): seed a shortlist, keep alpha probes in flight to the closest unqueried contacts, fold replies back, stop when the frontier closes on the true k-closest. O(log N) rounds. - FIND_VALUE/STORE for a real PUT/GET key-value store; contact learning from every message; bucket-refresh + republish maintenance; crash/restart (volatile routing rebuilt by a self-lookup, durable stored pairs). A dropped lookup reply never evicts a live contact. - Invariants: k-bucket well-formedness (safety) plus two eventual gauges; the headline re-runs the alpha-parallel search offline over the frozen tables from every node to every target and demands the exact XOR-nearest k. - Bespoke binary-trie visualiser (the canonical Figure-1): the selected node's k-buckets drawn as the shaded subtrees hanging off its root-to-leaf path, with a k slider, bucket inspector, stored pairs and key-placement panels. - 13 self-tests (suite 165 -> 178/178), incl. k=2 multi-hop lookups, PUT/GET retrieval from every node, heal-after-two-crashes, and convergence under 15% message loss. Verified live in headless Chromium; full verify-project gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XuHZAjE3TuggDG5k91gL6
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Kademlia — the XOR-metric distributed hash table — as a new lab in the Quorum distributed-systems simulator (
projects/quorum-distsys-k7r2/). It's the counterpoint to the existing Chord lab and the DHT that actually runs the internet (BitTorrent Mainline, IPFS, Ethereum discv5, Storj). Everything is implemented from scratch on the project's existing deterministic kernel; all changes live in the single project folder.Chord vs. Kademlia is a genuinely instructive pair:
Implementation (Maymounkov & Mazières, 2002)
xor.ts) —distance(a,b)=a⊕b;bucketIndex= most-significant differing bit (the law2^i ≤ a⊕b < 2^(i+1)is asserted in a test).routing.ts) — one bucket per bit; the real prefer-old-live-contacts rule: a full bucket pings its least-recently-seen head and evicts it only if it fails to answer (the anti-churn / anti-eclipse bias). Pure data + functions so the kernel can snapshot it.kademlia.ts) — seed a shortlist from own buckets, keep α=3FIND_NODEprobes in flight to the closest unqueried contacts, fold replies back, stop when the frontier closes on the true k-closest. Each hop moves ≥1 bit closer ⇒ O(log N) rounds.invariants.ts) — k-bucket well-formedness (always-on safety) plus two eventual gauges; the headline iterative lookup finds the true k-closest re-runs the α-parallel search offline over the frozen tables from every node to every target and demands the exact XOR-nearest k.ui/KademliaTree.tsx,labs/KademliaLab.tsx) — the canonical paper Figure-1: live ids as leaves of a compressed binary trie, and for the selected node its k-buckets drawn as the shaded subtrees hanging off its root-to-leaf path ("bucket i · known/total"), contacts gold-ringed, the lookup target dropped in along its bit-path. A k slider (shrink k to force real multi-hop lookups), a bucket inspector with XOR distances, a stored-pairs view, a key-placement table, and deep links.Verification
closest()ordering, 8-node convergence, live lookups exact at k=4, k=2 partial tables forcing genuine multi-hop yet staying exact, offline-replay matching truth from every node, PUT→GET retrievable from every node, absent-key not-found, heal-after-two-crashes, restart-rejoin, tables converging under 15% message loss, determinism.#/kademliaroute in headless Chromium: 8-node network converges, all three invariants green (26/26 required buckets populated, 64/64 (node, target) lookups exact), the trie renders the buckets-as-subtrees correctly, no page errors.node scripts/verify-project.mjs quorum-distsys-k7r2— scope + conformance + lint + build).Scope
Single project, single folder — every changed file is under
projects/quorum-distsys-k7r2/(Golden Rule). Journal updated with the design, a session-log entry, and a six-item Kademlia backlog for future sessions.🤖 Generated with Claude Code
https://claude.ai/code/session_019XuHZAjE3TuggDG5k91gL6
Generated by Claude Code