Partition Boxes by key hash with balanced, fenced per-partition ownership

[predatorray]

Summary

Removes the single-node write bottleneck per Box: every Box is now split into a creation-time-fixed number of hash partitions (CRC32C of the key, mod the count in the new ZK BoxDescriptor; default 8, partitions.per.box.default). Each partition is an independent LSM engine (WAL / memtable / manifest / SSTables / Syrups) under its own fenced ZooKeeper ownership lease, so the partitions of one Box are owned and served by multiple nodes concurrently. Design, decisions, and trade-offs are captured in BOX_PARTITIONING_PLAN.md.

Coordination

• Layout becomes boxes/<box>/meta (immutable descriptor), boxes/<box>/partitions/<p>/{owner,manifest}, plus cluster/{balancer,assignment}.
• New CoordinationService.children() (fake + ZooKeeper impl + shared contract test) for Box enumeration.

Server

• BoxOwnership → PartitionOwnership; the lease + recover + pointer-CAS handover sequence is unchanged, at partition granularity. CandyboxNode keys ownership, heartbeat, compaction/GC/multipart-sweep by (box, partition).
• createBox publishes the descriptor and owns all partitions initially; deleteBox takes over free partitions (or, with force, defers to the owners' deleted-box sweep); boxExists/listBoxes now answer cluster-wide from coordination.
• New PartitionBalancer (elected coordinator under the cluster/balancer lease): publishes a sticky, failover-eager assignment; migration away from live owners is rate-limited (balancer.max.moves.per.round, default 4) while dead-node failover is never limited; releases flush the memtable first so the next owner's WAL replay is tiny. This also adds automatic failover, which previously only existed as a manual/test path.

Protocol & client

• Partition fields on list / delete-range / list-uploads requests; CreateBox carries an optional partition count; new BoxInfo request/response; MOVED names the requested partition's owner.
• The client routes keyed ops (including all multipart ops — they already carry the object key) to the key's partition owner, caches descriptors, scatter-gathers listings in global key order (the lastKey cursor still works), fans out range deletes, and falls back to a client-side byte copy for cross-partition copy/rename/uploadPartCopy (same-partition stays zero-copy and atomic).

Bug fix

Recovering a partition whose current WAL is empty (now the common case thanks to the pre-handover flush, but reachable before this change too) called hlc.observe(Hlc.MIN), and the skew check in HybridLogicalClock.observe overflowed and threw. Fixed by skipping the observe on an empty replay and making the check overflow-safe.

Tests & docs

• New: PartitioningTest, BoxDescriptorTest, PartitionBalancerTest (even spread, rate limit, stickiness, failover, deleted-box sweep), and PartitionedBoxIT — a fakes-based multi-node IT driving a partition-spread Box through the cluster client (routing, merged listings, fan-out deletes, cross-partition copy fallbacks, multipart).
• Updated suites across coordination/protocol/server/client/admin + the BookKeeper-backed ITs (ServerClientLifecycleIT now runs with 2 partitions per Box over embedded BK).
• DESIGN.md, OPERATIONS.md, README, and the shipped candybox.properties.example (balancer enabled at 5 s) updated.

All unit suites and all 12 integration test classes pass locally (mvn test, mvn verify -pl candybox-integration-tests).

https://claude.ai/code/session_015ZRRX63Hie7PDKxsH4Zosn

---

Generated by Claude Code

[github-actions]

✅ S3 compatibility gate passed

• mode: gate
• result: 164 passed in 78.28s (0:01:18)

[codecov]

Codecov Report

❌ Patch coverage is 89.02439% with 72 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.05%. Comparing base (516507d) to head (fe084be).

Files with missing lines	Patch %	Lines
...a/me/predatorray/candybox/server/CandyboxNode.java	82.48%	16 Missing and 8 partials ⚠️
...me/predatorray/candybox/client/CandyboxClient.java	83.17%	5 Missing and 13 partials ⚠️
...predatorray/candybox/server/PartitionBalancer.java	86.61%	12 Missing and 5 partials ⚠️
...redatorray/candybox/server/PartitionOwnership.java	81.48%	4 Missing and 1 partial ⚠️
...redatorray/candybox/server/NodeRequestHandler.java	96.38%	2 Missing and 1 partial ⚠️
...coordination/fake/InMemoryCoordinationService.java	86.66%	0 Missing and 2 partials ⚠️
.../coordination/zk/ZooKeeperCoordinationService.java	71.42%	2 Missing ⚠️
...va/me/predatorray/candybox/client/CandyboxCli.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main      #19      +/-   ##
============================================
+ Coverage     83.01%   84.05%   +1.04%     
- Complexity      518      528      +10     
============================================
  Files           125      129       +4     
  Lines          6435     6925     +490     
  Branches        906     1014     +108     
============================================
+ Hits           5342     5821     +479     
+ Misses          726      719       -7     
- Partials        367      385      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.