From 074e46cf915053e48392a4341d3c907336277569 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:12:54 +0000 Subject: [PATCH 1/7] Cross-partition zero-copy: LSM core (manifest v3 rename intents, zeroCopyPut, conditional delete) Add the engine-level primitives both features build on: - ManifestEdit/State/Serializer v3 carry cross-partition RenameIntent records, replayed on handover exactly like in-flight multipart upload state. - BoxEngine.resolveLocator (relay the source parts), zeroCopyPut (reuse a foreign partition's segments verbatim), deleteCandyConditional (LWW-safe source delete), rename-intent record/list/clear, and a public referencedSyrups() for Box-global GC. Plan in CROSS_PARTITION_ZERO_COPY_PLAN.md. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- CROSS_PARTITION_ZERO_COPY_PLAN.md | 201 ++++++++++++++++++ .../candybox/lsm/engine/BoxEngine.java | 140 ++++++++++++ .../candybox/lsm/manifest/Manifest.java | 1 + .../candybox/lsm/manifest/ManifestEdit.java | 34 ++- .../lsm/manifest/ManifestSerializer.java | 59 ++++- .../candybox/lsm/manifest/ManifestState.java | 22 +- .../candybox/lsm/manifest/RenameIntent.java | 58 +++++ .../CrossPartitionZeroCopyEngineTest.java | 153 +++++++++++++ .../lsm/manifest/ManifestEditTest.java | 6 +- .../candybox/lsm/manifest/ManifestTest.java | 21 ++ 10 files changed, 682 insertions(+), 13 deletions(-) create mode 100644 CROSS_PARTITION_ZERO_COPY_PLAN.md create mode 100644 candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/RenameIntent.java create mode 100644 candybox-lsm/src/test/java/me/predatorray/candybox/lsm/engine/CrossPartitionZeroCopyEngineTest.java diff --git a/CROSS_PARTITION_ZERO_COPY_PLAN.md b/CROSS_PARTITION_ZERO_COPY_PLAN.md new file mode 100644 index 0000000..cc4b7f3 --- /dev/null +++ b/CROSS_PARTITION_ZERO_COPY_PLAN.md @@ -0,0 +1,201 @@ +# Cross-Partition Zero-Copy Copy/Rename — Implementation Plan + +## Goal + +Restore the **zero-copy** promise for `copy`/`rename` across partition boundaries, and make a +cross-partition `rename` **eventually atomic** (it converges to "destination present, source gone" +even across crashes) instead of the current best-effort "copy-then-delete that can leave both keys +live forever". + +Box partitioning (commit `7c1a70e`) split each Box into hash partitions, each an independent LSM +engine with its own fenced owner and its own reference-counted Syrup GC. That broke two things: + +1. **Zero-copy** — a same-partition `copy`/`rename` reuses the source locator's Syrup *segments* + verbatim (no byte copy). Across partitions the client falls back to `get`+`put` (a full byte + copy) because Syrup GC is reference-counted *within one manifest*: the source partition's GC + cannot see a destination partition's references, so segment sharing would let the source delete a + Syrup the destination still points at. +2. **Atomicity** — a cross-partition `rename` is `byteCopy` + `deleteCandy`, two RPCs to two + independently-fenced owners. A crash between them leaves both keys live, permanently. + +The key insight is that **the blob layer is already physically global**: a `SegmentRef` resolves +bytes purely by `ledgerStore.openLedger(syrupId)` (a raw BookKeeper ledger read keyed only on the +globally-unique Syrup id — see `SyrupReader`), with no Box/partition scoping. So no bytes ever need +to move across partitions; what broke is purely the *bookkeeping* (GC liveness) and the +*coordination* (cross-owner atomicity). This plan fixes exactly those two. + +## Decisions + +1. **Box-global Syrup GC (Option A).** A Syrup is physically reclaimed only once **no partition of + the Box references it**. Each partition owner publishes its referenced-Syrup set to coordination; + the Syrup's creating partition deletes it only when it is in no sibling partition's published set. + This subsumes the existing within-manifest reference counting and is safe by construction (the + liveness decision is always recomputed from currently-committed references). +2. **Cross-partition zero-copy via client relay.** The client fetches the source's `CandyLocator` + parts from the source owner (new `GetCandyLocator` op) and hands them to the destination owner + (new `ZeroCopyPut` op), which writes a destination locator reusing the source's segments + verbatim. No bytes move. Plain `copy` needs nothing more. +3. **Rename intent journal, owner-local completion via a ZK rendezvous.** A cross-partition `rename` + records a durable **rename intent** in the *source* partition's manifest (the owed conditional + delete), the destination writes a durable **completion marker** in coordination when its + zero-copy put commits, and the **source partition's owner finalizes its own intents** — on its + maintenance loop and after handover replay — by reading that marker and tombstoning the source + (LWW-conditioned on the source's HLC, so a legitimately recreated source is never clobbered). + Roll-forward only; no abort. + - *Why not the "coordinator-driven resumer" originally sketched:* completing a rename means + deleting the source key on a possibly-different partition's owner. The architecture has **no + server-to-server RPC** (the server module doesn't depend on the client, and `CandyboxNode` has + no outbound transport). Owner-local completion via a ZK rendezvous delivers the identical + roll-forward guarantee using only the existing `CoordinationService` primitives, with no new + cross-node call path, and is strictly simpler. The coordinator (balancer holder) retains a + minor janitor role: reaping abandoned intents/markers past a long timeout. + +## Design + +### Consistency + +- **Zero-copy correctness** is unchanged from the same-partition path: the destination locator + shares the source's segments; reads stream those segments directly from BookKeeper regardless of + which partition's owner serves the read. +- **Cross-partition `copy`** is atomic-enough already (no source mutation): a crash leaves the + destination present or absent, and the idempotency token makes retry safe. +- **Cross-partition `rename`** becomes **eventually atomic** (converges to completed). A reader + between the destination commit and the source delete may still momentarily observe both keys — + this is eventual, not linearizable, atomicity, and matches what "move" usually means. The one + residual non-atomic window (destination committed, completion marker not yet durable, then a + crash) degrades to "both keys live", i.e. exactly today's behavior — never data loss. +- **Recreate-safety:** the source delete is LWW-conditioned on the source HLC captured at intent + time, so a source key legitimately re-`put` after the rename began is never deleted by a delayed + completion. + +### Box-global Syrup GC (Option A) + +Today `BoxEngine.recomputeOrphanSyrupsLocked` computes a partition's *referenced* set (manifest +SSTable refs ∪ in-flight multipart refs ∪ memtable refs ∪ open write Syrup) and marks any +`liveSyrups` not in it as a pending orphan; `GarbageCollector.collect(engine)` deletes orphans aged +past the grace period. A Syrup only ever lives in `liveSyrups` of the partition that *created* it, +so only that partition ever tries to delete it. + +Changes: + +- **Publish:** each partition owner writes its referenced-Syrup set to + `boxes//partitions/

/refs` (a versioned KV) whenever the set changes materially — at + minimum once per GC tick, and **synchronously as part of committing a `ZeroCopyPut`** (so a new + cross-partition reference is visible before the rename's source delete can run). +- **Gate deletion globally:** `GarbageCollector` deletes an orphan Syrup `S` of partition `(box,p)` + only if `S` is in **no** sibling partition's published refs. The local manifest live-set drop + (`dropSyrups`) still happens per partition; only the physical `deleteLedger` is gated by the + Box-global union. + +Safety: a Syrup referenced cross-partition was referenced by the destination (and published) before +the rename's source delete committed; the grace period plus "delete only if globally unreferenced" +means a partial/crashed rename can only *retain* a Syrup (a leak), never delete a referenced one. +Leaks are already the accepted v1 failure mode (DESIGN §9d). A partition with no live owner keeps +its last-published refs in ZK (conservative), and a new owner republishes on takeover. + +### Cross-partition zero-copy protocol + +New wire ops (trusted intra-cluster client only): + +- `GetCandyLocatorRequest(box, key)` → `CandyLocatorResponse(parts, contentType, userMetadata, hlc, + createdAtMillis, acl)` — exposes the resolved locator's segment list so the client can relay it. +- `ZeroCopyPutRequest(box, dstKey, parts, contentType, userMetadata, acl, idempotencyToken, + renameToken?, srcKey?, srcPartition?, srcHlc?)` → `HeadCandyResponse` — the destination owner + writes a destination locator reusing `parts` verbatim, commits it to WAL + memtable, publishes its + refs, and (when `renameToken` is set) writes the completion marker + `boxes//renames/` = `{dstKey, srcKey, srcPartition, srcHlc}` after the locator is + durable. +- `CompleteRenameRequest(box, srcKey, srcPartition, renameToken, srcHlc)` → `OkResponse` — the + source owner finalizes the rename: LWW-conditioned tombstone of `srcKey`, clear the intent, delete + the marker. (Same code path the maintenance loop uses; this is just the synchronous fast path.) + +Client `renameCandy` (cross-partition): + +1. `GetCandyLocator(src)` from the source owner → parts + `srcHlc`. Also records the rename intent + on the source owner (folded into `GetCandyLocator` via a `forRename` flag, or a dedicated + `PrepareRename` — see work items). +2. `ZeroCopyPut(dst, parts, renameToken=token, src..., srcHlc)` on the destination owner. +3. `CompleteRename(token)` on the source owner. + +Client `copyCandy` (cross-partition): steps 1–2 only, no `renameToken`, no intent. + +### Rename intent journal + +A `RenameIntent { token, srcKey, srcHlc, dstKey, dstPartition }` is stored in the **source** +partition's `ManifestState`, serialized in `ManifestEdit` as a trailing field exactly parallel to +the existing in-flight multipart upload state (`addedUploads`/`upsertParts`/`removedUploads`), and +therefore replayed on handover — a new source owner re-acquires the obligation automatically. + +Lifecycle on the source owner: + +- **Record** at prepare (step 1) via a manifest edit. +- **Finalize** (step 3, or the maintenance loop, or post-handover replay): read + `boxes//renames/`. If present → tombstone `srcKey` LWW-conditioned on `srcHlc`, clear + the intent (manifest edit), delete the marker. If absent and the intent is older than + `rename.intent.abandon.millis` → drop the intent (the rename never reached the destination; the + source stays live). Idempotent throughout. + +Crash windows (all converge; none lose data): + +| Crash point | State | Resolution | +|---|---|---| +| before step 1 durable | src live, dst absent | client retry | +| after step 1, before step 2 | src live (intent PENDING), dst absent, no marker | maintenance abandons the intent; retry restarts | +| after step 2, before step 3 (old W2) | both keys live, intent PENDING, **marker present** | maintenance finalizes → src tombstoned | +| after step 3 | dst live, src gone | done | +| dst committed, marker not yet durable, crash | both keys live, no marker | abandoned → both live (== today's behavior, no data loss) | + +### Coordination layout (additions) + +``` +boxes//partitions/

/refs versioned KV: this partition's referenced-Syrup id set (Box-global GC) +boxes//renames/ rename completion marker: {dstKey, srcKey, srcPartition, srcHlc} +``` + +### Configuration + +| Key | Default | Meaning | +|---|---|---| +| `rename.intent.abandon.millis` | 60000 | Age after which a source-side rename intent with no completion marker is abandoned. | + +Box-global GC reuses the existing `ledger.gc.grace.millis`. + +## Work items + +1. **protocol** — `GetCandyLocatorRequest`/`CandyLocatorResponse`, `ZeroCopyPutRequest`, + `CompleteRenameRequest`, `PrepareRenameRequest` (or a `forRename` flag on `GetCandyLocator`); + opcodes + `MessageCodec` encode/decode for `Part`/`SegmentRef` lists; codec round-trip tests. +2. **coordination** — `CandyboxKeys.partitionRefsKey(box, p)` and `renameMarkerKey(box, token)`; + `children` already suffices for enumeration. +3. **lsm** — `ManifestEdit`/`ManifestState`/`ManifestSerializer` (v3): `addedRenameIntents` + + `removedRenameIntents`, replayed on handover; `BoxEngine`: `resolveLocator(key)` (for + `GetCandyLocator`), `zeroCopyPut(dstKey, parts, meta, acl, token)`, `recordRenameIntent`, + `clearRenameIntent`, `listRenameIntents`, `deleteCandyConditional(key, expectedHlc)`, and a + public `referencedSyrups()` accessor for publishing. +4. **server** — `CandyboxNode`: publish per-partition refs; Box-global GC gate in + `GarbageCollector` (consult sibling `refs`); a rename-intent maintenance sweep (finalize via + marker / abandon by age) on the existing maintenance worker; replay-driven finalize after + takeover. `NodeRequestHandler`: handlers for the new ops. `ServerConfig`/`CandyboxConfig`: + `renameIntentAbandonMillis`. +5. **client** — `CandyboxClient.renameCandy`/`copyCandy` cross-partition zero-copy via + `GetCandyLocator` + `ZeroCopyPut` (+ `CompleteRename` for rename); `Router` plumbing for the new + ops (they are partition-keyed). +6. **tests** — + - lsm: zero-copy put from relayed parts; conditional delete LWW; rename-intent record/clear/list + + serializer round-trip + handover replay; `referencedSyrups` accessor. + - server: Box-global GC keeps a cross-referenced Syrup alive then reclaims it once unreferenced; + intent finalize via marker; intent abandon by age; handover replays an intent and finalizes. + - protocol: codec round-trips for the new messages. + - client: cross-partition copy/rename now stay zero-copy (no `get`+`put`); the 3-step rename flow; + idempotent retry. + - integration: cross-partition zero-copy copy/rename through the routing transport (bytes shared, + not re-stored); a crash-injected rename (drop step 3) converges via the maintenance sweep; + Box-global GC across two partitions on a real BookKeeper/ZK cluster. + - **regression:** the full existing suite stays green (same-partition zero-copy, the prior + cross-partition byte-copy tests are updated to assert the new zero-copy behavior). +7. **docs** — DESIGN.md (§6 copy/rename, §9 GC, consistency §3), README.md, OPERATIONS.md (GC + + rename intents + new config), the Hugo site (`architecture`, `client`, `operations`, + `reference/configuration`), and this plan. + +Out of scope (unchanged v1 simplifications): zero-copy `UploadPartCopy` (still buffers, even +same-partition); read scaling off owners; Syrup defragmentation; watch-based coordination (polling). diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/engine/BoxEngine.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/engine/BoxEngine.java index 3a0465f..af50bca 100644 --- a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/engine/BoxEngine.java +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/engine/BoxEngine.java @@ -56,6 +56,7 @@ import me.predatorray.candybox.lsm.manifest.ManifestEdit; import me.predatorray.candybox.lsm.manifest.ManifestState; import me.predatorray.candybox.lsm.manifest.MultipartUploadState; +import me.predatorray.candybox.lsm.manifest.RenameIntent; import me.predatorray.candybox.lsm.memtable.Memtable; import me.predatorray.candybox.lsm.sstable.SSTableMeta; import me.predatorray.candybox.lsm.sstable.SSTableReader; @@ -708,6 +709,145 @@ private CandyMetadata copyOrRename(CandyKey src, CandyKey dst, String idempotenc } } + // ---- cross-partition zero-copy (copy/rename across partitions) --------------------------- + + /** + * Resolves a key to its live {@link CandyLocator}, for a cross-partition copy/rename: the client + * relays the returned parts to the destination partition's owner (which has no way to read this + * partition's manifest directly), where {@link #zeroCopyPut} reuses the segments verbatim. + * + * @throws CandyNotFoundException if there is no live Candy at {@code key} + */ + public CandyLocator resolveLocator(CandyKey key) { + Validation.checkCandyKey(key, config.sizeLimits()); + return resolveLive(key) + .orElseThrow(() -> new CandyNotFoundException(box.value(), key.value())); + } + + /** + * Writes a fresh PUT at {@code dst} that reuses {@code parts} verbatim — the zero-copy trick, but + * across partitions: the {@code parts} were resolved from a source Candy in another + * partition (via {@link #resolveLocator} relayed by the client). No Candy bytes are read or + * rewritten; the destination locator points at the very same Syrup segments, which stay alive + * Box-globally because every partition publishes its referenced-Syrup set (see the server's + * Box-global GC). Same-Box only. + */ + public CandyMetadata zeroCopyPut(CandyKey dst, List parts, String contentType, + Map userMetadata, long createdAtMillis, + ObjectAcl acl, String idempotencyToken) { + Validation.checkCandyKey(dst, config.sizeLimits()); + Map metadata = userMetadata == null ? Map.of() : Map.copyOf(userMetadata); + if (idempotencyToken != null) { + CandyMetadata cached = idempotencyCache.get(idempotencyToken); + if (cached != null) { + return cached; + } + } + lock.writeLock().lock(); + try { + rejectIfStalled(); + Hlc stamp = hlc.tick(); + CandyLocator dstLocator = new CandyLocator(stamp, LocatorType.PUT, contentType, metadata, + createdAtMillis > 0 ? createdAtMillis : clock.currentTimeMillis(), + List.copyOf(parts), acl == null ? ObjectAcl.NONE : acl); + Mutation mutation = new Mutation(dst, dstLocator); + wal.append(mutation); + active.put(mutation); + maybeFlushLocked(); + CandyMetadata result = CandyMetadata.from(dstLocator); + if (idempotencyToken != null) { + idempotencyCache.put(idempotencyToken, result); + } + putCount.incrementAndGet(); + return result; + } finally { + lock.writeLock().unlock(); + } + } + + /** + * Conditionally tombstones {@code key} only if its live locator's HLC equals {@code expectedHlc} + * — the LWW-safe source delete that finalizes a cross-partition rename. If the source was already + * deleted, or legitimately re-{@code put} after the rename began (a strictly newer HLC), the + * delete is a no-op so a delayed/duplicated finalize can never clobber a newer value. + * + * @return {@code true} if a tombstone was written, {@code false} if the condition did not hold + */ + public boolean deleteCandyConditional(CandyKey key, Hlc expectedHlc) { + Validation.checkCandyKey(key, config.sizeLimits()); + lock.writeLock().lock(); + try { + rejectIfStalled(); + CandyLocator live = resolveLiveLocked(key).orElse(null); + if (live == null || expectedHlc == null || !live.hlc().equals(expectedHlc)) { + return false; + } + Hlc stamp = hlc.tick(); + Mutation mutation = new Mutation(key, CandyLocator.tombstone(stamp, + clock.currentTimeMillis())); + wal.append(mutation); + active.put(mutation); + maybeFlushLocked(); + deleteCount.incrementAndGet(); + return true; + } finally { + lock.writeLock().unlock(); + } + } + + /** Records a cross-partition {@link RenameIntent} this (source) partition owes a delete for. */ + public void recordRenameIntent(RenameIntent intent) { + lock.writeLock().lock(); + try { + manifest.apply(ManifestEdit.builder().addRenameIntent(intent).build()); + } finally { + lock.writeLock().unlock(); + } + } + + /** Clears a recorded rename intent once it has been finalized or abandoned. */ + public void clearRenameIntent(String token) { + lock.writeLock().lock(); + try { + manifest.apply(ManifestEdit.builder() + .removedRenameIntents(java.util.Set.of(token)).build()); + } finally { + lock.writeLock().unlock(); + } + } + + /** The cross-partition rename intents this partition still owes a source delete for. */ + public List listRenameIntents() { + return new java.util.ArrayList<>(manifest.current().renameIntents().values()); + } + + /** + * The set of Syrup ids this partition currently references — manifest SSTable refs, in-flight + * multipart parts, the memtable, and the open write Syrup. Published to coordination so the + * Box-global GC never reclaims a Syrup a sibling partition still points at (cross-partition + * zero-copy copy/rename). + */ + public java.util.Set referencedSyrups() { + lock.readLock().lock(); + try { + ManifestState current = manifest.current(); + java.util.Set referenced = new java.util.HashSet<>(current.referencedSyrups()); + referenced.addAll(current.multipartReferencedSyrups()); + for (java.util.Iterator it = active.iterator(); it.hasNext(); ) { + for (SegmentRef seg : it.next().locator().segments()) { + referenced.add(seg.syrupId()); + } + } + long openSyrup = syrupManager.currentSyrupId(); + if (openSyrup >= 0) { + referenced.add(openSyrup); + } + return referenced; + } finally { + lock.readLock().unlock(); + } + } + /** * Deletes every live Candy whose key falls in {@code [startInclusive, endExclusive)} with a single * O(1) range tombstone — no per-key scan or write. Either bound may be null (null start = from the diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/Manifest.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/Manifest.java index 048a121..476e3d1 100644 --- a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/Manifest.java +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/Manifest.java @@ -136,6 +136,7 @@ private static ManifestEdit checkpoint(ManifestState state) { .addedSyrups(state.liveSyrups()) .newWalLedgerId(wal) .addedUploads(new ArrayList<>(state.multipartUploads().values())) + .addedRenameIntents(new ArrayList<>(state.renameIntents().values())) .build(); } } diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestEdit.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestEdit.java index 869e539..c41c655 100644 --- a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestEdit.java +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestEdit.java @@ -42,6 +42,8 @@ * @param addedUploads new multipart uploads created by this edit (CreateMultipartUpload) * @param upsertParts parts added/replaced under an existing upload (UploadPart) * @param removedUploads upload ids dropped by this edit (CompleteMultipartUpload / Abort) + * @param addedRenameIntents in-flight cross-partition rename intents recorded by this edit (v3) + * @param removedRenameIntents rename intent tokens finalized/abandoned by this edit (v3) * @param ownerFencingToken fencing token of the authoring owner ({@code 0} = "stamp at apply time") */ public record ManifestEdit( @@ -53,6 +55,8 @@ public record ManifestEdit( List addedUploads, List upsertParts, Set removedUploads, + List addedRenameIntents, + Set removedRenameIntents, long ownerFencingToken) { public ManifestEdit { @@ -63,6 +67,8 @@ public record ManifestEdit( addedUploads = addedUploads == null ? List.of() : List.copyOf(addedUploads); upsertParts = upsertParts == null ? List.of() : List.copyOf(upsertParts); removedUploads = removedUploads == null ? Set.of() : Set.copyOf(removedUploads); + addedRenameIntents = addedRenameIntents == null ? List.of() : List.copyOf(addedRenameIntents); + removedRenameIntents = removedRenameIntents == null ? Set.of() : Set.copyOf(removedRenameIntents); if (ownerFencingToken < 0) { throw new IllegalArgumentException("ownerFencingToken must be non-negative"); } @@ -75,13 +81,14 @@ public static Builder builder() { /** Convenience: a flush edit adding one table plus its syrups, optionally rotating the WAL. */ public static ManifestEdit flush(SSTableMeta table, Set syrups, Long newWalLedgerId) { return new ManifestEdit(List.of(table), Set.of(), syrups, Set.of(), newWalLedgerId, - List.of(), List.of(), Set.of(), 0L); + List.of(), List.of(), Set.of(), List.of(), Set.of(), 0L); } /** Returns a copy with the given owner fencing token (used by {@link Manifest#apply}). */ public ManifestEdit withOwnerFencingToken(long token) { return new ManifestEdit(addedTables, removedTableLedgerIds, addedSyrups, removedSyrups, - newWalLedgerId, addedUploads, upsertParts, removedUploads, token); + newWalLedgerId, addedUploads, upsertParts, removedUploads, addedRenameIntents, + removedRenameIntents, token); } /** @@ -112,6 +119,8 @@ public static final class Builder { private List addedUploads = List.of(); private List upsertParts = List.of(); private Set removedUploads = Set.of(); + private List addedRenameIntents = List.of(); + private Set removedRenameIntents = Set.of(); private long ownerFencingToken = 0L; public Builder addedTables(List v) { @@ -154,6 +163,24 @@ public Builder removedUploads(Set v) { return this; } + public Builder addedRenameIntents(List v) { + this.addedRenameIntents = v; + return this; + } + + public Builder removedRenameIntents(Set v) { + this.removedRenameIntents = v; + return this; + } + + /** Adds a single rename intent without disturbing any already-set ones. */ + public Builder addRenameIntent(RenameIntent intent) { + java.util.ArrayList next = new java.util.ArrayList<>(addedRenameIntents); + next.add(intent); + this.addedRenameIntents = next; + return this; + } + public Builder ownerFencingToken(long v) { this.ownerFencingToken = v; return this; @@ -177,7 +204,8 @@ public Builder addUpload(MultipartUploadState upload) { public ManifestEdit build() { return new ManifestEdit(addedTables, removedTableLedgerIds, addedSyrups, removedSyrups, - newWalLedgerId, addedUploads, upsertParts, removedUploads, ownerFencingToken); + newWalLedgerId, addedUploads, upsertParts, removedUploads, addedRenameIntents, + removedRenameIntents, ownerFencingToken); } } diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestSerializer.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestSerializer.java index ebee08f..2a9682c 100644 --- a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestSerializer.java +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestSerializer.java @@ -22,6 +22,7 @@ import java.util.Map; import java.util.Set; import me.predatorray.candybox.common.CandyKey; +import me.predatorray.candybox.common.Hlc; import me.predatorray.candybox.common.Part; import me.predatorray.candybox.common.SegmentRef; import me.predatorray.candybox.common.exception.SerializationException; @@ -34,12 +35,15 @@ * {@link SSTableMeta} they contain. * *

v2 layout adds the multipart-upload tracking fields ({@code addedUploads}, - * {@code upsertParts}, {@code removedUploads}) at the end of the v1 record. Older v1 records cannot - * be read back; this is acceptable because the project has no production data to migrate. + * {@code upsertParts}, {@code removedUploads}) at the end of the v1 record. v3 layout appends + * the cross-partition rename-intent fields ({@code addedRenameIntents}, {@code removedRenameIntents}). + * A v2 record (no rename-intent fields) still reads back as an edit with empty intent sets. Older v1 + * records cannot be read back; this is acceptable because the project has no production data to + * migrate. */ public final class ManifestSerializer { - public static final byte FORMAT_VERSION = 2; + public static final byte FORMAT_VERSION = 3; private ManifestSerializer() { } @@ -79,13 +83,23 @@ public static byte[] serialize(ManifestEdit edit) { for (String id : edit.removedUploads()) { w.writeString(id); } + + // ---- cross-partition rename intents (v3) --------------------------------------------- + w.writeVarInt(edit.addedRenameIntents().size()); + for (RenameIntent intent : edit.addedRenameIntents()) { + writeRenameIntent(w, intent); + } + w.writeVarInt(edit.removedRenameIntents().size()); + for (String token : edit.removedRenameIntents()) { + w.writeString(token); + } return w.toByteArray(); } public static ManifestEdit deserialize(byte[] data) { BinaryReader r = new BinaryReader(data); int version = r.readByte(); - if (version != FORMAT_VERSION) { + if (version != 2 && version != FORMAT_VERSION) { throw new SerializationException("Unsupported ManifestEdit version: " + version); } int tableCount = r.readVarInt(); @@ -118,8 +132,43 @@ public static ManifestEdit deserialize(byte[] data) { removedUploads.add(r.readString()); } + List addedIntents = new ArrayList<>(); + Set removedIntents = new LinkedHashSet<>(); + if (version >= 3) { + int addedIntentCount = r.readVarInt(); + for (int i = 0; i < addedIntentCount; i++) { + addedIntents.add(readRenameIntent(r)); + } + int removedIntentCount = r.readVarInt(); + for (int i = 0; i < removedIntentCount; i++) { + removedIntents.add(r.readString()); + } + } + return new ManifestEdit(tables, removedTables, addedSyrups, removedSyrups, newWal, - addedUploads, upserts, removedUploads, ownerFencingToken); + addedUploads, upserts, removedUploads, addedIntents, removedIntents, + ownerFencingToken); + } + + private static void writeRenameIntent(BinaryWriter w, RenameIntent intent) { + w.writeString(intent.token()); + w.writeString(intent.srcKey()); + w.writeVarLong(intent.srcHlc().physicalMillis()); + w.writeVarInt(intent.srcHlc().logicalCounter()); + w.writeInt(intent.srcHlc().nodeId()); + w.writeString(intent.dstKey()); + w.writeVarInt(intent.dstPartition()); + w.writeVarLong(Math.max(0, intent.createdAtMillis())); + } + + private static RenameIntent readRenameIntent(BinaryReader r) { + String token = r.readString(); + String srcKey = r.readString(); + Hlc srcHlc = new Hlc(r.readVarLong(), r.readVarInt(), r.readInt()); + String dstKey = r.readString(); + int dstPartition = r.readVarInt(); + long createdAtMillis = r.readVarLong(); + return new RenameIntent(token, srcKey, srcHlc, dstKey, dstPartition, createdAtMillis); } private static void writeTable(BinaryWriter w, SSTableMeta t) { diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestState.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestState.java index 8309040..dcfaecd 100644 --- a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestState.java +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/ManifestState.java @@ -37,19 +37,22 @@ public final class ManifestState { private static final ManifestState EMPTY = - new ManifestState(List.of(), Set.of(), -1L, Map.of()); + new ManifestState(List.of(), Set.of(), -1L, Map.of(), Map.of()); private final List tables; private final Set liveSyrups; private final long walLedgerId; private final Map multipartUploads; + private final Map renameIntents; private ManifestState(List tables, Set liveSyrups, long walLedgerId, - Map multipartUploads) { + Map multipartUploads, + Map renameIntents) { this.tables = List.copyOf(tables); this.liveSyrups = Set.copyOf(liveSyrups); this.walLedgerId = walLedgerId; this.multipartUploads = Collections.unmodifiableMap(new LinkedHashMap<>(multipartUploads)); + this.renameIntents = Collections.unmodifiableMap(new LinkedHashMap<>(renameIntents)); } public static ManifestState empty() { @@ -107,6 +110,11 @@ public Map multipartUploads() { return multipartUploads; } + /** Snapshot of in-flight cross-partition rename intents owed by this partition, keyed by token. */ + public Map renameIntents() { + return renameIntents; + } + /** * Syrups referenced by parts of in-flight multipart uploads. These must not be GC'd while the * upload is pending — even though no SSTable points at them yet. @@ -153,6 +161,14 @@ public ManifestState apply(ManifestEdit edit) { for (String dropped : edit.removedUploads()) { newUploads.remove(dropped); } - return new ManifestState(newTables, newSyrups, newWal, newUploads); + + Map newIntents = new LinkedHashMap<>(renameIntents); + for (RenameIntent intent : edit.addedRenameIntents()) { + newIntents.put(intent.token(), intent); + } + for (String token : edit.removedRenameIntents()) { + newIntents.remove(token); + } + return new ManifestState(newTables, newSyrups, newWal, newUploads, newIntents); } } diff --git a/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/RenameIntent.java b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/RenameIntent.java new file mode 100644 index 0000000..c95f05e --- /dev/null +++ b/candybox-lsm/src/main/java/me/predatorray/candybox/lsm/manifest/RenameIntent.java @@ -0,0 +1,58 @@ +/* + * Copyright (c) 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package me.predatorray.candybox.lsm.manifest; + +import me.predatorray.candybox.common.Hlc; + +/** + * A durable record, held in the source partition's manifest, of a cross-partition + * {@code rename} that is in flight: the destination's zero-copy {@code put} has been (or is being) + * written in another partition, and the source partition still owes the conditional delete of + * {@code srcKey}. + * + *

The intent is the cross-key analog of an in-flight {@link MultipartUploadState} — it is + * serialized into {@link ManifestEdit} and replayed on handover, so a new source owner re-acquires + * the obligation. The source owner finalizes it (on its maintenance loop, or synchronously via + * {@code CompleteRename}, or after handover replay) by checking the coordination rendezvous marker + * {@code boxes//renames/}: present ⇒ tombstone {@code srcKey} (LWW-conditioned on + * {@code srcHlc}, so a legitimately re-{@code put} source is never clobbered) and clear the intent; + * absent past the abandon window ⇒ drop the intent (the rename never reached the destination). + * + * @param token the rename's idempotency/rendezvous token (also the coordination marker key) + * @param srcKey the source key to delete once the destination is confirmed durable + * @param srcHlc the HLC of the source locator captured at prepare time (LWW delete guard) + * @param dstKey the destination key (informational; the marker confirms the destination put) + * @param dstPartition the destination key's partition (informational / diagnostics) + * @param createdAtMillis when the intent was recorded, used to abandon stale intents + */ +public record RenameIntent(String token, String srcKey, Hlc srcHlc, String dstKey, int dstPartition, + long createdAtMillis) { + + public RenameIntent { + if (token == null || token.isEmpty()) { + throw new IllegalArgumentException("token is required"); + } + if (srcKey == null || srcKey.isEmpty()) { + throw new IllegalArgumentException("srcKey is required"); + } + if (dstKey == null || dstKey.isEmpty()) { + throw new IllegalArgumentException("dstKey is required"); + } + if (srcHlc == null) { + throw new IllegalArgumentException("srcHlc is required"); + } + } +} diff --git a/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/engine/CrossPartitionZeroCopyEngineTest.java b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/engine/CrossPartitionZeroCopyEngineTest.java new file mode 100644 index 0000000..3e70e77 --- /dev/null +++ b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/engine/CrossPartitionZeroCopyEngineTest.java @@ -0,0 +1,153 @@ +/* + * Copyright (c) 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package me.predatorray.candybox.lsm.engine; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; +import me.predatorray.candybox.bookkeeper.fake.InMemoryLedgerStore; +import me.predatorray.candybox.common.BoxName; +import me.predatorray.candybox.common.CandyKey; +import me.predatorray.candybox.common.CandyLocator; +import me.predatorray.candybox.common.Hlc; +import me.predatorray.candybox.common.ManualClock; +import me.predatorray.candybox.common.auth.ObjectAcl; +import me.predatorray.candybox.common.config.CandyboxConfig; +import me.predatorray.candybox.common.exception.CandyNotFoundException; +import me.predatorray.candybox.lsm.manifest.RenameIntent; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; + +/** + * Engine-level coverage of the cross-partition zero-copy primitives: {@link BoxEngine#resolveLocator}, + * {@link BoxEngine#zeroCopyPut}, the LWW {@link BoxEngine#deleteCandyConditional}, the rename-intent + * journal, and the published referenced-Syrup set the Box-global GC relies on. + * + *

Two engines over one shared {@link InMemoryLedgerStore} stand in for two partitions of a Box: + * the destination engine reuses the source engine's Syrup segments verbatim, and reads stream those + * segments straight from the shared store — no bytes are copied. + */ +class CrossPartitionZeroCopyEngineTest { + + private final InMemoryLedgerStore store = new InMemoryLedgerStore(); + private final BoxName box = BoxName.of("my-box"); + private BoxEngine src; + private BoxEngine dst; + + @AfterEach + void tearDown() { + if (src != null) { + src.close(); + } + if (dst != null) { + dst.close(); + } + store.close(); + } + + private static byte[] bytes(String s) { + return s.getBytes(StandardCharsets.UTF_8); + } + + @Test + void zeroCopyPutReusesSourceSegmentsAcrossPartitions() { + src = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 1, new ManualClock(1000), 1L); + dst = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 2, new ManualClock(1000), 1L); + src.putCandy(CandyKey.of("src"), bytes("payload"), "text/plain", Map.of("k", "v"), null); + src.flush(); + + CandyLocator locator = src.resolveLocator(CandyKey.of("src")); + long srcSyrups = dst.manifestState().referencedSyrups().size(); + + dst.zeroCopyPut(CandyKey.of("dst"), locator.parts(), locator.contentType(), + locator.userMetadata(), locator.createdAtMillis(), locator.acl(), null); + + // Destination resolves the source's bytes without the source engine writing anything new, + // and the destination wrote no new Syrup of its own (it points at the source's segment). + assertThat(dst.getCandy(CandyKey.of("dst"))).isEqualTo(bytes("payload")); + assertThat(dst.headCandy(CandyKey.of("dst")).contentType()).isEqualTo("text/plain"); + dst.flush(); + assertThat(dst.manifestState().referencedSyrups()).hasSize((int) srcSyrups + 1); + // The destination references the source's Syrup id (the shared segment). + long sharedSyrup = locator.parts().get(0).segments().get(0).syrupId(); + assertThat(dst.referencedSyrups()).contains(sharedSyrup); + } + + @Test + void conditionalDeleteHonorsLwwGuard() { + src = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 1, new ManualClock(1000), 1L); + src.putCandy(CandyKey.of("k"), bytes("v1"), null, Map.of(), null); + Hlc v1 = src.resolveLocator(CandyKey.of("k")).hlc(); + + // A stale guard (a different HLC) is a no-op; the key stays live. + assertThat(src.deleteCandyConditional(CandyKey.of("k"), Hlc.MIN)).isFalse(); + assertThat(src.getCandy(CandyKey.of("k"))).isEqualTo(bytes("v1")); + + // The matching guard tombstones the key. + assertThat(src.deleteCandyConditional(CandyKey.of("k"), v1)).isTrue(); + assertThatThrownBy(() -> src.getCandy(CandyKey.of("k"))) + .isInstanceOf(CandyNotFoundException.class); + } + + @Test + void conditionalDeleteNeverClobbersARecreatedSource() { + src = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 1, new ManualClock(1000), 1L); + Hlc oldHlc = src.putCandy(CandyKey.of("k"), bytes("old"), null, Map.of(), null).hlc(); + // The key is legitimately re-put after the rename began (a strictly newer HLC). + src.putCandy(CandyKey.of("k"), bytes("new"), null, Map.of(), null); + + // A finalize keyed on the stale source HLC must not delete the fresh value. + assertThat(src.deleteCandyConditional(CandyKey.of("k"), oldHlc)).isFalse(); + assertThat(src.getCandy(CandyKey.of("k"))).isEqualTo(bytes("new")); + } + + @Test + void renameIntentsAreRecordedListedClearedAndSurviveHandover() { + src = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 1, new ManualClock(1000), 1L); + RenameIntent intent = new RenameIntent("tok-1", "src", new Hlc(1000, 0, 1), "dst", 3, 1000); + src.recordRenameIntent(intent); + assertThat(src.listRenameIntents()).containsExactly(intent); + + long manifestLedgerId = src.manifestLedgerId(); + src.close(); + src = null; + + // A new owner takes over and replays the intent from the manifest. + dst = BoxEngine.recover(box, CandyboxConfig.defaults(), store, 2, new ManualClock(1000), + manifestLedgerId, 2L); + assertThat(dst.listRenameIntents()).containsExactly(intent); + + dst.clearRenameIntent("tok-1"); + assertThat(dst.listRenameIntents()).isEmpty(); + } + + @Test + void zeroCopyPutIsIdempotentUnderRetryToken() { + src = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 1, new ManualClock(1000), 1L); + dst = BoxEngine.createNew(box, CandyboxConfig.defaults(), store, 2, new ManualClock(1000), 1L); + src.putCandy(CandyKey.of("src"), bytes("payload"), null, Map.of(), null); + List parts = src.resolveLocator(CandyKey.of("src")).parts(); + + CandyMetadata first = dst.zeroCopyPut(CandyKey.of("dst"), parts, null, Map.of(), 0L, + ObjectAcl.NONE, "tok"); + CandyMetadata retry = dst.zeroCopyPut(CandyKey.of("dst"), parts, null, Map.of(), 0L, + ObjectAcl.NONE, "tok"); + assertThat(retry.hlc()).isEqualTo(first.hlc()); + } +} diff --git a/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestEditTest.java b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestEditTest.java index 5f7365e..273f21f 100644 --- a/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestEditTest.java +++ b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestEditTest.java @@ -39,16 +39,18 @@ private static Part part() { @Test void constructorNormalizesNullCollectionsToEmpty() { ManifestEdit edit = new ManifestEdit(List.of(), Set.of(), Set.of(), Set.of(), null, - null, null, null, 0L); + null, null, null, null, null, 0L); assertThat(edit.addedUploads()).isEmpty(); assertThat(edit.upsertParts()).isEmpty(); assertThat(edit.removedUploads()).isEmpty(); + assertThat(edit.addedRenameIntents()).isEmpty(); + assertThat(edit.removedRenameIntents()).isEmpty(); } @Test void constructorRejectsNegativeFencingToken() { assertThatThrownBy(() -> new ManifestEdit(List.of(), Set.of(), Set.of(), Set.of(), null, - List.of(), List.of(), Set.of(), -1L)) + List.of(), List.of(), Set.of(), List.of(), Set.of(), -1L)) .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("ownerFencingToken"); } diff --git a/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestTest.java b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestTest.java index c6be741..adbeb97 100644 --- a/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestTest.java +++ b/candybox-lsm/src/test/java/me/predatorray/candybox/lsm/manifest/ManifestTest.java @@ -61,6 +61,27 @@ void editsSerializeAndReplayRoundTripIncludingToken() { assertThat(ManifestSerializer.deserialize(bytes)).isEqualTo(edit); } + @Test + void renameIntentsRoundTripThroughTheManifest() { + RenameIntent intent = new RenameIntent("tok-1", "src/key", + new me.predatorray.candybox.common.Hlc(12345L, 7, 2), "dst/key", 3, 12345L); + ManifestEdit addEdit = ManifestEdit.builder().addRenameIntent(intent).build(); + byte[] ab = ManifestSerializer.serialize(addEdit); + assertThat(ManifestSerializer.deserialize(ab)).isEqualTo(addEdit); + + ManifestEdit clearEdit = ManifestEdit.builder() + .removedRenameIntents(Set.of("tok-1")).build(); + byte[] cb = ManifestSerializer.serialize(clearEdit); + assertThat(ManifestSerializer.deserialize(cb)).isEqualTo(clearEdit); + + // An intent applied then cleared leaves the state empty. + Manifest m = Manifest.createNew(store, cfg, 1L); + m.apply(addEdit); + assertThat(m.current().renameIntents()).containsKey("tok-1"); + m.apply(clearEdit); + assertThat(m.current().renameIntents()).isEmpty(); + } + @Test void multipartUploadsRoundTripThroughTheManifest() { java.util.Map initialParts = new java.util.LinkedHashMap<>(); From 0c2201988d58cb2220c36d787e48a0202c107a7e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:15:06 +0000 Subject: [PATCH 2/7] Cross-partition zero-copy: protocol messages + coordination keys New wire ops for the client relay: GetCandyLocator/PrepareRename (-> CandyLocatorResponse carrying Parts + source HLC), ZeroCopyPut (-> HeadCandyResponse), CompleteRename (-> Ok), with Part/SegmentRef/Hlc codec helpers. New coordination keys: per-partition referenced-Syrup publication (partitionRefsKey) and the rename rendezvous marker (renameMarkerKey). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- .../candybox/coordination/CandyboxKeys.java | 17 +++ .../candybox/protocol/Message.java | 67 +++++++++++ .../candybox/protocol/MessageCodec.java | 108 ++++++++++++++++++ .../predatorray/candybox/protocol/Opcode.java | 13 ++- .../candybox/protocol/MessageCodecTest.java | 47 ++++++++ 5 files changed, 251 insertions(+), 1 deletion(-) diff --git a/candybox-coordination/src/main/java/me/predatorray/candybox/coordination/CandyboxKeys.java b/candybox-coordination/src/main/java/me/predatorray/candybox/coordination/CandyboxKeys.java index 97f8812..77ae011 100644 --- a/candybox-coordination/src/main/java/me/predatorray/candybox/coordination/CandyboxKeys.java +++ b/candybox-coordination/src/main/java/me/predatorray/candybox/coordination/CandyboxKeys.java @@ -53,4 +53,21 @@ public static String manifestKey(String boxName, int partition) { public static String boxAclKey(String boxName) { return "acls/" + boxName; } + + /** + * The versioned key holding one partition's published referenced-Syrup id set, consulted by the + * Box-global garbage collector so a Syrup shared cross-partition (zero-copy copy/rename) is never + * reclaimed while any partition still points at it. + */ + public static String partitionRefsKey(String boxName, int partition) { + return BOXES_ROOT + "/" + boxName + "/partitions/" + partition + "/refs"; + } + + /** + * The rendezvous marker a destination owner writes when a cross-partition rename's zero-copy put + * is durable; the source owner reads it to finalize (tombstone the source) the rename. + */ + public static String renameMarkerKey(String boxName, String renameToken) { + return BOXES_ROOT + "/" + boxName + "/renames/" + renameToken; + } } diff --git a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Message.java b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Message.java index 1202ab9..d33f28f 100644 --- a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Message.java +++ b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Message.java @@ -17,6 +17,8 @@ import java.util.List; import java.util.Map; +import me.predatorray.candybox.common.Hlc; +import me.predatorray.candybox.common.Part; /** * The typed protocol messages, mapped to/from {@link Frame}s by {@link MessageCodec}. A sealed @@ -141,6 +143,71 @@ public Opcode opcode() { } } + // ---- cross-partition zero-copy copy/rename --------------------------------------------- + + /** + * Resolves {@code key}'s live {@link me.predatorray.candybox.common.CandyLocator} parts so the + * client can relay them to the destination partition's owner for a zero-copy + * {@link ZeroCopyPutRequest}. Answered with a {@link CandyLocatorResponse}. + */ + record GetCandyLocatorRequest(String box, String key) implements Message { + public Opcode opcode() { + return Opcode.GET_CANDY_LOCATOR; + } + } + + /** + * Like {@link GetCandyLocatorRequest} but, on the source partition's owner, also records a durable + * cross-partition rename intent ({@code renameToken}) keyed on the resolved source HLC, so the + * owed source delete survives a crash/handover. Answered with a {@link CandyLocatorResponse}. + */ + record PrepareRenameRequest(String box, String srcKey, String dstKey, int dstPartition, + String renameToken) implements Message { + public Opcode opcode() { + return Opcode.PREPARE_RENAME; + } + } + + /** + * The resolved source locator relayed to the destination owner. {@code parts} are reused verbatim + * (zero byte copy); {@code hlc} is the source's LWW stamp (the rename's delete guard). + */ + record CandyLocatorResponse(List parts, String contentType, + Map userMetadata, Hlc hlc, long createdAtMillis, + String owner, List grants) implements Message { + public Opcode opcode() { + return Opcode.RESPONSE_CANDY_LOCATOR; + } + } + + /** + * Writes a destination Candy at {@code dstKey} reusing {@code parts} verbatim — the cross-partition + * zero-copy put. For a rename, {@code renameToken} is set and the destination owner writes the + * coordination completion marker once the locator is durable; {@code srcKey}/{@code srcPartition}/ + * {@code srcHlc} are recorded in that marker. For a plain copy they are null/absent. Answered with + * a {@link HeadCandyResponse}. + */ + record ZeroCopyPutRequest(String box, String dstKey, List parts, String contentType, + Map userMetadata, String owner, List grants, + String idempotencyToken, String renameToken, String srcKey, + int srcPartition, Hlc srcHlc) implements Message { + public Opcode opcode() { + return Opcode.ZERO_COPY_PUT; + } + } + + /** + * Finalizes a cross-partition rename on the source owner: an LWW-conditioned tombstone of + * {@code srcKey} (only if its live HLC still equals {@code srcHlc}) plus clearing the intent and + * the marker. Idempotent; answered with an {@link OkResponse}. + */ + record CompleteRenameRequest(String box, String srcKey, int srcPartition, String renameToken, + Hlc srcHlc) implements Message { + public Opcode opcode() { + return Opcode.COMPLETE_RENAME; + } + } + /** * Deletes a key range with a single range tombstone in one partition; the client fans * the request out to every partition of the Box. Exactly one of: a {@code prefix}, or a diff --git a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/MessageCodec.java b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/MessageCodec.java index 7dcd891..4209522 100644 --- a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/MessageCodec.java +++ b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/MessageCodec.java @@ -19,6 +19,9 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import me.predatorray.candybox.common.Hlc; +import me.predatorray.candybox.common.Part; +import me.predatorray.candybox.common.SegmentRef; import me.predatorray.candybox.common.serial.BinaryReader; import me.predatorray.candybox.common.serial.BinaryWriter; @@ -76,6 +79,41 @@ public Frame encode(Message message) { w.writeString(m.srcKey()); w.writeString(m.dstKey()); writeNullable(w, m.idempotencyToken()); + } else if (message instanceof Message.GetCandyLocatorRequest m) { + writeBoxKey(w, m.box(), m.key()); + } else if (message instanceof Message.PrepareRenameRequest m) { + w.writeString(m.box()); + w.writeString(m.srcKey()); + w.writeString(m.dstKey()); + w.writeVarInt(m.dstPartition()); + w.writeString(m.renameToken()); + } else if (message instanceof Message.CandyLocatorResponse m) { + writeParts(w, m.parts()); + writeNullable(w, m.contentType()); + writeMetadata(w, m.userMetadata()); + writeHlc(w, m.hlc()); + w.writeVarLong(Math.max(0, m.createdAtMillis())); + writeNullable(w, m.owner()); + writeStrings(w, m.grants()); + } else if (message instanceof Message.ZeroCopyPutRequest m) { + w.writeString(m.box()); + w.writeString(m.dstKey()); + writeParts(w, m.parts()); + writeNullable(w, m.contentType()); + writeMetadata(w, m.userMetadata()); + writeNullable(w, m.owner()); + writeStrings(w, m.grants()); + writeNullable(w, m.idempotencyToken()); + writeNullable(w, m.renameToken()); + writeNullable(w, m.srcKey()); + w.writeVarInt(m.srcPartition()); + writeNullableHlc(w, m.srcHlc()); + } else if (message instanceof Message.CompleteRenameRequest m) { + w.writeString(m.box()); + w.writeString(m.srcKey()); + w.writeVarInt(m.srcPartition()); + w.writeString(m.renameToken()); + writeNullableHlc(w, m.srcHlc()); } else if (message instanceof Message.DeleteRangeRequest m) { w.writeString(m.box()); w.writeVarInt(m.partition()); @@ -273,6 +311,19 @@ public Message decode(Frame frame) { r.readString(), readNullable(r), readNullable(r), readStrings(r)); case RENAME_CANDY -> new Message.RenameCandyRequest(r.readString(), r.readString(), r.readString(), readNullable(r)); + case GET_CANDY_LOCATOR -> new Message.GetCandyLocatorRequest(r.readString(), + r.readString()); + case PREPARE_RENAME -> new Message.PrepareRenameRequest(r.readString(), r.readString(), + r.readString(), r.readVarInt(), r.readString()); + case RESPONSE_CANDY_LOCATOR -> new Message.CandyLocatorResponse(readParts(r), + readNullable(r), readMetadata(r), readHlc(r), r.readVarLong(), readNullable(r), + readStrings(r)); + case ZERO_COPY_PUT -> new Message.ZeroCopyPutRequest(r.readString(), r.readString(), + readParts(r), readNullable(r), readMetadata(r), readNullable(r), readStrings(r), + readNullable(r), readNullable(r), readNullable(r), r.readVarInt(), + readNullableHlc(r)); + case COMPLETE_RENAME -> new Message.CompleteRenameRequest(r.readString(), r.readString(), + r.readVarInt(), r.readString(), readNullableHlc(r)); case DELETE_RANGE -> new Message.DeleteRangeRequest(r.readString(), r.readVarInt(), readNullable(r), readNullable(r), readNullable(r)); case LIST_CANDIES -> new Message.ListCandiesRequest(r.readString(), r.readVarInt(), @@ -410,6 +461,63 @@ private static Message decodeList(BinaryReader r) { return new Message.ListCandiesResponse(entries, readNullable(r)); } + private static void writeParts(BinaryWriter w, List parts) { + List list = parts == null ? List.of() : parts; + w.writeVarInt(list.size()); + for (Part p : list) { + w.writeVarLong(p.partLength()); + w.writeVarInt(p.chunkSize()); + w.writeInt(p.crc32c()); + List segs = p.segments(); + w.writeVarInt(segs.size()); + for (SegmentRef s : segs) { + w.writeVarLong(s.syrupId()); + w.writeVarLong(s.firstEntryId()); + w.writeVarLong(s.lastEntryId()); + } + } + } + + private static List readParts(BinaryReader r) { + int partCount = r.readVarInt(); + List parts = new ArrayList<>(partCount); + for (int i = 0; i < partCount; i++) { + long partLength = r.readVarLong(); + int chunkSize = r.readVarInt(); + int crc32c = r.readInt(); + int segCount = r.readVarInt(); + List segments = new ArrayList<>(segCount); + for (int j = 0; j < segCount; j++) { + segments.add(new SegmentRef(r.readVarLong(), r.readVarLong(), r.readVarLong())); + } + parts.add(new Part(partLength, chunkSize, crc32c, segments)); + } + return parts; + } + + private static void writeHlc(BinaryWriter w, Hlc hlc) { + w.writeVarLong(hlc.physicalMillis()); + w.writeVarInt(hlc.logicalCounter()); + w.writeInt(hlc.nodeId()); + } + + private static Hlc readHlc(BinaryReader r) { + return new Hlc(r.readVarLong(), r.readVarInt(), r.readInt()); + } + + private static void writeNullableHlc(BinaryWriter w, Hlc hlc) { + if (hlc == null) { + w.writeBoolean(false); + } else { + w.writeBoolean(true); + writeHlc(w, hlc); + } + } + + private static Hlc readNullableHlc(BinaryReader r) { + return r.readBoolean() ? readHlc(r) : null; + } + private static void writeBoxKey(BinaryWriter w, String box, String key) { w.writeString(box); w.writeString(key); diff --git a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Opcode.java b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Opcode.java index 75c49d2..3e27a71 100644 --- a/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Opcode.java +++ b/candybox-protocol/src/main/java/me/predatorray/candybox/protocol/Opcode.java @@ -48,6 +48,15 @@ public enum Opcode { /** Metadata-only locator rewrite replacing one object's owner/grants. */ SET_CANDY_ACL(45), + /** Resolves a key's {@code CandyLocator} parts for a cross-partition zero-copy copy/rename. */ + GET_CANDY_LOCATOR(46), + /** Like {@link #GET_CANDY_LOCATOR} but also records a source-side rename intent (cross-partition). */ + PREPARE_RENAME(47), + /** Writes a destination Candy reusing a source partition's segments verbatim (zero byte copy). */ + ZERO_COPY_PUT(48), + /** Finalizes a cross-partition rename: LWW-conditioned delete of the source key. */ + COMPLETE_RENAME(49), + /** Selects the SASL mechanism for this connection; must precede {@link #SASL_AUTHENTICATE}. */ SASL_HANDSHAKE(50), /** One step of the SASL exchange: an opaque, mechanism-defined client token. */ @@ -79,7 +88,9 @@ public enum Opcode { /** Authenticated but not authorized for the operation (S3 AccessDenied / HTTP 403). */ RESPONSE_ACCESS_DENIED(55), RESPONSE_BOX_ACL(56), - RESPONSE_CANDY_ACL(57); + RESPONSE_CANDY_ACL(57), + /** A resolved {@code CandyLocator}'s parts + metadata, for a cross-partition zero-copy relay. */ + RESPONSE_CANDY_LOCATOR(58); private final int code; diff --git a/candybox-protocol/src/test/java/me/predatorray/candybox/protocol/MessageCodecTest.java b/candybox-protocol/src/test/java/me/predatorray/candybox/protocol/MessageCodecTest.java index d4ecfe5..10d7f85 100644 --- a/candybox-protocol/src/test/java/me/predatorray/candybox/protocol/MessageCodecTest.java +++ b/candybox-protocol/src/test/java/me/predatorray/candybox/protocol/MessageCodecTest.java @@ -80,6 +80,53 @@ void copyRenameAndDeleteRangeRequestsRoundTrip() { assertThat(byWindow.endKey()).isEqualTo("e"); } + @Test + void crossPartitionZeroCopyMessagesRoundTrip() { + me.predatorray.candybox.common.Part part = new me.predatorray.candybox.common.Part( + 7L, 1 << 20, 0x1234abcd, + List.of(new me.predatorray.candybox.common.SegmentRef(42, 0, 3), + new me.predatorray.candybox.common.SegmentRef(43, 0, 1))); + me.predatorray.candybox.common.Hlc hlc = new me.predatorray.candybox.common.Hlc(999, 5, 7); + + Message.GetCandyLocatorRequest gl = (Message.GetCandyLocatorRequest) roundTrip( + new Message.GetCandyLocatorRequest("box", "src")); + assertThat(gl.key()).isEqualTo("src"); + + Message.PrepareRenameRequest pr = (Message.PrepareRenameRequest) roundTrip( + new Message.PrepareRenameRequest("box", "src", "dst", 3, "tok-1")); + assertThat(pr.dstPartition()).isEqualTo(3); + assertThat(pr.renameToken()).isEqualTo("tok-1"); + + Message.CandyLocatorResponse loc = (Message.CandyLocatorResponse) roundTrip( + new Message.CandyLocatorResponse(List.of(part), "text/plain", Map.of("m", "v"), hlc, + 12345L, "User:alice", List.of("User:bob:READ"))); + assertThat(loc.parts()).isEqualTo(List.of(part)); + assertThat(loc.hlc()).isEqualTo(hlc); + assertThat(loc.contentType()).isEqualTo("text/plain"); + assertThat(loc.owner()).isEqualTo("User:alice"); + assertThat(loc.grants()).containsExactly("User:bob:READ"); + + Message.ZeroCopyPutRequest zp = (Message.ZeroCopyPutRequest) roundTrip( + new Message.ZeroCopyPutRequest("box", "dst", List.of(part), "text/plain", + Map.of("m", "v"), "User:alice", List.of(), "idem", "tok-1", "src", 2, hlc)); + assertThat(zp.parts()).isEqualTo(List.of(part)); + assertThat(zp.renameToken()).isEqualTo("tok-1"); + assertThat(zp.srcPartition()).isEqualTo(2); + assertThat(zp.srcHlc()).isEqualTo(hlc); + + // A plain copy carries no rename token / source HLC. + Message.ZeroCopyPutRequest copy = (Message.ZeroCopyPutRequest) roundTrip( + new Message.ZeroCopyPutRequest("box", "dst", List.of(part), null, Map.of(), null, + List.of(), "idem", null, null, 0, null)); + assertThat(copy.renameToken()).isNull(); + assertThat(copy.srcHlc()).isNull(); + + Message.CompleteRenameRequest cr = (Message.CompleteRenameRequest) roundTrip( + new Message.CompleteRenameRequest("box", "src", 2, "tok-1", hlc)); + assertThat(cr.srcKey()).isEqualTo("src"); + assertThat(cr.srcHlc()).isEqualTo(hlc); + } + @Test void listCandiesRequestRoundTripsRangeAndDirection() { Message.ListCandiesRequest req = new Message.ListCandiesRequest("box", 2, "p/", "p/cursor", 50, From 7d456c9e5d51ea257327b2e6b0d8cb4a8e9a7cca Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:21:28 +0000 Subject: [PATCH 3/7] Cross-partition zero-copy: Box-global GC, rename rendezvous, client relay Server: per-partition referenced-Syrup publication + Box-global GC gate (a Syrup shared cross-partition is never reclaimed while any sibling references it); ZeroCopyPut/GetCandyLocator/ PrepareRename/CompleteRename handlers; owner-local rename-intent finalize sweep (marker present => LWW-conditioned source delete; abandoned past the window => drop). Client: cross-partition copy/rename now zero-copy via the locator relay, rename eventually-atomic via the rendezvous. New config rename.intent.abandon.millis (default 60s). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- .../candybox/client/CandyboxClient.java | 75 +++++--- .../common/config/CandyboxConfig.java | 17 ++ .../candybox/server/CandyboxNode.java | 176 +++++++++++++++++- .../candybox/server/GarbageCollector.java | 24 ++- .../candybox/server/NodeRequestHandler.java | 75 ++++++++ .../candybox/server/ServerConfig.java | 1 + 6 files changed, 340 insertions(+), 28 deletions(-) diff --git a/candybox-client/src/main/java/me/predatorray/candybox/client/CandyboxClient.java b/candybox-client/src/main/java/me/predatorray/candybox/client/CandyboxClient.java index 1b0f155..9f3e2b7 100644 --- a/candybox-client/src/main/java/me/predatorray/candybox/client/CandyboxClient.java +++ b/candybox-client/src/main/java/me/predatorray/candybox/client/CandyboxClient.java @@ -434,9 +434,10 @@ public PartListing listParts(String box, String key, String uploadId, int partNu } /** - * Server-side copy of {@code srcKey} to {@code dstKey} within the same Box. When both keys hash - * to the same partition the copy is zero-copy (the new key reuses the stored bytes); across - * partitions it degrades to a client-side byte copy. Returns the destination's metadata. + * Server-side copy of {@code srcKey} to {@code dstKey} within the same Box. Zero-copy in both + * cases: a same-partition copy reuses the stored bytes via one server-side call; a cross-partition + * copy relays the source's locator parts to the destination owner, which points the new key at the + * very same Syrup segments (no bytes are moved). Returns the destination's metadata. */ public CandyInfo copyCandy(String box, String srcKey, String dstKey, String idempotencyToken) { return copyCandy(box, srcKey, dstKey, idempotencyToken, null, List.of()); @@ -451,23 +452,47 @@ public CandyInfo copyCandy(String box, String srcKey, String dstKey, String idem CandyKey.of(srcKey).value(), CandyKey.of(dstKey).value(), idempotencyToken, owner, g)); } - return byteCopy(box, srcKey, dstKey, idempotencyToken, owner, g); + // Cross-partition zero-copy: resolve the source's parts, then point the destination at them. + Message.CandyLocatorResponse loc = resolveLocator(box, srcKey, + new Message.GetCandyLocatorRequest(BoxName.of(box).value(), + CandyKey.of(srcKey).value())); + return zeroCopyPut(box, dstKey, new Message.ZeroCopyPutRequest(BoxName.of(box).value(), + CandyKey.of(dstKey).value(), loc.parts(), loc.contentType(), loc.userMetadata(), + owner, g, idempotencyToken, null, null, 0, null)); } /** - * Rename/move of {@code srcKey} to {@code dstKey} within the same Box. Same-partition renames - * are zero-copy and atomic; a cross-partition rename is a byte copy followed by a delete of the - * source (not atomic — a failure in between can leave both keys live). Returns the destination's - * metadata. + * Rename/move of {@code srcKey} to {@code dstKey} within the same Box. A same-partition rename is + * zero-copy and atomic. A cross-partition rename is zero-copy and eventually atomic: it + * prepares a durable intent on the source owner, writes the destination via a zero-copy put, then + * finalizes the source delete (LWW-conditioned on the source's HLC). If the final step is lost, + * the source owner's maintenance sweep finalizes it via the coordination rendezvous marker — so a + * crash converges to "destination present, source gone" rather than leaving both keys live + * forever. Returns the destination's metadata. */ public CandyInfo renameCandy(String box, String srcKey, String dstKey, String idempotencyToken) { if (partitionFor(box, srcKey) == partitionFor(box, dstKey)) { return copyOrRename(box, srcKey, new Message.RenameCandyRequest(BoxName.of(box).value(), CandyKey.of(srcKey).value(), CandyKey.of(dstKey).value(), idempotencyToken)); } - CandyInfo copied = byteCopy(box, srcKey, dstKey, idempotencyToken); - deleteCandy(box, srcKey); - return copied; + int srcPartition = partitionFor(box, srcKey); + int dstPartition = partitionFor(box, dstKey); + String token = idempotencyToken != null ? idempotencyToken + : java.util.UUID.randomUUID().toString(); + + // 1. Prepare on the source owner: resolve the locator and durably record the rename intent. + Message.CandyLocatorResponse loc = resolveLocator(box, srcKey, + new Message.PrepareRenameRequest(BoxName.of(box).value(), CandyKey.of(srcKey).value(), + CandyKey.of(dstKey).value(), dstPartition, token)); + // 2. Zero-copy put on the destination owner (preserves the source's owner/grants). + CandyInfo result = zeroCopyPut(box, dstKey, new Message.ZeroCopyPutRequest( + BoxName.of(box).value(), CandyKey.of(dstKey).value(), loc.parts(), loc.contentType(), + loc.userMetadata(), loc.owner(), loc.grants(), token, token, + CandyKey.of(srcKey).value(), srcPartition, loc.hlc())); + // 3. Finalize the source delete (best-effort fast path; the maintenance sweep is the backstop). + router.callPartition(box, srcPartition, new Message.CompleteRenameRequest( + BoxName.of(box).value(), CandyKey.of(srcKey).value(), srcPartition, token, loc.hlc())); + return result; } private CandyInfo copyOrRename(String box, String srcKey, Message request) { @@ -479,21 +504,23 @@ private CandyInfo copyOrRename(String box, String srcKey, Message request) { throw mapUnexpected(response, box, srcKey); } - /** The cross-partition copy fallback: read the source whole, re-put it at the destination. */ - private CandyInfo byteCopy(String box, String srcKey, String dstKey, String idempotencyToken) { - return byteCopy(box, srcKey, dstKey, idempotencyToken, null, List.of()); + /** Sends a locator-resolving request to the source's partition owner and unwraps the response. */ + private Message.CandyLocatorResponse resolveLocator(String box, String srcKey, Message request) { + Message response = router.callPartition(box, partitionFor(box, srcKey), request); + if (response instanceof Message.CandyLocatorResponse loc) { + return loc; + } + throw mapUnexpected(response, box, srcKey); } - private CandyInfo byteCopy(String box, String srcKey, String dstKey, String idempotencyToken, - String owner, List grants) { - Message response = callKey(box, srcKey, new Message.GetCandyRequest(BoxName.of(box).value(), - CandyKey.of(srcKey).value())); - if (!(response instanceof Message.CandyDataResponse data)) { - throw mapUnexpected(response, box, srcKey); - } - putCandy(box, dstKey, data.data(), data.contentType(), data.userMetadata(), - idempotencyToken, owner, grants); - return headCandy(box, dstKey); + /** Sends a zero-copy put to the destination's partition owner and unwraps the head response. */ + private CandyInfo zeroCopyPut(String box, String dstKey, Message.ZeroCopyPutRequest request) { + Message response = router.callPartition(box, partitionFor(box, dstKey), request); + if (response instanceof Message.HeadCandyResponse head) { + return new CandyInfo(head.contentLength(), head.contentType(), head.userMetadata(), + head.crc32c(), head.createdAtMillis()); + } + throw mapUnexpected(response, box, dstKey); } /** diff --git a/candybox-common/src/main/java/me/predatorray/candybox/common/config/CandyboxConfig.java b/candybox-common/src/main/java/me/predatorray/candybox/common/config/CandyboxConfig.java index 234f7b8..70da374 100644 --- a/candybox-common/src/main/java/me/predatorray/candybox/common/config/CandyboxConfig.java +++ b/candybox-common/src/main/java/me/predatorray/candybox/common/config/CandyboxConfig.java @@ -48,6 +48,7 @@ public final class CandyboxConfig { private final int partitionsPerBoxDefault; private final long balancerIntervalMillis; private final int balancerMaxMovesPerRound; + private final long renameIntentAbandonMillis; private CandyboxConfig(Builder b) { this.sizeLimits = b.sizeLimits; @@ -72,6 +73,7 @@ private CandyboxConfig(Builder b) { this.partitionsPerBoxDefault = b.partitionsPerBoxDefault; this.balancerIntervalMillis = b.balancerIntervalMillis; this.balancerMaxMovesPerRound = b.balancerMaxMovesPerRound; + this.renameIntentAbandonMillis = b.renameIntentAbandonMillis; } public static CandyboxConfig defaults() { @@ -183,6 +185,15 @@ public int balancerMaxMovesPerRound() { return balancerMaxMovesPerRound; } + /** + * How long a source-side cross-partition rename intent may sit without its destination completion + * marker appearing before it is abandoned (the rename never reached the destination; the source + * stays live). Roll-forward otherwise: a present marker finalizes the source delete. + */ + public long renameIntentAbandonMillis() { + return renameIntentAbandonMillis; + } + public static final class Builder { private SizeLimits sizeLimits = SizeLimits.defaults(); private Map quorums = new EnumMap<>(QuorumConfig.defaults()); @@ -206,6 +217,7 @@ public static final class Builder { private int partitionsPerBoxDefault = 8; // write spread vs. per-engine cost private long balancerIntervalMillis = 0L; // balancing round; 0 disables private int balancerMaxMovesPerRound = 4; // migration rate limit + private long renameIntentAbandonMillis = 60_000L; // abandon a stuck rename intent public Builder sizeLimits(SizeLimits v) { this.sizeLimits = v; @@ -317,6 +329,11 @@ public Builder balancerMaxMovesPerRound(int v) { return this; } + public Builder renameIntentAbandonMillis(long v) { + this.renameIntentAbandonMillis = v; + return this; + } + public CandyboxConfig build() { if (l0StallThreshold < l0CompactionTrigger) { throw new IllegalArgumentException("l0StallThreshold must be >= l0CompactionTrigger"); diff --git a/candybox-server/src/main/java/me/predatorray/candybox/server/CandyboxNode.java b/candybox-server/src/main/java/me/predatorray/candybox/server/CandyboxNode.java index aa23d79..1562bbb 100644 --- a/candybox-server/src/main/java/me/predatorray/candybox/server/CandyboxNode.java +++ b/candybox-server/src/main/java/me/predatorray/candybox/server/CandyboxNode.java @@ -17,8 +17,10 @@ import java.nio.charset.StandardCharsets; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.Optional; +import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.Executors; @@ -26,7 +28,9 @@ import java.util.concurrent.TimeUnit; import me.predatorray.candybox.bookkeeper.LedgerStore; import me.predatorray.candybox.common.BoxName; +import me.predatorray.candybox.common.CandyKey; import me.predatorray.candybox.common.Clock; +import me.predatorray.candybox.common.Hlc; import me.predatorray.candybox.common.SystemClock; import me.predatorray.candybox.common.auth.Authorizer; import me.predatorray.candybox.common.config.CandyboxConfig; @@ -40,7 +44,10 @@ import me.predatorray.candybox.coordination.CasConflictException; import me.predatorray.candybox.coordination.CoordinationService; import me.predatorray.candybox.coordination.VersionedValue; +import me.predatorray.candybox.common.serial.BinaryReader; +import me.predatorray.candybox.common.serial.BinaryWriter; import me.predatorray.candybox.lsm.engine.BoxEngine; +import me.predatorray.candybox.lsm.manifest.RenameIntent; import me.predatorray.candybox.protocol.transport.RequestHandler; import me.predatorray.candybox.server.PartitionAssignment.BoxPartition; import org.slf4j.Logger; @@ -142,6 +149,7 @@ private void runMaintenance() { compactOwnedBoxesOnce(); collectGarbageOnce(); sweepStaleMultipartUploadsOnce(); + finalizeRenameIntentsOnce(); } private static ScheduledExecutorService daemonScheduler(String name) { @@ -509,13 +517,23 @@ public int compactOwnedBoxesOnce() { * @return the number of ledgers deleted */ public int collectGarbageOnce() { + // Publish every owned partition's referenced-Syrup set first, so this pass's Box-global gate + // sees up-to-date references from this node's partitions (cross-node freshness rides the other + // owners' own publishes plus the GC grace period). + for (PartitionOwnership ownership : partitions.values()) { + if (ownership.isOwner()) { + publishPartitionRefs(ownership); + } + } int deleted = 0; for (PartitionOwnership ownership : partitions.values()) { if (!ownership.isOwner()) { continue; } try { - deleted += garbageCollector.collect(ownership.engine()); + Set foreign = syrupsReferencedByOtherPartitions(ownership.box().value(), + ownership.partition()); + deleted += garbageCollector.collect(ownership.engine(), foreign); } catch (NotOwnerException lostOwnership) { LOG.info("Skipping GC of a partition on node {}: {}", nodeId, lostOwnership.getMessage()); @@ -526,6 +544,50 @@ public int collectGarbageOnce() { return deleted; } + /** + * Finalizes (or abandons) the cross-partition rename intents owed by every partition this node + * owns: a present coordination rendezvous marker ⇒ LWW-conditioned delete of the source key, then + * clear the intent and the marker; no marker past {@code renameIntentAbandonMillis} ⇒ drop the + * intent (the rename never reached the destination — the source stays live). Idempotent; also runs + * after a handover replays an intent. Exposed for manual/operational triggering and tests. + * + * @return the number of intents finalized (source actually tombstoned) + */ + public int finalizeRenameIntentsOnce() { + long abandon = config.renameIntentAbandonMillis(); + int finalized = 0; + for (PartitionOwnership ownership : partitions.values()) { + if (!ownership.isOwner()) { + continue; + } + BoxEngine engine; + String box = ownership.box().value(); + try { + engine = ownership.engine(); + } catch (NotOwnerException lost) { + continue; + } + for (RenameIntent intent : engine.listRenameIntents()) { + try { + if (renameMarkerPresent(box, intent.token())) { + engine.deleteCandyConditional(CandyKey.of(intent.srcKey()), intent.srcHlc()); + engine.clearRenameIntent(intent.token()); + deleteRenameMarker(box, intent.token()); + finalized++; + } else if (clock.currentTimeMillis() - intent.createdAtMillis() > abandon) { + engine.clearRenameIntent(intent.token()); + } + } catch (NotOwnerException lost) { + break; + } catch (RuntimeException e) { + LOG.warn("Rename-intent finalize error on node {} for token {}: {}", nodeId, + intent.token(), e.getMessage()); + } + } + } + return finalized; + } + /** * Aborts in-flight multipart uploads whose {@code createdAtMillis + multipartUploadTtlMillis} is * already in the past. Reuses the engine's {@link me.predatorray.candybox.lsm.engine.BoxEngine @@ -562,6 +624,118 @@ public int sweepStaleMultipartUploadsOnce() { return aborted; } + // ---- Box-global GC support + cross-partition rename rendezvous --------------------------- + + long currentTimeMillis() { + return clock.currentTimeMillis(); + } + + /** Publishes one owned partition's referenced-Syrup set to coordination (Box-global GC input). */ + void publishPartitionRefs(BoxName box, int partition) { + PartitionOwnership ownership = partitions.get(new BoxPartition(box.value(), partition)); + if (ownership != null && ownership.isOwner()) { + publishPartitionRefs(ownership); + } + } + + private void publishPartitionRefs(PartitionOwnership ownership) { + Set refs; + try { + refs = ownership.engine().referencedSyrups(); + } catch (NotOwnerException lost) { + return; + } + casPut(CandyboxKeys.partitionRefsKey(ownership.box().value(), ownership.partition()), + encodeLongSet(refs)); + } + + /** The union of every other partition's published referenced-Syrup set for a Box. */ + private Set syrupsReferencedByOtherPartitions(String box, int excludePartition) { + BoxDescriptor descriptor; + try { + descriptor = descriptor(BoxName.of(box)); + } catch (BoxNotFoundException gone) { + return Set.of(); + } + Set referenced = new HashSet<>(); + for (int p = 0; p < descriptor.partitionCount(); p++) { + if (p == excludePartition) { + continue; + } + coordination.get(CandyboxKeys.partitionRefsKey(box, p)) + .ifPresent(v -> referenced.addAll(decodeLongSet(v.value()))); + } + return referenced; + } + + /** + * Writes the rendezvous marker telling the source partition's owner that a cross-partition + * rename's destination put is durable. Idempotent (a retried write is a no-op). + */ + void writeRenameMarker(String box, String token, String srcKey, int srcPartition, Hlc srcHlc, + String dstKey) { + BinaryWriter w = new BinaryWriter(64); + w.writeString(srcKey); + w.writeVarInt(srcPartition); + w.writeVarLong(srcHlc == null ? 0L : srcHlc.physicalMillis()); + w.writeVarInt(srcHlc == null ? 0 : srcHlc.logicalCounter()); + w.writeInt(srcHlc == null ? 0 : srcHlc.nodeId()); + w.writeString(dstKey == null ? "" : dstKey); + String key = CandyboxKeys.renameMarkerKey(box, token); + try { + coordination.create(key, w.toByteArray()); + } catch (CasConflictException alreadyThere) { + // Idempotent: the marker is already present. + } + } + + boolean renameMarkerPresent(String box, String token) { + return coordination.get(CandyboxKeys.renameMarkerKey(box, token)).isPresent(); + } + + void deleteRenameMarker(String box, String token) { + String key = CandyboxKeys.renameMarkerKey(box, token); + coordination.get(key).ifPresent(v -> { + try { + coordination.delete(key, v.version()); + } catch (CasConflictException raced) { + // Already deleted (or rewritten); nothing to do. + } + }); + } + + private void casPut(String key, byte[] value) { + Optional current = coordination.get(key); + try { + if (current.isEmpty()) { + coordination.create(key, value); + } else { + coordination.compareAndSet(key, value, current.get().version()); + } + } catch (CasConflictException raced) { + // A concurrent writer won; the next pass republishes. + } + } + + private static byte[] encodeLongSet(Set values) { + BinaryWriter w = new BinaryWriter(Math.max(16, values.size() * 2)); + w.writeVarInt(values.size()); + for (long v : values) { + w.writeVarLong(v); + } + return w.toByteArray(); + } + + private static Set decodeLongSet(byte[] data) { + BinaryReader r = new BinaryReader(data); + int count = r.readVarInt(); + Set values = new HashSet<>(Math.max(4, count * 2)); + for (int i = 0; i < count; i++) { + values.add(r.readVarLong()); + } + return values; + } + @Override public void close() { if (leaseHeartbeat != null) { diff --git a/candybox-server/src/main/java/me/predatorray/candybox/server/GarbageCollector.java b/candybox-server/src/main/java/me/predatorray/candybox/server/GarbageCollector.java index 32884d0..d1229a1 100644 --- a/candybox-server/src/main/java/me/predatorray/candybox/server/GarbageCollector.java +++ b/candybox-server/src/main/java/me/predatorray/candybox/server/GarbageCollector.java @@ -16,6 +16,7 @@ package me.predatorray.candybox.server; import java.util.List; +import java.util.Set; import me.predatorray.candybox.bookkeeper.LedgerStore; import me.predatorray.candybox.common.Clock; import me.predatorray.candybox.common.exception.StorageException; @@ -64,9 +65,20 @@ public GarbageCollector(LedgerStore ledgerStore, long graceMillis, Clock clock) * @return the number of ledgers deleted */ public int collect(BoxEngine engine) { + return collect(engine, Set.of()); + } + + /** + * As {@link #collect(BoxEngine)}, but a Syrup is only physically reclaimed when it is referenced + * by no partition of the Box: {@code foreignReferencedSyrups} is the union of every + * sibling partition's published referenced-Syrup set, so a Syrup shared cross-partition by a + * zero-copy copy/rename is never deleted out from under the partition that points at it. This is + * the Box-global garbage collection of DESIGN §9. + */ + public int collect(BoxEngine engine, Set foreignReferencedSyrups) { long cutoff = clock.currentTimeMillis() - graceMillis; int deleted = collectSSTables(engine, cutoff); - deleted += collectSyrups(engine, cutoff); + deleted += collectSyrups(engine, cutoff, foreignReferencedSyrups); deleted += collectWals(engine, cutoff); return deleted; } @@ -99,8 +111,14 @@ private int collectSSTables(BoxEngine engine, long cutoff) { return deleted; } - private int collectSyrups(BoxEngine engine, long cutoff) { - List orphans = engine.reclaimableSyrups(cutoff); + private int collectSyrups(BoxEngine engine, long cutoff, Set foreignReferencedSyrups) { + List orphans = new java.util.ArrayList<>(engine.reclaimableSyrups(cutoff)); + if (!foreignReferencedSyrups.isEmpty()) { + // Box-global gate: keep a Syrup alive (neither dropped from the live set nor deleted) while + // any sibling partition still references it — it stays a pending orphan and is retried on a + // later pass once the cross-partition reference is gone. + orphans.removeIf(foreignReferencedSyrups::contains); + } if (orphans.isEmpty()) { return 0; } diff --git a/candybox-server/src/main/java/me/predatorray/candybox/server/NodeRequestHandler.java b/candybox-server/src/main/java/me/predatorray/candybox/server/NodeRequestHandler.java index 344b419..9d2a84a 100644 --- a/candybox-server/src/main/java/me/predatorray/candybox/server/NodeRequestHandler.java +++ b/candybox-server/src/main/java/me/predatorray/candybox/server/NodeRequestHandler.java @@ -36,7 +36,9 @@ import me.predatorray.candybox.coordination.BoxDescriptor; import me.predatorray.candybox.lsm.engine.BoxEngine; import me.predatorray.candybox.lsm.engine.CandyMetadata; +import me.predatorray.candybox.common.CandyLocator; import me.predatorray.candybox.common.Part; +import me.predatorray.candybox.lsm.manifest.RenameIntent; import me.predatorray.candybox.lsm.engine.ListResult; import me.predatorray.candybox.lsm.engine.ScanDirection; import me.predatorray.candybox.lsm.engine.ScanQuery; @@ -212,6 +214,14 @@ private static String boxOf(Message message) { return m.box(); } else if (message instanceof Message.RenameCandyRequest m) { return m.box(); + } else if (message instanceof Message.GetCandyLocatorRequest m) { + return m.box(); + } else if (message instanceof Message.PrepareRenameRequest m) { + return m.box(); + } else if (message instanceof Message.ZeroCopyPutRequest m) { + return m.box(); + } else if (message instanceof Message.CompleteRenameRequest m) { + return m.box(); } else if (message instanceof Message.DeleteRangeRequest m) { return m.box(); } else if (message instanceof Message.ListCandiesRequest m) { @@ -287,6 +297,14 @@ private static String routingKeyOf(Message message) { return m.dstKey(); } else if (message instanceof Message.RenameCandyRequest m) { return m.dstKey(); + } else if (message instanceof Message.GetCandyLocatorRequest m) { + return m.key(); + } else if (message instanceof Message.PrepareRenameRequest m) { + return m.srcKey(); + } else if (message instanceof Message.ZeroCopyPutRequest m) { + return m.dstKey(); + } else if (message instanceof Message.CompleteRenameRequest m) { + return m.srcKey(); } else if (message instanceof Message.CreateMultipartUploadRequest m) { return m.key(); } else if (message instanceof Message.UploadPartRequest m) { @@ -392,6 +410,46 @@ private Message dispatch(Message message, Principal principal) { m.idempotencyToken()); return new Message.HeadCandyResponse(meta.contentLength(), meta.contentType(), meta.userMetadata(), meta.crc32c(), meta.createdAtMillis()); + } else if (message instanceof Message.GetCandyLocatorRequest m) { + CandyLocator locator = node.engine(BoxName.of(m.box()), m.key()) + .resolveLocator(CandyKey.of(m.key())); + return toCandyLocatorResponse(locator); + } else if (message instanceof Message.PrepareRenameRequest m) { + // The source partition's owner: resolve the locator (relayed to the destination) and + // durably record the rename intent so the owed source delete survives a crash/handover. + BoxEngine engine = node.engine(BoxName.of(m.box()), m.srcKey()); + CandyLocator locator = engine.resolveLocator(CandyKey.of(m.srcKey())); + engine.recordRenameIntent(new RenameIntent(m.renameToken(), m.srcKey(), locator.hlc(), + m.dstKey(), m.dstPartition(), node.currentTimeMillis())); + return toCandyLocatorResponse(locator); + } else if (message instanceof Message.ZeroCopyPutRequest m) { + BoxEngine engine = node.engine(BoxName.of(m.box()), m.dstKey()); + boolean isRename = m.renameToken() != null; + // A rename keeps the object's identity, so the source's owner/grants travel verbatim; a + // copy belongs to the requester (effectiveAcl applies the super-user owner rule). + ObjectAcl acl = isRename + ? parseObjectAcl(m.owner(), m.grants()) + : effectiveAcl(principal, m.owner(), m.grants()); + CandyMetadata meta = engine.zeroCopyPut(CandyKey.of(m.dstKey()), m.parts(), + m.contentType(), m.userMetadata(), 0L, acl, m.idempotencyToken()); + // Publish this partition's refs before the source delete can run, so the Box-global GC + // already sees the new cross-partition reference (no premature reclaim of the shared Syrup). + node.publishPartitionRefs(BoxName.of(m.box()), + node.descriptor(BoxName.of(m.box())).partitionOf(m.dstKey())); + if (isRename) { + node.writeRenameMarker(m.box(), m.renameToken(), m.srcKey(), m.srcPartition(), + m.srcHlc(), m.dstKey()); + } + return new Message.HeadCandyResponse(meta.contentLength(), meta.contentType(), + meta.userMetadata(), meta.crc32c(), meta.createdAtMillis()); + } else if (message instanceof Message.CompleteRenameRequest m) { + // The source partition's owner finalizes the rename (also reachable via the maintenance + // sweep / after handover): LWW-conditioned delete of the source, then clear intent + marker. + BoxEngine engine = node.engine(BoxName.of(m.box()), m.srcKey()); + engine.deleteCandyConditional(CandyKey.of(m.srcKey()), m.srcHlc()); + engine.clearRenameIntent(m.renameToken()); + node.deleteRenameMarker(m.box(), m.renameToken()); + return new Message.OkResponse(); } else if (message instanceof Message.DeleteRangeRequest m) { BoxEngine engine = node.enginePartition(BoxName.of(m.box()), m.partition()); if (m.prefix() != null) { @@ -547,6 +605,23 @@ private BoxEngine samePartitionEngine(String box, String srcKey, String dstKey) * S3 gateway writing on behalf of its authenticated end user. Request grants (the S3 canned-ACL * path) are accepted from any writer. */ + private static Message.CandyLocatorResponse toCandyLocatorResponse(CandyLocator locator) { + return new Message.CandyLocatorResponse(locator.parts(), locator.contentType(), + locator.userMetadata(), locator.hlc(), locator.createdAtMillis(), + locator.acl().owner(), locator.acl().grants().stream().map(Grant::toText).toList()); + } + + /** Parses an object ACL from its wire text form verbatim (rename preserves the source's ACL). */ + private static ObjectAcl parseObjectAcl(String owner, List grantTexts) { + try { + List grants = grantTexts == null ? List.of() + : grantTexts.stream().map(Grant::parse).toList(); + return new ObjectAcl(owner == null ? null : Principal.parse(owner).toString(), grants); + } catch (IllegalArgumentException e) { + throw new ValidationException("Invalid object ACL: " + e.getMessage()); + } + } + private ObjectAcl effectiveAcl(Principal principal, String requestedOwner, List grantTexts) { List grants; diff --git a/candybox-server/src/main/java/me/predatorray/candybox/server/ServerConfig.java b/candybox-server/src/main/java/me/predatorray/candybox/server/ServerConfig.java index 311e18e..e44916c 100644 --- a/candybox-server/src/main/java/me/predatorray/candybox/server/ServerConfig.java +++ b/candybox-server/src/main/java/me/predatorray/candybox/server/ServerConfig.java @@ -309,6 +309,7 @@ CandyboxConfig buildTuning() { applyInt("partitions.per.box.default", b::partitionsPerBoxDefault); applyLong("balancer.interval.millis", b::balancerIntervalMillis); applyInt("balancer.max.moves.per.round", b::balancerMaxMovesPerRound); + applyLong("rename.intent.abandon.millis", b::renameIntentAbandonMillis); // Per-role BookKeeper quorum overrides, "E/Qw/Qa" (e.g. 1/1/1 for a single-bookie dev box). applyQuorum("quorum.wal", LedgerRole.WAL, b); applyQuorum("quorum.manifest", LedgerRole.MANIFEST, b); From 10d72abf433b59631c96f0a1afabb6211700f282 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:23:43 +0000 Subject: [PATCH 4/7] Update cross-partition routing test to assert zero-copy relay (was byte-copy fallback) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- .../client/PartitionedRoutingClientTest.java | 53 +++++++++++++------ 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/candybox-client/src/test/java/me/predatorray/candybox/client/PartitionedRoutingClientTest.java b/candybox-client/src/test/java/me/predatorray/candybox/client/PartitionedRoutingClientTest.java index 3227989..828ff5b 100644 --- a/candybox-client/src/test/java/me/predatorray/candybox/client/PartitionedRoutingClientTest.java +++ b/candybox-client/src/test/java/me/predatorray/candybox/client/PartitionedRoutingClientTest.java @@ -32,8 +32,8 @@ /** * Pins {@link CandyboxClient}'s partition-aware behaviour against a recording two-partition stub * node: descriptor caching, scatter-gather list merging (order, truncation, reverse), fanned-out - * range deletes, merged multipart-upload listings, and the cross-partition byte-copy fallbacks for - * copy/rename/uploadPartCopy. + * range deletes, merged multipart-upload listings, the cross-partition zero-copy relay for + * copy/rename, and the (still byte-copy) uploadPartCopy fallback. */ class PartitionedRoutingClientTest { @@ -92,6 +92,16 @@ private Message dispatch(Message message) { } else if (message instanceof Message.CopyCandyRequest || message instanceof Message.RenameCandyRequest) { return new Message.HeadCandyResponse(7, "text/plain", Map.of(), 9, 1); + } else if (message instanceof Message.GetCandyLocatorRequest + || message instanceof Message.PrepareRenameRequest) { + me.predatorray.candybox.common.Part part = new me.predatorray.candybox.common.Part( + 7, 1 << 20, 9, + List.of(new me.predatorray.candybox.common.SegmentRef(1, 0, 0))); + return new Message.CandyLocatorResponse(List.of(part), "text/plain", Map.of("m", "x"), + new me.predatorray.candybox.common.Hlc(1000, 0, 1), 1, "User:alice", + List.of()); + } else if (message instanceof Message.ZeroCopyPutRequest) { + return new Message.HeadCandyResponse(7, "text/plain", Map.of("m", "x"), 9, 1); } return new Message.OkResponse(); } @@ -196,26 +206,39 @@ void listMultipartUploadsMergesAndPaginatesAcrossPartitions() { } @Test - void crossPartitionCopyAndRenameFallBackToGetPutDelete() { + void crossPartitionCopyAndRenameStayZeroCopyViaTheLocatorRelay() { String src = keyIn(0, "src"); String dst = keyIn(1, "dst"); StubNode node = new StubNode(); try (CandyboxClient client = new CandyboxClient(new LoopbackTransport(node), "x", 0)) { CandyboxClient.CandyInfo copied = client.copyCandy("box", src, dst, "tok"); assertThat(copied.contentLength()).isEqualTo(7); - // No server-side copy was attempted; the bytes and metadata were re-put at dst. - assertThat(node.recorded(Message.CopyCandyRequest.class)).isEmpty(); - Message.PutCandyRequest put = node.recorded(Message.PutCandyRequest.class).get(0); - assertThat(put.key()).isEqualTo(dst); - assertThat(put.contentType()).isEqualTo("text/plain"); - assertThat(put.userMetadata()).containsEntry("m", "x"); - assertThat(put.idempotencyToken()).isEqualTo("tok"); - assertThat(node.recorded(Message.DeleteCandyRequest.class)).isEmpty(); + // No byte copy: the source's locator parts were relayed and reused at the destination. + assertThat(node.recorded(Message.GetCandyRequest.class)).isEmpty(); + assertThat(node.recorded(Message.PutCandyRequest.class)).isEmpty(); + Message.GetCandyLocatorRequest gl = + node.recorded(Message.GetCandyLocatorRequest.class).get(0); + assertThat(gl.key()).isEqualTo(src); + Message.ZeroCopyPutRequest copyPut = node.recorded(Message.ZeroCopyPutRequest.class).get(0); + assertThat(copyPut.dstKey()).isEqualTo(dst); + assertThat(copyPut.idempotencyToken()).isEqualTo("tok"); + assertThat(copyPut.renameToken()).isNull(); // a plain copy carries no rename intent - client.renameCandy("box", src, dst, null); - assertThat(node.recorded(Message.RenameCandyRequest.class)).isEmpty(); - Message.DeleteCandyRequest deleted = node.recorded(Message.DeleteCandyRequest.class).get(0); - assertThat(deleted.key()).isEqualTo(src); // rename = copy + delete source + client.renameCandy("box", src, dst, "rtok"); + // Rename = prepare-intent on the source, zero-copy put on the destination, finalize delete. + Message.PrepareRenameRequest pr = node.recorded(Message.PrepareRenameRequest.class).get(0); + assertThat(pr.srcKey()).isEqualTo(src); + assertThat(pr.renameToken()).isEqualTo("rtok"); + Message.ZeroCopyPutRequest renamePut = + node.recorded(Message.ZeroCopyPutRequest.class).get(1); + assertThat(renamePut.renameToken()).isEqualTo("rtok"); + assertThat(renamePut.srcKey()).isEqualTo(src); + Message.CompleteRenameRequest cr = + node.recorded(Message.CompleteRenameRequest.class).get(0); + assertThat(cr.srcKey()).isEqualTo(src); + assertThat(cr.renameToken()).isEqualTo("rtok"); + // Still no whole-object byte copy anywhere. + assertThat(node.recorded(Message.PutCandyRequest.class)).isEmpty(); } } From ba405d67038d1979d58db16f5f2133aae9052b4b Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:27:58 +0000 Subject: [PATCH 5/7] Tests: Box-global GC, rename finalize/abandon, cross-partition zero-copy + crash/resume IT Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- .../candybox/it/PartitionedBoxIT.java | 44 ++++- .../server/CrossPartitionGcAndRenameTest.java | 174 ++++++++++++++++++ 2 files changed, 215 insertions(+), 3 deletions(-) create mode 100644 candybox-server/src/test/java/me/predatorray/candybox/server/CrossPartitionGcAndRenameTest.java diff --git a/candybox-integration-tests/src/test/java/me/predatorray/candybox/it/PartitionedBoxIT.java b/candybox-integration-tests/src/test/java/me/predatorray/candybox/it/PartitionedBoxIT.java index aac199a..525d7b5 100644 --- a/candybox-integration-tests/src/test/java/me/predatorray/candybox/it/PartitionedBoxIT.java +++ b/candybox-integration-tests/src/test/java/me/predatorray/candybox/it/PartitionedBoxIT.java @@ -34,6 +34,8 @@ import me.predatorray.candybox.coordination.fake.InMemoryCoordinationService; import me.predatorray.candybox.protocol.Frame; import me.predatorray.candybox.protocol.FrameCodec; +import me.predatorray.candybox.protocol.Message; +import me.predatorray.candybox.protocol.MessageCodec; import me.predatorray.candybox.protocol.transport.Connection; import me.predatorray.candybox.protocol.transport.RequestHandler; import me.predatorray.candybox.protocol.transport.Transport; @@ -45,8 +47,9 @@ /** * End-to-end behaviour of a hash-partitioned Box whose partitions are owned by two different * nodes, driven through the cluster-aware {@link CandyboxClient}: keyed routing, scatter-gather - * listing with pagination, fanned-out range deletes, cross-partition copy/rename byte-copy - * fallbacks, and multipart uploads. Runs entirely on the in-memory fakes (no BookKeeper/ZooKeeper / + * listing with pagination, fanned-out range deletes, cross-partition zero-copy copy/rename (and its + * crash-and-resume convergence), and multipart uploads. Runs entirely on the in-memory fakes (no + * BookKeeper/ZooKeeper / * sockets) with a port-routing in-JVM transport, so it is fast yet exercises the real codec, request * handler, and routing stack. */ @@ -54,6 +57,7 @@ class PartitionedBoxIT { private static final int PARTITIONS = 4; private static final String BOX = "parted-box"; + private static final MessageCodec MSG_CODEC = new MessageCodec(); /** An in-JVM {@link Transport} that routes by port to the matching node's request handler. */ private static final class RoutingTransport implements Transport { @@ -195,18 +199,20 @@ void deleteRangeFansOutToEveryPartition() { } @Test - void crossPartitionCopyAndRenameFallBackToByteCopy() { + void crossPartitionCopyAndRenameAreZeroCopy() { String src = keyIn(0, "src"); // owned by node 1 String dstCopy = keyIn(3, "copy"); // owned by node 2 String dstMove = keyIn(2, "move"); // owned by node 2 client.putCandy(BOX, src, bytes("payload"), "text/plain", Map.of("m", "x"), null); + // Copy: the destination (on the other node) reads the very bytes the source stored. CandyboxClient.CandyInfo copied = client.copyCandy(BOX, src, dstCopy, null); assertThat(copied.contentLength()).isEqualTo(7); assertThat(client.getCandy(BOX, dstCopy)).isEqualTo(bytes("payload")); assertThat(client.headCandy(BOX, dstCopy).userMetadata()).containsEntry("m", "x"); assertThat(client.getCandy(BOX, src)).isEqualTo(bytes("payload")); // copy keeps the source + // Rename converges to destination-present, source-gone across the two owners. CandyboxClient.CandyInfo moved = client.renameCandy(BOX, src, dstMove, null); assertThat(moved.contentLength()).isEqualTo(7); assertThat(client.getCandy(BOX, dstMove)).isEqualTo(bytes("payload")); @@ -214,6 +220,38 @@ void crossPartitionCopyAndRenameFallBackToByteCopy() { .isInstanceOf(CandyNotFoundException.class); // rename removed the source } + @Test + void crashedCrossPartitionRenameConvergesViaTheMaintenanceSweep() { + String src = keyIn(0, "rsrc"); // owned by node 1 + String dst = keyIn(2, "rdst"); // owned by node 2 + client.putCandy(BOX, src, bytes("payload"), "text/plain", Map.of(), null); + String token = "it-rename-token"; + + // Drive the rename's first two steps by hand, then "crash" before the finalize: + // 1. prepare on the source owner (records the intent + returns the locator parts) + Message prepared = send(nodeA, new Message.PrepareRenameRequest(BOX, src, dst, 2, token)); + Message.CandyLocatorResponse loc = (Message.CandyLocatorResponse) prepared; + // 2. zero-copy put on the destination owner (writes the rendezvous marker) + Message put = send(nodeB, new Message.ZeroCopyPutRequest(BOX, dst, loc.parts(), + loc.contentType(), loc.userMetadata(), loc.owner(), loc.grants(), token, token, + src, 0, loc.hlc())); + assertThat(put).isInstanceOf(Message.HeadCandyResponse.class); + + // The finalize (step 3) is lost: both keys are live (the transient W2 state). + assertThat(client.getCandy(BOX, src)).isEqualTo(bytes("payload")); + assertThat(client.getCandy(BOX, dst)).isEqualTo(bytes("payload")); + + // The source owner's maintenance sweep finds the marker and finalizes the rename. + assertThat(nodeA.finalizeRenameIntentsOnce()).isEqualTo(1); + assertThatThrownBy(() -> client.getCandy(BOX, src)) + .isInstanceOf(CandyNotFoundException.class); + assertThat(client.getCandy(BOX, dst)).isEqualTo(bytes("payload")); + } + + private Message send(CandyboxNode node, Message message) { + return MSG_CODEC.decode(node.requestHandler().handle(MSG_CODEC.encode(message))); + } + @Test void samePartitionCopyStaysServerSide() { String src = keyIn(1, "zsrc"); diff --git a/candybox-server/src/test/java/me/predatorray/candybox/server/CrossPartitionGcAndRenameTest.java b/candybox-server/src/test/java/me/predatorray/candybox/server/CrossPartitionGcAndRenameTest.java new file mode 100644 index 0000000..15c316e --- /dev/null +++ b/candybox-server/src/test/java/me/predatorray/candybox/server/CrossPartitionGcAndRenameTest.java @@ -0,0 +1,174 @@ +/* + * Copyright (c) 2026 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package me.predatorray.candybox.server; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +import java.nio.charset.StandardCharsets; +import java.util.Map; +import me.predatorray.candybox.bookkeeper.fake.InMemoryLedgerStore; +import me.predatorray.candybox.common.BoxName; +import me.predatorray.candybox.common.CandyKey; +import me.predatorray.candybox.common.CandyLocator; +import me.predatorray.candybox.common.ManualClock; +import me.predatorray.candybox.common.Partitioning; +import me.predatorray.candybox.common.auth.ObjectAcl; +import me.predatorray.candybox.common.config.CandyboxConfig; +import me.predatorray.candybox.common.exception.CandyNotFoundException; +import me.predatorray.candybox.coordination.fake.InMemoryCoordinationService; +import me.predatorray.candybox.lsm.engine.BoxEngine; +import me.predatorray.candybox.lsm.manifest.RenameIntent; +import org.junit.jupiter.api.Test; + +/** + * Server-level coverage of the cross-partition zero-copy machinery on a single node owning several + * partitions of a Box: the Box-global garbage collector keeps a Syrup shared across partitions alive, + * and the rename-intent maintenance sweep finalizes (via the rendezvous marker) or abandons (past the + * window) the source-side delete. + */ +class CrossPartitionGcAndRenameTest { + + private static final BoxName BOX = BoxName.of("xp-box"); + private static final int PARTITIONS = 4; + + private static byte[] bytes(String s) { + return s.getBytes(StandardCharsets.UTF_8); + } + + private static String keyIn(int partition, String tag) { + for (int i = 0; i < 10_000; i++) { + String candidate = tag + "-" + i; + if (Partitioning.partitionOf(candidate, PARTITIONS) == partition) { + return candidate; + } + } + throw new AssertionError("no key for partition " + partition); + } + + private static CandyboxConfig aggressiveGcConfig() { + return CandyboxConfig.builder() + .memtableFlushThresholdBytes(1) // every write flushes => its own L0 table + WAL + .syrupRolloverBytes(1) // every Candy in its own Syrup ledger + .l0CompactionTrigger(2) + .ledgerGcGraceMillis(0) // reclaim immediately + .leaseRenewIntervalMillis(0) + .compactionIntervalMillis(0) // manual control + .build(); + } + + @Test + void boxGlobalGcKeepsASyrupSharedAcrossPartitionsAlive() { + InMemoryLedgerStore store = new InMemoryLedgerStore(); + try (CandyboxNode node = new CandyboxNode(1, aggressiveGcConfig(), store, + new InMemoryCoordinationService(), new ManualClock(1000))) { + node.createBox(BOX, PARTITIONS); + String src = keyIn(0, "src"); + String dst = keyIn(1, "dst"); + BoxEngine p0 = node.enginePartition(BOX, 0); + BoxEngine p1 = node.enginePartition(BOX, 1); + + p0.putCandy(CandyKey.of(src), bytes("payload"), "text/plain", Map.of(), null); + p0.flush(); + CandyLocator locator = p0.resolveLocator(CandyKey.of(src)); + long sharedSyrup = locator.parts().get(0).segments().get(0).syrupId(); + + // Partition 1 zero-copy-references partition 0's Syrup (a cross-partition copy). + p1.zeroCopyPut(CandyKey.of(dst), locator.parts(), "text/plain", Map.of(), 0L, + ObjectAcl.NONE, null); + assertThat(p1.referencedSyrups()).contains(sharedSyrup); + + // Delete the source and compact partition 0 so its Syrup becomes a local orphan. + node.collectGarbageOnce(); // publishes every partition's refs (incl. p1 -> sharedSyrup) + p0.deleteCandy(CandyKey.of(src)); + p0.flush(); + for (int i = 0; i < 5; i++) { + node.compactOwnedBoxesOnce(); + } + + // Box-global GC must NOT reclaim the shared Syrup — partition 1 still points at it. + node.collectGarbageOnce(); + assertThat(p1.getCandy(CandyKey.of(dst))).isEqualTo(bytes("payload")); + } + store.close(); + } + + @Test + void renameIntentIsFinalizedWhenTheRendezvousMarkerIsPresent() { + InMemoryLedgerStore store = new InMemoryLedgerStore(); + ManualClock clock = new ManualClock(1000); + try (CandyboxNode node = new CandyboxNode(1, CandyboxConfig.builder() + .leaseRenewIntervalMillis(0).build(), store, + new InMemoryCoordinationService(clock), clock)) { + node.createBox(BOX, PARTITIONS); + String src = keyIn(0, "src"); + String dst = keyIn(1, "dst"); + BoxEngine p0 = node.enginePartition(BOX, 0); + BoxEngine p1 = node.enginePartition(BOX, 1); + String token = "rename-token-1"; + + // Simulate a cross-partition rename that crashed after the destination put (W2: both live). + p0.putCandy(CandyKey.of(src), bytes("payload"), null, Map.of(), null); + CandyLocator locator = p0.resolveLocator(CandyKey.of(src)); + p0.recordRenameIntent(new RenameIntent(token, src, locator.hlc(), dst, 1, + node.currentTimeMillis())); + p1.zeroCopyPut(CandyKey.of(dst), locator.parts(), null, Map.of(), 0L, ObjectAcl.NONE, + token); + node.writeRenameMarker(BOX.value(), token, src, 0, locator.hlc(), dst); + + // Both keys are live before finalization. + assertThat(p0.getCandy(CandyKey.of(src))).isEqualTo(bytes("payload")); + assertThat(p1.getCandy(CandyKey.of(dst))).isEqualTo(bytes("payload")); + + int finalized = node.finalizeRenameIntentsOnce(); + + assertThat(finalized).isEqualTo(1); + assertThatThrownBy(() -> p0.getCandy(CandyKey.of(src))) + .isInstanceOf(CandyNotFoundException.class); // source tombstoned + assertThat(p1.getCandy(CandyKey.of(dst))).isEqualTo(bytes("payload")); // destination kept + assertThat(p0.listRenameIntents()).isEmpty(); // intent cleared + assertThat(node.renameMarkerPresent(BOX.value(), token)).isFalse(); // marker deleted + } + store.close(); + } + + @Test + void renameIntentIsAbandonedWhenNoMarkerAppearsWithinTheWindow() { + InMemoryLedgerStore store = new InMemoryLedgerStore(); + ManualClock clock = new ManualClock(1_000_000); + try (CandyboxNode node = new CandyboxNode(1, CandyboxConfig.builder() + .leaseRenewIntervalMillis(0).renameIntentAbandonMillis(60_000).build(), store, + new InMemoryCoordinationService(clock), clock)) { + node.createBox(BOX, PARTITIONS); + String src = keyIn(0, "src"); + BoxEngine p0 = node.enginePartition(BOX, 0); + String token = "stuck-token"; + + p0.putCandy(CandyKey.of(src), bytes("payload"), null, Map.of(), null); + CandyLocator locator = p0.resolveLocator(CandyKey.of(src)); + // An intent recorded well outside the abandon window, with no destination marker ever set. + p0.recordRenameIntent(new RenameIntent(token, src, locator.hlc(), "dst", 1, + node.currentTimeMillis() - 120_000)); + + int finalized = node.finalizeRenameIntentsOnce(); + + assertThat(finalized).isEqualTo(0); // nothing was completed + assertThat(p0.listRenameIntents()).isEmpty(); // the stuck intent was dropped + assertThat(p0.getCandy(CandyKey.of(src))).isEqualTo(bytes("payload")); // source stays live + } + store.close(); + } +} From b9e9547a056a72e1c210db1d57a0cc49a70e1d5e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 16:34:52 +0000 Subject: [PATCH 6/7] docs+conf: cross-partition zero-copy & Box-global GC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update DESIGN.md (§3/§5/§6/§7/§9/§11/§12), README, OPERATIONS, the Hugo site (architecture/client/operations/reference), and the shipped candybox.properties.example to describe the restored cross-partition zero-copy copy/rename, Box-global Syrup GC, the rename intent journal + rendezvous (eventually-atomic rename), the new wire ops and coordination keys, and the rename.intent.abandon.millis config. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- DESIGN.md | 69 ++++++++++++++----- OPERATIONS.md | 16 ++++- README.md | 11 +-- .../src/conf/candybox.properties.example | 6 ++ docs/content/docs/architecture/_index.md | 6 +- docs/content/docs/client/_index.md | 10 +-- docs/content/docs/operations/_index.md | 10 +++ docs/content/docs/reference/configuration.md | 1 + 8 files changed, 99 insertions(+), 30 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index e5ab59f..fa2a0ec 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -66,7 +66,11 @@ Dependency rule: `candybox-lsm` depends **only** on the two SPIs (`bookkeeper`, serve a stale memtable read, but its *writes* fail). Operations spanning partitions are weaker by construction: `deleteRange` is one tombstone per partition (idempotent, not atomic across them), listings are scatter-gather merges (each partition's page is consistent; the merged page is not a - snapshot), and a cross-partition `rename` is copy-then-delete (not atomic). This is the guarantee + snapshot), and a cross-partition `rename` is copy-then-delete across two owners and only + **eventually atomic** — it converges to "source gone, destination present" via a durable + rename-intent journal plus a coordination rendezvous marker (§6), so a reader can momentarily + observe both keys, but the rename never leaves both keys live forever and never loses data. A + same-partition `rename` is still fully atomic (one owner, one write lock). This is the guarantee Candybox documents — not general eventual consistency. ### HLC recovery on handover (critical correctness point) @@ -143,14 +147,17 @@ RangeTombstone payload>`. Replay reports the max HLC across **both** kinds, so h **Syrup chunk entry**: `int crc32c | payload[<= chunkSize]`. The crc covers the payload only. -**ManifestEdit** (`ManifestSerializer`, **version 2**): `version | addedTables[] +**ManifestEdit** (`ManifestSerializer`, **version 3**): `version | addedTables[] | removedTableLedgerIds[] | addedSyrups[] | removedSyrups[] | (bool, varlong) newWalLedgerId -| ownerFencingToken | addedUploads[] | upsertParts[] | removedUploads[]`, where each `SSTableMeta` +| ownerFencingToken | addedUploads[] | upsertParts[] | removedUploads[] | addedRenameIntents[] +| removedRenameIntents[]`, where each `SSTableMeta` is `varlong ledgerId | varint level | bytes minKey | bytes maxKey | varlong entryCount`. The multipart-tracking trailing fields (Phase 5) carry in-flight upload state: a `MultipartUploadState` record per `addedUpload`, a `(uploadId, partNumber, Part)` triple per `upsertParts`, and a string -per `removedUpload`. Replayed on handover so a takeover sees the upload exactly where the prior -owner left it. +per `removedUpload`. The v3 trailing fields carry in-flight **rename intents** (one per cross-partition +`rename` the partition owns the source of, each pinning the source key and its HLC); like upload +state they are replayed on handover, so a takeover finalizes (or abandons) a rename exactly where +the prior owner left it. **Protocol frame** (`FrameCodec`): `magic(2)=0xCB0F | version(1)=1 | opcode(1) | length(4) | payload`. **Message body** (`MessageCodec`): `bodyVersion(1) | `. @@ -202,11 +209,22 @@ consults range tombstones across all tables rather than pruning by point range. - **copy / rename**: when source and destination hash to the **same partition**, write a fresh PUT at the destination reusing the source locator's *parts* verbatim (zero byte copy and zero locator rebuild — a multipart source becomes a multipart-shaped copy); `rename` also tombstones the - source. Both commit atomically under the partition owner's write lock (same Box only). Because GC - is reference-counted by Syrup id *within one manifest*, several keys may share one segment set - safely — which is also why a **cross-partition** copy/rename cannot share segments (the source - partition's GC would not see the destination's references) and instead degrades to a client-side - byte copy (`rename`: copy then delete, not atomic). + source. Both commit atomically under the partition owner's write lock (same Box only). Several + keys may share one segment set safely because GC reference-counts Syrups by actual `SegmentRef` + (§9). **Cross-partition** copy/rename is **zero-copy too**: the client fetches the source's + `CandyLocator` parts from the source partition's owner (`GetCandyLocator` / `PrepareRename` → + `CandyLocatorResponse`) and relays them to the destination owner (`ZeroCopyPut`), which writes a + destination locator reusing the very same Syrup segments — no object bytes are ever moved or + re-stored. This is safe because the blob layer is physically global (a `SegmentRef` resolves bytes + by `syrupId` alone, no partition scoping) and because **Box-global GC** (§9) keeps a Syrup alive + while *any* partition references it. A cross-partition `rename` is still copy-then-delete across + two owners, but now **eventually atomic**: the source owner records a durable rename intent (§5), + the destination owner writes a rendezvous marker once its zero-copy put is durable, and the source + owner — synchronously via the client's `CompleteRename`, on its maintenance sweep, or after a + handover replays the intent — reads the marker and LWW-conditionally tombstones the source (so a + re-PUT source is never clobbered), then clears the intent and marker. A reader may momentarily see + both keys; the one residual crash window degrades to "both keys live" (the old observable outcome), + never data loss. - **range get**: `getCandyRange(key, firstByte, lastByte)` over HTTP `Range: bytes=…` semantics (inclusive on both ends). Prefix-sums the part lengths to find the first/last parts, then indexes into chunks by `byteWithinPart / part.chunkSize`. Chunks at the slice boundary are read whole — @@ -258,6 +276,8 @@ The coordination layout: boxes//meta BoxDescriptor {partitionCount} — immutable, the routing truth boxes//partitions/

/owner partition p's ownership lease (fenced, TTL'd) boxes//partitions/

/manifest partition p's manifest-ledger pointer (versioned CAS) +boxes//partitions/

/refs partition p's published referenced-Syrup set (Box-global GC, §9(f)) +boxes//renames/ cross-partition rename rendezvous marker (§6); cleared on finalize cluster/balancer the balancer's coordinator-election lease cluster/assignment the desired partition→node assignment table (versioned CAS) members/ membership (advertised host:port) @@ -334,7 +354,16 @@ The detailed rules: segment set (a `copyCandy`, or the in-flight state of a `renameCandy`) keep that Syrup live until *all* of them are gone — no per-key refcount table is needed. A range tombstone holds no segments, so a Candy it shadows is reclaimed only once compaction drops the covered point locator (as with a - point delete). + point delete); +- (f) **Box-global GC** (what makes cross-partition zero-copy safe): a destination partition can now + reuse a source partition's Syrup segments (§6), so within-manifest reference counting is + generalized Box-wide. Each partition owner publishes its referenced-Syrup set to coordination at + `boxes//partitions/

/refs`; the creating partition's GC skips any Syrup a sibling partition + still references (consulting the published sets, gated by the same `ledgerGcGraceMillis`), so a + Syrup is physically reclaimed only once **no** partition of the Box references it. This supersedes + the prior claim (under (e)) that cross-partition copy/rename cannot share segments. Safe by + construction: a partial/crashed rename can only *retain* a Syrup (a leak — the accepted v1 failure + mode), never delete a referenced one. The full compaction-then-GC cycle is covered end-to-end by `CompactionGcCycleIT`. One gap: deleting a Box drops its manifest pointer but does not yet reclaim that Box's SSTable/Syrup/WAL/manifest ledgers @@ -378,11 +407,12 @@ Per-role BK quorum (E/Qw/Qa) defaults: **WAL 3/3/2, Manifest 3/3/2, SSTable 3/2/ | Balancer round interval / move rate | 5 s in shipped conf (0 = off, the unit-test default) / 4 moves per round | Frequent enough to converge quickly after joins/failures; the move cap keeps a node join from stampeding handovers (failover is never rate-limited). | | Compaction + GC worker interval | configurable; 0 disables | Background maintenance cadence on the owner. | | Tombstone-GC time bound | 24 h | Covers in-flight late writes before a delete is reclaimable. | -| Ledger-GC grace | 5 min | Margin for in-flight readers / continuation tokens before a physical delete. | +| Ledger-GC grace | 5 min | Margin for in-flight readers / continuation tokens before a physical delete; also gates Box-global GC (§9(f)). | +| Rename-intent abandon | 60 s (`rename.intent.abandon.millis`) | A cross-partition rename intent whose rendezvous marker never appears is dropped after this (the source stays live). | | Client router cache TTL | 5 s | How long the client caches a Box→owner mapping before re-resolving. | | Continuation token | `lastKey` | `lastKey` alone resumes a range/reverse scan, exclusive in the scan direction. | | LWW tiebreaker | `nodeId` (locked) | Deterministic, coordination-free. | -| TCP opcodes | incl. dedicated `RESPONSE_BUSY` and `RESPONSE_MOVED` | Backpressure and re-routing are first-class signals. | +| TCP opcodes | incl. dedicated `RESPONSE_BUSY` and `RESPONSE_MOVED`, plus the cross-partition zero-copy ops `GET_CANDY_LOCATOR` / `PREPARE_RENAME` / `ZERO_COPY_PUT` / `COMPLETE_RENAME` and response `RESPONSE_CANDY_LOCATOR` | Backpressure, re-routing, and the cross-partition locator relay are first-class signals. | ## 12. Deliberate v1 simplifications (escape hatches) @@ -391,9 +421,13 @@ Per-role BK quorum (E/Qw/Qa) defaults: **WAL 3/3/2, Manifest 3/3/2, SSTable 3/2/ listing a scatter-gather fan-out (each page queries all partitions). Key-range partitions with splits, and re-partitioning, are future work. A partition is still unavailable during its own handover fence+replay window (shrunk by the pre-handover flush). -- **Cross-partition copy/rename byte-copies through the client** — zero-copy segment sharing is only - safe within one partition's manifest (§6); a cross-partition `rename` is copy-then-delete and not - atomic. S3 semantics (CopyObject, no rename) are unaffected. +- **Cross-partition copy/rename is zero-copy (resolved), but its rename is only *eventually* atomic** + — the former byte-copy-through-the-client simplification is gone: cross-partition copy/rename now + reuse the source's Syrup segments via the locator relay, kept safe by Box-global GC (§6, §9(f)). A + cross-partition `rename` is copy-then-delete across two owners and converges via the rename-intent + journal + rendezvous marker — *eventually* (not linearizably) atomic; a reader may briefly see both + keys. **`UploadPartCopy` across partitions still byte-copies** (out of scope, see below). S3 + semantics (CopyObject, no rename) are unaffected. - **Box-level `deleteBox` needs takeover or the balancer** — the deleting node takes over partitions whose leases are free; partitions held by other live nodes fail a non-force delete, while a force delete removes the descriptor and lets each owner's balancer sweep drop its partitions. @@ -419,7 +453,8 @@ path, range tombstones, zero-copy copy/rename); ZooKeeper-backed coordination ownership/handover; **hash-partitioned Boxes** — per-partition fenced ownership spread across the cluster by the elected-coordinator `PartitionBalancer` (sticky, failover-eager, rate-limited moves), with partition-aware client routing (keyed ops to the key's partition owner; scatter-gather -listing; fanned-out range deletes; cross-partition copy fallback); cluster membership + client-side +listing; fanned-out range deletes; cross-partition zero-copy copy/rename via the locator relay); +cluster membership + client-side routing over the framed TCP protocol (`RESPONSE_MOVED`); multi-level leveled compaction on a background worker; reference-counted GC of SSTable/Syrup/WAL ledgers; the storage node with health/metrics, the `candybox` CLI, and a packaged distribution (Docker image + Compose + diff --git a/OPERATIONS.md b/OPERATIONS.md index ab05078..38df6c4 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -55,7 +55,8 @@ All knobs are configurable; pick defaults unless a workload demands otherwise. | `l0StallThreshold` | 12 | L0 SSTable count at which writes are rejected with `BUSY`. | | `maxClockSkewMillis` | 5 min | HLC skew-rejection bound on observed timestamps. | | `tombstoneGcGraceMillis` | 24 h | Late-write window before a bottommost tombstone may be dropped. | -| `ledgerGcGraceMillis` | 5 min | Grace before an obsoleted ledger (compaction input, dead Syrup, rotated WAL) is deleted. | +| `ledgerGcGraceMillis` | 5 min | Grace before an obsoleted ledger (compaction input, dead Syrup, rotated WAL) is deleted; also gates Box-global GC of cross-partition-shared Syrups. | +| `rename.intent.abandon.millis` | 60 s | Cross-partition rename: a rename intent whose rendezvous marker never appears is dropped after this (the source stays live, the rename never reached the destination). Env `CANDYBOX_RENAME_INTENT_ABANDON_MILLIS`. | Leveled compaction also takes a per-level byte budget (`levelBaseBytes` 10 MiB, `levelMultiplier` 10): level N's budget is `levelBaseBytes × multiplier^(N-1)`. @@ -72,7 +73,10 @@ level N's budget is `levelBaseBytes × multiplier^(N-1)`. to the highest HLC (LWW, nodeId tiebreaker). The only per-key eventual-consistency window is ownership handover. Cross-partition operations are weaker: listings are scatter-gather merges, `deleteRange` is one tombstone per partition (idempotent, not atomic), and a cross-partition - rename is copy-then-delete. + rename is copy-then-delete across two owners — only **eventually atomic**, converging to "source + gone, destination present" via a durable rename-intent journal plus a coordination rendezvous + marker (so a reader may briefly see both keys, but the rename never strands both keys forever). A + same-partition rename is fully atomic. - **Backpressure.** When a Box accumulates `l0StallThreshold` L0 SSTables, writes return a retriable `BUSY` (protocol `RESPONSE_BUSY`) instead of blocking. Clients should back off and retry; the stall clears once compaction drains L0. @@ -88,6 +92,7 @@ level N's budget is `levelBaseBytes × multiplier^(N-1)`. | **Zombie owner / zombie compactor** | A fencing token on every manifest append, plus BookKeeper recover-open, reject any commit from a superseded owner. | | **Client hits the wrong node** | The node replies `RESPONSE_MOVED(ownerNodeId)`; the client re-resolves via coordination and retries. | | **Clock skew** | HLC ordering survives a regressing wall clock; an observed HLC leading local time by more than `maxClockSkewMillis` is rejected. | +| **Crash mid cross-partition rename** | Converges by roll-forward: the source owner finalizes its durable rename intent against the destination's rendezvous marker — synchronously, on its background maintenance sweep, or after a handover replays the intent. Marker present ⇒ source tombstoned; absent past `rename.intent.abandon.millis` ⇒ intent dropped, source stays live. It never leaves both keys live forever and never loses data. | With the balancer enabled (`balancerIntervalMillis > 0`), a dead node's partitions are reassigned and taken over automatically within a round or two of its leases expiring. With it disabled, @@ -104,6 +109,13 @@ Run only by a Box's owner, against the committed manifest, after `ledgerGcGraceM so deletes/overwrites cause Syrup space amplification until the whole ledger dies). - **WAL** ledgers rotated out at flush are deleted once their mutations are durable in an SSTable. +**Box-global GC.** Because cross-partition zero-copy copy/rename lets a destination partition share a +source partition's Syrup segments, each partition owner publishes its referenced-Syrup set to +coordination (`boxes//partitions/

/refs`). The creating partition's GC skips any Syrup a +sibling partition still references, so a shared Syrup is reclaimed only once **no** partition of the +Box references it (still gated by `ledgerGcGraceMillis`). A partial or crashed rename can therefore +only *retain* a Syrup (a leak), never delete one another partition still points at. + GC tracking is in-memory, so ledgers orphaned by a prior owner that crashed before GC leak until an enumeration backstop is added (future). diff --git a/README.md b/README.md index c2384bd..e465015 100644 --- a/README.md +++ b/README.md @@ -130,17 +130,18 @@ operations an S3-style store cannot do cheaply: ```bash candybox list photos --start a --end m --reverse # bounded, reverse-order range scan candybox copy photos cat.jpg cat-copy.jpg # zero-copy: shares the stored bytes -candybox rename photos cat.jpg pets/cat.jpg # zero-copy move (same Box) +candybox rename photos cat.jpg pets/cat.jpg # zero-copy move (within a Box) candybox delete-range photos thumbnails/ # one O(1) range tombstone, not N deletes candybox delete-range photos --start a --end m # delete a half-open [start, end) key window ``` - **Bounded / reverse range scans** walk a `[start, end)` window in either direction (`list --start K --end K --reverse`), paging with `--start-after`. -- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved — and - `rename` removes the source atomically (same Box; when source and destination land in different - hash partitions the client transparently falls back to a byte copy, and the rename is no longer - atomic). +- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved, even + when source and destination land in different hash partitions (the stored bytes are shared across + partitions). A same-partition `rename` removes the source atomically; a cross-partition `rename` is + *eventually* atomic — it converges to "source gone, destination present" (a reader may briefly see + both keys, but the rename never strands both keys forever). - **`delete-range`** deletes a whole prefix or key window with a single range tombstone (constant work regardless of how many keys it covers); the bytes are reclaimed lazily by compaction. diff --git a/candybox-dist/src/conf/candybox.properties.example b/candybox-dist/src/conf/candybox.properties.example index 783b379..8e1f4ed 100644 --- a/candybox-dist/src/conf/candybox.properties.example +++ b/candybox-dist/src/conf/candybox.properties.example @@ -88,11 +88,17 @@ data.dir=./data # balancer — partitions then stay where they were created until moved manually. # balancer.max.moves.per.round: at most this many partitions are migrated away from live owners # per round (failover of a dead node's partitions is never rate-limited). +# rename.intent.abandon.millis: how long a source-side cross-partition rename intent may sit +# without its destination completion marker before it is abandoned (the rename never reached the +# destination; the source stays live). Roll-forward otherwise: a present marker finalizes the +# source delete. Cross-partition copy/rename are zero-copy (no bytes are moved); rename is +# eventually atomic — see CROSS_PARTITION_ZERO_COPY_PLAN.md. # --------------------------------------------------------------------------- # partitions.per.box.default=8 balancer.interval.millis=5000 # balancer.max.moves.per.round=4 +# rename.intent.abandon.millis=60000 # --------------------------------------------------------------------------- # Multipart upload tuning (S3-style): diff --git a/docs/content/docs/architecture/_index.md b/docs/content/docs/architecture/_index.md index 62bdea7..8752f3e 100644 --- a/docs/content/docs/architecture/_index.md +++ b/docs/content/docs/architecture/_index.md @@ -64,8 +64,10 @@ operations an S3-style store cannot do cheaply: - **Bounded / reverse range scans** walk a `[start, end)` window in either direction, paging with `--start-after`. -- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved — and - `rename` removes the source atomically (within a Box). +- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved, even + across hash partitions (the stored bytes are shared cluster-wide). A same-partition `rename` removes + the source atomically; a cross-partition `rename` is *eventually* atomic, converging to "source gone, + destination present". - **`delete-range`** deletes a whole prefix or key window with a single range tombstone (constant work regardless of how many keys it covers); the bytes are reclaimed lazily by compaction. diff --git a/docs/content/docs/client/_index.md b/docs/content/docs/client/_index.md index 1114eea..e6d7ba1 100644 --- a/docs/content/docs/client/_index.md +++ b/docs/content/docs/client/_index.md @@ -31,16 +31,18 @@ standard output. ```bash candybox list photos --start a --end m --reverse # bounded, reverse-order range scan candybox copy photos cat.jpg cat-copy.jpg # zero-copy: shares the stored bytes -candybox rename photos cat.jpg pets/cat.jpg # zero-copy move (same Box) +candybox rename photos cat.jpg pets/cat.jpg # zero-copy move (within a Box) candybox delete-range photos thumbnails/ # one O(1) range tombstone, not N deletes candybox delete-range photos --start a --end m # delete a half-open [start, end) key window ``` - **Bounded / reverse range scans** walk a `[start, end)` window in either direction (`list --start K --end K --reverse`), paging with `--start-after`. -- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved — and - `rename` removes the source atomically (same Box; when source and destination land in different hash - partitions the client transparently falls back to a byte copy, and the rename is no longer atomic). +- **Zero-copy `copy` / `rename`** point a new key at the *same* stored bytes — no data is moved, even + when source and destination land in different hash partitions (the stored bytes are shared across + partitions). A same-partition `rename` removes the source atomically; a cross-partition `rename` is + *eventually* atomic — it converges to "source gone, destination present" (a reader may briefly see + both keys, but the rename never strands both keys forever). - **`delete-range`** deletes a whole prefix or key window with a single range tombstone (constant work regardless of how many keys it covers); the bytes are reclaimed lazily by compaction. diff --git a/docs/content/docs/operations/_index.md b/docs/content/docs/operations/_index.md index 35f674d..527366e 100644 --- a/docs/content/docs/operations/_index.md +++ b/docs/content/docs/operations/_index.md @@ -21,6 +21,16 @@ A Candybox deployment is a set of identical storage nodes plus their dependencie - Optionally the **S3 gateway** and the **admin / dashboard API**, both stateless and horizontally scalable behind a load balancer. +## Garbage collection + +Each Box owner reclaims obsoleted ledgers (compaction-input SSTables, dead Syrups, rotated WALs) +against its committed manifest after a grace period. Because cross-partition zero-copy copy/rename +lets a destination partition share a source partition's stored bytes, GC is **Box-global**: every +partition owner publishes its referenced-Syrup set to coordination, and a shared Syrup is reclaimed +only once **no** partition of the Box references it. See the +[Garbage collection section of `OPERATIONS.md`](https://github.com/predatorray/candybox/blob/main/OPERATIONS.md#garbage-collection) +for the full rules. + ## Observability Each node exposes an HTTP endpoint (default port `9710`) with `/healthz`, `/readyz`, and `/metrics`. diff --git a/docs/content/docs/reference/configuration.md b/docs/content/docs/reference/configuration.md index b5e64e1..1faf358 100644 --- a/docs/content/docs/reference/configuration.md +++ b/docs/content/docs/reference/configuration.md @@ -21,6 +21,7 @@ and configure each instance through the environment. | `health.port` | HTTP port for `/healthz`, `/readyz`, `/metrics`. | `9710` | | `quorum.*` | BookKeeper replication per ledger role (`E/Qw/Qa`). | `3/3/2` (WAL, manifest), `3/2/2` (data) | | `multipart.upload.ttl.millis` | How long an abandoned multipart upload is kept before a background sweep aborts it. | 7 days | +| `rename.intent.abandon.millis` | Cross-partition rename: how long the source owner keeps a rename intent whose destination rendezvous marker never appears before dropping it (the source stays live). | 60000 (60 s) | ## Full reference From f5ec19f2c4ead14c662dd5fe46ecaa03d734d909 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 20 Jun 2026 00:03:08 +0000 Subject: [PATCH 7/7] docs: annotate BOX_PARTITIONING_PLAN decision #5 as superseded by cross-partition zero-copy Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MKLpqCjqn5Pt8dEiXuSTwE --- BOX_PARTITIONING_PLAN.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/BOX_PARTITIONING_PLAN.md b/BOX_PARTITIONING_PLAN.md index 1376e0c..2cc7591 100644 --- a/BOX_PARTITIONING_PLAN.md +++ b/BOX_PARTITIONING_PLAN.md @@ -19,6 +19,12 @@ Decisions locked with the project owner: 5. Cross-partition `copy`/`rename` degrade to a **client-side byte copy** (rename additionally deletes the source afterwards, i.e. it is no longer atomic across partitions). Same-partition copy/rename keep the zero-copy server-side path. + > **Superseded** by the cross-partition zero-copy work (see + > [`CROSS_PARTITION_ZERO_COPY_PLAN.md`](CROSS_PARTITION_ZERO_COPY_PLAN.md)): cross-partition + > copy/rename are now **zero-copy too** (the client relays the source locator's segments to the + > destination owner, kept safe by Box-global GC), and a cross-partition `rename` is **eventually + > atomic** (a durable rename-intent journal + coordination rendezvous marker converge to "source + > gone, destination present"). `UploadPartCopy` across partitions still byte-copies. 6. **No migration / backward compatibility** with the pre-partitioning ZK layout or wire format (not in production yet). 7. Scope is **write scaling only** — reads still go to each partition's owner; compaction still