diff --git a/extensions/s3/API.md b/extensions/s3/API.md index 0a0dfa38..abecf8d5 100644 --- a/extensions/s3/API.md +++ b/extensions/s3/API.md @@ -25,7 +25,7 @@ operations from administrative maintenance. | Restore logical state as new history | `start_restore`, `advance_restore` | | Move a branch administratively | `reset_branch` | | Merge branches | `start_merge`, `advance_merge`, `publish_merge` | -| Check repository integrity | `start_fsck`, `advance_fsck` | +| Check repository integrity | `start_fsck`, `advance_fsck`, `resume_fsck`, `start_fsck_cleanup`, `advance_fsck_cleanup` | | Reclaim unreachable immutable data | `start_gc`, `advance_gc`, `sweep_gc` | | Synchronize only one logical snapshot | `start_repair_from`, `start_clone_from`, `start_fetch_from`, `start_push_to` | | Preserve a complete source commit DAG | `start_history_clone_from`, `start_history_fetch_from`, `start_history_push_to` | @@ -56,6 +56,7 @@ Start with `Client::builder` (the `builder` constructor). The builder exposes: |---|---| | AWS transport and location | `aws_client`, `bucket`, `repository_prefix`, `default_branch` | | Writer identity and fencing | `writer`, `authority_lease_duration`, `read_only` | +| Persisted metadata-tree geometry | `state_tree_format` (initialization-time; must match on reopen) | | Provider qualification | `provider_identity`, `attestation_signer`, `provider_attestation`, `provider_attestation_validity`, `provider_per_key_version_limit` | | Immutable-node caching | `node_cache`, `max_cached_node_pack_bytes`, `max_cached_node_locations`, `max_cached_node_bytes` | | Index maintenance | `background_index_maintenance`, `journal_index_max_unindexed_events`, `operation_index_limits` | @@ -114,8 +115,28 @@ When an object listing is truncated, pass the last returned logical key as values into durable atomic batches, uploads each checkpoint window with bounded concurrency, and returns one receipt per published batch. `put_object_stream` accepts a fallible `Stream` plus `BulkWriteOptions` for bounded-memory ingestion -from an unbounded source. Completed checkpoint windows remain resumable after -cancellation or a source/object failure. +from an unbounded source. Durable checkpoints append only mutations changed +since the preceding sequence; resume validates and folds the windows by key. +Completed windows remain resumable after cancellation or a source/object +failure without rewriting earlier payload bindings. + +`ordered_publication_queue` is the concurrent-caller group-commit path. +`OrderedPublicationOptions` independently bounds channel capacity, unique keys +per publication, whole-object upload concurrency, coalescing wait, and durable +checkpoint-window size. Producers await channel capacity. Unique keys are +prepared concurrently and published in canonical order; repeated submissions +for one key are split across consecutive commits so version order is retained. +One failed object returns its own error without discarding valid objects in the +same group. Successful callers receive a constant-size +`OrderedPublicationReceipt` only after the grouped ref CAS succeeds. + +Every distinct payload is stored as one complete immutable provider object. +Built-in streaming uses one bounded disk spool followed by one conditional +`PutObject`, so it is limited by the provider single-PUT maximum. For larger or +resumable transfers, call `prepare_external_object_upload`, complete the one +final object with a provider transfer manager, and then call +`stage_external_object_upload`. Prolly never persists upload IDs, parts, or +payload extents. For explicit control, call `begin_commit`. `CommitSessionBuilder` supports: @@ -186,8 +207,15 @@ pruned without loading full commit node packs. `start_fsck(false)` validates metadata and immutable structure. `start_fsck(true)` additionally downloads and hashes reachable payload bytes. -Advance either mode with `advance_fsck`, persisting the cursor after every -page. +Advance either mode with `advance_fsck`. The repository durably checkpoints +every returned page. After process loss, call `resume_fsck(job)`; a stale worker +is fenced by the checkpoint generation. After retaining the completed report, +use `start_fsck_cleanup(job)` and `advance_fsck_cleanup` to exact-delete the +job's payload-dedup tree, closure tree, and checkpoint in bounded pages. + +`FsckReport` separately counts logical payload references, distinct complete +physical objects, verified bytes, and deep content bytes. Payload bodies are +never packed or split by Prolly. ### GC @@ -196,8 +224,10 @@ Start with `start_gc(grace_millis)`, advance marking and discovery with Retention pins are roots. The grace period must exceed the longest possible unpublished upload, commit session, merge, repair, or transfer. -Concurrent GC coordinates all writer handles inside the authoritative process. -Quiesce separately running writer processes before GC. +GC closes durable repository-wide publication admission. Branch and tag CAS +operations in every process hold expiring publication tickets; marking starts +only after all pre-maintenance tickets finish or expire. New publications fail +with `PreconditionFailed` until GC cleanup reopens admission. ## Repair, clone, fetch, push, and backup @@ -240,15 +270,21 @@ These methods are operational controls, not normal foreground request paths: - `node_cache_snapshot` returns immutable-node cache counters. - `prewarm_node_cache` traverses both state trees for one snapshot. +- `prewarm_node_cache_levels` loads only a bounded number of shared upper + levels, avoiding a full scan before point-read traffic. - `s3_operation_metrics` returns provider operation and wire-attempt counters. - `reset_s3_operation_metrics` atomically returns and resets those counters. Metrics are process-local. Export them before process termination and correlate them with provider request IDs and service-side metrics. +Prolly reports only operations it owns. Multipart create, part, complete, abort, +and resumable-transfer metrics belong to the external provider transfer manager +and are intentionally absent from Prolly metrics. Journal-derived node indexes are built from compact commit descriptors and -node-pack tables of contents. Payload sections are range-fetched only when a -referenced node is actually read. +metadata node-pack tables of contents. Encoded node regions are range-fetched +only when a referenced node is actually read; node packs never contain user +object bytes. ## Error and consistency model diff --git a/extensions/s3/ENTERPRISE-READINESS-AUDIT.md b/extensions/s3/ENTERPRISE-READINESS-AUDIT.md index 5fddc8d9..b10976ca 100644 --- a/extensions/s3/ENTERPRISE-READINESS-AUDIT.md +++ b/extensions/s3/ENTERPRISE-READINESS-AUDIT.md @@ -1,6 +1,6 @@ # Prolly S3 enterprise-readiness audit -Audit date: 2026-08-12 +Audit date: 2026-08-13 Baseline: `ff6beb10` (`origin/main` at audit start) @@ -31,7 +31,7 @@ contract, update `API.md`, and add runnable scenario examples. | Writer fencing | Ready with runbook | Branch-scoped leases, renewal, takeover barrier, fenced-branch reporting | | History and merge | Ready | Bounded log/diff/reflog, restartable restore and structural merge, DAG-preserving transfer | | Integrity and repair | Ready | Metadata/deep fsck, logical repair, downloaded-content backup verification | -| Garbage collection | Conditional | Bounded exact-version sweep and dirty roots; separately running writers must be quiesced | +| Garbage collection | Ready pending live fault injection | Bounded exact-version sweep plus durable cross-process publication admission and expiring tickets | | Local provider compatibility | Ready | Eight live RustFS integration tests and all six runnable examples passed against the pinned image | | AWS compatibility | Not yet evidenced here | Operator-owned general-purpose versioned bucket qualification is required | | Performance and scale | Not generally qualified | 10K and AWS SLO gates exist but require recorded workload-specific runs | @@ -133,14 +133,14 @@ read so the memory provider can no longer mask this behavior. passed, but the 10K RustFS commit and 4K graph gates did not complete inside this audit's 15-minute limit. Even passing those regression tests would not prove millions or billions of files, commits, or refs. -3. **GC does not fence separate operating-system processes.** It coordinates - writer handles in the authoritative process. A production runbook must - quiesce or revoke other writers before GC, or the protocol must gain a - durable cross-process publication barrier. -4. **Large-file behavior is deliberately limited.** One logical file must fit - the repository limit, the provider's single `PutObject` limit, and local - spool capacity for streamed sessions. There is no multipart logical-file - representation. +3. **Cross-process GC needs provider fault evidence.** The protocol now closes + durable publication admission and drains expiring per-publication tickets, + but crash/timeout races still require live multi-process fault injection on + every supported provider. +4. **Large-file transfer needs provider-manager evidence.** Prolly deliberately + owns no multipart or chunk lifecycle. The embedding service must qualify its + provider transfer manager's resume, abort cleanup, retry cost, throughput, + encryption, and completed whole-object handoff on RustFS and AWS. 5. **Disaster recovery needs provider-level drills.** History transfer and logical verification do not prove that bucket replication, lifecycle, encryption-key recovery, retention, or regional failover are configured diff --git a/extensions/s3/OPERATIONS.md b/extensions/s3/OPERATIONS.md index 5f4895c3..37220f1a 100644 --- a/extensions/s3/OPERATIONS.md +++ b/extensions/s3/OPERATIONS.md @@ -62,6 +62,21 @@ hit ratio, bytes, eviction, validation failures, and provider range reads. Prewarm current branch roots and upper levels after deployment or failover when tail latency matters. +## Integrity checks + +`start_fsck` creates a snapshot-bound durable job. Every `advance_fsck` page +updates its repository checkpoint with a monotonic generation. After process +loss, reopen the repository, catch up the source branch index if necessary, +then call `resume_fsck(job)`. Never continue a locally saved cursor after +another worker has advanced the job; stale generations fail with `RefConflict`. + +Keep the completed report for audit, then call `start_fsck_cleanup(job)` and +advance its cursor in bounded pages. Cleanup removes the distinct-payload work +tree, commit-closure work tree, older checkpoint versions, and finally the +current checkpoint. It is permitted only after completion and can restart from +the beginning after process loss. Metadata mode verifies bindings and provider +metadata; deep mode downloads and hashes each distinct complete payload object. + ## Backups Provider bucket replication or object backup must preserve all repository @@ -73,14 +88,18 @@ backup verification before declaring a backup usable. ## Storage retention Use bounded `start_gc`, `advance_gc`, and `sweep_gc` jobs to reclaim unreachable -immutable commit, direct-node, and payload versions. Persist the cursor after -every page. Set the grace period longer than the maximum duration of any -unpublished commit, merge, repair, or transfer. Retention pins are GC roots. - -GC journals branch/tag changes and fences deletion batches against concurrent -publication in the authoritative process. Quiesce separately running writer -processes before GC. Never delete payload, commit, node, publication, index, or -administration keys manually. +immutable commit, direct-node, and payload versions. The repository durably +checkpoints every returned page. Set the grace period longer than the maximum +duration of any unpublished commit, merge, repair, or transfer. Retention pins +are GC roots. + +GC closes a durable repository-wide publication-admission epoch. Every branch +or tag CAS owns an expiring publication ticket, including writers in separate +processes. Marking waits for pre-epoch tickets to finish or expire; new +publications receive `PreconditionFailed` until cleanup reopens admission. +Alert on tickets that approach the authority-lease duration. Never delete +payload, commit, node, publication, index, ticket, or administration keys +manually. Expired mutable commit-session checkpoints can be removed through `cleanup_expired_commit_sessions`. diff --git a/extensions/s3/PERFORMANCE-ENVELOPE-2026-08-13.md b/extensions/s3/PERFORMANCE-ENVELOPE-2026-08-13.md new file mode 100644 index 00000000..5fd14a99 --- /dev/null +++ b/extensions/s3/PERFORMANCE-ENVELOPE-2026-08-13.md @@ -0,0 +1,312 @@ +# Prolly S3 measured performance envelope + +Date: 2026-08-13 + +Provider: local RustFS 1.0.0-beta.10, versioned bucket, Apple Silicon Docker +Desktop, 32-way client concurrency. These numbers are regression evidence for +this machine, not AWS SLO evidence. + +> Architecture notice: the historical results table was collected with an +> experimental payload-packing and Prolly-owned multipart implementation that +> has since been removed. It remains useful as a metadata-tree baseline but does +> not qualify the current architecture. The corrected 10K, 100K, 500K, and 1M +> measurements in this section use one complete provider object per distinct +> payload. Historical packed-payload rows remain labeled and must not be used +> to qualify the current build. + +## Corrected whole-object baseline + +After removing payload packs and Prolly-owned multipart state, the 10K +tiny-file RustFS gate stored every distinct 16-byte body as one complete +provider object. Eliminating a redundant authority read per staged object and +replacing cumulative checkpoints with append-only mutation windows reduced the +run from 35.19 seconds, 10,097 calls, and 44.1 MB uploaded to 20.64 seconds, +10,079 calls, and 19.43 MB. With 128 bounded uploads it reached 484.6 files/s. +The 64-way and 256-way probes reached 461.8 and 477.1 files/s respectively, so +128 is the measured local configuration. It remains just below the unchanged +500 files/s release gate. Remaining amplification is complete logical/version +metadata plus bounded checkpoint windows, not payload chunks or packs. + +The revised 10K same-branch gate now records per-commit p50/p95/p99 as well as +throughput. A debug-build diagnostic at concurrency 32 was deliberately stopped +after 1,000 commits took 247.35 seconds (4.04 commits/s); completing 10K would +have consumed roughly 41 minutes before allowing for history growth. This is a +failed scalability signal, not a completed release measurement. It confirms +that independent one-object commits must not be the bulk-ingestion model: +ordered group commit or multiple ingestion branches followed by structural +merge is required. The earlier 13.44 commits/s result remains historical and +must not be used as evidence for the current whole-object build. + +The metadata-only deep-history gate was rerun in release mode with 4,096 +first-parent commits. Indexed merge-base selection completed in 1.578 ms using +27 object-plane calls and zero fallback commit visits, confirming that graph +skip pointers avoid linear history traversal in this case. + +An ordered publication queue now coalesces independently submitted unique keys +into one commit, splits repeated keys across commits, returns per-object staging +errors, and uses constant-size acknowledgements. The first implementation +revalidated the session per object and copied the full batch receipt to every +caller; those defects produced 2.057 calls/file and O(N²) acknowledgement work. +After correction, a fresh-bucket release run published 10K submissions as one +commit with ten durable checkpoint windows, 10,083 S3 calls (1.008/file), and a +24.84-second p99. Best throughput was 402.6 files/s at concurrency 512; 256 and +1,024 reached 376.6 and 335.4 files/s. The unchanged 500 files/s gate therefore +still fails. The p99 is cohort latency: every caller is acknowledged after the +single grouped ref CAS. A same-host bulk run reached 342.0 files/s, showing the +queue is now near the provider's current whole-object PUT envelope rather than +limited by publication or acknowledgement copying. + +### Corrected whole-object 100K gate + +A fresh 100K run stored 100K unique 35-byte bodies in ten 10K-mutation commits. +It completed in 250.117 seconds at 399.8 files/s, issued 101,133 calls including +100,334 PUTs (1.0113 calls/file), uploaded 199.81 MB, downloaded 124.73 MB, and +reached 287,328 KiB RSS. The 3.50 MB logical input therefore incurred 57.1x +upload amplification. This is an honest one-object-per-file result and does not +meet the unchanged 500 files/s release target. + +The same process listed 100K entries in 1.555 seconds at 64.3K entries/s with a +25.9 ms page p99 and 118 KB downloaded. A fresh process with an empty 64 MiB +node cache took 6.307 seconds at 15.9K entries/s, 138.1 ms p99, and 60.30 MB; +its second pass took 1.466 seconds at 68.2K entries/s, 18.0 ms p99, and 121 KB. +After populating a Foyer cache, another process retained the warm result: +1.616 seconds, 61.9K entries/s, 19.4 ms p99, and 121 KB downloaded. + +A separate fresh process ran point reads before any list traversal. One thousand +random reads completed at 340.7 reads/s with 224.5 ms p99, 4,104 GETs, and +58.32 MB downloaded for 35 KB of logical bodies: 4.104 calls/read and about +1,666x download amplification. This fails the cold-read target. After metadata +was warm, the same repository reached 1,712 reads/s with 23.5 ms p99 and 1.23 +MB downloaded. A new persistent-Foyer process reached 1,594 reads/s with 49.0 +ms p99 and 1.23 MB downloaded. + +Tree-geometry probes at 10K rejected smaller target leaf counts. The default +128-entry geometry reached 925 cold reads/s with 91.2 ms p99, 3,169 calls, and +7.33 MB downloaded. Targets 64, 32, and 16 produced p99 values of 151, 120, +and 455 ms respectively while adding traversal requests; the default remains +canonical. A two-level 100K prewarm loaded 23 shared nodes with 64 calls and +701 KB but reduced cold ranged leaf fetches only from 563 to 536, so bounded +prewarm remains an opt-in deployment tool rather than the default. + +An exact-target local readiness marker now avoids rereading the durable journal +index head after this process completes catch-up. A regression test fixes the +steady read shape at one branch-ref GET plus one complete-payload GET. On the +100K RustFS repository, a warm 1K-read pass fell from 3,000 to 2,002 calls and +from about 1.23 MB to 744 KB downloaded; it reached 1,212 reads/s with 54.3 ms +p99 on the now-heavily loaded shared provider. A fresh cold run likewise fell +from 4,104 to 3,116 calls, although host contention made its latency unsuitable +for before/after comparison. A changed branch target cannot use the marker and +falls back to durable index validation. + +Branch creation took 157 ms for the 100-key case and 270 ms for the 10K-key +case. Direct-child diff took 10.4 ms for 100 keys and 131 ms for 10K keys; +merge took 469 ms and 1.087 seconds respectively. Preparing the 10K whole-object +branch delta took 25.45 seconds at 393 files/s. A restartable index rebuild over +16 publications and 2,627 nodes completed in 462 ms and subsequent catch-up +reported zero lag. + +Deep fsck over the resulting 110K-object snapshot completed correctly but took +272.9 seconds, 119,251 calls, 142.68 MB downloaded, and 269.01 MB uploaded. +It verified 110K physical payloads totaling 3.69 MB. The 5,452 administrative +PUTs show that the cumulative fsck dedup tree needs append-only checkpoint +segments and bounded merging; payload storage remains whole-object. + +Repository-wide GC also completed correctly with zero dirty roots or reachable +deletions, but took 512.4 seconds and 1,259 pages. It issued 17,409 calls, +downloaded 247.29 MB, uploaded 30.44 MB, and deleted four unreachable objects +totaling 77.6 KB. Candidate discovery performs a near-full provider inventory +even when almost nothing is reclaimable. A durable append-only physical-object +journal should replace repeated namespace inventory while preserving the +maintenance barrier and whole-object payload model. + +### Corrected whole-object 500K gate + +The qualified 100K repository was extended by 400K unique 35-byte objects in +40 grouped commits. The extension took 1,167.97 seconds at 342.5 files/s, +issued 408,411 calls with zero retries (1.021 calls/file), uploaded 871.38 MB, +downloaded 545.77 MB, and reached 295,888 KiB RSS. Upload amplification was +62.2x over 14.0 MB of logical input. Correctness held across the sustained +19.5-minute session, but throughput and metadata amplification both fail the +release targets. + +With an insufficient 256 MiB in-process cache, the first full 500K list took +26.71 seconds at 18.7K entries/s, 133 ms page p99, and 276.4 MB downloaded. A +second pass still took 29.10 seconds and 269.9 MB because the traversal working +set churned out of cache. A new process backed by a populated 2 GiB Foyer cache +listed 500K in 8.18 seconds at 61.2K entries/s with 26.3 ms page p99 and only +600 KB downloaded. It retained 5,231 node hits with zero misses. + +A genuine fresh-process cold point-read pass reached only 162 reads/s with 529 +ms p99, 4.061 calls/read, and 152.57 MB downloaded for 35 KB of logical bodies. +After the working set was resident, point reads reached 1,775 reads/s with +21.8 ms p99. A separately reopened persistent-Foyer process reached 1,825 +reads/s with 29.2 ms p99, exactly two GETs/read, and 743 KB downloaded. + +Branch creation took 151 ms for the 100-key case and 566 ms for the 10K-key +case. Direct-child diff took 12.6 ms and 93.7 ms; merge took 766 ms and 2.33 +seconds respectively. Preparing the 10K whole-object delta took 30.54 seconds +at 327 files/s. Index rebuild over 58 publications and 12,320 nodes completed +in 1.388 seconds, after which catch-up reported zero lag. + +The 100K deep-fsck and GC algorithms already failed their amplification gates; +running the unchanged algorithms over 500K would multiply provider cost without +adding design evidence. Their next qualification follows append-only fsck work +segments and journal-driven GC candidate enumeration. + +### Corrected whole-object 1M gate + +The qualified 500K repository was extended by another 500K unique 35-byte +objects in 50 grouped commits. The extension took 2,093.90 seconds at 238.8 +files/s, issued 519,185 calls with zero retries (1.0384 calls/file), uploaded +1.190 GB, downloaded 692.83 MB, and reached 254,656 KiB RSS at the stage +boundary. Upload amplification was 68.0x over 17.5 MB of new logical input. +Mean 10K-object commit time grew to 41.88 seconds. One complete conditional +payload PUT remained mandatory per distinct body; the growing excess is +metadata publication and maintenance work, not payload packing. + +The first full 1M list took 255.17 seconds at 3.92K entries/s with a 2.725 +second page p99, 13,337 GETs, and 578.20 MB downloaded. A cold Foyer fill was +similarly expensive at 237.48 seconds, 4.21K entries/s, and 611.47 MB. A new +process reopening that persistent metadata cache listed the same snapshot in +24.11 seconds at 41.48K entries/s with a 146 ms page p99, 2,027 GETs, and 1.21 +MB downloaded. Persistence therefore removes most metadata transfer after +warmup but does not fix the first-instance traversal envelope. + +A fresh process performed one thousand random reads before listing. It reached +101.7 reads/s with 768 ms p99, 4,158 GETs, and 156.62 MB downloaded for 35 KB +of logical bodies: 4.158 calls/read and about 4,475x download amplification. +After the in-process metadata working set was warm, reads reached 1,467/s with +34.5 ms p99, exactly two GETs/read, and 727 KB downloaded. Reopening the fully +populated Foyer cache eliminated metadata misses and transferred only 757 KB, +but this heavily loaded local provider delivered 100 reads/s with 2.38 second +p99. The remaining branch/ref and whole-payload requests, plus provider behavior +at one million physical objects, need an isolated real-provider matrix. + +For a 100-key branch, creation, write, direct-child diff, and merge took 454 +ms, 2.35 seconds, 11.7 ms, and 2.01 seconds. For a 10K-key branch they took +1.39 seconds, 78.45 seconds, 206.5 ms, and 4.18 seconds. Diff and merge retain +structural pruning; whole-object branch preparation remains request-rate bound. +A restartable index rebuild over 110 publications and 24,584 nodes completed in +9.32 seconds and subsequent catch-up reported zero lag. + +The unchanged deep-fsck and GC algorithms were not rerun at 1M. Their 100K +request and byte amplification already rejected the algorithms; a larger run +would add provider cost without testing a revised implementation. + +## Results + +| Workload | Result | +|---|---:| +| 4K first-parent commits plus indexed merge-base selection | 11.56 s; exact base, zero fallback commit visits | +| 10K one-file commits on one branch | 743.9 s, 13.44 commits/s, 339,427 S3 calls, 33.94 calls/commit | +| 10K files, one grouped commit | 2.252 s, 4,441 files/s, 65 S3 calls | +| 10K warm random reads (1K samples) | 1,699 reads/s, p99 28.0 ms, 1.159 MB downloaded | +| 10K full list | 38,994 entries/s, page p99 30.8 ms, 19.2 KB downloaded | +| 10K branch / 100-key sparse diff / merge | 230 ms / 20.7 ms / 329 ms | +| 100K files, ten grouped commits | 25.65 s, 3,899 files/s, 910 S3 calls | +| 100K warm random reads (1K samples) | 1,738 reads/s, p99 25.4 ms, 1.701 MB downloaded | +| 100K full list | 13,876 entries/s, page p99 113 ms, 40.7 MB downloaded | +| 100K branch creation | 98.9 ms, 8.7 KB downloaded | +| 100K snapshot, 100-key sparse diff | 214 ms, 2.52 MB downloaded | +| 100K snapshot, 100-key merge | 1.148 s, 48.3 MB downloaded | +| 500K files, 50 grouped commits | 142.0 s, 3,521 files/s, 7,416 S3 calls | +| 500K full list, opaque snapshot cursor (clean runs) | 26.3–29.1 s, 17.2K–19.0K entries/s, page p99 97–178 ms, 343.8 MB downloaded | +| 500K full list, opaque snapshot cursor (contended runs) | 61.8–75.5 s, 6.6K–8.1K entries/s, page p99 423–631 ms | +| 500K random reads (100 samples, restart-cold process) | 441–545 reads/s, p99 89–105 ms, 15.4–15.7 MB downloaded | +| 500K full list, 512 MiB in-process warm cache | 9.21 s, 54.3K entries/s, page p99 28.4 ms, 607 KB downloaded, 391 MiB RSS | +| 500K random reads, 512 MiB in-process warm cache (1K samples) | 1,469 reads/s, p99 38.6 ms, 1.23 MB downloaded | +| 500K full list, reopened 128 MiB + 2 GiB Foyer cache | 10.59 s, 47.2K entries/s, page p99 31.4 ms, 602 KB downloaded, 168 MiB RSS after list | +| 500K random reads, reopened Foyer cache (1K samples) | 1,467 reads/s, p99 37.2 ms, 1.21 MB downloaded, 192 MiB RSS | +| 500K branch creation | 72–120 ms in clean runs | +| 500K snapshot, 100-key direct-child diff, before / after | 7.42 s, 659 MB, 7,284 calls / 9–12 ms, 22 KB, 4 calls | +| 500K snapshot, 100-key merge, before / after | 17.28 s, 1.37 GB / 474 ms, 88.9 KB | +| 500K snapshot, 10K-key external-delta diff | 83–208 ms, 12.9 KB, 22 calls | +| 500K snapshot, 10K-key external-delta merge, before / after promotion | 29.6 s, 60.4 MB, 879 calls / 8.38 s, 3.53 MB, 232 calls | +| 500K → 1M extension, 50 grouped commits | 220.3 s, 2,270 files/s, 16,075 calls, 2.68 GB uploaded, 763 MB downloaded, 480 MiB peak observed RSS | +| 1M full list, indexed-head barrier + persistent-warm Foyer | 20.92 s, 47.8K entries/s, page p99 33.2 ms, 1.19 MB downloaded, 2,027 GETs, zero PUTs | +| 1M random reads, persistent-warm Foyer (1K samples) | 936 reads/s, p99 76.7 ms, 1.21 MB downloaded | +| 1M branch / 100-key direct diff / merge | 87.7 ms / 8.24 ms / 582 ms | + +The original 500K list failure used the compatibility API that rebuilt a +key-based traversal on every page. The benchmark now uses one immutable, +opaque snapshot cursor and completes all 500 pages without retries. Clean runs +clear the 10K entries/s throughput goal; Docker/host contention can still cut +throughput below that goal and materially widen p99. The traversal downloads +343.8 MB and performs about 8.6K GETs, so request and byte amplification remain +the next listing/cache target. + +A restartable branch-index rebuild over 56 publications and 12,404 indexed +nodes completed in 1.42 seconds. The 512 MiB memory-cache pass had 5,231 node +hits and zero misses on its second full traversal. A separately reopened Foyer +process retained the same 5,231 hits and zero misses, reducing metadata download +from 340.3 MB cold to 0.60 MB persistent-warm. Point reads still perform about +three provider GETs each for mutable/ref and payload metadata even when every +Prolly node hits cache; that is now the dominant warm-read request cost. + +The 1M extension completed every foreground functional phase. Its first list +overlapped background index catch-up (150 PUTs), so the benchmark now executes +and reports an explicit indexed-head barrier before resetting foreground list +metrics. The clean rerun found zero lag in 15.3 ms, then listed 1M entries with +2,027 GETs and zero PUTs. Peak observed RSS was 213 MiB during the read/branch +phases of that reopened 128 MiB-memory Foyer process. + +Deep fsck now binds the snapshot's object/version roots into a repository-owned +durable cursor, uses range-readable commit metadata during DAG discovery, and +deduplicates checks by complete physical-object identity. Every returned page +is CAS-checkpointed with a monotonic generation; a cross-process test resumes +the work, fences a stale worker, bounds retained checkpoint versions, and +restart-cleans the payload work tree, closure work tree, and checkpoint in +one-object pages after completion. The former packed-payload numbers are not +evidence for this revised whole-object fsck path; the 100K, 500K, and 1M +performance gates must be rerun. + +GC now checkpoints its cursor under the repository-wide maintenance epoch at +start and after every returned page. A new process resumed the measured 1M run +after two forced terminations; an explicit recovery operation can abandon only +legacy/crashed epochs that never published a cursor. Node marking dequeues a +bounded wave, batch-checks marks, fetches nodes with concurrency 32, deduplicates +physical payload marks within version leaves, and publishes one work-tree mutation +per wave. Candidate scanning batch-probes reachability and publishes candidates +once per provider page. The former packed-object GC timing is not evidence for +the whole-object layout. Cross-process fencing and restart semantics remain, +but clean end-to-end 100K–1M timing and deliberate mid-sweep restart must be +rerun. + +The 500K grouped ingest uploaded 2.56 GB for 17.5 MB of logical content, about +146x byte amplification. Direct-child diff now pages the exact commit delta +rather than allowing tree boundary shifts to trigger a collected structural +fallback. Merge uses the same restartable delta frontier and promotes an +external direct-child state/delta without rebuilding identical object and +version trees. + +## Supported envelope + +- Whole-object ingest, metadata listing, point reads, branch creation, and + 100/10K-key direct-child diff/merge now have local evidence through 1M. + Ingest, first-instance list/read latency, and metadata byte amplification + fail their production gates; this is a measured ceiling, not a support claim. +- Do not claim 1M production support until cold/warm/persistent reads, grouped + whole-object ingest, interrupted fsck, clean full GC, lifecycle/fencing, and + the real-provider cost/throttling matrix pass on the revised format. +- One-file-per-commit history is reliable through 10K and repeated authority + renewal, but 13.44 commits/s and 33.94 calls/commit make it unsuitable for + bulk ingest. +- Multiple ingestion branches remove same-ref serialization when independent + histories are acceptable, but final structural-merge cost must be included. + +## Remaining release gates + +1. Repeat 100K, 500K, and 1M on AWS with cold/warm/Foyer-cache matrices and + provider cost/throttling data. +2. Reduce listing GET/byte amplification, stabilize page p99 under contention, + remove foreground index-maintenance PUTs during traversal, and avoid the + remaining two mutable/control GETs per warm page. +3. Keep delta-driven diff/merge for direct children; add an equivalent bounded + change index for arbitrary non-parent snapshot pairs and divergent merges. +4. Keep multipart and resumable-transfer state outside Prolly. A provider + transfer manager uploads the final content-addressed object, then hands its + whole-object identity back for verification and publication. Prolly must + not own upload IDs, part geometry, part ETags, chunk manifests, or chunk + garbage collection. Use native ranged GETs and provider encryption controls. +5. Fault-inject cross-process GC ticket expiry, process death, lifecycle, + replication, retention, and Object Lock behavior on real providers. diff --git a/extensions/s3/PROLLY-S3-DESIGN.md b/extensions/s3/PROLLY-S3-DESIGN.md index 426edc98..d2926cf6 100644 --- a/extensions/s3/PROLLY-S3-DESIGN.md +++ b/extensions/s3/PROLLY-S3-DESIGN.md @@ -114,10 +114,15 @@ See [the path specification](spec/prolly-s3/paths.md) for exact templates. ## Deliberate limits -There is no chunked payload format and no multipart ingestion. One file must fit -the repository and provider single-PUT limit. Garbage collection and -cross-repository transfer are not production APIs. Immutable unreachable data -therefore accumulates and must be included in storage planning. +There is no chunked or packed payload format and Prolly owns no multipart +ingestion state. The built-in upload path requires one provider `PutObject`. +Larger or resumable files are uploaded as one final object by an external +provider transfer manager, then handed to Prolly for whole-object verification +and publication. The transfer manager alone owns incomplete-upload cleanup. +Repair and cross-repository history import also request complete-object +transfers through the provider boundary. Repository code never reconstructs a +body from parts; provider adapters may use native copy or an external transfer +manager. The design has no fixed file, commit, or ref count in its logical structures, but real deployments are bounded by provider quotas, request cost, latency, diff --git a/extensions/s3/QUALIFICATION.md b/extensions/s3/QUALIFICATION.md index 4ed40c1e..43e73186 100644 --- a/extensions/s3/QUALIFICATION.md +++ b/extensions/s3/QUALIFICATION.md @@ -58,6 +58,14 @@ Run ignored scale tests explicitly and record: - cold, prewarmed, and persistent-cache reads; - sparse diff and merge at 10K+ keys; - restart during staging, merge, index rebuild, and publication response loss. +- external provider transfer-manager completion followed by whole-object + verification, while retaining exactly one physical S3 object per logical + large object; +- crash/restart between external upload and Prolly publication, with no upload + ID or part lifecycle persisted by Prolly; +- cross-process publication fencing throughout a complete GC epoch; +- one complete provider object per distinct logical payload, with no payload + packs, chunk manifests, or Prolly-owned multipart state. Results must include p50/p95/p99 latency, S3 calls by operation, bytes transferred, CAS retries, cache hit ratio, index lag, wall time, provider @@ -125,6 +133,29 @@ Do not promote on RustFS results alone. AWS qualification must cover: - cache-loss cold start; - lifecycle and replication interactions. +Optional dedicated policy buckets extend the AWS matrix without changing +bucket configuration during the test: + +- `PROLLY_S3_AWS_BUCKET_LIFECYCLE` must contain a lifecycle rule and must be + rejected before any probe object is written. +- `PROLLY_S3_AWS_BUCKET_OBJECT_LOCK_DEFAULT` must have default Object Lock + retention and must be rejected before probe writes. +- `PROLLY_S3_AWS_BUCKET_REPLICATED` must have active replication and must be + accepted with `replication_enabled=true`; destination lifecycle, ownership, + KMS, and replication lag remain operator assertions. + +The signed provider capability profile records whether replication +configuration was readable and whether replication was enabled. Replication is +reported rather than rejected because it does not mutate source objects, but +its destination lifecycle, replica ownership, KMS grants, latency, and request +cost must be included in the deployment runbook. + +GC treats an exact-delete `PermissionDenied` as a provider-protected physical +version, records its count, bytes, and object kind, and continues the bounded +sweep. This lets Object Lock retention or legal hold preserve the version +without leaving repository publication admission closed indefinitely. Any +other deletion error remains terminal and the durable epoch stays resumable. + ## Promotion criteria Promote only when: @@ -136,9 +167,66 @@ Promote only when: - no test relies on bucket listing for normal node lookup; - operation response loss reconciles before fencing; - authority renewal survives the intended deployment duration; -- GC grace periods, external-writer quiescence, and retention policies are - tested in the deployment runbook. +- GC grace periods, durable cross-process admission fencing, crash/resume, and + retention policies are tested in the deployment runbook. External writer + quiescence is not a correctness requirement for clients using the repository + protocol; bypass writers remain unsupported. Million- or billion-object claims require measurements at representative cardinality and traffic. Passing a 10K test is a regression gate, not proof of unbounded production capacity. + +The staged benchmark defaults to 10K, 20K, 50K, 100K, 500K, and 1M objects and +records bounded whole-object ingest, point reads, full and prefix listing, +branch creation, sparse diff, and merge for every stage. Comma-separated branch +change counts exercise both sparse and larger deltas without changing payload +storage: + +```bash +PROLLY_RUSTFS_PERF_STAGES=100000,500000,1000000 \ +PROLLY_RUSTFS_PERF_WRITE_CONCURRENCY=512 \ +PROLLY_RUSTFS_PERF_BRANCH_CHANGES=100,10000 \ +PROLLY_RUSTFS_PERF_LIST_PREFIXES=repo/files/000,repo/files/009 \ + cargo run --release --manifest-path extensions/s3/Cargo.toml \ + -p prolly-s3-client --example rustfs_small_files_benchmark +``` + +Keep the printed repository prefix, then run a fresh process against the same +prefix to measure reopen-cold behavior. Set `EXISTING_FILES` and a single stage +to the repository cardinality so ingestion is skipped: + +```bash +PROLLY_RUSTFS_PERF_OPEN_MODE=open \ +PROLLY_RUSTFS_PERF_PREFIX=benchmarks/small-files/ \ +PROLLY_RUSTFS_PERF_EXISTING_FILES=1000000 \ +PROLLY_RUSTFS_PERF_STAGES=1000000 \ +PROLLY_RUSTFS_PERF_READ_PASSES=2 \ +PROLLY_RUSTFS_PERF_LIST_PASSES=2 \ + cargo run --release --manifest-path extensions/s3/Cargo.toml \ + -p prolly-s3-client --example rustfs_small_files_benchmark +``` + +Set `PROLLY_RUSTFS_PERF_LIST_PASSES=0` and +`PROLLY_RUSTFS_PERF_BRANCH=false` in a fresh process to measure genuinely cold +point reads before a full traversal warms metadata. Read or list pass counts may +be zero so each cache phase can be isolated. + +Use the same reopen form with `PROLLY_RUSTFS_PERF_FOYER_DIR` to qualify a +persistent cache, `PROLLY_RUSTFS_PERF_REBUILD_INDEX=true` for index rebuild, +and `PROLLY_RUSTFS_PERF_FSCK=deep` or `PROLLY_RUSTFS_PERF_GC=true` for the +restartable maintenance gates. Every phase prints provider calls and bytes, +wire attempts, cache behavior, and resident memory. Set the provider-specific +`PROLLY_RUSTFS_PERF_{READ,WRITE,LIST,DELETE}_USD_PER_1000` rates to include an +estimated request cost in every metrics row; zero is the default so the harness +never assumes an AWS or S3-compatible provider's pricing. + +`PROLLY_RUSTFS_PERF_TREE_TARGET_ENTRIES` creates a new repository with a +bounded experimental metadata-tree geometry (maximum eight times the target +and a 1 MiB hard node limit). Omit it to use the canonical default. Reopen runs +must pass the same value because tree geometry is part of repository identity; +compare cold reads, warm full listing, write amplification, and RSS before +promoting a new default. + +Set `PROLLY_RUSTFS_PERF_PREWARM_LEVELS` on a fresh reopen process to measure +the bounded startup cost and the resulting point-read tail latency separately. +This preloads only shared metadata-tree levels and never reads payload bodies. diff --git a/extensions/s3/client/Cargo.toml b/extensions/s3/client/Cargo.toml index fa59bf75..efb040d7 100644 --- a/extensions/s3/client/Cargo.toml +++ b/extensions/s3/client/Cargo.toml @@ -28,7 +28,7 @@ prolly-s3-core = { path = "../core", version = "0.1.0" } serde = { version = "1.0", features = ["derive"] } sha2 = "0.10" tempfile = "3.23" -tokio = { version = "1.45", features = ["io-util", "macros", "net", "rt-multi-thread", "sync", "time"] } +tokio = { version = "1.45", features = ["fs", "io-util", "macros", "net", "rt-multi-thread", "sync", "time"] } [dev-dependencies] aws-config = { version = "1.8", features = ["behavior-version-latest"] } diff --git a/extensions/s3/client/README.md b/extensions/s3/client/README.md index 9b89a0f3..8e157ae6 100644 --- a/extensions/s3/client/README.md +++ b/extensions/s3/client/README.md @@ -120,11 +120,10 @@ assert_eq!(current.bytes, b"second revision\n"); assert_eq!(historical.bytes, b"first revision\n"); ``` -A standalone write uploads one whole content-addressed payload. Bulk ingestion -packs non-empty files up to 4 KiB into immutable segments capped at 4 MiB; -logical bindings retain checksums and byte extents, while empty and larger -files keep the direct representation. Identical bytes in a segment share one -extent, and replaying the same segment reuses its content-addressed object. +A standalone write and bulk ingestion both upload one whole content-addressed +payload object for every distinct non-empty file body. Prolly does not combine +small files, split large files, or persist byte extents or multipart state. +Identical complete bodies reuse the same content-addressed object. For safe retries, generate and persist one operation ID: @@ -378,12 +377,22 @@ payload metadata. Deep mode also downloads and hashes payload bytes: ```rust let mut fsck = client.start_fsck(true).await?; +let fsck_job = fsck.job; while fsck.phase != prolly_s3_client::core::FsckPhase::Complete { fsck = client.advance_fsck(&fsck, 1_000).await?.cursor; } println!("verified {} commits", fsck.report.commits); +let mut cleanup = client.start_fsck_cleanup(fsck_job).await?; +while cleanup.phase != prolly_s3_client::core::FsckCleanupPhase::Complete { + cleanup = client.advance_fsck_cleanup(&cleanup, 1_000).await?.cursor; +} ``` +Every page is checkpointed in the repository. A new process resumes with +`client.resume_fsck(fsck_job)`; checkpoint generations reject stale workers. +Cleanup is permitted only after completion, is bounded, and can restart from +`start_fsck_cleanup` after process loss. + Cross-provider repair is logical. It does not copy provider version IDs. It downloads verified source payloads, rebinds them at the destination, preserves logical metadata, and removes destination-only keys: @@ -488,8 +497,8 @@ while gc.phase != GcPhase::Complete { } ``` -The collector sweeps only immutable commit, direct-node, direct-payload, and -payload-pack objects. +The collector sweeps only immutable commit, direct-node, and whole-payload +objects. It never sweeps mutable refs, derived indexes, publication journals, format markers, or administration data. Branch and tag updates during an epoch write dirty-root records before their CAS; sweep batches fence publication and catch @@ -534,16 +543,20 @@ that fits the configured disk block; larger nodes are deliberately rejected from cache admission and continue to use the provider fallback. Use `prewarm_node_cache(snapshot)` during startup to traverse both state trees. +Use `prewarm_node_cache_levels(snapshot, levels)` when startup should load only +the roots and shared upper paths instead of scanning every leaf. Use `node_cache_snapshot()` before and after to observe hits, misses, insertions, corruptions, coalesced waits, ranged fetches, fetched and avoided bytes, and admission rejections. ## Performance model -- A single-file write uploads one payload and publishes immutable metadata. -- A tiny-file batch uploads one payload pack per segment; direct payloads use - bounded parallel uploads. Tree and publication requests are amortized across - the batch. +- Every distinct logical payload is one complete immutable provider object. + Exact duplicate bodies may reuse that complete object by content hash. +- Tiny-file batches upload those whole objects with bounded concurrency. Tree, + append-only checkpoint, and publication requests are amortized across the + batch; Prolly never combines file bodies or records payload extents. Each + checkpoint window contains only keys changed since its predecessor. - A current or historical read resolves a ref/commit/tree path and one payload. - Warm immutable-node caches remove most repeated metadata reads. - Branch creation inherits immutable derived-index roots from its source and is @@ -551,8 +564,19 @@ bytes, and admission rejections. descriptors and only the node ranges on their traversal frontier. - Large commit deltas are external Prolly trees, keeping commit descriptors and restart metadata bounded independently of batch size. +- Callers that need provider-native multipart or resumable transfer use + `prepare_external_object_upload`, upload the single final object with their + provider transfer manager, then call `stage_external_object_upload`. Prolly + persists no upload ID, part geometry, part ETags, chunks, or chunk manifest. +- Metadata node packs contain only Prolly index nodes. They never contain user + object bodies and are not a payload storage or transfer format. - Same-branch writers contend on one ref CAS; different branches publish independently. +- Concurrent independent callers can use `ordered_publication_queue` to apply + bounded backpressure, prepare complete objects concurrently, and coalesce + unique keys into one deterministic commit. Duplicate-key submissions cross a + commit boundary to preserve version order. Acknowledgements are constant-size + and are delivered only after the grouped ref CAS succeeds. Measure with `s3_operation_metrics` on the provider and workload you will run. The repository enforces configured request-shape limits, but no universal @@ -561,13 +585,23 @@ latency or cost claim substitutes for AWS qualification. ## Limitations - The client must be the exclusive authority for its repository prefix. -- One file must fit the repository and provider single-object PUT limits. -- There is no chunked or multipart file representation. +- One file must fit the repository and provider single-object size limits. +- Prolly does not define or manage a chunked file representation. Its built-in + upload path uses one `PutObject` up to the provider's 5 GiB single-PUT limit. +- Larger or resumable transfers must be completed by an external provider + transfer manager and handed back as one whole object for verification and + publication. +- The convenience `put_stream` path uses a bounded disk spool because it learns + content identity at end-of-stream. The external-upload handoff avoids Prolly + transfer state but requires the complete size and whole-object checksums. - Concurrent writes to one branch can conflict; batch related changes. -- Concurrent GC coordinates all writer handles in the authoritative process. - Do not run GC while a separately running process can publish to the same - repository prefix; use one authoritative writer process or quiesce external - writers first. +- Concurrent GC closes publication admission through the durable repository + coordinator, fences writer handles in other processes, checkpoints its epoch, + and resumes after process restart. Writers bypassing the repository protocol + remain unsupported. +- Provider-retained or legal-held versions that reject exact deletion are + counted in `GcReport::protected_versions`/`protected_bytes`; they remain + physically present while GC completes and publication admission reopens. - Snapshot clone/fetch/push preserves only the selected logical state. The `history_` variants preserve the source commit DAG, but not source commit IDs or reflog identity. diff --git a/extensions/s3/client/examples/rustfs_small_files_benchmark.rs b/extensions/s3/client/examples/rustfs_small_files_benchmark.rs index 5ab7dfb2..b2a97896 100644 --- a/extensions/s3/client/examples/rustfs_small_files_benchmark.rs +++ b/extensions/s3/client/examples/rustfs_small_files_benchmark.rs @@ -16,9 +16,18 @@ use aws_sdk_s3::{ }; use futures_util::{stream, StreamExt}; use prolly_s3_client::{ - core::ProviderPerKeyVersionLimit, Client, HmacAttestationSigner, ProviderIdentity, + core::{ + BoundaryRule, GcPhase, MergePhase, MergePolicy, NodeCacheSnapshot, ObjectHeaders, + ProviderPerKeyVersionLimit, TreeFormat, + }, + BulkWriteOptions, CheckoutRef, Client, HmacAttestationSigner, ProviderIdentity, PutObjectInput, S3OperationMetrics, S3WireAttemptInterceptor, S3WireAttemptMetrics, }; +#[cfg(feature = "foyer-cache")] +use prolly_s3_client::{FoyerNodeCache, FoyerNodeCacheConfig}; +use std::collections::BTreeMap; +#[cfg(feature = "foyer-cache")] +use std::path::PathBuf; type BenchResult = Result>; @@ -57,6 +66,35 @@ fn parse_stages() -> BenchResult> { Ok(stages) } +fn parse_positive_sizes(name: &str, default: &str) -> BenchResult> { + let mut values = env(name, default) + .split(',') + .map(str::trim) + .filter(|part| !part.is_empty()) + .map(str::parse::) + .collect::, _>>()?; + if values.is_empty() || values.contains(&0) { + return Err(format!("{name} must contain positive integers").into()); + } + values.sort_unstable(); + values.dedup(); + Ok(values) +} + +fn parse_prefixes(name: &str) -> Vec { + std::env::var(name) + .ok() + .into_iter() + .flat_map(|raw| { + raw.split(',') + .map(str::trim) + .filter(|prefix| !prefix.is_empty()) + .map(ToOwned::to_owned) + .collect::>() + }) + .collect() +} + fn percentile(sorted: &[Duration], percentile: usize) -> Duration { sorted[(sorted.len() * percentile).div_ceil(100) - 1] } @@ -90,8 +128,19 @@ fn key(index: usize) -> String { } fn print_provider_metrics(label: &str, metrics: S3OperationMetrics) { + let read_requests = metrics.get_object + metrics.head_object; + let write_requests = metrics.put_object; + let list_requests = metrics.list_objects_v2 + metrics.list_object_versions; + let delete_requests = metrics.delete_object + metrics.delete_objects; + let estimated_cost = request_price("PROLLY_RUSTFS_PERF_READ_USD_PER_1000") + * read_requests as f64 + / 1_000.0 + + request_price("PROLLY_RUSTFS_PERF_WRITE_USD_PER_1000") * write_requests as f64 / 1_000.0 + + request_price("PROLLY_RUSTFS_PERF_LIST_USD_PER_1000") * list_requests as f64 / 1_000.0 + + request_price("PROLLY_RUSTFS_PERF_DELETE_USD_PER_1000") * delete_requests as f64 + / 1_000.0; println!( - "METRICS phase={label} s3_calls={} get={} head={} put={} list={} list_versions={} delete={} delete_batch={} uploaded_bytes={} downloaded_bytes={}", + "METRICS phase={label} s3_calls={} get={} head={} put={} list={} list_versions={} delete={} delete_batch={} uploaded_bytes={} downloaded_bytes={} estimated_request_cost_usd={estimated_cost:.8}", metrics.total_calls(), metrics.get_object, metrics.head_object, @@ -105,6 +154,13 @@ fn print_provider_metrics(label: &str, metrics: S3OperationMetrics) { ); } +fn request_price(name: &str) -> f64 { + std::env::var(name) + .ok() + .and_then(|value| value.parse().ok()) + .unwrap_or(0.0) +} + fn print_wire_metrics(label: &str, metrics: S3WireAttemptMetrics) { println!( "WIRE phase={label} executions={} transmissions={} retries={} success={} client_errors={} server_errors={} no_response={}", @@ -118,6 +174,35 @@ fn print_wire_metrics(label: &str, metrics: S3WireAttemptMetrics) { ); } +fn print_cache_metrics(label: &str, before: NodeCacheSnapshot, after: NodeCacheSnapshot) { + println!( + "CACHE phase={label} hits={} misses={} insertions={} coalesced_waits={} ranged_fetches={} fetched_bytes={} avoided_bytes={} admission_rejections={} rss_kib={}", + after.hits.saturating_sub(before.hits), + after.misses.saturating_sub(before.misses), + after.insertions.saturating_sub(before.insertions), + after.coalesced_waits.saturating_sub(before.coalesced_waits), + after.ranged_fetches.saturating_sub(before.ranged_fetches), + after.fetched_bytes.saturating_sub(before.fetched_bytes), + after.avoided_bytes.saturating_sub(before.avoided_bytes), + after + .admission_rejections + .saturating_sub(before.admission_rejections), + resident_kib().map_or_else(|| "unknown".to_string(), |rss| rss.to_string()), + ); +} + +fn resident_kib() -> Option { + let output = std::process::Command::new("ps") + .args(["-o", "rss=", "-p", &std::process::id().to_string()]) + .output() + .ok()?; + std::str::from_utf8(&output.stdout) + .ok()? + .trim() + .parse() + .ok() +} + async fn ensure_versioned_bucket(aws: &aws_sdk_s3::Client, bucket: &str) -> BenchResult { if let Err(error) = aws.create_bucket().bucket(bucket).send().await { let text = format!("{error:?}"); @@ -145,9 +230,54 @@ async fn main() -> BenchResult { let bucket = env("PROLLY_RUSTFS_BUCKET", "prolly"); let read_sample_size = env("PROLLY_RUSTFS_PERF_READ_SAMPLES", "1000").parse::()?; let read_concurrency = env("PROLLY_RUSTFS_PERF_READ_CONCURRENCY", "32").parse::()?; + let write_concurrency = env("PROLLY_RUSTFS_PERF_WRITE_CONCURRENCY", "32").parse::()?; + let existing_files = env("PROLLY_RUSTFS_PERF_EXISTING_FILES", "0").parse::()?; + let branch_changes = parse_positive_sizes("PROLLY_RUSTFS_PERF_BRANCH_CHANGES", "100")?; + let list_prefixes = parse_prefixes("PROLLY_RUSTFS_PERF_LIST_PREFIXES"); + let open_mode = env("PROLLY_RUSTFS_PERF_OPEN_MODE", "initialize"); + if !matches!(open_mode.as_str(), "initialize" | "open") { + return Err("open mode must be initialize or open".into()); + } + let list_passes = env("PROLLY_RUSTFS_PERF_LIST_PASSES", "1").parse::()?; + let read_passes = env("PROLLY_RUSTFS_PERF_READ_PASSES", "1").parse::()?; + let run_branches = env("PROLLY_RUSTFS_PERF_BRANCH", "true").parse::()?; + let node_cache_mib = env("PROLLY_RUSTFS_PERF_NODE_CACHE_MIB", "64").parse::()?; + let prewarm_levels = env("PROLLY_RUSTFS_PERF_PREWARM_LEVELS", "0").parse::()?; + let tree_target_entries = std::env::var("PROLLY_RUSTFS_PERF_TREE_TARGET_ENTRIES") + .ok() + .map(|value| value.parse::()) + .transpose()?; + let rebuild_index = env("PROLLY_RUSTFS_PERF_REBUILD_INDEX", "false").parse::()?; + let run_gc = env("PROLLY_RUSTFS_PERF_GC", "false").parse::()?; + let abandon_incomplete_gc = + env("PROLLY_RUSTFS_PERF_GC_ABANDON_INCOMPLETE", "false").parse::()?; + let gc_grace_millis = env("PROLLY_RUSTFS_PERF_GC_GRACE_MILLIS", "1").parse::()?; + let run_foreground = env("PROLLY_RUSTFS_PERF_FOREGROUND", "true").parse::()?; + let fsck_mode = env("PROLLY_RUSTFS_PERF_FSCK", "none"); + if !matches!(fsck_mode.as_str(), "none" | "shallow" | "deep") { + return Err("fsck mode must be none, shallow, or deep".into()); + } + #[cfg(feature = "foyer-cache")] + let foyer_directory = std::env::var_os("PROLLY_RUSTFS_PERF_FOYER_DIR").map(PathBuf::from); + #[cfg(not(feature = "foyer-cache"))] + if std::env::var_os("PROLLY_RUSTFS_PERF_FOYER_DIR").is_some() { + return Err("Foyer cache requested but the foyer-cache feature is disabled".into()); + } let stages = parse_stages()?; - if read_sample_size == 0 || read_concurrency == 0 { - return Err("read sample size and concurrency must be positive".into()); + if read_sample_size == 0 + || read_concurrency == 0 + || write_concurrency == 0 + || branch_changes + .iter() + .any(|changes| *changes > MUTATIONS_PER_COMMIT) + || node_cache_mib == 0 + || prewarm_levels > 64 + || tree_target_entries.is_some_and(|target| !(4..=65_536).contains(&target)) + { + return Err("read sample size and read/write concurrency must be positive".into()); + } + if existing_files > *stages.first().expect("non-empty stages") { + return Err("existing file count cannot exceed the first benchmark stage".into()); } let wire = S3WireAttemptInterceptor::new(); @@ -173,11 +303,31 @@ async fn main() -> BenchResult { "PROLLY_RUSTFS_PERF_PREFIX", &format!("benchmarks/small-files/{run_id}"), ); - let client = Client::builder() + let writer = env( + "PROLLY_RUSTFS_PERF_WRITER", + &format!("small-files-benchmark-{run_id}"), + ); + let branch_suffix = env("PROLLY_RUSTFS_PERF_BRANCH_SUFFIX", &run_id.to_string()); + #[cfg(feature = "foyer-cache")] + let foyer_cache = match foyer_directory.as_ref() { + Some(directory) => Some( + FoyerNodeCache::open(FoyerNodeCacheConfig { + directory: directory.clone(), + memory_capacity_bytes: node_cache_mib * 1024 * 1024, + disk_capacity_bytes: 2 * 1024 * 1024 * 1024, + disk_block_size_bytes: 1024 * 1024, + memory_shards: 64, + }) + .await?, + ), + None => None, + }; + #[allow(unused_mut)] + let mut builder = Client::builder() .aws_client(aws) .bucket(&bucket) .repository_prefix(&prefix) - .writer(format!("small-files-benchmark-{run_id}")) + .writer(&writer) // A sequential 10K-file session outlives the short interactive default. // Keep one authority permit valid for the full qualification run. .authority_lease_duration(Duration::from_secs(12 * 60 * 60)) @@ -187,150 +337,499 @@ async fn main() -> BenchResult { vec![0x62; 32], )?)) .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) - .initialize() - .await?; + .max_cached_node_bytes(node_cache_mib * 1024 * 1024); + if let Some(target) = tree_target_entries { + let mut tree_format = TreeFormat::default(); + tree_format.chunking.rule = BoundaryRule::HashThreshold { factor: target }; + tree_format.chunking.target = u64::from(target); + tree_format.chunking.max = u64::from(target) * 8; + tree_format.chunking.hard_max_node_bytes = 1024 * 1024; + builder = builder.state_tree_format(tree_format); + } + #[cfg(feature = "foyer-cache")] + if let Some(cache) = &foyer_cache { + builder = builder.node_cache(cache.clone()); + } + let client = match open_mode.as_str() { + "initialize" => builder.initialize().await?, + "open" => builder.open().await?, + _ => unreachable!("open mode validated above"), + }; println!( - "CONFIG endpoint={endpoint} bucket={bucket} prefix={prefix} stages={stages:?} object_bytes={} mutations_per_commit={MUTATIONS_PER_COMMIT} read_samples={read_sample_size} read_concurrency={read_concurrency}", + "CONFIG endpoint={endpoint} bucket={bucket} prefix={prefix} writer={writer} open_mode={open_mode} stages={stages:?} existing_files={existing_files} branch_changes={branch_changes:?} list_prefixes={list_prefixes:?} list_passes={list_passes} read_passes={read_passes} branch={run_branches} node_cache_mib={node_cache_mib} prewarm_levels={prewarm_levels} tree_target_entries={} foyer={} rebuild_index={rebuild_index} foreground={run_foreground} fsck={fsck_mode} gc={run_gc} gc_grace_millis={gc_grace_millis} object_bytes={} mutations_per_commit={MUTATIONS_PER_COMMIT} write_concurrency={write_concurrency} read_samples={read_sample_size} read_concurrency={read_concurrency}", + tree_target_entries.map_or_else(|| "default".to_string(), |target| target.to_string()), + if cfg!(feature = "foyer-cache") + && std::env::var_os("PROLLY_RUSTFS_PERF_FOYER_DIR").is_some() + { + "enabled" + } else { + "disabled" + }, payload(0).len(), ); - let mut prior_target = 0_usize; + if rebuild_index { + let started = Instant::now(); + let mut cursor = client.start_branch_index_rebuild().await?; + let mut indexed_publications = 0usize; + let mut indexed_nodes = 0usize; + loop { + let step = client.advance_branch_index_rebuild(&cursor, 1_000).await?; + indexed_publications += step.indexed_publications; + indexed_nodes += step.indexed_nodes; + cursor = step.cursor; + if step.complete { + break; + } + } + println!( + "INDEX_REBUILD wall_ms={:.3} indexed_publications={indexed_publications} indexed_nodes={indexed_nodes}", + millis(started.elapsed()), + ); + } + + let mut prior_target = existing_files; for target in stages { let added = target - prior_target; + if added == 0 { + println!("STAGE target={target} added=0 write=skipped_existing"); + } else { + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let stage_started = Instant::now(); + let receipts = client + .put_object_stream( + stream::iter((prior_target..target).map(|index| { + Ok(PutObjectInput { + key: key(index), + bytes: payload(index), + headers: ObjectHeaders::default(), + user_metadata: BTreeMap::new(), + }) + })), + BulkWriteOptions { + batch_size: MUTATIONS_PER_COMMIT, + concurrency: write_concurrency, + checkpoint_every: 1_000, + }, + ) + .await?; + + let write_wall = stage_started.elapsed(); + let write_metrics = client.reset_s3_operation_metrics(); + let write_wire = wire.reset(); + let logical_bytes = added as u64 * payload(0).len() as u64; + println!( + "STAGE target={target} added={added} logical_bytes={logical_bytes} write_wall_ms={:.3} files_per_second={:.2} commit_count={} mean_commit_ms={:.3} calls_per_file={:.4} uploaded_byte_amplification={:.2}", + millis(write_wall), + added as f64 / write_wall.as_secs_f64(), + receipts.len(), + millis(write_wall) / receipts.len() as f64, + write_metrics.total_calls() as f64 / added as f64, + write_metrics.uploaded_body_bytes as f64 / logical_bytes as f64, + ); + print_provider_metrics("write", write_metrics); + print_wire_metrics("write", write_wire); + print_cache_metrics("write", cache_before, client.node_cache_snapshot()); + } + + let cache_before = client.node_cache_snapshot(); client.reset_s3_operation_metrics(); wire.reset(); - let stage_started = Instant::now(); - let mut write_latencies = Vec::with_capacity(added); - let mut publish_latencies = Vec::with_capacity(added.div_ceil(MUTATIONS_PER_COMMIT)); - - for batch_start in (prior_target..target).step_by(MUTATIONS_PER_COMMIT) { - let batch_end = (batch_start + MUTATIONS_PER_COMMIT).min(target); - let mut session = client - .begin_commit() - .message(format!("small-file benchmark through {batch_end}")) - .checkpoint_every(1_000) - .start() + let index_started = Instant::now(); + let index = client.advance_branch_indexes().await?; + println!( + "INDEX_CATCHUP target={target} wall_ms={:.3} journal_publications={} journal_commits={} journal_nodes={} operation_events={}", + millis(index_started.elapsed()), + index.journal.indexed_publications, + index.journal.indexed_commits, + index.journal.indexed_nodes, + index.operations.indexed_events, + ); + print_provider_metrics("index_catchup", client.reset_s3_operation_metrics()); + print_wire_metrics("index_catchup", wire.reset()); + print_cache_metrics("index_catchup", cache_before, client.node_cache_snapshot()); + if prewarm_levels > 0 { + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let started = Instant::now(); + let snapshot = client.head().await?; + let report = client + .prewarm_node_cache_levels(snapshot, prewarm_levels) .await?; - for index in batch_start..batch_end { + println!( + "PREWARM target={target} levels={prewarm_levels} wall_ms={:.3} object_nodes={} version_nodes={}", + millis(started.elapsed()), + report.object_nodes, + report.version_nodes, + ); + print_provider_metrics("prewarm", client.reset_s3_operation_metrics()); + print_wire_metrics("prewarm", wire.reset()); + print_cache_metrics("prewarm", cache_before, client.node_cache_snapshot()); + } + if !run_foreground { + prior_target = target; + continue; + } + + for pass in 1..=list_passes { + let label = format!("list_pass_{pass}"); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let list_started = Instant::now(); + let mut continuation = None; + let mut listed = 0_usize; + let mut list_page_latencies = Vec::new(); + loop { + let page_started = Instant::now(); + let page = client + .list_objects_page("repo/files/", continuation.as_deref(), LIST_PAGE_SIZE) + .await?; + list_page_latencies.push(page_started.elapsed()); + listed += page.objects.len(); + continuation = page.continuation; + if continuation.is_none() { + break; + } + } + let list_wall = list_started.elapsed(); + if listed != target { + return Err(format!("expected {target} listed files, found {listed}").into()); + } + let list_summary = summarize(list_page_latencies); + println!( + "LIST target={target} pass={pass} wall_ms={:.3} files_per_second={:.2} pages={} page_mean_ms={:.3} page_p50_ms={:.3} page_p95_ms={:.3} page_p99_ms={:.3} page_max_ms={:.3}", + millis(list_wall), + target as f64 / list_wall.as_secs_f64(), + list_summary.count, + list_summary.mean_ms, + list_summary.p50_ms, + list_summary.p95_ms, + list_summary.p99_ms, + list_summary.max_ms, + ); + print_provider_metrics(&label, client.reset_s3_operation_metrics()); + print_wire_metrics(&label, wire.reset()); + print_cache_metrics(&label, cache_before, client.node_cache_snapshot()); + + for prefix in &list_prefixes { + let prefix_label = format!("prefix_list_pass_{pass}_{}", prefix.replace('/', "_")); + let expected = (0..target) + .filter(|index| key(*index).starts_with(prefix)) + .count(); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); let started = Instant::now(); - session.put_object(key(index), payload(index)).await?; - write_latencies.push(started.elapsed()); + let mut continuation = None; + let mut listed = 0usize; + let mut latencies = Vec::new(); + loop { + let page_started = Instant::now(); + let page = client + .list_objects_page(prefix, continuation.as_deref(), LIST_PAGE_SIZE) + .await?; + latencies.push(page_started.elapsed()); + listed += page.objects.len(); + continuation = page.continuation; + if continuation.is_none() { + break; + } + } + if listed != expected { + return Err(format!( + "expected {expected} files under prefix {prefix:?}, found {listed}" + ) + .into()); + } + let wall = started.elapsed(); + let summary = summarize(latencies); + println!( + "PREFIX_LIST target={target} pass={pass} prefix={prefix:?} objects={listed} wall_ms={:.3} files_per_second={:.2} pages={} page_p50_ms={:.3} page_p95_ms={:.3} page_p99_ms={:.3} page_max_ms={:.3}", + millis(wall), + listed as f64 / wall.as_secs_f64(), + summary.count, + summary.p50_ms, + summary.p95_ms, + summary.p99_ms, + summary.max_ms, + ); + print_provider_metrics(&prefix_label, client.reset_s3_operation_metrics()); + print_wire_metrics(&prefix_label, wire.reset()); + print_cache_metrics(&prefix_label, cache_before, client.node_cache_snapshot()); } - let publish_started = Instant::now(); - session.publish().await?; - publish_latencies.push(publish_started.elapsed()); + } + + let samples = read_sample_size.min(target); + for pass in 1..=read_passes { + let label = format!("read_pass_{pass}"); + let indexes = (0..samples) + .map(|sample| sample * target / samples) + .collect::>(); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let read_started = Instant::now(); + let results = stream::iter(indexes) + .map(|index| { + let client = client.clone(); + async move { + let started = Instant::now(); + let object = client.get_object(key(index)).await?; + if object.as_ref().map(|value| value.bytes.as_slice()) + != Some(payload(index).as_slice()) + { + return Err( + format!("read verification failed for index {index}").into() + ); + } + Ok::>(started.elapsed()) + } + }) + .buffer_unordered(read_concurrency) + .collect::>() + .await; + let read_wall = read_started.elapsed(); + let read_latencies = results.into_iter().collect::>>()?; + let read_summary = summarize(read_latencies); println!( - "PROGRESS target={target} committed={batch_end} added={}", - batch_end - prior_target + "READ target={target} pass={pass} samples={samples} concurrency={read_concurrency} wall_ms={:.3} reads_per_second={:.2} mean_ms={:.3} p50_ms={:.3} p95_ms={:.3} p99_ms={:.3} max_ms={:.3}", + millis(read_wall), + samples as f64 / read_wall.as_secs_f64(), + read_summary.mean_ms, + read_summary.p50_ms, + read_summary.p95_ms, + read_summary.p99_ms, + read_summary.max_ms, ); + print_provider_metrics(&label, client.reset_s3_operation_metrics()); + print_wire_metrics(&label, wire.reset()); + print_cache_metrics(&label, cache_before, client.node_cache_snapshot()); } - let write_wall = stage_started.elapsed(); - let write_summary = summarize(write_latencies); - let publish_summary = summarize(publish_latencies); - let write_metrics = client.reset_s3_operation_metrics(); - let write_wire = wire.reset(); - println!( - "STAGE target={target} added={added} write_wall_ms={:.3} files_per_second={:.2} write_count={} write_mean_ms={:.3} write_p50_ms={:.3} write_p95_ms={:.3} write_p99_ms={:.3} write_max_ms={:.3} publish_count={} publish_mean_ms={:.3} publish_p50_ms={:.3} publish_p95_ms={:.3} publish_p99_ms={:.3} publish_max_ms={:.3}", - millis(write_wall), - added as f64 / write_wall.as_secs_f64(), - write_summary.count, - write_summary.mean_ms, - write_summary.p50_ms, - write_summary.p95_ms, - write_summary.p99_ms, - write_summary.max_ms, - publish_summary.count, - publish_summary.mean_ms, - publish_summary.p50_ms, - publish_summary.p95_ms, - publish_summary.p99_ms, - publish_summary.max_ms, + if run_branches { + for configured_changes in &branch_changes { + let base = client.head().await?; + let branch = format!("scale-{target}-{configured_changes}-{branch_suffix}"); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let branch_started = Instant::now(); + client.create_branch(&branch, Some(base)).await?; + let branch_elapsed = branch_started.elapsed(); + println!( + "BRANCH target={target} changes={configured_changes} wall_ms={:.3}", + millis(branch_elapsed) + ); + print_provider_metrics("branch", client.reset_s3_operation_metrics()); + print_wire_metrics("branch", wire.reset()); + print_cache_metrics("branch", cache_before, client.node_cache_snapshot()); + + let feature = client.checkout(CheckoutRef::Branch(branch.clone())).await?; + let sparse_changes = target.min(*configured_changes); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let feature_write_started = Instant::now(); + feature + .put_object_stream( + stream::iter((0..sparse_changes).map(|index| { + Ok(PutObjectInput { + key: format!("repo/branch-probes/{target}/{index:03}.txt"), + bytes: format!("branch-{target}-{index}\n").into_bytes(), + headers: ObjectHeaders::default(), + user_metadata: BTreeMap::new(), + }) + })), + BulkWriteOptions { + batch_size: sparse_changes, + concurrency: write_concurrency, + checkpoint_every: sparse_changes, + }, + ) + .await?; + let feature_head = feature.head().await?; + println!( + "BRANCH_WRITE target={target} changes={sparse_changes} wall_ms={:.3} changes_per_second={:.2}", + millis(feature_write_started.elapsed()), + sparse_changes as f64 / feature_write_started.elapsed().as_secs_f64(), ); - print_provider_metrics("write", write_metrics); - print_wire_metrics("write", write_wire); + print_provider_metrics("branch_write", client.reset_s3_operation_metrics()); + print_wire_metrics("branch_write", wire.reset()); + print_cache_metrics("branch_write", cache_before, client.node_cache_snapshot()); + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let diff_started = Instant::now(); + let mut diff_cursor = None; + let mut changed = 0usize; + loop { + let page = client + .diff_bounded(base, feature_head, diff_cursor.as_ref(), 1_000) + .await?; + changed += page.changes.len(); + diff_cursor = page.continuation; + if diff_cursor.is_none() { + break; + } + } + let diff_elapsed = diff_started.elapsed(); + println!( + "DIFF target={target} changes={changed} wall_ms={:.3} changes_per_second={:.2}", + millis(diff_elapsed), + changed as f64 / diff_elapsed.as_secs_f64(), + ); + print_provider_metrics("diff", client.reset_s3_operation_metrics()); + print_wire_metrics("diff", wire.reset()); + print_cache_metrics("diff", cache_before, client.node_cache_snapshot()); + + let cache_before = client.node_cache_snapshot(); + client.reset_s3_operation_metrics(); + wire.reset(); + let merge_started = Instant::now(); + let mut merge = client + .start_merge( + &branch, + Some(base), + MergePolicy::Theirs, + format!("merge scale branch at {target}"), + ) + .await?; + let mut merge_processed = 0usize; + while merge.phase != MergePhase::ReadyToPublish { + let page = client.advance_merge(&merge, 1_000).await?; + merge_processed += page.processed; + merge = page.cursor; + } + let merged = client.publish_merge(&merge).await?; + let merge_elapsed = merge_started.elapsed(); + println!( + "MERGE target={target} changes={} processed={} wall_ms={:.3}", + merged.changed_keys, + merge_processed, + millis(merge_elapsed), + ); + print_provider_metrics("merge", client.reset_s3_operation_metrics()); + print_wire_metrics("merge", wire.reset()); + print_cache_metrics("merge", cache_before, client.node_cache_snapshot()); + client.delete_branch(&branch, feature_head).await?; + } + } + println!("STAGE_COMPLETE target={target}"); + prior_target = target; + } + if fsck_mode != "none" { + let cache_before = client.node_cache_snapshot(); client.reset_s3_operation_metrics(); wire.reset(); - let list_started = Instant::now(); - let mut after = None; - let mut listed = 0_usize; - let mut list_page_latencies = Vec::new(); + let started = Instant::now(); + let mut cursor = client.start_fsck(fsck_mode == "deep").await?; + let mut pages = 0usize; loop { - let page_started = Instant::now(); - let (_, page, truncated) = client - .list_objects("repo/files/", after.as_deref(), LIST_PAGE_SIZE) - .await?; - list_page_latencies.push(page_started.elapsed()); - listed += page.len(); - after = page - .last() - .map(|item| String::from_utf8_lossy(&item.key).into_owned()); - if !truncated { + let page = client.advance_fsck(&cursor, 10_000).await?; + pages += 1; + cursor = page.cursor; + if pages % 10 == 0 { + println!( + "FSCK_PROGRESS mode={fsck_mode} pages={pages} phase={:?} commits={} current_objects={} logical_versions={}", + cursor.phase, + cursor.report.commits, + cursor.report.current_objects, + cursor.report.logical_versions, + ); + } + if page.complete { break; } } - let list_wall = list_started.elapsed(); - if listed != target { - return Err(format!("expected {target} listed files, found {listed}").into()); - } - let list_summary = summarize(list_page_latencies); println!( - "LIST target={target} wall_ms={:.3} files_per_second={:.2} pages={} page_mean_ms={:.3} page_p50_ms={:.3} page_p95_ms={:.3} page_p99_ms={:.3} page_max_ms={:.3}", - millis(list_wall), - target as f64 / list_wall.as_secs_f64(), - list_summary.count, - list_summary.mean_ms, - list_summary.p50_ms, - list_summary.p95_ms, - list_summary.p99_ms, - list_summary.max_ms, + "FSCK mode={fsck_mode} wall_ms={:.3} pages={pages} commits={} reachable_nodes={} current_objects={} logical_versions={} payloads_verified={} payload_bytes_verified={} deep_content_bytes_verified={} physical_payloads_verified={} physical_payload_bytes_verified={} deep_physical_bytes_read={}", + millis(started.elapsed()), + cursor.report.commits, + cursor.report.reachable_nodes, + cursor.report.current_objects, + cursor.report.logical_versions, + cursor.report.payloads_verified, + cursor.report.payload_bytes_verified, + cursor.report.deep_content_bytes_verified, + cursor.report.physical_payloads_verified, + cursor.report.physical_payload_bytes_verified, + cursor.report.deep_physical_bytes_read, ); - print_provider_metrics("list", client.reset_s3_operation_metrics()); - print_wire_metrics("list", wire.reset()); - - let samples = read_sample_size.min(target); - let indexes = (0..samples) - .map(|sample| sample * target / samples) - .collect::>(); + print_provider_metrics("fsck", client.reset_s3_operation_metrics()); + print_wire_metrics("fsck", wire.reset()); + print_cache_metrics("fsck", cache_before, client.node_cache_snapshot()); + } + if run_gc { + let cache_before = client.node_cache_snapshot(); client.reset_s3_operation_metrics(); wire.reset(); - let read_started = Instant::now(); - let results = stream::iter(indexes) - .map(|index| { - let client = client.clone(); - async move { - let started = Instant::now(); - let object = client.get_object(key(index)).await?; - if object.as_ref().map(|value| value.bytes.as_slice()) - != Some(payload(index).as_slice()) - { - return Err(format!("read verification failed for index {index}").into()); - } - Ok::>(started.elapsed()) - } - }) - .buffer_unordered(read_concurrency) - .collect::>() - .await; - let read_wall = read_started.elapsed(); - let read_latencies = results.into_iter().collect::>>()?; - let read_summary = summarize(read_latencies); + let started = Instant::now(); + if abandon_incomplete_gc { + match client.abandon_incomplete_gc().await { + Ok(epoch) => println!("GC_ABANDONED_INCOMPLETE epoch={epoch}"), + Err(error) + if error.code == prolly_s3_client::core::ErrorCode::PreconditionFailed => {} + Err(error) => return Err(error.into()), + } + } + let mut cursor = match client.resume_gc().await? { + Some(cursor) => cursor, + None => client.start_gc(gc_grace_millis).await?, + }; + let mut pages = 0usize; + loop { + let page = match cursor.phase { + GcPhase::Ready | GcPhase::Sweeping => client.sweep_gc(&cursor, 1_000).await?, + GcPhase::Complete => break, + GcPhase::MarkNodes => client.advance_gc(&cursor, 100).await?, + GcPhase::ScanCandidates => client.advance_gc(&cursor, 100).await?, + _ => client.advance_gc(&cursor, 100).await?, + }; + pages += 1; + cursor = page.cursor; + if pages % 100 == 0 { + println!( + "GC_PROGRESS pages={pages} phase={:?} commits={} nodes={} logical_versions={} candidates={} deleted_versions={}", + cursor.phase, + cursor.report.commits, + cursor.report.nodes, + cursor.report.logical_versions, + cursor.report.candidates, + cursor.report.deleted_versions, + ); + } + } println!( - "READ target={target} samples={samples} concurrency={read_concurrency} wall_ms={:.3} reads_per_second={:.2} mean_ms={:.3} p50_ms={:.3} p95_ms={:.3} p99_ms={:.3} max_ms={:.3}", - millis(read_wall), - samples as f64 / read_wall.as_secs_f64(), - read_summary.mean_ms, - read_summary.p50_ms, - read_summary.p95_ms, - read_summary.p99_ms, - read_summary.max_ms, + "GC wall_ms={:.3} pages={pages} roots={} commits={} nodes={} logical_versions={} candidates={} candidate_bytes={} dirty_roots={} deleted_versions={} deleted_bytes={} protected_versions={} protected_bytes={} already_missing={} skipped_reachable={}", + millis(started.elapsed()), + cursor.report.roots, + cursor.report.commits, + cursor.report.nodes, + cursor.report.logical_versions, + cursor.report.candidates, + cursor.report.candidate_bytes, + cursor.report.dirty_roots, + cursor.report.deleted_versions, + cursor.report.deleted_bytes, + cursor.report.protected_versions, + cursor.report.protected_bytes, + cursor.report.already_missing, + cursor.report.skipped_reachable, ); - print_provider_metrics("read", client.reset_s3_operation_metrics()); - print_wire_metrics("read", wire.reset()); - println!("STAGE_COMPLETE target={target}"); - prior_target = target; + print_provider_metrics("gc", client.reset_s3_operation_metrics()); + print_wire_metrics("gc", wire.reset()); + print_cache_metrics("gc", cache_before, client.node_cache_snapshot()); + } + drop(client); + #[cfg(feature = "foyer-cache")] + if let Some(cache) = foyer_cache { + cache.close().await?; } Ok(()) } diff --git a/extensions/s3/client/src/aws_object.rs b/extensions/s3/client/src/aws_object.rs index ffda3366..b305666b 100644 --- a/extensions/s3/client/src/aws_object.rs +++ b/extensions/s3/client/src/aws_object.rs @@ -21,6 +21,7 @@ use prolly_s3_core::{ StoredMetadata, StoredObject, }; use sha2::{Digest, Sha256}; +const MAX_SINGLE_PUT_BYTES: u64 = 5 * 1_024 * 1_024 * 1_024; /// Object-plane calls issued to the AWS SDK and body bytes handed to or /// collected from it. SDK-internal HTTP retries are intentionally not counted; @@ -312,6 +313,12 @@ impl ObjectPlane for AwsS3ObjectPlane { "immutable spool size changed before upload", )); } + if request.size > MAX_SINGLE_PUT_BYTES { + return Err(Error::new( + ErrorCode::EntityTooLarge, + "object exceeds S3 PutObject limits; upload the single final object with an external provider transfer manager and stage its handoff", + )); + } self.metrics.put_object.fetch_add(1, Ordering::Relaxed); self.metrics .uploaded_body_bytes diff --git a/extensions/s3/client/src/client.rs b/extensions/s3/client/src/client.rs index 433f7c0f..d4bb4d8f 100644 --- a/extensions/s3/client/src/client.rs +++ b/extensions/s3/client/src/client.rs @@ -1,5 +1,5 @@ use std::{ - collections::{BTreeMap, VecDeque}, + collections::{BTreeMap, BTreeSet, VecDeque}, io::Write as _, str::FromStr as _, sync::{Arc, Mutex}, @@ -12,21 +12,26 @@ use md5::Md5; use prolly_s3_core::{ BackupVerificationCursor, BackupVerificationPage, BatchId, BranchCatalogPage, BranchHead, BranchIndexAdvanceReport, BranchIndexHealth, CommitId, CommitPage, CommitReceipt, - CommitSessionManifest, DelimitedObjectPage, Error, ErrorCode, FsckCursor, FsckPage, GcCursor, - GcPage, HistoryCursor, HistoryTransferCursor, HistoryTransferMapping, HistoryTransferPage, - JournalIndexRebuildCleanup, JournalIndexRebuildCursor, JournalIndexRebuildStep, - ListObjectsPage, MergeAdvancePage, MergeBaseCursor, MergeBasePage, MergeChangeCursor, - MergeChangePage, MergeCleanupCursor, MergeCleanupPage, MergeConflictCursor, MergeConflictPage, - MergeCursor, MergePolicy, MergeReceipt, NodeCachePrewarmReport, ObjectData, ObjectDiff, - ObjectDiffCursor, ObjectDiffPage, ObjectHeaders, ObjectRangeData, ObjectSummary, ObjectVersion, - OperationId, OperationIndexRebuildCursor, OperationIndexRebuildStep, ProviderAttestation, + CommitSessionManifest, DelimitedObjectPage, Error, ErrorCode, FsckCleanupCursor, + FsckCleanupPage, FsckCursor, FsckPage, GcCursor, GcPage, HistoryCursor, HistoryTransferCursor, + HistoryTransferMapping, HistoryTransferPage, JournalIndexRebuildCleanup, + JournalIndexRebuildCursor, JournalIndexRebuildStep, ListObjectsPage, MergeAdvancePage, + MergeBaseCursor, MergeBasePage, MergeChangeCursor, MergeChangePage, MergeCleanupCursor, + MergeCleanupPage, MergeConflictCursor, MergeConflictPage, MergeCursor, MergePolicy, + MergeReceipt, NodeCachePrewarmReport, ObjectData, ObjectDiff, ObjectDiffCursor, ObjectDiffPage, + ObjectHeaders, ObjectPath, ObjectRangeData, ObjectSummary, ObjectVersion, OperationId, + OperationIndexRebuildCursor, OperationIndexRebuildStep, ProviderAttestation, ProviderPerKeyVersionLimit, ProviderProfileId, PublicationJournalCursor, PublicationJournalPage, RefCatalogCursor, RefCatalogRepairPage, RefKind, RefMoveReceipt, RepairCursor, RepairPage, Repository, RepositoryOptions, RestoreCursor, RestorePage, Result, RetentionPin, RetentionPinPage, StagedMutation, Tag, TagCatalogPage, TraversalBudget, - VersionSummary, + TreeFormat, VersionSummary, }; use sha2::{Digest as _, Sha256}; +use tokio::{ + sync::{mpsc, oneshot}, + time::Instant, +}; use crate::{ ensure_attestation_current, load_valid_attestation, qualify_and_store, @@ -51,6 +56,21 @@ pub struct Client { _branch_index_maintenance: Arc>>, } +/// Durable handoff describing one complete provider object. +/// +/// Upload this exact object with any S3 transfer manager, including the +/// `prolly-sha256` metadata value. Prolly never observes upload IDs, parts, or +/// part checksums; it only verifies the completed whole object before publish. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct ExternalObjectUpload { + pub batch: BatchId, + pub key: Vec, + pub path: ObjectPath, + pub size: u64, + pub checksum_sha256: [u8; 32], + pub checksum_md5: [u8; 16], +} + #[derive(Default)] pub struct ClientBuilder { aws_client: Option, @@ -58,6 +78,7 @@ pub struct ClientBuilder { repository_prefix: Option, default_branch: Option, writer: Option, + state_tree_format: Option, authority_lease_duration: Option, read_only: bool, max_cached_node_pack_bytes: Option, @@ -106,6 +127,114 @@ impl Default for BulkWriteOptions { } } +/// Controls an [`OrderedPublicationQueue`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct OrderedPublicationOptions { + /// Maximum unique logical keys published by one atomic commit. + pub max_group_size: usize, + /// Maximum complete-object uploads prepared concurrently for one group. + pub upload_concurrency: usize, + /// Maximum time the worker waits for more submissions after receiving the + /// first item in a group. + pub max_wait: Duration, + /// Bounded channel capacity. Producers await capacity and therefore apply + /// backpressure rather than growing process memory without limit. + pub queue_capacity: usize, + /// Maximum successfully staged mutations appended to one durable recovery + /// checkpoint window. This is independent of the final publication group. + pub checkpoint_every: usize, + /// Persist one recovery checkpoint before each grouped publication. + pub durable: bool, +} + +impl Default for OrderedPublicationOptions { + fn default() -> Self { + Self { + max_group_size: 1_000, + upload_concurrency: 128, + max_wait: Duration::from_millis(2), + queue_capacity: 10_000, + checkpoint_every: 1_000, + durable: true, + } + } +} + +/// Bounded ordered submission queue that coalesces independent callers into +/// atomic branch publications. +/// +/// Every user body remains one complete immutable provider object. Only +/// candidate metadata and the final branch CAS are grouped. Repeated writes to +/// the same key are split across consecutive commits so logical version order +/// is preserved. +#[derive(Clone)] +pub struct OrderedPublicationQueue { + sender: mpsc::Sender, +} + +/// Constant-size acknowledgement for one object accepted by an ordered group +/// commit. The full batch receipt remains an internal publication result so +/// acknowledging N callers does not clone an N-entry version vector N times. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct OrderedPublicationReceipt { + pub commit: CommitId, + pub operation: OperationId, + pub group_size: u64, +} + +struct QueuedPut { + object: PutObjectInput, + response: oneshot::Sender>, +} + +impl OrderedPublicationQueue { + pub async fn put_object( + &self, + key: impl Into, + bytes: Vec, + ) -> Result { + self.put_object_with_metadata(key, bytes, ObjectHeaders::default(), BTreeMap::new()) + .await + } + + pub async fn put_object_with_metadata( + &self, + key: impl Into, + bytes: Vec, + headers: ObjectHeaders, + user_metadata: BTreeMap, + ) -> Result { + let (response, receipt) = oneshot::channel(); + self.sender + .send(QueuedPut { + object: PutObjectInput { + key: key.into(), + bytes, + headers, + user_metadata, + }, + response, + }) + .await + .map_err(|_| { + Error::new( + ErrorCode::OperationCanceled, + "ordered publication queue is closed", + ) + })?; + receipt.await.map_err(|_| { + Error::new( + ErrorCode::OperationCanceled, + "ordered publication worker stopped before acknowledging the object", + ) + })? + } + + pub fn remaining_capacity(&self) -> usize { + self.sender.capacity() + } +} + /// A revision accepted by [`Client::checkout`]. /// /// Unqualified names resolve branches before tags. Use `Branch` or `Tag` (or @@ -202,6 +331,17 @@ impl Client { .await } + pub async fn prewarm_node_cache_levels( + &self, + snapshot: CommitId, + levels: usize, + ) -> Result { + self.ensure_provider_qualified()?; + self.repository + .prewarm_node_cache_levels(&self.branch, snapshot, levels) + .await + } + /// Select a branch, tag, or immutable commit, following Git-like ref /// precedence. Branch checkouts are writable; tag and commit checkouts are /// detached and reject mutation APIs with `InvalidRevision`. @@ -463,12 +603,54 @@ impl Client { self.repository.advance_fsck(cursor, max_steps).await } + pub async fn resume_fsck(&self, job: OperationId) -> Result> { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository.resume_fsck(job).await + } + + pub async fn start_fsck_cleanup(&self, job: OperationId) -> Result { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository.start_fsck_cleanup(job).await + } + + pub async fn advance_fsck_cleanup( + &self, + cursor: &FsckCleanupCursor, + max_objects: usize, + ) -> Result { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository + .advance_fsck_cleanup(cursor, max_objects) + .await + } + pub async fn start_gc(&self, grace_millis: u64) -> Result { self.ensure_provider_qualified()?; self.attached_branch()?; self.repository.start_gc(grace_millis).await } + pub async fn resume_gc(&self) -> Result> { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository.resume_gc().await + } + + pub async fn abandon_gc(&self, expected_epoch: prolly_s3_core::OperationId) -> Result<()> { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository.abandon_gc(expected_epoch).await + } + + pub async fn abandon_incomplete_gc(&self) -> Result { + self.ensure_provider_qualified()?; + self.attached_branch()?; + self.repository.abandon_incomplete_gc().await + } + pub async fn advance_gc(&self, cursor: &GcCursor, max_steps: usize) -> Result { self.ensure_provider_qualified()?; self.attached_branch()?; @@ -886,6 +1068,51 @@ impl Client { .await } + /// Start a bounded ordered queue that coalesces independent submissions + /// into grouped branch commits. The worker lives until every queue handle + /// is dropped and drains submissions already accepted by the channel. + pub fn ordered_publication_queue( + &self, + options: OrderedPublicationOptions, + ) -> Result { + self.ensure_provider_qualified()?; + self.attached_branch()?; + let max = self + .repository + .format() + .canonical_limits + .max_mutations_per_commit as usize; + if options.max_group_size == 0 || options.max_group_size > max { + return Err(invalid( + "ordered publication group size exceeds the canonical mutation limit", + )); + } + if options.upload_concurrency == 0 || options.upload_concurrency > 1_024 { + return Err(invalid( + "ordered publication upload concurrency must be 1..=1024", + )); + } + if options.queue_capacity == 0 || options.queue_capacity > 1_000_000 { + return Err(invalid( + "ordered publication queue capacity must be 1..=1000000", + )); + } + if options.checkpoint_every == 0 || options.checkpoint_every > options.max_group_size { + return Err(invalid( + "ordered publication checkpoint interval must be within the group size", + )); + } + let runtime = tokio::runtime::Handle::try_current() + .map_err(|_| invalid("ordered publication queue requires an active Tokio runtime"))?; + let (sender, receiver) = mpsc::channel(options.queue_capacity); + runtime.spawn(run_ordered_publication_queue( + self.clone(), + receiver, + options, + )); + Ok(OrderedPublicationQueue { sender }) + } + /// Put objects through durable atomic batches. This is the recommended /// path for bulk loading because publication and checkpoint costs are /// amortized across each batch. @@ -1142,6 +1369,7 @@ impl Client { checkpoint_every: 256, checkpoint_sequence: checkpoint.sequence, dirty_mutations: 0, + dirty_keys: BTreeSet::new(), }) } @@ -1521,6 +1749,10 @@ impl Client { self.repository.plane().reset_metrics() } + pub fn provider_capabilities(&self) -> &prolly_s3_core::ProviderCapabilities { + &self.provider_attestation.body.capabilities + } + pub fn fenced_branches(&self) -> Result> { self.repository.fenced_branches() } @@ -1545,6 +1777,179 @@ impl Client { } } +async fn run_ordered_publication_queue( + client: Client, + mut receiver: mpsc::Receiver, + options: OrderedPublicationOptions, +) { + let mut pending = None; + loop { + let first = match pending.take() { + Some(first) => first, + None => match receiver.recv().await { + Some(first) => first, + None => return, + }, + }; + if first.response.is_closed() { + continue; + } + + let mut keys = BTreeSet::from([first.object.key.clone()]); + let mut group = vec![first]; + let deadline = Instant::now() + options.max_wait; + while group.len() < options.max_group_size { + let received = if options.max_wait.is_zero() { + receiver.try_recv().ok() + } else { + tokio::time::timeout_at(deadline, receiver.recv()) + .await + .ok() + .flatten() + }; + let Some(next) = received else { + break; + }; + if next.response.is_closed() { + continue; + } + if !keys.insert(next.object.key.clone()) { + pending = Some(next); + break; + } + group.push(next); + } + publish_ordered_group(&client, group, options).await; + } +} + +async fn publish_ordered_group( + client: &Client, + group: Vec, + options: OrderedPublicationOptions, +) { + let mut group = group + .into_iter() + .filter(|queued| !queued.response.is_closed()) + .collect::>(); + if group.is_empty() { + return; + } + group.sort_by(|left, right| left.object.key.cmp(&right.object.key)); + let group_len = group.len(); + let mut builder = client + .begin_commit() + .message("ordered grouped publication") + .checkpoint_every(options.checkpoint_every.min(group_len)); + if !options.durable { + builder = builder.ephemeral(); + } + let mut session = match builder.start().await { + Ok(session) => session, + Err(error) => { + acknowledge_group_error(group, error); + return; + } + }; + let mut pending = VecDeque::from(group); + let mut responses = Vec::with_capacity(group_len); + while !pending.is_empty() { + let mut window_responses = Vec::with_capacity(options.checkpoint_every); + let mut objects = Vec::with_capacity(options.checkpoint_every); + for _ in 0..options.checkpoint_every { + let Some(queued) = pending.pop_front() else { + break; + }; + if queued.response.is_closed() { + continue; + } + let PutObjectInput { + key, + bytes, + headers, + user_metadata, + } = queued.object; + window_responses.push(queued.response); + objects.push((key.into_bytes(), bytes, headers, user_metadata)); + } + let staged = match client + .repository + .stage_commit_session_put_batch_results( + &session.manifest, + objects, + options.upload_concurrency, + ) + .await + { + Ok(staged) => staged, + Err(error) => { + let _ = session.abort().await; + for response in responses + .into_iter() + .chain(window_responses) + .chain(pending.into_iter().map(|queued| queued.response)) + { + let _ = response.send(Err(error.clone())); + } + return; + } + }; + for (response, result) in window_responses.into_iter().zip(staged) { + if response.is_closed() { + continue; + } + match result { + Ok(mutation) => { + session.insert_staged(mutation); + responses.push(response); + } + Err(mut error) => { + error.operation_id = Some(session.id().to_string()); + let _ = response.send(Err(error)); + } + } + } + if options.durable { + if let Err(error) = session.checkpoint().await { + for response in responses + .into_iter() + .chain(pending.into_iter().map(|queued| queued.response)) + { + let _ = response.send(Err(error.clone())); + } + return; + } + } + } + if responses.is_empty() { + let _ = session.abort().await; + return; + } + match session.publish().await { + Ok(receipt) => { + let acknowledgement = OrderedPublicationReceipt { + commit: receipt.id, + operation: receipt.operation, + group_size: receipt.changed_keys, + }; + for response in responses { + let _ = response.send(Ok(acknowledgement)); + } + } + Err(error) => { + for response in responses { + let _ = response.send(Err(error.clone())); + } + } + } +} + +fn acknowledge_group_error(group: Vec, error: Error) { + for queued in group { + let _ = queued.response.send(Err(error.clone())); + } +} + pub struct CommitSessionBuilder { client: Client, message: String, @@ -1564,8 +1969,9 @@ impl CommitSessionBuilder { self } - /// Disable remote checkpoints for the minimum N + 3 S3 PUT shape. The - /// session cannot then be resumed after process loss. + /// Disable remote checkpoints for the minimum N + 4 S3 PUT shape: N + /// payloads, commit/event/ref publication, and one short-lived durable GC + /// admission ticket. The session cannot then be resumed after process loss. pub fn ephemeral(mut self) -> Self { self.durable = false; self @@ -1610,6 +2016,7 @@ impl CommitSessionBuilder { checkpoint_every: self.checkpoint_every, checkpoint_sequence, dirty_mutations: 0, + dirty_keys: BTreeSet::new(), }) } } @@ -1622,6 +2029,7 @@ pub struct CommitSession { checkpoint_every: usize, checkpoint_sequence: u64, dirty_mutations: usize, + dirty_keys: BTreeSet>, } impl CommitSession { @@ -1646,7 +2054,9 @@ impl CommitSession { } fn insert_staged(&mut self, staged: StagedMutation) { - self.staged.insert(staged.key().to_vec(), staged); + let key = staged.key().to_vec(); + self.staged.insert(key.clone(), staged); + self.dirty_keys.insert(key); self.dirty_mutations = self.dirty_mutations.saturating_add(1); } @@ -1758,13 +2168,76 @@ impl CommitSession { Ok(()) } + /// Prepare a durable whole-object handoff for an external S3 uploader. + /// The uploader owns multipart, retry, and resume behavior entirely. + pub async fn prepare_external_object_upload( + &self, + key: impl Into, + size: u64, + checksum_sha256: [u8; 32], + checksum_md5: [u8; 16], + ) -> Result { + self.client.ensure_provider_qualified()?; + let key = key.into().into_bytes(); + let path = self + .client + .repository + .commit_session_payload_path(&self.manifest, &key, size, checksum_sha256) + .await?; + Ok(ExternalObjectUpload { + batch: self.id(), + key, + path, + size, + checksum_sha256, + checksum_md5, + }) + } + + /// Verify and stage a completed whole object uploaded to the handoff path. + pub async fn stage_external_object_upload( + &mut self, + upload: &ExternalObjectUpload, + headers: ObjectHeaders, + metadata: BTreeMap, + ) -> Result<()> { + upload.validate_for(self)?; + let expected_path = self + .client + .repository + .commit_session_payload_path( + &self.manifest, + &upload.key, + upload.size, + upload.checksum_sha256, + ) + .await?; + if upload.path != expected_path { + return Err(invalid("external object handoff path was modified")); + } + let staged = self + .client + .repository + .stage_commit_session_existing_object( + &self.manifest, + upload.key.clone(), + upload.size, + upload.checksum_sha256, + upload.checksum_md5, + headers, + metadata, + ) + .await?; + self.insert_staged(staged); + self.mark_staged_and_checkpoint_if_due().await + } + pub fn delete_object(&mut self, key: impl Into) -> Result<()> { let key = key.into().into_bytes(); if key.is_empty() { return Err(invalid("commit-session delete key is empty")); } - self.staged.insert(key.clone(), StagedMutation::delete(key)); - self.dirty_mutations = self.dirty_mutations.saturating_add(1); + self.insert_staged(StagedMutation::delete(key)); Ok(()) } @@ -1776,18 +2249,25 @@ impl CommitSession { .checkpoint_sequence .checked_add(1) .ok_or_else(|| invalid("commit-session checkpoint sequence overflow"))?; + let delta = self + .dirty_keys + .iter() + .map(|key| { + self.staged + .get(key) + .cloned() + .expect("dirty commit-session key is staged") + }) + .collect(); let checkpoint = self .client .repository - .checkpoint_commit_session( - &self.manifest, - self.staged.values().cloned().collect(), - sequence, - ) + .checkpoint_commit_session(&self.manifest, delta, self.staged.len(), sequence) .await?; self.manifest = checkpoint.session; self.checkpoint_sequence = checkpoint.sequence; self.dirty_mutations = 0; + self.dirty_keys.clear(); Ok(()) } @@ -1802,17 +2282,18 @@ impl CommitSession { /// Mark a durable session aborted. Immutable payload candidates remain /// deduplicated and bounded staging cleanup removes expired checkpoints. - pub async fn abort(self) -> Result<()> { + pub async fn abort(mut self) -> Result<()> { if !self.durable { return Ok(()); } + self.checkpoint().await?; let sequence = self .checkpoint_sequence .checked_add(1) .ok_or_else(|| invalid("commit-session checkpoint sequence overflow"))?; self.client .repository - .abort_commit_session(self.manifest, self.staged.into_values().collect(), sequence) + .abort_commit_session(self.manifest, Vec::new(), self.staged.len(), sequence) .await } @@ -1824,6 +2305,21 @@ impl CommitSession { } } +impl ExternalObjectUpload { + fn validate_for(&self, session: &CommitSession) -> Result<()> { + if self.batch != session.id() + || self.key.is_empty() + || self.size == 0 + || self.checksum_sha256 == [0; 32] + { + return Err(invalid( + "external object handoff is malformed or belongs to another session", + )); + } + Ok(()) + } +} + impl ClientBuilder { pub fn aws_client(mut self, client: aws_sdk_s3::Client) -> Self { self.aws_client = Some(client); @@ -1882,6 +2378,14 @@ impl ClientBuilder { self } + /// Select the persisted tree geometry when initializing a repository. + /// Opening an existing repository must supply the identical format. Shape + /// experiments therefore require a new repository prefix. + pub fn state_tree_format(mut self, format: TreeFormat) -> Self { + self.state_tree_format = Some(format); + self + } + pub fn read_only(mut self, read_only: bool) -> Self { self.read_only = read_only; self @@ -2027,6 +2531,9 @@ impl ClientBuilder { options.authority_lease_millis = u64::try_from(duration.as_millis()) .map_err(|_| invalid("authority lease duration exceeds u64 milliseconds"))?; } + if let Some(format) = self.state_tree_format { + options.state_tree_format = format; + } if let Some(bytes) = self.max_cached_node_pack_bytes { options.max_cached_node_pack_bytes = bytes; } @@ -2054,9 +2561,9 @@ impl ClientBuilder { options.node_cache = self.node_cache; let branch = options.default_branch.clone(); let repository = if initialize { - Repository::initialize(plane, options).await? + Repository::initialize(plane.clone(), options).await? } else { - Repository::open(plane, options).await? + Repository::open(plane.clone(), options).await? }; let repository = Arc::new(repository); let shard_authority_maintenance = if self.read_only { diff --git a/extensions/s3/client/src/provider.rs b/extensions/s3/client/src/provider.rs index 5fe9ed3f..f56bf540 100644 --- a/extensions/s3/client/src/provider.rs +++ b/extensions/s3/client/src/provider.rs @@ -212,7 +212,8 @@ pub(crate) async fn qualify_and_store( .map_err(|_| invalid("provider attestation validity exceeds millisecond range"))?, ) .ok_or_else(|| invalid("provider attestation expiry overflow"))?; - let capabilities = probe_provider(plane.clone(), repository_prefix).await?; + let capabilities = + probe_provider(plane.clone(), repository_prefix, identity.bucket_class()).await?; capabilities.validate_prolly_s3()?; let body = ProviderAttestationBody { endpoint_fingerprint: identity.endpoint_fingerprint()?, @@ -346,7 +347,28 @@ pub(crate) fn ensure_attestation_current(attestation: &ProviderAttestation) -> R async fn probe_provider( plane: Arc, repository_prefix: &str, + bucket_class: BucketClass, ) -> Result { + // Inspect policies before creating probe objects. A default retention rule + // could otherwise lock qualification debris, and a lifecycle rule could + // mutate probes while their consistency semantics are being measured. + let conflicting_lifecycle_rule = lifecycle_conflicts(plane.client(), plane.bucket()).await?; + if conflicting_lifecycle_rule { + return Err(not_qualified( + "bucket lifecycle rules can mutate repository objects", + )); + } + let default_object_lock_retention = + object_lock_conflicts(plane.client(), plane.bucket()).await?; + if default_object_lock_retention { + return Err(not_qualified( + "bucket default Object Lock retention would retain qualification and repository objects", + )); + } + let (replication_configuration_readable, replication_enabled) = + replication_status(plane.client(), plane.bucket(), bucket_class).await?; + let physical_versioning = bucket_versioning(plane.client(), plane.bucket()).await?; + let probe = format!( "{repository_prefix}/probes/{}/", prolly_s3_core::OperationId::new() @@ -473,7 +495,6 @@ async fn probe_provider( )); } - let physical_versioning = bucket_versioning(plane.client(), plane.bucket()).await?; if matches!( physical_versioning, PhysicalVersioning::Enabled | PhysicalVersioning::Suspended @@ -516,9 +537,6 @@ async fn probe_provider( )); } - let conflicting_lifecycle_rule = lifecycle_conflicts(plane.client(), plane.bucket()).await?; - let default_object_lock_retention = - object_lock_conflicts(plane.client(), plane.bucket()).await?; Ok(ProviderCapabilities { conditional_create: true, conditional_update: true, @@ -532,11 +550,51 @@ async fn probe_provider( physical_versioning, conflicting_lifecycle_rule, default_object_lock_retention, + replication_configuration_readable, + replication_enabled, max_object_bytes: 5 * 1_024 * 1_024 * 1_024 * 1_024, max_single_put_bytes: 5 * 1_024 * 1_024 * 1_024, }) } +async fn replication_status( + client: &aws_sdk_s3::Client, + bucket: &str, + bucket_class: BucketClass, +) -> Result<(bool, bool)> { + match client.get_bucket_replication().bucket(bucket).send().await { + Ok(output) => Ok(( + true, + output + .replication_configuration() + .is_some_and(|configuration| !configuration.rules().is_empty()), + )), + Err(error) + if matches!( + error.code(), + Some( + "ReplicationConfigurationNotFoundError" + | "NoSuchReplicationConfiguration" + | "NoSuchConfiguration" + | "NotFound" + ) + ) => + { + Ok((true, false)) + } + Err(error) + if bucket_class == BucketClass::S3Compatible + && matches!( + error.code(), + Some("NotImplemented" | "UnsupportedOperation" | "MethodNotAllowed") + ) => + { + Ok((false, false)) + } + Err(error) => Err(provider_error("GetBucketReplication", error.code(), &error)), + } +} + async fn delete_metadata_exact( plane: &AwsS3ObjectPlane, path: &ObjectPath, @@ -692,6 +750,8 @@ mod tests { physical_versioning: PhysicalVersioning::Unversioned, conflicting_lifecycle_rule: false, default_object_lock_retention: false, + replication_configuration_readable: true, + replication_enabled: false, max_object_bytes: 1, max_single_put_bytes: 1, } @@ -750,6 +810,15 @@ mod tests { lifecycle.validate_required().unwrap_err().code, ErrorCode::ProviderNotQualified ); + let mut object_lock = capabilities(); + object_lock.default_object_lock_retention = true; + assert_eq!( + object_lock.validate_required().unwrap_err().code, + ErrorCode::ProviderNotQualified + ); + let mut replicated = capabilities(); + replicated.replication_enabled = true; + replicated.validate_required().unwrap(); } #[test] diff --git a/extensions/s3/client/tests/aws_qualification.rs b/extensions/s3/client/tests/aws_qualification.rs index 7f449a13..35a5b7f2 100644 --- a/extensions/s3/client/tests/aws_qualification.rs +++ b/extensions/s3/client/tests/aws_qualification.rs @@ -32,6 +32,29 @@ fn signer() -> Arc { Arc::new(HmacAttestationSigner::single("aws-qualification", vec![0x51; 32]).unwrap()) } +async fn assert_policy_bucket_rejected( + aws: &aws_sdk_s3::Client, + bucket: &str, + region: &str, + profile: &str, +) { + let result = Client::builder() + .aws_client(aws.clone()) + .bucket(bucket) + .repository_prefix(unique_prefix(profile)) + .writer(format!("aws-{profile}-qualification")) + .provider_identity(ProviderIdentity::aws_region(region.to_string())) + .attestation_signer(signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + .initialize() + .await; + let error = match result { + Ok(_) => panic!("policy-conflicting bucket {bucket} was accepted"), + Err(error) => error, + }; + assert_eq!(error.code, ErrorCode::ProviderNotQualified); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn aws_general_purpose_bucket_qualification_matrix() { if !enabled() { @@ -48,6 +71,27 @@ async fn aws_general_purpose_bucket_qualification_matrix() { .await; let aws = aws_sdk_s3::Client::new(&shared); + if let Ok(bucket) = std::env::var("PROLLY_S3_AWS_BUCKET_LIFECYCLE") { + assert_policy_bucket_rejected(&aws, &bucket, ®ion_name, "lifecycle").await; + } + if let Ok(bucket) = std::env::var("PROLLY_S3_AWS_BUCKET_OBJECT_LOCK_DEFAULT") { + assert_policy_bucket_rejected(&aws, &bucket, ®ion_name, "object-lock-default").await; + } + if let Ok(bucket) = std::env::var("PROLLY_S3_AWS_BUCKET_REPLICATED") { + let replicated = Client::builder() + .aws_client(aws.clone()) + .bucket(&bucket) + .repository_prefix(unique_prefix("replicated")) + .writer("aws-replicated-qualification") + .provider_identity(ProviderIdentity::aws_region(region_name.clone())) + .attestation_signer(signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + .initialize() + .await + .unwrap(); + assert!(replicated.provider_capabilities().replication_enabled); + } + if let Ok(bucket) = std::env::var("PROLLY_AWS_BUCKET_UNVERSIONED") { let result = Client::builder() .aws_client(aws.clone()) @@ -89,6 +133,12 @@ async fn aws_general_purpose_bucket_qualification_matrix() { .initialize() .await .unwrap(); + assert!( + client + .provider_capabilities() + .replication_configuration_readable, + "AWS qualification must inspect the bucket replication policy" + ); let first = client .put_object("matrix/history.bin", b"first".to_vec()) diff --git a/extensions/s3/client/tests/rustfs_repository.rs b/extensions/s3/client/tests/rustfs_repository.rs index 295310b7..e017d642 100644 --- a/extensions/s3/client/tests/rustfs_repository.rs +++ b/extensions/s3/client/tests/rustfs_repository.rs @@ -1,5 +1,9 @@ use std::{ - sync::Arc, + collections::BTreeSet, + sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }, time::{Duration, SystemTime, UNIX_EPOCH}, }; @@ -11,13 +15,16 @@ use aws_sdk_s3::{ types::{BucketVersioningStatus, VersioningConfiguration}, }; use futures_util::{stream, StreamExt}; +use md5::{Digest as _, Md5}; use prolly_s3_client::{ core::{ - decode_canonical, encode_canonical, BatchId, Error, ErrorCode, LogicalObjectVersionKind, - MergeCursor, MergePhase, MergePolicy, ProviderPerKeyVersionLimit, + decode_canonical, encode_canonical, BatchId, Error, ErrorCode, GcPhase, + LogicalObjectVersionKind, MergeCursor, MergePhase, MergePolicy, ProviderPerKeyVersionLimit, }, - BulkWriteOptions, CheckoutRef, Client, HmacAttestationSigner, ProviderIdentity, PutObjectInput, + BulkWriteOptions, CheckoutRef, Client, HmacAttestationSigner, OrderedPublicationOptions, + ProviderIdentity, PutObjectInput, }; +use sha2::Sha256; fn rustfs_enabled() -> bool { std::env::var("PROLLY_S3_RUSTFS").as_deref() == Ok("1") @@ -31,6 +38,10 @@ fn unique_name(label: &str) -> String { format!("integration/{label}/{nanos}") } +fn percentile(sorted: &[Duration], percentile: usize) -> Duration { + sorted[(sorted.len() * percentile).div_ceil(100) - 1] +} + fn provider_identity() -> ProviderIdentity { ProviderIdentity::s3_compatible( std::env::var("PROLLY_RUSTFS_ENDPOINT") @@ -475,7 +486,7 @@ async fn rustfs_detached_paged_and_streamed_lists_stay_on_the_selected_commit() } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn rustfs_commit_session_preserves_n_plus_three_puts() { +async fn rustfs_commit_session_accounts_for_publication_ticket() { if !rustfs_enabled() { eprintln!("set PROLLY_S3_RUSTFS=1 to run RustFS integration tests"); return; @@ -515,8 +526,12 @@ async fn rustfs_commit_session_preserves_n_plus_three_puts() { assert_eq!(receipt.changed_keys, 3); let calls = client.reset_s3_operation_metrics(); assert_eq!( - calls.put_object, 5, - "two immutable payload PUTs plus commit/event/ref publication must preserve N + 3: {calls:?}" + calls.put_object, 6, + "two payloads, commit/event/ref publication, and one GC admission ticket require N + 4 PUTs: {calls:?}" + ); + assert_eq!( + calls.delete_object, 1, + "successful publication must exact-delete its admission ticket: {calls:?}" ); assert_eq!( calls.head_object, 0, @@ -915,10 +930,11 @@ async fn rustfs_streaming_bulk_write_is_bounded_batched_and_ordered() { return; } let (aws, bucket) = rustfs_client().await; + let repository_prefix = unique_name("streaming-bulk-write"); let client = Client::builder() - .aws_client(aws) + .aws_client(aws.clone()) .bucket(&bucket) - .repository_prefix(unique_name("streaming-bulk-write")) + .repository_prefix(&repository_prefix) .writer("rustfs-streaming-bulk-writer") .provider_identity(provider_identity()) .attestation_signer(attestation_signer()) @@ -966,9 +982,19 @@ async fn rustfs_streaming_bulk_write_is_bounded_batched_and_ordered() { .unwrap(); let first_binding = first.version.binding.unwrap(); let second_binding = second.version.binding.unwrap(); - assert!(first_binding.is_packed()); - assert_eq!(first_binding.path, second_binding.path); - assert_ne!(first_binding.pack_range, second_binding.pack_range); + assert_ne!(first_binding.path, second_binding.path); + assert!(!first_binding.path.as_str().contains("payload-packs")); + assert!(!second_binding.path.as_str().contains("payload-packs")); + let physical_payloads = aws + .list_objects_v2() + .bucket(&bucket) + .prefix(format!("{repository_prefix}/payloads/")) + .max_keys(1_000) + .send() + .await + .unwrap(); + assert_eq!(physical_payloads.key_count(), Some(257)); + assert!(!physical_payloads.is_truncated().unwrap_or(false)); assert_eq!( client .get_object_range(receipts[0].id, "stream/0000.txt", 0..=u64::MAX) @@ -991,6 +1017,104 @@ async fn rustfs_streaming_bulk_write_is_bounded_batched_and_ordered() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn rustfs_ordered_publication_queue_groups_unique_keys_and_orders_duplicates() { + if !rustfs_enabled() { + eprintln!("set PROLLY_S3_RUSTFS=1 to run RustFS integration tests"); + return; + } + let (aws, bucket) = rustfs_client().await; + let client = Client::builder() + .aws_client(aws) + .bucket(&bucket) + .repository_prefix(unique_name("ordered-publication-queue")) + .writer("rustfs-ordered-publication-writer") + .provider_identity(provider_identity()) + .attestation_signer(attestation_signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + .initialize() + .await + .unwrap(); + let queue = client + .ordered_publication_queue(OrderedPublicationOptions { + max_group_size: 128, + upload_concurrency: 32, + max_wait: Duration::from_millis(25), + queue_capacity: 512, + checkpoint_every: 32, + durable: true, + }) + .unwrap(); + + let receipts = stream::iter(0..257) + .map(|index| { + let queue = queue.clone(); + async move { + queue + .put_object( + format!("queue/{index:04}.txt"), + format!("value-{index}").into_bytes(), + ) + .await + } + }) + .buffer_unordered(257) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .unwrap(); + let commits = receipts + .iter() + .map(|receipt| receipt.commit) + .collect::>(); + assert_eq!(commits.len(), 3); + assert_eq!( + commits + .iter() + .map(|commit| receipts + .iter() + .find(|receipt| receipt.commit == *commit) + .unwrap()) + .map(|receipt| receipt.group_size) + .sum::(), + 257 + ); + + let (valid, invalid) = tokio::join!( + queue.put_object("queue/valid-after-error.txt", b"valid".to_vec()), + queue.put_object("", b"invalid".to_vec()), + ); + assert_eq!(invalid.unwrap_err().code, ErrorCode::InvalidKey); + assert_eq!(valid.unwrap().group_size, 1); + assert_eq!( + client + .get_object("queue/valid-after-error.txt") + .await + .unwrap() + .unwrap() + .bytes, + b"valid" + ); + + let (first, second) = tokio::join!( + queue.put_object("queue/repeated.txt", b"first".to_vec()), + queue.put_object("queue/repeated.txt", b"second".to_vec()), + ); + let first = first.unwrap(); + let second = second.unwrap(); + assert_ne!(first.commit, second.commit); + assert_eq!( + client + .get_object("queue/repeated.txt") + .await + .unwrap() + .unwrap() + .bytes, + b"second" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 8)] async fn rustfs_streaming_bulk_failure_preserves_completed_checkpoint() { if !rustfs_enabled() { @@ -1071,16 +1195,25 @@ async fn rustfs_streaming_bulk_exceeds_500_files_per_second() { client.reset_s3_operation_metrics(); let started = std::time::Instant::now(); let receipts = client - .put_object_stream(objects, BulkWriteOptions::default()) + .put_object_stream( + objects, + BulkWriteOptions { + concurrency: 128, + ..BulkWriteOptions::default() + }, + ) .await .unwrap(); let elapsed = started.elapsed(); let throughput = FILES as f64 / elapsed.as_secs_f64(); let metrics = client.reset_s3_operation_metrics(); eprintln!( - "RUSTFS_STREAMING_BULK files={FILES} wall_ms={} files_per_second={throughput:.2} s3_calls={} uploaded_bytes={}", + "RUSTFS_STREAMING_BULK files={FILES} wall_ms={} files_per_second={throughput:.2} s3_calls={} put_object={} get_object={} head_object={} uploaded_bytes={}", elapsed.as_millis(), metrics.total_calls(), + metrics.put_object, + metrics.get_object, + metrics.head_object, metrics.uploaded_body_bytes, ); assert_eq!(receipts.len(), 1); @@ -1089,8 +1222,8 @@ async fn rustfs_streaming_bulk_exceeds_500_files_per_second() { "throughput was {throughput:.2} files/s" ); assert!( - metrics.put_object < 100, - "packing should require fewer than 100 physical PUTs: {metrics:?}" + metrics.put_object >= FILES as u64 && metrics.put_object <= FILES as u64 + 256, + "whole-object ingestion should use one payload PUT per file plus bounded metadata PUTs: {metrics:?}" ); } @@ -1436,6 +1569,101 @@ async fn rustfs_20k_branch_diff_merge_meets_amplification_slos() { /// Reproducible release gate for the same-branch publication lane. /// +/// Independent callers enqueue 10K whole-object candidates. The queue prepares +/// payloads concurrently and acknowledges each caller after its ordered group +/// commit publishes. +#[tokio::test(flavor = "multi_thread", worker_threads = 16)] +#[ignore = "10K ordered group-commit RustFS release gate"] +async fn rustfs_10k_ordered_publication_queue_gate() { + assert!( + rustfs_enabled(), + "set PROLLY_S3_RUSTFS=1 to run the ordered publication gate" + ); + const FILES: usize = 10_000; + let upload_concurrency = std::env::var("PROLLY_S3_ORDERED_CONCURRENCY") + .ok() + .and_then(|value| value.parse::().ok()) + .unwrap_or(128); + assert!((1..=1_024).contains(&upload_concurrency)); + let (aws, bucket) = rustfs_client().await; + let client = Client::builder() + .aws_client(aws) + .bucket(&bucket) + .repository_prefix(unique_name("10k-ordered-publication")) + .writer("rustfs-10k-ordered-writer") + .provider_identity(provider_identity()) + .attestation_signer(attestation_signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + .initialize() + .await + .unwrap(); + let queue = client + .ordered_publication_queue(OrderedPublicationOptions { + max_group_size: 10_000, + upload_concurrency, + max_wait: Duration::from_millis(50), + queue_capacity: 10_000, + checkpoint_every: 1_000, + durable: true, + }) + .unwrap(); + + client.reset_s3_operation_metrics(); + let started = std::time::Instant::now(); + let results = stream::iter(0..FILES) + .map(|index| { + let queue = queue.clone(); + async move { + let operation_started = std::time::Instant::now(); + queue + .put_object( + format!("ordered/{index:05}.bin"), + index.to_be_bytes().to_vec(), + ) + .await + .map(|receipt| (receipt, operation_started.elapsed())) + } + }) + .buffer_unordered(FILES) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .unwrap(); + let elapsed = started.elapsed(); + let throughput = FILES as f64 / elapsed.as_secs_f64(); + let mut latencies = results + .iter() + .map(|(_, latency)| *latency) + .collect::>(); + latencies.sort_unstable(); + let commits = results + .iter() + .map(|(receipt, _)| receipt.commit) + .collect::>(); + let p50 = percentile(&latencies, 50); + let p95 = percentile(&latencies, 95); + let p99 = percentile(&latencies, 99); + let metrics = client.reset_s3_operation_metrics(); + eprintln!( + "RUSTFS_10K_ORDERED files={FILES} commits={} upload_concurrency={upload_concurrency} wall_ms={} files_per_second={throughput:.2} p50_ms={:.2} p95_ms={:.2} p99_ms={:.2} s3_calls={} calls_per_file={:.3}", + commits.len(), + elapsed.as_millis(), + p50.as_secs_f64() * 1_000.0, + p95.as_secs_f64() * 1_000.0, + p99.as_secs_f64() * 1_000.0, + metrics.total_calls(), + metrics.total_calls() as f64 / FILES as f64, + ); + assert_eq!(commits.len(), 1); + assert!( + throughput >= 500.0, + "ordered publication achieved {throughput:.2} files/s" + ); +} + +/// Reproducible release gate for deliberately distinct same-branch commits. +/// /// This is intentionally ignored because it creates exactly 10,000 commits /// and is meant for an isolated RustFS qualification run. Client clones issue /// work concurrently; the branch-local lane serializes the linear ref history @@ -1470,24 +1698,34 @@ async fn rustfs_10k_concurrent_commit_regression_gate() { client.reset_s3_operation_metrics(); let started = std::time::Instant::now(); + let completed = Arc::new(AtomicUsize::new(0)); let results = stream::iter(0..COMMITS) .map(|index| { let client = client.clone(); + let completed = completed.clone(); async move { + let operation_started = std::time::Instant::now(); client .put_object( format!("concurrent/{index:05}.bin"), index.to_be_bytes().to_vec(), ) - .await + .await?; + let count = completed.fetch_add(1, Ordering::Relaxed) + 1; + if count.is_multiple_of(1_000) { + eprintln!( + "RUSTFS_10K_PROGRESS completed={count} wall_ms={}", + started.elapsed().as_millis() + ); + } + Ok::<_, Error>(operation_started.elapsed()) } }) .buffer_unordered(concurrency) .collect::>() .await; - for result in results { - result.unwrap(); - } + let mut latencies = results.into_iter().collect::, _>>().unwrap(); + latencies.sort_unstable(); let mut after = None; let mut files = 0_usize; @@ -1506,11 +1744,217 @@ async fn rustfs_10k_concurrent_commit_regression_gate() { } assert_eq!(files, COMMITS); + let elapsed = started.elapsed(); + let throughput = COMMITS as f64 / elapsed.as_secs_f64(); + let p50 = percentile(&latencies, 50); + let p95 = percentile(&latencies, 95); + let p99 = percentile(&latencies, 99); let metrics = client.reset_s3_operation_metrics(); eprintln!( - "RUSTFS_10K_COMMITS commits={COMMITS} concurrency={concurrency} wall_ms={} s3_calls={} calls_per_commit={:.2}", - started.elapsed().as_millis(), + "RUSTFS_10K_COMMITS commits={COMMITS} concurrency={concurrency} wall_ms={} commits_per_second={throughput:.2} p50_ms={:.2} p95_ms={:.2} p99_ms={:.2} s3_calls={} calls_per_commit={:.2}", + elapsed.as_millis(), + p50.as_secs_f64() * 1_000.0, + p95.as_secs_f64() * 1_000.0, + p99.as_secs_f64() * 1_000.0, metrics.total_calls(), metrics.total_calls() as f64 / COMMITS as f64, ); } + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +#[ignore = "interrupts and resumes a repository-wide GC epoch on live RustFS"] +async fn rustfs_gc_restart_keeps_independent_writer_fenced() { + assert!( + rustfs_enabled(), + "set PROLLY_S3_RUSTFS=1 to run the GC restart gate" + ); + let (aws, bucket) = rustfs_client().await; + let prefix = unique_name("gc-restart-fence"); + let builder = |aws: aws_sdk_s3::Client| { + Client::builder() + .aws_client(aws) + .bucket(&bucket) + .repository_prefix(&prefix) + .writer("rustfs-gc-writer") + .background_index_maintenance(false) + .provider_identity(provider_identity()) + .attestation_signer(attestation_signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + }; + let owner = builder(aws.clone()).initialize().await.unwrap(); + owner + .put_object("live.txt", b"reachable".to_vec()) + .await + .unwrap(); + let independent = builder(aws.clone()).open().await.unwrap(); + tokio::time::sleep(Duration::from_millis(2)).await; + let gc = owner.start_gc(1).await.unwrap(); + let expected_epoch = gc.epoch; + assert_eq!( + independent + .put_object("fenced-before-restart.txt", b"blocked".to_vec()) + .await + .unwrap_err() + .code, + ErrorCode::PreconditionFailed + ); + drop(owner); + + let resumed_owner = builder(aws).open().await.unwrap(); + let mut gc = resumed_owner.resume_gc().await.unwrap().unwrap(); + assert_eq!(gc.epoch, expected_epoch); + assert_eq!( + independent + .put_object("fenced-after-restart.txt", b"blocked".to_vec()) + .await + .unwrap_err() + .code, + ErrorCode::PreconditionFailed + ); + for _ in 0..100_000 { + gc = match gc.phase { + GcPhase::Ready | GcPhase::Sweeping => { + resumed_owner.sweep_gc(&gc, 100).await.unwrap().cursor + } + GcPhase::Complete => break, + _ => resumed_owner.advance_gc(&gc, 100).await.unwrap().cursor, + }; + } + assert_eq!(gc.phase, GcPhase::Complete); + independent.advance_branch_indexes().await.unwrap(); + independent + .put_object("after-gc.txt", b"admission reopened".to_vec()) + .await + .unwrap(); + assert_eq!( + independent + .get_object("live.txt") + .await + .unwrap() + .unwrap() + .bytes, + b"reachable" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 12)] +#[ignore = "uploads a 65 MiB single object to live RustFS"] +async fn rustfs_streamed_large_object_uses_one_provider_object() { + assert!( + rustfs_enabled(), + "set PROLLY_S3_RUSTFS=1 to run the large-object RustFS gate" + ); + const SIZE: usize = 65 * 1_024 * 1_024; + let (aws, bucket) = rustfs_client().await; + let client = Client::builder() + .aws_client(aws) + .bucket(&bucket) + .repository_prefix(unique_name("single-object-stream")) + .writer("rustfs-single-object-writer") + .provider_identity(provider_identity()) + .attestation_signer(attestation_signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + .initialize() + .await + .unwrap(); + + client.reset_s3_operation_metrics(); + let mut session = client + .begin_commit() + .message("single-object stream") + .start() + .await + .unwrap(); + session + .put_stream("large/payload.bin", ByteStream::from(vec![0x5a; SIZE])) + .await + .unwrap(); + let receipt = session.publish().await.unwrap(); + let metrics = client.reset_s3_operation_metrics(); + // One payload PutObject plus bounded repository metadata publications. + assert!(metrics.put_object >= 1); + assert!(metrics.uploaded_body_bytes >= SIZE as u64); + assert!(metrics.uploaded_body_bytes < SIZE as u64 + 1024 * 1024); + + let tail = client + .get_object_range( + receipt.id, + "large/payload.bin", + (SIZE as u64 - 32)..=(SIZE as u64 - 1), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(tail.bytes, vec![0x5a; 32]); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +#[ignore = "hands off and publishes a 33 MiB externally uploaded object"] +async fn rustfs_external_uploader_handoff_survives_restart() { + assert!( + rustfs_enabled(), + "set PROLLY_S3_RUSTFS=1 to run the external-uploader gate" + ); + const MIB: usize = 1_024 * 1_024; + let (aws, bucket) = rustfs_client().await; + let prefix = unique_name("external-upload-resume"); + let builder = |aws: aws_sdk_s3::Client| { + Client::builder() + .aws_client(aws) + .bucket(&bucket) + .repository_prefix(&prefix) + .writer("rustfs-external-upload-writer") + .provider_identity(provider_identity()) + .attestation_signer(attestation_signer()) + .provider_per_key_version_limit(ProviderPerKeyVersionLimit::Finite(10_000)) + }; + let client = builder(aws.clone()).initialize().await.unwrap(); + let mut body = vec![0x11; 16 * MIB]; + body.extend(vec![0x22; 16 * MIB]); + body.extend(vec![0x33; MIB]); + let session = client + .begin_commit() + .message("external upload resume") + .start() + .await + .unwrap(); + let batch = session.id(); + let upload = session + .prepare_external_object_upload( + "large/resumable.bin", + body.len() as u64, + Sha256::digest(&body).into(), + Md5::digest(&body).into(), + ) + .await + .unwrap(); + aws.put_object() + .bucket(&bucket) + .key(upload.path.as_str()) + .metadata("prolly-sha256", hex::encode(upload.checksum_sha256)) + .body(ByteStream::from(body.clone())) + .send() + .await + .unwrap(); + let persisted = serde_json::to_vec(&upload).unwrap(); + drop(session); + drop(client); + + let reopened = builder(aws).open().await.unwrap(); + let mut session = reopened.resume_commit(batch).await.unwrap(); + let upload = serde_json::from_slice(&persisted).unwrap(); + session + .stage_external_object_upload(&upload, Default::default(), Default::default()) + .await + .unwrap(); + session.publish().await.unwrap(); + assert_eq!( + reopened + .get_object("large/resumable.bin") + .await + .unwrap() + .unwrap() + .bytes, + body + ); +} diff --git a/extensions/s3/core/src/authority.rs b/extensions/s3/core/src/authority.rs index 00f684fa..2290d4c6 100644 --- a/extensions/s3/core/src/authority.rs +++ b/extensions/s3/core/src/authority.rs @@ -299,8 +299,13 @@ impl ShardWriterAuthority

{ })?; let current: AuthorityLease = decode_canonical(&stored.bytes)?; current.validate(self.repository, &permit.lease.scope)?; - if current != permit.lease - || stored.metadata.token != permit.token + // The immutable authority epoch, not the mutable lease object's + // storage token or expiry, fences publication. Ordinary renewal keeps + // the epoch stable, so an operation/session that began before one or + // many renewals remains valid while the current lease for that epoch + // is active. A real takeover advances the stamp generation and is + // rejected here. + if current.stamp() != permit.lease.stamp() || current.expires_at_millis <= now_millis || !matches!(current.state, AuthorityLeaseState::Active) { diff --git a/extensions/s3/core/src/commit_session.rs b/extensions/s3/core/src/commit_session.rs index 8e43467b..481c9ee8 100644 --- a/extensions/s3/core/src/commit_session.rs +++ b/extensions/s3/core/src/commit_session.rs @@ -1,4 +1,4 @@ -use std::sync::Arc; +use std::{collections::BTreeMap, sync::Arc}; use crate::{ decode_canonical, encode_canonical, BatchId, CommitSessionCheckpoint, @@ -76,7 +76,7 @@ impl CommitSessionStore

{ pub async fn latest(&self, batch: BatchId) -> Result> { let prefix = self.batch_prefix(batch); let mut continuation = None; - let mut latest: Option<(u64, ObjectPath)> = None; + let mut paths = BTreeMap::new(); let mut scanned = 0_usize; loop { let page = self @@ -99,11 +99,11 @@ impl CommitSessionStore

{ )); } let sequence = parse_checkpoint_sequence(&prefix, &entry.path)?; - if latest - .as_ref() - .is_none_or(|(current, _)| sequence > *current) - { - latest = Some((sequence, entry.path)); + if paths.insert(sequence, entry.path).is_some() { + return Err(Error::new( + ErrorCode::CorruptContent, + "commit session has duplicate checkpoint sequences", + )); } } continuation = page.continuation; @@ -111,38 +111,78 @@ impl CommitSessionStore

{ break; } } - let Some((sequence, path)) = latest else { + if paths.is_empty() { return Ok(None); - }; - let stored = self - .plane - .get(GetRequest { - path, - range: None, - physical_version: None, - }) - .await? - .ok_or_else(|| { - Error::new( - ErrorCode::OutcomeUnknown, - "latest commit-session checkpoint disappeared during resume", - ) - })?; - let checkpoint: CommitSessionCheckpoint = decode_canonical(&stored.bytes)?; - checkpoint.validate(self.repository, self.max_mutations)?; - if checkpoint.session.id != batch { - return Err(Error::new( - ErrorCode::CorruptContent, - "checkpoint path and embedded batch ID disagree", - )); } - if checkpoint.sequence != sequence { - return Err(Error::new( - ErrorCode::CorruptContent, - "checkpoint path and embedded sequence disagree", - )); + + let mut aggregate = BTreeMap::, crate::StagedMutation>::new(); + let mut latest = None; + let checkpoint_count = paths.len(); + for (index, (sequence, path)) in paths.into_iter().enumerate() { + if sequence != index as u64 { + return Err(Error::new( + ErrorCode::MissingClosure, + "commit-session checkpoint sequence has a gap", + )); + } + let stored = self + .plane + .get(GetRequest { + path, + range: None, + physical_version: None, + }) + .await? + .ok_or_else(|| { + Error::new( + ErrorCode::OutcomeUnknown, + "commit-session checkpoint disappeared during resume", + ) + })?; + let checkpoint: CommitSessionCheckpoint = decode_canonical(&stored.bytes)?; + checkpoint.validate(self.repository, self.max_mutations)?; + if checkpoint.session.id != batch || checkpoint.sequence != sequence { + return Err(Error::new( + ErrorCode::CorruptContent, + "checkpoint path and embedded identity disagree", + )); + } + if let Some(previous) = latest.as_ref() { + if !same_session_lineage(previous, &checkpoint) { + return Err(Error::new( + ErrorCode::CorruptContent, + "commit-session checkpoint lineage changed", + )); + } + if previous.state == crate::CommitSessionState::Aborted { + return Err(Error::new( + ErrorCode::CorruptContent, + "commit session continues after an aborted checkpoint", + )); + } + } + if checkpoint.state == crate::CommitSessionState::Aborted + && index + 1 != checkpoint_count + { + return Err(Error::new( + ErrorCode::CorruptContent, + "aborted commit-session checkpoint is not terminal", + )); + } + for mutation in &checkpoint.mutations { + aggregate.insert(mutation.key().to_vec(), mutation.clone()); + } + if aggregate.len() as u64 != checkpoint.total_mutations { + return Err(Error::new( + ErrorCode::MissingClosure, + "commit-session checkpoint total does not match its windows", + )); + } + latest = Some(checkpoint); } - Ok(Some(checkpoint)) + let mut latest = latest.expect("nonempty checkpoint paths produced a checkpoint"); + latest.mutations = aggregate.into_values().collect(); + Ok(Some(latest)) } pub async fn cleanup_expired_page( @@ -235,6 +275,23 @@ impl CommitSessionStore

{ } } +fn same_session_lineage( + previous: &CommitSessionCheckpoint, + next: &CommitSessionCheckpoint, +) -> bool { + let previous = &previous.session; + let next = &next.session; + previous.id == next.id + && previous.branch == next.branch + && previous.base_commit == next.base_commit + && previous.identity.repository == next.identity.repository + && previous.identity.operation == next.identity.operation + && previous.identity.authority.writer_id == next.identity.authority.writer_id + && previous.message == next.message + && previous.created_at_millis == next.created_at_millis + && previous.expires_at_millis == next.expires_at_millis +} + fn parse_checkpoint_sequence(prefix: &str, path: &ObjectPath) -> Result { let suffix = path .as_str() diff --git a/extensions/s3/core/src/control_versions.rs b/extensions/s3/core/src/control_versions.rs index eded2957..81d5c38d 100644 --- a/extensions/s3/core/src/control_versions.rs +++ b/extensions/s3/core/src/control_versions.rs @@ -5,9 +5,9 @@ use std::{ use crate::{ decode_canonical, CommitGraphHead, CompareExchange, CompareExchangeOutcome, DeleteOutcome, - Error, ErrorCode, JournalDerivedIndexHead, ListRequest, NodeIndexHead, ObjectPath, ObjectPlane, - OperationIndexHead, PhysicalVersion, RefCatalogHead, RefCatalogShardHead, RefValue, Result, - StorageToken, + Error, ErrorCode, FsckCursor, JournalDerivedIndexHead, ListRequest, NodeIndexHead, ObjectPath, + ObjectPlane, OperationIndexHead, PhysicalVersion, RefCatalogHead, RefCatalogShardHead, + RefValue, Result, StorageToken, }; pub const DEFAULT_MUTABLE_CONTROL_VERSIONS_TO_RETAIN: usize = 100; @@ -27,6 +27,8 @@ pub enum MutableControlKind { OperationIndexHead, JournalDerivedIndexHead, GcCoordinator, + GcCursor, + FsckCursor, } #[derive(Clone, Debug, Default, PartialEq, Eq)] @@ -58,6 +60,14 @@ pub trait MutableControlObserver: Send + Sync { kind: MutableControlKind, request: &CompareExchange, ) -> Result<()>; + + async fn after_compare_exchange( + &self, + _kind: MutableControlKind, + _request: &CompareExchange, + ) -> Result<()> { + Ok(()) + } } impl MutableControlStore

{ @@ -115,9 +125,6 @@ impl MutableControlStore

{ "compare_exchange through the mutable-control store requires a registered control path", ) })?; - if let Some(observer) = self.observer.as_ref() { - observer.before_compare_exchange(kind, &request).await?; - } if let Some(expected) = request.expected.as_ref() { let first_update = self.is_unseen(&request.path)?; let ref_interval = (self.versions_to_retain / 2).max(1) as u64; @@ -150,6 +157,10 @@ impl MutableControlStore

{ let head: JournalDerivedIndexHead = decode_canonical(&request.bytes)?; first_update || head.generation.is_multiple_of(ref_interval) } + MutableControlKind::FsckCursor => { + let cursor: FsckCursor = decode_canonical(&request.bytes)?; + first_update || cursor.checkpoint_generation.is_multiple_of(ref_interval) + } _ => true, }; if scheduled { @@ -160,17 +171,23 @@ impl MutableControlStore

{ | MutableControlKind::RefCatalogShardHead | MutableControlKind::CommitGraphHead | MutableControlKind::OperationIndexHead - | MutableControlKind::JournalDerivedIndexHead => { - (self.versions_to_retain / 2).max(1) - } + | MutableControlKind::JournalDerivedIndexHead + | MutableControlKind::FsckCursor => (self.versions_to_retain / 2).max(1), _ => self.versions_to_retain.saturating_sub(1), }; self.compact_if_current(&request.path, expected, target) .await?; } } + if let Some(observer) = self.observer.as_ref() { + observer.before_compare_exchange(kind, &request).await?; + } let path = request.path.clone(); - let outcome = self.plane.compare_exchange(request).await?; + let outcome = self.plane.compare_exchange(request.clone()).await; + if let Some(observer) = self.observer.as_ref() { + observer.after_compare_exchange(kind, &request).await?; + } + let outcome = outcome?; if matches!(outcome, CompareExchangeOutcome::Applied(_)) { self.mark_seen(path)?; } @@ -374,6 +391,8 @@ pub fn classify_mutable_control_path( ["operation-index", "heads", _] => Some(MutableControlKind::OperationIndexHead), ["journal-index", "heads", _] => Some(MutableControlKind::JournalDerivedIndexHead), ["gc", "coordinator.cbor"] => Some(MutableControlKind::GcCoordinator), + ["gc", "epochs", _, "cursor.cbor"] => Some(MutableControlKind::GcCursor), + ["administration", "fsck", _, "cursor.cbor"] => Some(MutableControlKind::FsckCursor), _ => None, } } diff --git a/extensions/s3/core/src/error.rs b/extensions/s3/core/src/error.rs index 5d684580..d5e6a2fa 100644 --- a/extensions/s3/core/src/error.rs +++ b/extensions/s3/core/src/error.rs @@ -59,7 +59,7 @@ pub enum RetryAdvice { ReconcileOperation, } -#[derive(Debug, thiserror::Error)] +#[derive(Debug, Clone, thiserror::Error)] #[error("{code:?}: {message}")] pub struct Error { pub code: ErrorCode, diff --git a/extensions/s3/core/src/gc.rs b/extensions/s3/core/src/gc.rs index 6a7470d9..8c07654e 100644 --- a/extensions/s3/core/src/gc.rs +++ b/extensions/s3/core/src/gc.rs @@ -31,6 +31,8 @@ pub struct GcCursor { pub dirty_sequence: u64, pub dirty_target_sequence: u64, pub initial_scan_complete: bool, + #[serde(default)] + pub publication_barrier_drained: bool, pub sweep_after: Option>, pub report: GcReport, } @@ -48,8 +50,14 @@ pub struct GcReport { pub deleted_bytes: u64, pub already_missing: u64, pub skipped_reachable: u64, + #[serde(default)] + pub protected_versions: u64, + #[serde(default)] + pub protected_bytes: u64, pub candidates_by_kind: BTreeMap, pub deleted_by_kind: BTreeMap, + #[serde(default)] + pub protected_by_kind: BTreeMap, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -65,9 +73,19 @@ pub(crate) struct GcCoordinator { pub repository: RepositoryId, pub generation: u64, pub active_epoch: Option, + #[serde(default)] + pub admission_closed: bool, pub updated_at_millis: u64, } +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct GcPublicationTicket { + pub repository: RepositoryId, + pub instance: OperationId, + pub request_digest: [u8; 32], + pub expires_at_millis: u64, +} + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub(crate) struct GcDirtyRoot { pub repository: RepositoryId, diff --git a/extensions/s3/core/src/journal_indexes.rs b/extensions/s3/core/src/journal_indexes.rs index a8d3c270..327320b6 100644 --- a/extensions/s3/core/src/journal_indexes.rs +++ b/extensions/s3/core/src/journal_indexes.rs @@ -1,5 +1,9 @@ -use std::sync::Arc; +use std::{ + collections::{BTreeMap, BTreeSet}, + sync::Arc, +}; +use futures_util::{stream, StreamExt}; use prolly::{AsyncProlly, Config, Mutation, RuntimeConfig, Tree, TreeFormat}; use serde::{Deserialize, Serialize}; @@ -15,6 +19,7 @@ use crate::{ }; pub const DEFAULT_JOURNAL_INDEX_MAX_UNINDEXED_EVENTS: usize = 4_096; +const COMMIT_INDEX_LOAD_CONCURRENCY: usize = 32; #[derive(Clone, Debug, PartialEq, Eq)] pub struct JournalIndexAdvanceReport { @@ -274,31 +279,9 @@ impl JournalDerivedIndexes

{ runtime: RuntimeConfig::default(), }, }; - let mut indexed_commits = 0usize; - let mut indexed_nodes = 0usize; - for event in events { - if self - .graph_engine - .get(&graph_tree, event.new_target.as_bytes()) - .await? - .is_some() - { - continue; - } - let object = publisher.load_commit_index(event.new_target).await?; - self.index_node_pack( - &mut node_tree, - event.new_target, - object.commit.node_pack.as_ref().map(|pack| pack.id), - object.toc.as_ref(), - object.payload_offset, - &mut indexed_nodes, - ) + let (indexed_commits, indexed_nodes) = self + .index_events(publisher, &mut node_tree, &mut graph_tree, events) .await?; - self.index_commit_graph(&mut graph_tree, event.new_target, object.commit) - .await?; - indexed_commits += 1; - } let next = JournalDerivedIndexHead { repository: self.repository, @@ -632,31 +615,14 @@ impl JournalDerivedIndexes

{ } let mut node_tree = self.tree_from_root(&cursor.node_root); let mut graph_tree = self.tree_from_root(&cursor.commit_graph_root); - let mut indexed_commits = 0usize; - let mut indexed_nodes = 0usize; - for event in chunk.events.iter().rev() { - if self - .graph_engine - .get(&graph_tree, event.new_target.as_bytes()) - .await? - .is_some() - { - continue; - } - let object = publisher.load_commit_index(event.new_target).await?; - self.index_node_pack( + let (indexed_commits, indexed_nodes) = self + .index_events( + publisher, &mut node_tree, - event.new_target, - object.commit.node_pack.as_ref().map(|pack| pack.id), - object.toc.as_ref(), - object.payload_offset, - &mut indexed_nodes, + &mut graph_tree, + chunk.events.iter().rev().cloned().collect(), ) .await?; - self.index_commit_graph(&mut graph_tree, event.new_target, object.commit) - .await?; - indexed_commits += 1; - } let mut next = cursor.clone(); next.node_root.root = node_tree.root; next.commit_graph_root.root = graph_tree.root; @@ -1039,17 +1005,15 @@ impl JournalDerivedIndexes

{ )) } - async fn index_node_pack( + fn node_pack_mutations( &self, - tree: &mut Tree, commit: CommitId, pack_id: Option, toc: Option<&NodePackToc>, - payload_offset: Option, - indexed_nodes: &mut usize, - ) -> Result<()> { + node_region_offset: Option, + ) -> Result> { let Some(toc) = toc else { - return Ok(()); + return Ok(Vec::new()); }; let pack_id = pack_id.ok_or_else(|| { Error::new( @@ -1057,17 +1021,20 @@ impl JournalDerivedIndexes

{ "journal-indexed node pack has no logical reference", ) })?; - let payload_offset = payload_offset.ok_or_else(|| { + let node_region_offset = node_region_offset.ok_or_else(|| { Error::new( ErrorCode::CorruptCommit, - "journal-indexed node pack has no payload offset", + "journal-indexed node pack has no node-region offset", ) })?; let mut mutations = Vec::with_capacity(toc.entries.len()); for entry in &toc.entries { - let absolute_offset = payload_offset.checked_add(entry.offset).ok_or_else(|| { - Error::new(ErrorCode::CorruptNode, "journal node offset overflow") - })?; + let absolute_offset = + node_region_offset + .checked_add(entry.offset) + .ok_or_else(|| { + Error::new(ErrorCode::CorruptNode, "journal node offset overflow") + })?; mutations.push(Mutation::Upsert { key: entry.cid.as_bytes().to_vec(), val: encode_canonical(&JournalNodeIndexEntry { @@ -1080,53 +1047,126 @@ impl JournalDerivedIndexes

{ })?, }); } - *indexed_nodes = indexed_nodes.checked_add(mutations.len()).ok_or_else(|| { - Error::new(ErrorCode::InternalInvariant, "indexed node count overflow") - })?; - if !mutations.is_empty() { - *tree = self.node_engine.batch(tree, mutations).await?; - } - Ok(()) + Ok(mutations) } - async fn index_commit_graph( + async fn build_commit_graph_entry( &self, - tree: &mut Tree, + tree: &Tree, + pending: &BTreeMap, id: CommitId, commit: BucketCommit, - ) -> Result<()> { + ) -> Result { let mut jumps = Vec::new(); if let Some(first_parent) = commit.parents.first().copied() { jumps.push(first_parent); for level in 1..64usize { let ancestor = jumps[level - 1]; - let Some(encoded) = self.graph_engine.get(tree, ancestor.as_bytes()).await? else { - break; + let entry = if let Some(entry) = pending.get(&ancestor) { + entry.clone() + } else { + let Some(encoded) = self.graph_engine.get(tree, ancestor.as_bytes()).await? + else { + break; + }; + decode_canonical(&encoded)? }; - let entry: JournalCommitGraphEntry = decode_canonical(&encoded)?; let Some(next) = entry.first_parent_jumps.get(level - 1).copied() else { break; }; jumps.push(next); } } - let entry = JournalCommitGraphEntry { + Ok(JournalCommitGraphEntry { commit: id, generation: commit.generation, parents: commit.parents, first_parent_jumps: jumps, - }; - *tree = self - .graph_engine - .batch( - tree, - vec![Mutation::Upsert { - key: id.as_bytes().to_vec(), - val: encode_canonical(&entry)?, - }], - ) - .await?; - Ok(()) + }) + } + + /// Apply a chronological journal page with one tree rewrite per derived + /// index. Newly calculated graph entries remain available in memory while + /// later entries build their binary-lifting skip pointers. + async fn index_events( + &self, + publisher: &ShardedBranchPublisher

, + node_tree: &mut Tree, + graph_tree: &mut Tree, + events: Vec, + ) -> Result<(usize, usize)> { + let keys = events + .iter() + .map(|event| event.new_target.as_bytes().to_vec()) + .collect::>(); + let existing = self.graph_engine.get_many(graph_tree, &keys).await?; + let mut scheduled = BTreeSet::new(); + let missing = events + .into_iter() + .zip(existing) + .filter_map(|(event, existing)| { + (existing.is_none() && scheduled.insert(event.new_target)).then_some(event) + }) + .collect::>(); + let loaded = stream::iter(missing.into_iter().map(|event| async move { + let object = publisher.load_commit_index(event.new_target).await?; + // A merge may publish roots containing nodes introduced by a + // secondary parent. Import those immutable pack locations into + // the target branch index with the merge event so the merged + // snapshot remains readable after source-branch deletion/reopen. + let mut secondary_parents = Vec::new(); + for parent in object.commit.parents.iter().skip(1).copied() { + secondary_parents.push((parent, publisher.load_commit_index(parent).await?)); + } + Ok::<_, Error>((event, object, secondary_parents)) + })) + .buffered(COMMIT_INDEX_LOAD_CONCURRENCY) + .collect::>() + .await; + + let mut node_mutations = Vec::new(); + let mut graph_mutations = Vec::new(); + let mut pending = BTreeMap::new(); + let mut indexed_commits = 0usize; + for loaded in loaded { + let (event, object, secondary_parents) = loaded?; + node_mutations.extend(self.node_pack_mutations( + event.new_target, + object.commit.node_pack.as_ref().map(|pack| pack.id), + object.toc.as_ref(), + object.node_region_offset, + )?); + for (parent, parent_index) in secondary_parents { + node_mutations.extend(self.node_pack_mutations( + parent, + parent_index.commit.node_pack.as_ref().map(|pack| pack.id), + parent_index.toc.as_ref(), + parent_index.node_region_offset, + )?); + } + let entry = self + .build_commit_graph_entry(graph_tree, &pending, event.new_target, object.commit) + .await?; + graph_mutations.push(Mutation::Upsert { + key: event.new_target.as_bytes().to_vec(), + val: encode_canonical(&entry)?, + }); + pending.insert(event.new_target, entry); + indexed_commits = indexed_commits.checked_add(1).ok_or_else(|| { + Error::new( + ErrorCode::InternalInvariant, + "indexed commit count overflow", + ) + })?; + } + let indexed_nodes = node_mutations.len(); + if !node_mutations.is_empty() { + *node_tree = self.node_engine.batch(node_tree, node_mutations).await?; + } + if !graph_mutations.is_empty() { + *graph_tree = self.graph_engine.batch(graph_tree, graph_mutations).await?; + } + Ok((indexed_commits, indexed_nodes)) } async fn collect_unindexed_events( diff --git a/extensions/s3/core/src/lib.rs b/extensions/s3/core/src/lib.rs index c056f9f6..89b121b9 100644 --- a/extensions/s3/core/src/lib.rs +++ b/extensions/s3/core/src/lib.rs @@ -42,7 +42,7 @@ pub use journal_indexes::{ pub use merge::{ MergeAdvancePage, MergeBaseCursor, MergeBasePage, MergeChange, MergeChangeCursor, MergeChangePage, MergeCleanupCursor, MergeCleanupPage, MergeConflict, MergeConflictCursor, - MergeConflictPage, MergeCursor, MergePhase, MergePolicy, MergeReceipt, + MergeConflictPage, MergeCursor, MergeDiffCursor, MergePhase, MergePolicy, MergeReceipt, }; pub use model::*; pub use object_plane::*; @@ -52,7 +52,10 @@ pub use operation_index::{ DEFAULT_OPERATION_INDEX_MAX_UNINDEXED_EVENTS, DEFAULT_OPERATION_INDEX_MERGE_FANOUT, }; pub use payload::ImmutablePayloadStore; -pub use prolly::Cid; +pub use prolly::{ + BoundaryInput, BoundaryRule, ChunkMeasure, ChunkingSpec, Cid, HashAlgorithm, NodeLayoutSpec, + TreeFormat, +}; pub use publication::{ AppliedBranchBarrier, CommitPublication, LoadedRef, PublicationJournalCursor, PublicationJournalEntry, PublicationJournalPage, ShardedBranchPublisher, @@ -65,12 +68,13 @@ pub use repository::{ validate_branch, BackupVerificationCursor, BackupVerificationPage, BackupVerificationReport, BranchCatalogPage, BranchHead, BranchIndexAdvanceReport, BranchIndexHealth, BranchIndexMaintenance, CommitClosureCursor, CommitClosurePage, CommitPage, CommitReceipt, - CommitSessionPutInput, DelimitedObjectPage, FsckCursor, FsckPage, FsckPhase, FsckReport, - HistoryCursor, ListObjectsPage, NodeCachePrewarmReport, ObjectData, ObjectDiff, - ObjectDiffCursor, ObjectDiffPage, ObjectRangeData, ObjectSummary, RefCatalogRepairPage, - RefMoveReceipt, RepairCursor, RepairPage, RepairPhase, RepairReport, Repository, - RepositoryOptions, RestoreCursor, RestorePage, RetentionPin, RetentionPinPage, - ShardAuthorityMaintenance, Tag, TagCatalogPage, TraversalBudget, VersionSummary, + CommitSessionPutInput, DelimitedObjectPage, FsckCleanupCursor, FsckCleanupPage, + FsckCleanupPhase, FsckCursor, FsckPage, FsckPhase, FsckReport, HistoryCursor, ListObjectsPage, + NodeCachePrewarmReport, ObjectData, ObjectDiff, ObjectDiffCursor, ObjectDiffPage, + ObjectRangeData, ObjectSummary, RefCatalogRepairPage, RefMoveReceipt, RepairCursor, RepairPage, + RepairPhase, RepairReport, Repository, RepositoryOptions, RestoreCursor, RestorePage, + RetentionPin, RetentionPinPage, ShardAuthorityMaintenance, Tag, TagCatalogPage, + TraversalBudget, VersionSummary, }; pub use runtime::*; pub use store::{NodeCacheSnapshot, ProllyObjectStore}; diff --git a/extensions/s3/core/src/merge.rs b/extensions/s3/core/src/merge.rs index 65342429..acd61846 100644 --- a/extensions/s3/core/src/merge.rs +++ b/extensions/s3/core/src/merge.rs @@ -27,6 +27,14 @@ pub enum MergePhase { ReadyToPublish, } +/// Restartable frontier for either a general structural comparison or the +/// exact ordered transition tree of a direct child commit. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub enum MergeDiffCursor { + Structural(prolly::StructuralDiffCursor), + Delta { after: Vec }, +} + /// Constant-size handle for a restartable repository merge. /// /// The potentially unbounded graph frontier, planned changes, conflicts, and @@ -47,13 +55,13 @@ pub struct MergeCursor { pub created_at_millis: u64, pub phase: MergePhase, pub plan_root: RootManifest, - pub ours_diff: Option, - pub theirs_diff: Option, + pub ours_diff: Option, + pub theirs_diff: Option, pub ours_pending: Option, pub theirs_pending: Option, pub ours_finished: bool, pub theirs_finished: bool, - pub version_diff: Option, + pub version_diff: Option, pub version_diff_finished: bool, pub build_after: Option>, pub final_objects: Option, diff --git a/extensions/s3/core/src/model.rs b/extensions/s3/core/src/model.rs index e097dd2b..5ed0fc3e 100644 --- a/extensions/s3/core/src/model.rs +++ b/extensions/s3/core/src/model.rs @@ -249,6 +249,13 @@ pub struct ProviderCapabilities { pub physical_versioning: PhysicalVersioning, pub conflicting_lifecycle_rule: bool, pub default_object_lock_retention: bool, + /// Whether qualification could inspect the bucket replication policy. + #[serde(default)] + pub replication_configuration_readable: bool, + /// Replication is operationally relevant but does not mutate the source + /// repository, so it is reported rather than rejected. + #[serde(default)] + pub replication_enabled: bool, pub max_object_bytes: u64, pub max_single_put_bytes: u64, } @@ -526,11 +533,14 @@ pub struct NodePackAttachment { #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub struct NodePack { + /// Metadata-only Prolly node container. User object bytes are never stored + /// in a node pack. pub format_digest: TreeFormatDigest, /// Sorted strictly by CID. pub entries: Vec, pub attachments: Vec, - /// Concatenated canonical node and attachment bytes. + /// Concatenated canonical metadata-node and attachment bytes. The field + /// name is retained as part of the version-1 wire format. pub payload: Vec, } @@ -791,7 +801,7 @@ impl NodePack { Ok(pack) } - pub fn object_payload_offset(object_prefix: &[u8]) -> Result { + pub fn object_node_region_offset(object_prefix: &[u8]) -> Result { if object_prefix.len() < 12 || &object_prefix[..8] != NODE_PACK_MAGIC { return Err(Error::new( ErrorCode::CorruptNode, @@ -876,7 +886,7 @@ impl NodePack { .map_err(|_| Error::new(ErrorCode::CorruptNode, "node-pack payload exceeds u64"))?, )?; for entry in &self.entries { - let bytes = self.payload_slice(entry.offset, entry.len)?; + let bytes = self.node_region_slice(entry.offset, entry.len)?; if sha256(bytes) != entry.sha256 || entry.cid.as_bytes() != entry.sha256 { return Err(Error::new( ErrorCode::CorruptNode, @@ -885,7 +895,7 @@ impl NodePack { } } for attachment in &self.attachments { - let bytes = self.payload_slice(attachment.offset, attachment.len)?; + let bytes = self.node_region_slice(attachment.offset, attachment.len)?; if sha256(bytes) != attachment.digest { return Err(Error::new( ErrorCode::CorruptCommit, @@ -901,10 +911,10 @@ impl NodePack { return Ok(None); }; let entry = &self.entries[index]; - Ok(Some(self.payload_slice(entry.offset, entry.len)?)) + Ok(Some(self.node_region_slice(entry.offset, entry.len)?)) } - fn payload_slice(&self, offset: u64, len: u32) -> Result<&[u8]> { + fn node_region_slice(&self, offset: u64, len: u32) -> Result<&[u8]> { let start = usize::try_from(offset) .map_err(|_| Error::new(ErrorCode::CorruptNode, "node pack offset overflow"))?; let end = start @@ -1527,12 +1537,12 @@ impl CommitObject { Self::new(commit, node_pack) } - pub fn node_payload_offset(encoded: &[u8]) -> Result> { + pub fn node_region_offset(encoded: &[u8]) -> Result> { let (_, pack_range) = Self::ranges(encoded)?; if pack_range.is_empty() { return Ok(None); } - let relative = NodePack::object_payload_offset(&encoded[pack_range.start..])?; + let relative = NodePack::object_node_region_offset(&encoded[pack_range.start..])?; Ok(Some(pack_range.start as u64 + relative)) } @@ -1571,9 +1581,10 @@ pub struct MutationIdentity { pub authority: AuthorityStamp, } -/// Repository payload binding. The physical path is explicit because -/// stores content under immutable derived keys instead of accumulating -/// provider versions at the logical user key. +/// Repository binding for one complete immutable payload object. The physical +/// path is explicit because Prolly stores whole content under derived keys +/// instead of accumulating provider versions at the logical user key. This +/// format deliberately has no chunk, segment, or byte-extent fields. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub struct PayloadBinding { pub path: ObjectPath, @@ -1581,22 +1592,11 @@ pub struct PayloadBinding { pub provider_etag: String, /// SHA-256 of this logical object's bytes. pub checksum_sha256: [u8; 32], - /// SHA-256 of the physical pack. Absent for legacy/direct payloads. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub pack_checksum_sha256: Option<[u8; 32]>, - /// Inclusive logical extent inside the physical pack. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub pack_range: Option<(u64, u64)>, } impl PayloadBinding { pub fn validate(&self) -> Result<()> { - let pack_shape_valid = match (self.pack_checksum_sha256, self.pack_range) { - (None, None) => true, - (Some(physical), Some((start, end))) => physical != [0; 32] && start <= end, - _ => false, - }; - if self.provider_etag.is_empty() || self.checksum_sha256 == [0; 32] || !pack_shape_valid { + if self.provider_etag.is_empty() || self.checksum_sha256 == [0; 32] { return Err(Error::new( ErrorCode::CorruptCommit, "physical payload binding is malformed", @@ -1604,14 +1604,6 @@ impl PayloadBinding { } Ok(()) } - - pub fn physical_checksum_sha256(&self) -> [u8; 32] { - self.pack_checksum_sha256.unwrap_or(self.checksum_sha256) - } - - pub fn is_packed(&self) -> bool { - self.pack_range.is_some() - } } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -1815,6 +1807,12 @@ pub enum CommitSessionState { pub struct CommitSessionCheckpoint { pub session: CommitSessionManifest, pub sequence: u64, + /// Number of unique logical mutations represented after applying this + /// checkpoint window and every preceding window in sequence order. + pub total_mutations: u64, + /// Canonically ordered mutations changed since the preceding checkpoint. + /// Resume folds these bounded windows by key; earlier windows are never + /// serialized again. pub mutations: Vec, pub state: CommitSessionState, } @@ -1822,12 +1820,21 @@ pub struct CommitSessionCheckpoint { impl CommitSessionCheckpoint { pub fn validate(&self, repository: RepositoryId, max_mutations: usize) -> Result<()> { self.session.validate(repository)?; - if self.mutations.len() > max_mutations { + if self.total_mutations > max_mutations as u64 + || self.mutations.len() > max_mutations + || self.mutations.len() as u64 > self.total_mutations + { return Err(Error::new( ErrorCode::InvalidLimit, "commit-session checkpoint exceeds the mutation limit", )); } + if self.sequence == 0 && (self.total_mutations != 0 || !self.mutations.is_empty()) { + return Err(Error::new( + ErrorCode::CorruptCommit, + "initial commit-session checkpoint must be empty", + )); + } let mut previous = None; for mutation in &self.mutations { let key = mutation.key(); diff --git a/extensions/s3/core/src/object_plane.rs b/extensions/s3/core/src/object_plane.rs index 858a7179..bfb5a0d9 100644 --- a/extensions/s3/core/src/object_plane.rs +++ b/extensions/s3/core/src/object_plane.rs @@ -93,6 +93,20 @@ pub struct ImmutableFilePut { pub expected_sha256: [u8; 32], } +/// Transfer one complete immutable object between object planes. +/// +/// The provider adapter owns the transfer mechanism. Repository code supplies +/// only whole-object identity and never observes multipart IDs, parts, ranges, +/// or resumable-transfer state. +#[derive(Clone, Debug)] +pub struct ImmutableTransfer { + pub source_path: ObjectPath, + pub source_physical_version: Option, + pub destination_path: ObjectPath, + pub size: u64, + pub expected_sha256: [u8; 32], +} + #[derive(Clone, Debug, PartialEq, Eq)] pub enum ImmutablePutOutcome { Created(StoredMetadata), @@ -170,6 +184,21 @@ pub trait ObjectPlane: Send + Sync + 'static { }) .await } + + /// Transfer a complete immutable object without exposing transfer parts to + /// repository code. Providers should override this with a native copy or + /// external transfer-manager handoff when available. + async fn transfer_immutable_from( + &self, + source: &Q, + request: ImmutableTransfer, + ) -> Result + where + Self: Sized, + { + transfer_immutable_via_get(self, source, request).await + } + async fn load_mutable(&self, path: &ObjectPath) -> Result>; async fn compare_exchange(&self, request: CompareExchange) -> Result; async fn list(&self, request: ListRequest) -> Result; @@ -200,15 +229,50 @@ pub trait ObjectPlane: Send + Sync + 'static { } } +async fn transfer_immutable_via_get( + destination: &P, + source: &Q, + request: ImmutableTransfer, +) -> Result { + let stored = source + .get(GetRequest { + path: request.source_path, + range: None, + physical_version: request.source_physical_version, + }) + .await? + .ok_or_else(|| Error::new(ErrorCode::MissingClosure, "transfer source is missing"))?; + if stored.metadata.delete_marker + || stored.bytes.len() as u64 != request.size + || sha256(&stored.bytes) != request.expected_sha256 + { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "transfer source does not match its whole-object identity", + )); + } + destination + .put_immutable(ImmutablePut { + path: request.destination_path, + bytes: stored.bytes, + expected_sha256: request.expected_sha256, + }) + .await +} + #[derive(Clone)] pub struct MemoryObjectPlane { inner: Arc>, versioned: bool, requests: Arc, conflict_after_next_compare_exchange: Arc, + immutable_put_delay_millis: Arc, + immutable_put_in_flight: Arc, + immutable_put_max_in_flight: Arc, compare_exchange_delay_millis: Arc, compare_exchange_in_flight: Arc, compare_exchange_max_in_flight: Arc, + delete_protected_paths: Arc>>, } struct InFlightGuard(Arc); @@ -224,6 +288,7 @@ struct MemoryRequestCounters { get: AtomicU64, head: AtomicU64, immutable_put: AtomicU64, + immutable_transfer: AtomicU64, compare_exchange: AtomicU64, list: AtomicU64, delete_exact: AtomicU64, @@ -234,6 +299,7 @@ pub struct MemoryRequestSnapshot { pub get: u64, pub head: u64, pub immutable_put: u64, + pub immutable_transfer: u64, pub compare_exchange: u64, pub list: u64, pub delete_exact: u64, @@ -244,6 +310,7 @@ impl MemoryRequestSnapshot { self.get + self.head + self.immutable_put + + self.immutable_transfer + self.compare_exchange + self.list + self.delete_exact @@ -269,12 +336,31 @@ impl MemoryObjectPlane { versioned, requests: Arc::new(MemoryRequestCounters::default()), conflict_after_next_compare_exchange: Arc::new(AtomicBool::new(false)), + immutable_put_delay_millis: Arc::new(AtomicU64::new(0)), + immutable_put_in_flight: Arc::new(AtomicU64::new(0)), + immutable_put_max_in_flight: Arc::new(AtomicU64::new(0)), compare_exchange_delay_millis: Arc::new(AtomicU64::new(0)), compare_exchange_in_flight: Arc::new(AtomicU64::new(0)), compare_exchange_max_in_flight: Arc::new(AtomicU64::new(0)), + delete_protected_paths: Arc::new(RwLock::new(std::collections::BTreeSet::new())), } } + /// Test hook for proving immutable payload preparation overlaps before an + /// ordered publication lane. + pub fn set_immutable_put_delay_millis(&self, delay_millis: u64) { + self.immutable_put_delay_millis + .store(delay_millis, Ordering::Relaxed); + } + + pub fn max_immutable_puts_in_flight(&self) -> u64 { + self.immutable_put_max_in_flight.load(Ordering::Relaxed) + } + + pub fn reset_immutable_put_concurrency(&self) { + self.immutable_put_max_in_flight.store(0, Ordering::Relaxed); + } + /// Test hook for proving independent mutable-object publications overlap. pub fn set_compare_exchange_delay_millis(&self, delay_millis: u64) { self.compare_exchange_delay_millis @@ -290,6 +376,21 @@ impl MemoryObjectPlane { .store(0, Ordering::Relaxed); } + /// Test hook modeling provider retention or Object Lock legal hold. + pub fn protect_exact_deletes(&self, path: ObjectPath) { + self.delete_protected_paths + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .insert(path); + } + + pub fn allow_exact_deletes(&self, path: &ObjectPath) { + self.delete_protected_paths + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .remove(path); + } + /// Test hook that applies the next successful CAS, then reports a /// conflict containing the applied value. This models a provider or SDK /// retry whose first wire attempt committed but whose response was lost. @@ -304,6 +405,7 @@ impl MemoryObjectPlane { get: load(&self.requests.get), head: load(&self.requests.head), immutable_put: load(&self.requests.immutable_put), + immutable_transfer: load(&self.requests.immutable_transfer), compare_exchange: load(&self.requests.compare_exchange), list: load(&self.requests.list), delete_exact: load(&self.requests.delete_exact), @@ -315,6 +417,7 @@ impl MemoryObjectPlane { &self.requests.get, &self.requests.head, &self.requests.immutable_put, + &self.requests.immutable_transfer, &self.requests.compare_exchange, &self.requests.list, &self.requests.delete_exact, @@ -416,6 +519,14 @@ impl ObjectPlane for MemoryObjectPlane { async fn put_immutable(&self, request: ImmutablePut) -> Result { self.requests.immutable_put.fetch_add(1, Ordering::Relaxed); + let in_flight = self.immutable_put_in_flight.fetch_add(1, Ordering::Relaxed) + 1; + self.immutable_put_max_in_flight + .fetch_max(in_flight, Ordering::Relaxed); + let _in_flight = InFlightGuard(self.immutable_put_in_flight.clone()); + let delay = self.immutable_put_delay_millis.load(Ordering::Relaxed); + if delay > 0 { + tokio::time::sleep(std::time::Duration::from_millis(delay)).await; + } if sha256(&request.bytes) != request.expected_sha256 { return Err(Error::new( ErrorCode::ChecksumMismatch, @@ -453,6 +564,17 @@ impl ObjectPlane for MemoryObjectPlane { Ok(ImmutablePutOutcome::Created(metadata)) } + async fn transfer_immutable_from( + &self, + source: &Q, + request: ImmutableTransfer, + ) -> Result { + self.requests + .immutable_transfer + .fetch_add(1, Ordering::Relaxed); + transfer_immutable_via_get(self, source, request).await + } + async fn load_mutable(&self, path: &ObjectPath) -> Result> { self.get(GetRequest { path: path.clone(), @@ -598,6 +720,17 @@ impl ObjectPlane for MemoryObjectPlane { version: PhysicalVersion, ) -> Result { self.requests.delete_exact.fetch_add(1, Ordering::Relaxed); + if self + .delete_protected_paths + .read() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "memory lock poisoned"))? + .contains(path) + { + return Err(Error::new( + ErrorCode::PermissionDenied, + "exact delete is blocked by provider retention", + )); + } let mut state = self .inner .write() diff --git a/extensions/s3/core/src/operation_index.rs b/extensions/s3/core/src/operation_index.rs index 8555442c..d2c91dc6 100644 --- a/extensions/s3/core/src/operation_index.rs +++ b/extensions/s3/core/src/operation_index.rs @@ -1,4 +1,7 @@ -use std::{collections::BTreeMap, sync::Arc}; +use std::{ + collections::BTreeMap, + sync::{Arc, RwLock}, +}; use serde::{Deserialize, Serialize}; @@ -58,6 +61,14 @@ pub struct SegmentedOperationIndex { leaf_entries: usize, merge_fanout: usize, max_unindexed_events: usize, + suffix_cache: Arc>>, +} + +#[derive(Clone)] +struct OperationSuffixCache { + checkpoint: PublicationEventId, + frontier: PublicationEventId, + entries: BTreeMap, } struct LoadedHead { @@ -118,6 +129,7 @@ impl SegmentedOperationIndex

{ leaf_entries, merge_fanout, max_unindexed_events, + suffix_cache: Arc::new(RwLock::new(BTreeMap::new())), }) } @@ -493,30 +505,25 @@ impl SegmentedOperationIndex

{ }; self.validate_head(&head.value)?; if head.value.checkpoint != current.value.publication { - let events = self - .collect_unindexed_events( + if let Some(entry) = self + .lookup_unindexed_cached( publisher, branch, head.value.checkpoint, current.value.publication, + operation, ) - .await?; - if let Some(event) = events.into_iter().find(|event| { - event.operation == operation - && self.retention.contains( + .await? + { + return Ok(self + .retention + .contains( current.value.generation, now_millis, - event.generation, - event.created_at_millis, + entry.generation, + entry.created_at_millis, ) - }) { - return Ok(Some(IndexedOperation { - operation, - publication: event.id()?, - target: event.new_target, - generation: event.generation, - created_at_millis: event.created_at_millis, - })); + .then_some(entry)); } } for level in &head.value.levels { @@ -549,6 +556,66 @@ impl SegmentedOperationIndex

{ Ok(None) } + /// Cache the branch-local journal suffix that has not reached the durable + /// operation index yet. A hot writer normally adds one event between + /// lookups, so retry reconciliation reads only that new event instead of + /// repeatedly walking the complete maintenance lag. + async fn lookup_unindexed_cached( + &self, + publisher: &ShardedBranchPublisher

, + branch: &str, + checkpoint: PublicationEventId, + current: PublicationEventId, + operation: OperationId, + ) -> Result> { + let cached = self + .suffix_cache + .read() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "operation cache poisoned"))? + .get(branch) + .cloned(); + let (mut entries, frontier) = match cached { + Some(cached) if cached.checkpoint == checkpoint => (cached.entries, cached.frontier), + _ => (BTreeMap::new(), checkpoint), + }; + if frontier != current { + let events = self + .collect_unindexed_events(publisher, branch, frontier, current) + .await?; + for event in events { + entries.insert( + event.operation, + IndexedOperation { + operation: event.operation, + publication: event.id()?, + target: event.new_target, + generation: event.generation, + created_at_millis: event.created_at_millis, + }, + ); + } + if entries.len() > self.max_unindexed_events { + return Err(Error::new( + ErrorCode::HistoryLimitExceeded, + "operation-index lag exceeds its bounded catch-up window; run resumable rebuild", + )); + } + } + let found = entries.get(&operation).cloned(); + self.suffix_cache + .write() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "operation cache poisoned"))? + .insert( + branch.to_string(), + OperationSuffixCache { + checkpoint, + frontier: current, + entries, + }, + ); + Ok(found) + } + async fn collect_unindexed_events( &self, publisher: &ShardedBranchPublisher

, diff --git a/extensions/s3/core/src/payload.rs b/extensions/s3/core/src/payload.rs index d0a2a813..b4d8919b 100644 --- a/extensions/s3/core/src/payload.rs +++ b/extensions/s3/core/src/payload.rs @@ -1,8 +1,9 @@ use std::{path::PathBuf, sync::Arc}; use crate::{ - codec::sha256, GetRequest, ImmutableFilePut, ImmutablePut, ObjectPath, ObjectPlane, - PayloadBinding, PhysicalVersion, RepositoryId, Result, + codec::sha256, GetRequest, ImmutableFilePut, ImmutablePut, ImmutablePutOutcome, + ImmutableTransfer, ObjectPath, ObjectPlane, PayloadBinding, PhysicalVersion, RepositoryId, + Result, }; #[derive(Clone)] @@ -41,8 +42,6 @@ impl ImmutablePayloadStore

{ provider_version_id: metadata.token.version_id, provider_etag: metadata.token.etag, checksum_sha256, - pack_checksum_sha256: None, - pack_range: None, }; binding.validate()?; Ok(binding) @@ -79,8 +78,6 @@ impl ImmutablePayloadStore

{ provider_version_id: metadata.token.version_id, provider_etag: metadata.token.etag, checksum_sha256, - pack_checksum_sha256: None, - pack_range: None, }; binding.validate()?; Ok(binding) @@ -88,11 +85,7 @@ impl ImmutablePayloadStore

{ pub async fn get(&self, binding: &PayloadBinding) -> Result> { binding.validate()?; - let expected_path = if binding.is_packed() { - self.pack_path(binding.physical_checksum_sha256())? - } else { - self.path(binding.checksum_sha256)? - }; + let expected_path = self.path(binding.checksum_sha256)?; if binding.path != expected_path { return Err(crate::Error::new( crate::ErrorCode::CorruptContent, @@ -110,7 +103,7 @@ impl ImmutablePayloadStore

{ .plane .get(GetRequest { path: binding.path.clone(), - range: binding.pack_range.map(|(start, end)| start..=end), + range: None, physical_version, }) .await? @@ -126,77 +119,60 @@ impl ImmutablePayloadStore

{ Ok(stored.bytes) } - pub async fn put_pack(&self, objects: Vec<([u8; 32], Vec)>) -> Result> { - if objects.is_empty() { - return Ok(Vec::new()); - } - let total = objects.iter().try_fold(0_usize, |total, (_, bytes)| { - total.checked_add(bytes.len()).ok_or_else(|| { - crate::Error::new( - crate::ErrorCode::EntityTooLarge, - "payload pack size overflow", - ) - }) - })?; - let mut packed = Vec::with_capacity(total); - let mut ranges = Vec::with_capacity(objects.len()); - let mut extents = std::collections::BTreeMap::new(); - for (logical_checksum, bytes) in objects { - if sha256(&bytes) != logical_checksum { - return Err(crate::Error::new( - crate::ErrorCode::ChecksumMismatch, - "payload pack input does not match its logical checksum", - )); - } - if let Some((start, end)) = extents.get(&logical_checksum).copied() { - ranges.push((logical_checksum, start, end)); - continue; - } - let start = u64::try_from(packed.len()).map_err(|_| { - crate::Error::new( - crate::ErrorCode::EntityTooLarge, - "payload pack offset overflow", - ) - })?; - packed.extend_from_slice(&bytes); - let end = u64::try_from(packed.len() - 1).map_err(|_| { - crate::Error::new( - crate::ErrorCode::EntityTooLarge, - "payload pack extent overflow", - ) - })?; - extents.insert(logical_checksum, (start, end)); - ranges.push((logical_checksum, start, end)); + /// Transfer one complete immutable payload through the provider boundary. + /// The repository never observes transfer parts or reconstructs the body. + pub async fn transfer_from( + &self, + source: &ImmutablePayloadStore, + binding: &PayloadBinding, + size: u64, + ) -> Result { + binding.validate()?; + if binding.path != source.expected_path(binding)? { + return Err(crate::Error::new( + crate::ErrorCode::CorruptContent, + "transfer source binding path does not match its content checksum", + )); } - let pack_checksum = sha256(&packed); - let path = self.pack_path(pack_checksum)?; + let path = self.path(binding.checksum_sha256)?; + let source_physical_version = + binding + .provider_version_id + .as_ref() + .map(|version_id| PhysicalVersion::Versioned { + version_id: version_id.clone(), + }); let outcome = self .plane - .put_immutable(ImmutablePut { - path: path.clone(), - expected_sha256: pack_checksum, - bytes: packed, - }) + .transfer_immutable_from( + source.plane.as_ref(), + ImmutableTransfer { + source_path: binding.path.clone(), + source_physical_version, + destination_path: path.clone(), + size, + expected_sha256: binding.checksum_sha256, + }, + ) .await?; let metadata = match outcome { - crate::ImmutablePutOutcome::Created(metadata) - | crate::ImmutablePutOutcome::AlreadyPresent(metadata) => metadata, + ImmutablePutOutcome::Created(metadata) + | ImmutablePutOutcome::AlreadyPresent(metadata) => metadata, }; - ranges - .into_iter() - .map(|(logical_checksum, start, end)| { - let binding = PayloadBinding { - path: path.clone(), - provider_version_id: metadata.token.version_id.clone(), - provider_etag: metadata.token.etag.clone(), - checksum_sha256: logical_checksum, - pack_checksum_sha256: Some(pack_checksum), - pack_range: Some((start, end)), - }; - binding.validate()?; - Ok(binding) - }) - .collect() + if metadata.len != size || metadata.sha256 != binding.checksum_sha256 { + return Err(crate::Error::new( + crate::ErrorCode::ChecksumMismatch, + "transferred object does not match its whole-object identity", + )); + } + let transferred = PayloadBinding { + path, + provider_version_id: metadata.token.version_id, + provider_etag: metadata.token.etag, + checksum_sha256: binding.checksum_sha256, + }; + transferred.validate()?; + Ok(transferred) } pub fn path(&self, checksum_sha256: [u8; 32]) -> Result { @@ -211,23 +187,7 @@ impl ImmutablePayloadStore

{ )) } - pub fn pack_path(&self, checksum_sha256: [u8; 32]) -> Result { - let encoded = hex::encode(checksum_sha256); - ObjectPath::new(format!( - "{}/payload-packs/{}/sha256/{}/{}/{}", - self.prefix, - hex::encode(self.repository.as_bytes()), - &encoded[..2], - &encoded[2..4], - encoded - )) - } - pub fn expected_path(&self, binding: &PayloadBinding) -> Result { - if binding.is_packed() { - self.pack_path(binding.physical_checksum_sha256()) - } else { - self.path(binding.checksum_sha256) - } + self.path(binding.checksum_sha256) } } diff --git a/extensions/s3/core/src/publication.rs b/extensions/s3/core/src/publication.rs index 31b71418..c524ab90 100644 --- a/extensions/s3/core/src/publication.rs +++ b/extensions/s3/core/src/publication.rs @@ -18,11 +18,11 @@ pub struct LoadedRef { } /// Commit data needed by derived indexes without downloading the immutable -/// node-pack payload. Every node remains independently CID-verified when read. +/// node region. Every node remains independently CID-verified when read. pub(crate) struct CommitIndexView { pub(crate) commit: BucketCommit, pub(crate) toc: Option, - pub(crate) payload_offset: Option, + pub(crate) node_region_offset: Option, } #[derive(Clone, Debug)] @@ -889,7 +889,7 @@ impl ShardedBranchPublisher

{ return Ok(CommitIndexView { commit, toc: None, - payload_offset: None, + node_region_offset: None, }); }; if pack_len != expected.object_len { @@ -953,7 +953,7 @@ impl ShardedBranchPublisher

{ Ok(CommitIndexView { commit, toc: Some(toc), - payload_offset: Some(toc_end), + node_region_offset: Some(toc_end), }) } @@ -975,6 +975,7 @@ impl ShardedBranchPublisher

{ }) .await; match outcome { + Err(error) if error.code == ErrorCode::PreconditionFailed => Err(error), Err(error) => { if let Some(current) = self.plane.load_mutable(&path).await? { if current.bytes == bytes { diff --git a/extensions/s3/core/src/repository.rs b/extensions/s3/core/src/repository.rs index 397abcd4..a79e90b1 100644 --- a/extensions/s3/core/src/repository.rs +++ b/extensions/s3/core/src/repository.rs @@ -3,7 +3,7 @@ use std::{ path::PathBuf, sync::{ atomic::{AtomicBool, AtomicU64, Ordering}, - Arc, OnceLock, RwLock, Weak, + Arc, Mutex, OnceLock, RwLock, Weak, }, time::Duration, }; @@ -15,7 +15,7 @@ use prolly::{ }; use sha2::Sha256; -use crate::gc::{GcCandidate, GcCoordinator, GcDirtyRoot, GcNodeWork}; +use crate::gc::{GcCandidate, GcCoordinator, GcDirtyRoot, GcNodeWork, GcPublicationTicket}; use crate::merge::{MergeBaseCandidate, MergePlanEntry, MergeQueueEntry, MergeSeenEntry}; use crate::publication::BranchMovement; use crate::store::{LocatedPackedNode, NodeCacheNamespace, NodeLocator, PreparedNodePack}; @@ -34,15 +34,15 @@ use crate::{ LogicalObjectVersionBody, LogicalObjectVersionKind, MemoryNodeCache, MergeAdvancePage, MergeBaseCursor, MergeBasePage, MergeChange, MergeChangeCursor, MergeChangePage, MergeCleanupCursor, MergeCleanupPage, MergeConflict, MergeConflictCursor, MergeConflictPage, - MergeCursor, MergePhase, MergePolicy, MergeReceipt, MutableControlKind, MutableControlObserver, - MutableControlStore, MutationIdentity, NodeCache, ObjectHeaders, ObjectPath, ObjectPlane, - ObjectTransition, ObjectVersion, ObjectVersionId, ObjectVersionOrder, OperationId, - OperationIndexAdvanceReport, OperationIndexRebuildCursor, OperationIndexRebuildStep, - PendingHistoryTransferCommit, PhysicalVersion, ProllyObjectStore, ProviderPerKeyVersionLimit, - RandomIdSource, RefCatalogCursor, RefGeneration, RefKind, RepositoryFormat, Result, - RootManifest, SegmentedOperationIndex, ShardWriterAuthority, ShardedBranchPublisher, - ShardedRefCatalog, StagedMutation, StagedMutationBody, StagedPut, SystemClock, TagStore, - TakeoverRequest, + MergeCursor, MergeDiffCursor, MergePhase, MergePolicy, MergeReceipt, MutableControlKind, + MutableControlObserver, MutableControlStore, MutationIdentity, NodeCache, ObjectHeaders, + ObjectPath, ObjectPlane, ObjectTransition, ObjectVersion, ObjectVersionId, ObjectVersionOrder, + OperationId, OperationIndexAdvanceReport, OperationIndexRebuildCursor, + OperationIndexRebuildStep, PendingHistoryTransferCommit, PhysicalVersion, ProllyObjectStore, + ProviderPerKeyVersionLimit, RandomIdSource, RefCatalogCursor, RefGeneration, RefKind, + RepositoryFormat, Result, RootManifest, SegmentedOperationIndex, ShardWriterAuthority, + ShardedBranchPublisher, ShardedRefCatalog, StagedMutation, StagedMutationBody, StagedPut, + SystemClock, TagStore, TakeoverRequest, }; /// Keep ordinary commit descriptors small enough for one bounded metadata @@ -230,7 +230,13 @@ pub struct ObjectDiffCursor { branch: String, from: CommitId, to: CommitId, - traversal: prolly::StructuralDiffCursor, + traversal: ObjectDiffTraversal, +} + +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +enum ObjectDiffTraversal { + Structural(prolly::StructuralDiffCursor), + Delta { after: Vec }, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -305,6 +311,7 @@ pub enum FsckPhase { DiscoverCommits, VerifyObjects, VerifyVersions, + VerifyPayloads, Complete, } @@ -317,6 +324,12 @@ pub struct FsckReport { pub payloads_verified: u64, pub payload_bytes_verified: u64, pub deep_content_bytes_verified: u64, + #[serde(default)] + pub physical_payloads_verified: u64, + #[serde(default)] + pub physical_payload_bytes_verified: u64, + #[serde(default)] + pub deep_physical_bytes_read: u64, } #[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] @@ -324,6 +337,14 @@ pub struct FsckCursor { pub repository: crate::RepositoryId, pub branch: String, pub snapshot: CommitId, + pub snapshot_objects: RootManifest, + pub snapshot_versions: RootManifest, + pub job: OperationId, + /// Monotonic generation of the durable checkpoint. This fences stale + /// workers after another process resumes and advances the same job. + #[serde(default)] + pub checkpoint_generation: u64, + pub payloads: RootManifest, pub closure: CommitClosureCursor, pub phase: FsckPhase, pub after: Option>, @@ -331,6 +352,12 @@ pub struct FsckCursor { pub report: FsckReport, } +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +struct FsckPhysicalPayload { + binding: crate::PayloadBinding, + size: u64, +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct FsckPage { pub cursor: FsckCursor, @@ -338,6 +365,33 @@ pub struct FsckPage { pub complete: bool, } +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub enum FsckCleanupPhase { + PayloadWork, + ClosureWork, + CheckpointHistory, + CheckpointCurrent, + Complete, +} + +/// Restart-safe cleanup descriptor for one completed fsck job. Cleanup may be +/// restarted from this initial descriptor because every page exact-deletes +/// immutable physical versions and the durable checkpoint is removed last. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct FsckCleanupCursor { + pub repository: crate::RepositoryId, + pub job: OperationId, + pub closure_traversal: OperationId, + pub phase: FsckCleanupPhase, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FsckCleanupPage { + pub cursor: FsckCleanupCursor, + pub deleted: usize, + pub complete: bool, +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct RefMoveReceipt { pub branch: String, @@ -585,8 +639,16 @@ struct GcDirtyRootObserver { plane: Arc

, prefix: String, repository: crate::RepositoryId, - active_epoch: Arc>>, - sequence: Arc, + instance: OperationId, + clock: Arc, + ticket_ttl_millis: u64, + tickets: Mutex>, +} + +struct HeldPublicationTicket { + path: ObjectPath, + version: PhysicalVersion, + references: usize, } struct GcProcessState { @@ -622,68 +684,128 @@ impl MutableControlObserver for GcDirtyRootObserver

{ kind: MutableControlKind, request: &CompareExchange, ) -> Result<()> { - let epoch = *self - .active_epoch - .read() - .map_err(|_| Error::new(ErrorCode::InternalInvariant, "active GC lock poisoned"))?; - let Some(epoch) = epoch else { + if !matches!( + kind, + MutableControlKind::BranchRef | MutableControlKind::TagRef + ) { return Ok(()); - }; - let (namespace, name, target, previous_target, generation, operation, created_at_millis) = - match kind { - MutableControlKind::BranchRef => { - let value: crate::RefValue = decode_canonical(&request.bytes)?; - ( - "branch", - value.inline_reflog.branch, - value.target, - value.previous_target, - value.generation, - value.operation, - value.updated_at_millis, - ) - } - MutableControlKind::TagRef => { - let value: crate::TagValue = decode_canonical(&request.bytes)?; - ( - "tag", - value.inline_reflog.branch, - value.target, - value.previous_target, - value.generation, - value.operation, - value.updated_at_millis, - ) - } - _ => return Ok(()), - }; - let sequence = self - .sequence - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { - current.checked_add(1) - }) - .map(|previous| previous + 1) - .map_err(|_| Error::new(ErrorCode::InternalInvariant, "GC sequence overflow"))?; - let event = GcDirtyRoot { + } + let request_digest = publication_ticket_digest(request); + let now = self.clock.now_millis()?; + let ticket = GcPublicationTicket { repository: self.repository, - epoch, - sequence, - namespace: namespace.to_string(), - name, - target, - previous_target, - generation, - operation, - created_at_millis, + instance: self.instance, + request_digest, + expires_at_millis: now.checked_add(self.ticket_ttl_millis).ok_or_else(|| { + Error::new( + ErrorCode::InvalidLimit, + "publication ticket expiry overflow", + ) + })?, }; - let bytes = encode_canonical(&event)?; - self.plane + let bytes = encode_canonical(&ticket)?; + let path = gc_publication_ticket_path( + &self.prefix, + self.repository, + self.instance, + request_digest, + )?; + let stored = self + .plane .put_immutable(crate::ImmutablePut { - path: gc_dirty_root_path(&self.prefix, &event, crate::codec::sha256(&bytes))?, + path: path.clone(), expected_sha256: crate::codec::sha256(&bytes), bytes, }) .await?; + let metadata = match stored { + crate::ImmutablePutOutcome::Created(metadata) + | crate::ImmutablePutOutcome::AlreadyPresent(metadata) => metadata, + }; + let version = metadata.token.version_id.clone().map_or_else( + || PhysicalVersion::Unversioned { + token: Some(metadata.token.clone()), + }, + |version_id| PhysicalVersion::Versioned { version_id }, + ); + + let coordinator = self + .plane + .load_mutable(&gc_coordinator_path(&self.prefix)?) + .await? + .map(|stored| decode_canonical::(&stored.bytes)) + .transpose()?; + if coordinator.as_ref().is_some_and(|coordinator| { + coordinator.repository != self.repository + || coordinator.active_epoch.is_some() + || coordinator.admission_closed + }) { + let _ = self.plane.delete_exact(&path, version).await; + return Err(Error::new( + ErrorCode::PreconditionFailed, + "repository publication admission is closed for maintenance", + ) + .retry(crate::RetryAdvice::After(Duration::from_millis(250)))); + } + let mut tickets = self.tickets.lock().map_err(|_| { + Error::new( + ErrorCode::InternalInvariant, + "publication-ticket lock poisoned", + ) + })?; + tickets + .entry(request_digest) + .and_modify(|held| held.references = held.references.saturating_add(1)) + .or_insert(HeldPublicationTicket { + path, + version, + references: 1, + }); + Ok(()) + } + + async fn after_compare_exchange( + &self, + kind: MutableControlKind, + request: &CompareExchange, + ) -> Result<()> { + if !matches!( + kind, + MutableControlKind::BranchRef | MutableControlKind::TagRef + ) { + return Ok(()); + } + let digest = publication_ticket_digest(request); + let release = { + let mut tickets = self.tickets.lock().map_err(|_| { + Error::new( + ErrorCode::InternalInvariant, + "publication-ticket lock poisoned", + ) + })?; + let Some(ticket) = tickets.get_mut(&digest) else { + return Ok(()); + }; + ticket.references = ticket.references.saturating_sub(1); + (ticket.references == 0) + .then(|| tickets.remove(&digest)) + .flatten() + }; + if let Some(ticket) = release { + match self + .plane + .delete_exact(&ticket.path, ticket.version) + .await? + { + DeleteOutcome::Deleted | DeleteOutcome::NotFound => {} + DeleteOutcome::TokenMismatch => { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "publication ticket changed before release", + )); + } + } + } Ok(()) } } @@ -706,6 +828,7 @@ pub struct Repository { ref_catalog: Arc>, operation_index: SegmentedOperationIndex

, journal_indexes: Arc>, + maintenance_controls: MutableControlStore

, locator: Arc>, permits: RwLock>, fenced_scopes: RwLock>, @@ -925,8 +1048,10 @@ impl Repository

{ plane: plane.clone(), prefix: options.repository_prefix.clone(), repository: format.repository_id, - active_epoch: active_gc_epoch.clone(), - sequence: gc_dirty_sequence.clone(), + instance: options.ids.operation(), + clock: options.clock.clone(), + ticket_ttl_millis: options.authority_lease_millis, + tickets: Mutex::new(BTreeMap::new()), }); let publisher = ShardedBranchPublisher::new_with_gc_controls( plane.clone(), @@ -984,6 +1109,11 @@ impl Repository

{ options.journal_index_max_unindexed_events, options.mutable_control_versions_to_retain, )?); + let maintenance_controls = MutableControlStore::new( + plane.clone(), + options.repository_prefix.clone(), + options.mutable_control_versions_to_retain, + )?; let locator = Arc::new(JournalNodeLocator { indexes: journal_indexes.clone(), branches: RwLock::new(BTreeSet::new()), @@ -1005,6 +1135,7 @@ impl Repository

{ ref_catalog, operation_index, journal_indexes, + maintenance_controls, locator, permits: RwLock::new(BTreeMap::new()), fenced_scopes: RwLock::new(BTreeSet::new()), @@ -1076,6 +1207,86 @@ impl Repository

{ }) } + /// Load only the root and a bounded number of upper tree levels. This + /// avoids a full-snapshot scan while removing repeated cold traversal of + /// shared internal paths before point-read traffic begins. + pub async fn prewarm_node_cache_levels( + &self, + branch: &str, + snapshot: CommitId, + levels: usize, + ) -> Result { + if !(1..=64).contains(&levels) { + return Err(Error::new( + ErrorCode::InvalidLimit, + "node-cache prewarm levels must be within 1..=64", + )); + } + validate_branch(branch)?; + self.locator.register(branch)?; + self.require_branch_indexes_ready(branch).await?; + let before = self.node_store.node_cache_snapshot(); + let commit = self.load_commit_object(snapshot).await?.commit; + let object_nodes = self + .prewarm_root_levels(&commit.state.objects, levels) + .await?; + let version_nodes = self + .prewarm_root_levels(&commit.state.versions, levels) + .await?; + Ok(NodeCachePrewarmReport { + snapshot, + object_nodes, + version_nodes, + before, + after: self.node_store.node_cache_snapshot(), + }) + } + + async fn prewarm_root_levels(&self, root: &RootManifest, levels: usize) -> Result { + let mut frontier = root.root.iter().cloned().collect::>(); + let mut seen = BTreeSet::new(); + let mut loaded_nodes = 0usize; + for _ in 0..levels { + frontier.retain(|cid| seen.insert(cid.clone())); + if frontier.is_empty() { + break; + } + let loaded = stream::iter(frontier.into_iter().map(|cid| async move { + let bytes = self.node_store.get(cid.as_bytes()).await?.ok_or_else(|| { + Error::new(ErrorCode::MissingClosure, "prewarm node is missing") + })?; + let node = Node::from_bytes_with_format(&bytes, &self.format.state_tree_format) + .map_err(|error| { + Error::new( + ErrorCode::CorruptNode, + format!("prewarm node could not be decoded: {error}"), + ) + })?; + Ok::<_, Error>(node) + })) + .buffered(32) + .collect::>() + .await; + let mut next = Vec::new(); + for node in loaded { + let node = node?; + loaded_nodes = loaded_nodes.saturating_add(1); + if !node.leaf { + for child in node.vals { + next.push(prolly::Cid(child.try_into().map_err(|_| { + Error::new( + ErrorCode::CorruptNode, + "prewarm internal child CID is invalid", + ) + })?)); + } + } + } + frontier = next; + } + Ok(loaded_nodes) + } + pub async fn head(&self, branch: &str) -> Result { self.locator.register(branch)?; Ok(self.publisher.load(branch).await?.value.target) @@ -1477,6 +1688,7 @@ impl Repository

{ let checkpoint = CommitSessionCheckpoint { session, sequence: 0, + total_mutations: 0, mutations: Vec::new(), state: CommitSessionState::Open, }; @@ -1488,12 +1700,19 @@ impl Repository

{ &self, session: &CommitSessionManifest, mutations: Vec, + total_mutations: usize, sequence: u64, ) -> Result { self.validate_commit_session(session).await?; let checkpoint = CommitSessionCheckpoint { session: session.clone(), sequence, + total_mutations: u64::try_from(total_mutations).map_err(|_| { + Error::new( + ErrorCode::InvalidLimit, + "checkpoint mutation count exceeds u64", + ) + })?, mutations: self.canonical_session_mutations(mutations, true)?, state: CommitSessionState::Open, }; @@ -1548,7 +1767,9 @@ impl Repository

{ Error::new(ErrorCode::InvalidLimit, "checkpoint sequence overflow") })?; checkpoint.session.identity.authority = permit.stamp(); - self.commit_sessions.save(&checkpoint).await?; + let mut adoption = checkpoint.clone(); + adoption.mutations.clear(); + self.commit_sessions.save(&adoption).await?; } Ok(checkpoint) } @@ -1557,12 +1778,19 @@ impl Repository

{ &self, session: CommitSessionManifest, mutations: Vec, + total_mutations: usize, sequence: u64, ) -> Result<()> { self.validate_commit_session(&session).await?; let checkpoint = CommitSessionCheckpoint { session, sequence, + total_mutations: u64::try_from(total_mutations).map_err(|_| { + Error::new( + ErrorCode::InvalidLimit, + "checkpoint mutation count exceeds u64", + ) + })?, mutations: self.canonical_session_mutations(mutations, true)?, state: CommitSessionState::Aborted, }; @@ -1588,6 +1816,17 @@ impl Repository

{ user_metadata: BTreeMap, ) -> Result { self.validate_commit_session(session).await?; + self.stage_commit_session_put_validated(key, bytes, headers, user_metadata) + .await + } + + async fn stage_commit_session_put_validated( + &self, + key: Vec, + bytes: Vec, + headers: ObjectHeaders, + user_metadata: BTreeMap, + ) -> Result { self.validate_key(&key)?; if bytes.len() as u64 > self.format.canonical_limits.max_object_bytes { return Err(Error::new( @@ -1617,9 +1856,9 @@ impl Repository

{ }) } - /// Stage a bounded input window, packing non-empty payloads up to 4 KiB - /// into deterministic immutable segments no larger than 4 MiB. Larger and - /// empty payloads retain the direct content-addressed representation. + /// Stage a bounded input window as independent whole provider objects. + /// Uploads run concurrently, while the returned mutations are ordered by + /// logical key so publication remains deterministic. pub async fn stage_commit_session_put_batch( &self, session: &CommitSessionManifest, @@ -1627,10 +1866,10 @@ impl Repository

{ concurrency: usize, ) -> Result> { self.validate_commit_session(session).await?; - if concurrency == 0 { + if concurrency == 0 || concurrency > 1_024 { return Err(Error::new( ErrorCode::InvalidLimit, - "payload staging concurrency is zero", + "payload staging concurrency is outside 1..=1024", )); } objects.sort_by(|left, right| left.0.cmp(&right.0)); @@ -1652,52 +1891,51 @@ impl Repository

{ } } - const SMALL_OBJECT_MAX: usize = 4 * 1024; - const PACK_MAX: usize = 4 * 1024 * 1024; - let mut packed_groups = Vec::new(); - let mut current = Vec::new(); - let mut current_bytes = 0_usize; - let mut direct = Vec::new(); - for object in objects { - if object.1.is_empty() || object.1.len() > SMALL_OBJECT_MAX { - direct.push(object); - continue; - } - if !current.is_empty() && current_bytes.saturating_add(object.1.len()) > PACK_MAX { - packed_groups.push(std::mem::take(&mut current)); - current_bytes = 0; - } - current_bytes = current_bytes.saturating_add(object.1.len()); - current.push(object); - } - if !current.is_empty() { - packed_groups.push(current); - } - - let mut staged = Vec::new(); - for group in packed_groups { - let pack_inputs = group - .iter() - .map(|(_, bytes, _, _)| (crate::codec::sha256(bytes), bytes.clone())) - .collect(); - let bindings = self.payloads.put_pack(pack_inputs).await?; - for ((key, bytes, headers, user_metadata), binding) in group.into_iter().zip(bindings) { - staged.push(staged_put(key, bytes, headers, user_metadata, binding)); - } - } - let direct = stream::iter(direct) + let staged = stream::iter(objects) .map(|(key, bytes, headers, user_metadata)| async move { - self.stage_commit_session_put(session, key, bytes, headers, user_metadata) + self.stage_commit_session_put_validated(key, bytes, headers, user_metadata) .await }) .buffer_unordered(concurrency) .collect::>() .await; - staged.extend(direct.into_iter().collect::>>()?); + let mut staged = staged.into_iter().collect::>>()?; staged.sort_by(|left, right| left.key().cmp(right.key())); Ok(staged) } + /// Stage one bounded group after validating its session once, while + /// preserving an independent result for every input. This is used by + /// ordered publication queues so one failed object does not discard + /// successfully prepared whole-object candidates. + pub async fn stage_commit_session_put_batch_results( + &self, + session: &CommitSessionManifest, + objects: Vec, + concurrency: usize, + ) -> Result>> { + self.validate_commit_session(session).await?; + if concurrency == 0 || concurrency > 1_024 { + return Err(Error::new( + ErrorCode::InvalidLimit, + "payload staging concurrency is outside 1..=1024", + )); + } + let mut staged = stream::iter(objects.into_iter().enumerate()) + .map(|(index, (key, bytes, headers, user_metadata))| async move { + ( + index, + self.stage_commit_session_put_validated(key, bytes, headers, user_metadata) + .await, + ) + }) + .buffer_unordered(concurrency) + .collect::>() + .await; + staged.sort_by_key(|(index, _)| *index); + Ok(staged.into_iter().map(|(_, result)| result).collect()) + } + #[allow(clippy::too_many_arguments)] pub async fn stage_commit_session_file( &self, @@ -1740,6 +1978,86 @@ impl Repository

{ }) } + /// Resolve the immutable destination for one externally uploaded provider + /// object. Prolly defines the whole-object identity and never observes or + /// manages the uploader's transfer parts. + pub async fn commit_session_payload_path( + &self, + session: &CommitSessionManifest, + key: &[u8], + size: u64, + checksum_sha256: [u8; 32], + ) -> Result { + self.validate_commit_session(session).await?; + self.validate_key(key)?; + if size == 0 || size > self.format.canonical_limits.max_object_bytes { + return Err(Error::new( + ErrorCode::EntityTooLarge, + "external object size is outside repository limits", + )); + } + if checksum_sha256 == [0; 32] { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "external object requires its complete SHA-256", + )); + } + self.payloads.path(checksum_sha256) + } + + /// Stage one completed provider object. The caller may use any provider + /// transfer manager, but only the final object enters the repository. + #[allow(clippy::too_many_arguments)] + pub async fn stage_commit_session_existing_object( + &self, + session: &CommitSessionManifest, + key: Vec, + size: u64, + checksum_sha256: [u8; 32], + checksum_md5: [u8; 16], + headers: ObjectHeaders, + user_metadata: BTreeMap, + ) -> Result { + let path = self + .commit_session_payload_path(session, &key, size, checksum_sha256) + .await?; + let current = self.plane.head(&path).await?.ok_or_else(|| { + Error::new( + ErrorCode::MissingClosure, + "completed external object is not readable", + ) + })?; + if current.len != size || current.sha256 != checksum_sha256 || current.delete_marker { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "completed external object does not match its whole-object identity", + )); + } + let binding = crate::PayloadBinding { + path, + provider_version_id: current.token.version_id, + provider_etag: current.token.etag, + checksum_sha256, + }; + binding.validate()?; + Ok(StagedMutation { + body: StagedMutationBody::Put(Box::new(StagedPut { + key, + size, + logical_etag: format!("\"{}\"", hex::encode(checksum_md5)), + checksums: Checksums { + md5: Some(checksum_md5), + sha256: Some(checksum_sha256), + algorithm_values: BTreeMap::new(), + }, + headers, + user_metadata, + tags: BTreeMap::new(), + binding, + })), + }) + } + pub async fn publish_commit_session( &self, session: CommitSessionManifest, @@ -1777,6 +2095,7 @@ impl Repository

{ { return Ok(receipt); } + self.require_publication_admission_open().await?; let permit = self.active_permit(&session.branch, now).await?; if permit.stamp() != session.identity.authority { return Err(Error::new( @@ -1960,6 +2279,7 @@ impl Repository

{ idempotent_replay: false, }) } + Err(error) if error.code != ErrorCode::OutcomeUnknown => Err(error), Err(error) => { if let Some(receipt) = self .reconcile_operation( @@ -1988,7 +2308,11 @@ impl Repository

{ user_metadata: BTreeMap, ) -> Result { let operation = self.options.ids.operation(); - self.put_object_with_operation(branch, key, bytes, headers, user_metadata, operation) + // The operation ID was allocated inside this process and cannot be a + // retry of an earlier publication. Avoid walking the unindexed journal + // to prove absence on every hot-branch write; caller-stable operation + // IDs still take the full reconciliation path below. + self.put_object_inner(branch, key, bytes, headers, user_metadata, operation, false) .await } @@ -2067,6 +2391,20 @@ impl Repository

{ headers: ObjectHeaders, user_metadata: BTreeMap, operation: OperationId, + ) -> Result { + self.put_object_inner(branch, key, bytes, headers, user_metadata, operation, true) + .await + } + + async fn put_object_inner( + &self, + branch: &str, + key: Vec, + bytes: Vec, + headers: ObjectHeaders, + user_metadata: BTreeMap, + operation: OperationId, + reconcile_before_publication: bool, ) -> Result { if !self.writable.load(Ordering::Acquire) { return Err(Error::new( @@ -2074,6 +2412,9 @@ impl Repository

{ "repository is read-only", )); } + if !reconcile_before_publication { + self.require_publication_admission_open().await?; + } self.validate_key(&key)?; if bytes.len() as u64 > self.format.canonical_limits.max_object_bytes { return Err(Error::new( @@ -2094,20 +2435,41 @@ impl Repository

{ &headers_bytes, &metadata_bytes, ]); - let _lane = self.lock_branch(branch).await; - let now = self.options.clock.now_millis()?; - if let Some(receipt) = self - .reconcile_operation(branch, operation, input_digest, now) - .await? - { - return Ok(receipt); + // Resolve idempotent replays and reject a stale authority before any + // payload bytes enter the object plane. Payload upload is deliberately + // outside the branch publication lane: independent callers can hash + // and upload immutable content concurrently while only the tree update + // and ref CAS remain serialized. + let prepare_now = self.options.clock.now_millis()?; + let prepare_permit = self.active_permit(branch, prepare_now).await?; + self.authority + .validate_active(&prepare_permit, prepare_now) + .await?; + self.require_branch_indexes_ready(branch).await?; + if reconcile_before_publication { + if let Some(receipt) = self + .reconcile_operation(branch, operation, input_digest, prepare_now) + .await? + { + return Ok(receipt); + } + self.require_publication_admission_open().await?; } + let binding = self.payloads.put(bytes).await?; + + let _lane = self.lock_branch(branch).await; + let now = self.options.clock.now_millis()?; let permit = self.active_permit(branch, now).await?; - // The authority check happens before payload bytes enter the object - // plane. A stale process therefore fails before its payload PUT. self.authority.validate_active(&permit, now).await?; + if reconcile_before_publication { + if let Some(receipt) = self + .reconcile_operation(branch, operation, input_digest, now) + .await? + { + return Ok(receipt); + } + } self.require_branch_indexes_ready(branch).await?; - let binding = self.payloads.put(bytes).await?; let current = self.publisher.load(branch).await?; let base = self.load_commit_object(current.value.target).await?.commit; @@ -2382,33 +2744,16 @@ impl Repository

{ version_id: version_id.clone(), }); let logical_range = *range.start()..=(*range.end()).min(size - 1); - let translated = if let Some((offset, pack_end)) = binding.pack_range { - let start = offset.checked_add(*logical_range.start()).ok_or_else(|| { - Error::new(ErrorCode::InvalidRange, "packed payload range overflow") - })?; - let end = offset - .checked_add(*logical_range.end()) - .filter(|end| *end <= pack_end) - .ok_or_else(|| { - Error::new( - ErrorCode::CorruptContent, - "packed payload range exceeds its logical extent", - ) - })?; - start..=end - } else { - logical_range.clone() - }; let stored = self .plane .get(GetRequest { path: binding.path.clone(), - range: Some(translated), + range: Some(logical_range.clone()), physical_version, }) .await? .ok_or_else(|| Error::new(ErrorCode::MissingClosure, "payload is missing"))?; - if stored.metadata.sha256 != binding.physical_checksum_sha256() + if stored.metadata.sha256 != binding.checksum_sha256 || stored.metadata.token.etag != binding.provider_etag || stored.metadata.token.version_id != binding.provider_version_id { @@ -2527,6 +2872,7 @@ impl Repository

{ { return Ok(receipt); } + self.require_publication_admission_open().await?; let permit = self.active_permit(branch, now).await?; self.authority.validate_active(&permit, now).await?; self.require_branch_indexes_ready(branch).await?; @@ -3197,7 +3543,7 @@ impl Repository

{ match state.as_deref() { Some([1]) => {} Some([0]) => { - let commit = self.load_commit_object(work.commit).await?.commit; + let commit = (*self.load_commit_metadata(work.commit).await?).clone(); mutations.push(Mutation::Upsert { key: seen_key, val: vec![1], @@ -3221,7 +3567,7 @@ impl Repository

{ )); } None => { - let commit = self.load_commit_object(work.commit).await?.commit; + let commit = (*self.load_commit_metadata(work.commit).await?).clone(); mutations.push(Mutation::Upsert { key: seen_key, val: vec![0], @@ -3264,6 +3610,30 @@ impl Repository

{ }) } + /// Fail fast before payload preparation or tree construction when another + /// process owns a durable repository-wide maintenance epoch. The mutable + /// control observer repeats this check under the ref-CAS barrier. + async fn require_publication_admission_open(&self) -> Result<()> { + let coordinator = self + .plane + .load_mutable(&gc_coordinator_path(&self.options.repository_prefix)?) + .await? + .map(|stored| decode_canonical::(&stored.bytes)) + .transpose()?; + if coordinator.as_ref().is_some_and(|coordinator| { + coordinator.repository != self.format.repository_id + || coordinator.active_epoch.is_some() + || coordinator.admission_closed + }) { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "repository publication admission is closed for maintenance", + ) + .retry(crate::RetryAdvice::After(Duration::from_millis(250)))); + } + Ok(()) + } + /// Start a bounded concurrent collector for immutable repository data. /// `grace_millis` must exceed the longest allowed unpublished operation. pub async fn start_gc(&self, grace_millis: u64) -> Result { @@ -3308,7 +3678,7 @@ impl Repository

{ return Err(error); } let work = self.gc_work_engine(epoch)?.create(); - Ok(GcCursor { + let cursor = GcCursor { repository: self.format.repository_id, epoch, cutoff_millis, @@ -3318,9 +3688,91 @@ impl Repository

{ dirty_sequence: self.gc_dirty_sequence.load(Ordering::Acquire), dirty_target_sequence: 0, initial_scan_complete: false, + publication_barrier_drained: false, sweep_after: None, report: crate::GcReport::default(), - }) + }; + self.persist_gc_cursor(&cursor).await?; + Ok(cursor) + } + + /// Resume the active repository-wide GC epoch after a worker restart. + pub async fn resume_gc(&self) -> Result> { + let Some(stored) = self + .plane + .load_mutable(&gc_coordinator_path(&self.options.repository_prefix)?) + .await? + else { + return Ok(None); + }; + let coordinator: GcCoordinator = decode_canonical(&stored.bytes)?; + let Some(epoch) = coordinator.active_epoch else { + return Ok(None); + }; + let stored = self + .plane + .load_mutable(&gc_cursor_path(&self.options.repository_prefix, epoch)?) + .await? + .ok_or_else(|| { + Error::new( + ErrorCode::MissingClosure, + "active GC epoch has no durable cursor checkpoint", + ) + })?; + let cursor: GcCursor = decode_canonical(&stored.bytes)?; + self.validate_gc_cursor(&cursor)?; + Ok(Some(cursor)) + } + + /// Explicitly release a known active epoch when its durable checkpoint is + /// irrecoverably missing. The expected epoch prevents abandoning another + /// maintenance worker's run. + pub async fn abandon_gc(&self, expected_epoch: OperationId) -> Result<()> { + let active = *self + .active_gc_epoch + .read() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "active GC lock poisoned"))?; + if active != Some(expected_epoch) { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "active GC epoch does not match the requested abandonment", + )); + } + self.publish_gc_coordinator(None, self.options.clock.now_millis()?) + .await?; + *self + .active_gc_epoch + .write() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "active GC lock poisoned"))? = + None; + Ok(()) + } + + /// Release an active legacy/crashed epoch only when no durable cursor was + /// ever published for it. A resumable epoch must use `resume_gc` instead. + pub async fn abandon_incomplete_gc(&self) -> Result { + let stored = self + .plane + .load_mutable(&gc_coordinator_path(&self.options.repository_prefix)?) + .await? + .ok_or_else(|| Error::new(ErrorCode::InvalidRevision, "GC coordinator is missing"))?; + let coordinator: GcCoordinator = decode_canonical(&stored.bytes)?; + let epoch = coordinator + .active_epoch + .ok_or_else(|| Error::new(ErrorCode::PreconditionFailed, "no GC epoch is active"))?; + if self + .plane + .load_mutable(&gc_cursor_path(&self.options.repository_prefix, epoch)?) + .await? + .is_some() + { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "active GC epoch has a durable cursor and must be resumed", + )); + } + self.abandon_gc(epoch).await?; + Ok(epoch) } /// Advance root discovery, reachability marking, candidate discovery, or @@ -3350,6 +3802,21 @@ impl Repository

{ }); } let mut next = cursor.clone(); + if !next.publication_barrier_drained { + let (processed, continuation, drained) = self + .gc_drain_publication_tickets(next.continuation.clone(), max_steps) + .await?; + next.continuation = continuation; + next.publication_barrier_drained = drained; + let page = GcPage { + complete: false, + cursor: next, + processed, + restarted_for_new_roots: false, + }; + self.persist_gc_cursor(&page.cursor).await?; + return Ok(page); + } let processed = match next.phase { GcPhase::DiscoverBranches => self.gc_discover_refs(&mut next, false, max_steps).await?, GcPhase::DiscoverTags => self.gc_discover_refs(&mut next, true, max_steps).await?, @@ -3362,12 +3829,14 @@ impl Repository

{ GcPhase::Cleanup => self.gc_cleanup(&mut next, max_steps).await?, GcPhase::Ready | GcPhase::Sweeping | GcPhase::Complete => 0, }; - Ok(GcPage { + let page = GcPage { complete: next.phase == GcPhase::Complete, cursor: next, processed, restarted_for_new_roots: false, - }) + }; + self.persist_gc_cursor(&page.cursor).await?; + Ok(page) } /// Delete at most `max_candidates` exact immutable physical versions. @@ -3393,12 +3862,14 @@ impl Repository

{ next.phase = GcPhase::CatchUpDirtyRoots; next.dirty_target_sequence = dirty_target; next.continuation = None; - return Ok(GcPage { + let page = GcPage { cursor: next, processed: 0, complete: false, restarted_for_new_roots: true, - }); + }; + self.persist_gc_cursor(&page.cursor).await?; + return Ok(page); } let engine = self.gc_work_engine(cursor.epoch)?; let tree = self.tree_from_root(&cursor.work)?; @@ -3439,12 +3910,24 @@ impl Repository

{ if retained { next.report.skipped_reachable = next.report.skipped_reachable.saturating_add(1); } else { - match self + let deletion = self .plane .delete_exact(&candidate.path, candidate.physical_version) - .await? - { - DeleteOutcome::Deleted => { + .await; + match deletion { + Err(error) if error.code == ErrorCode::PermissionDenied => { + next.report.protected_versions = + next.report.protected_versions.saturating_add(1); + next.report.protected_bytes = + next.report.protected_bytes.saturating_add(candidate.len); + *next + .report + .protected_by_kind + .entry(candidate.kind) + .or_default() += 1; + } + Err(error) => return Err(error), + Ok(DeleteOutcome::Deleted) => { next.report.deleted_versions = next.report.deleted_versions.saturating_add(1); next.report.deleted_bytes = @@ -3455,10 +3938,10 @@ impl Repository

{ .entry(candidate.kind) .or_default() += 1; } - DeleteOutcome::NotFound => { + Ok(DeleteOutcome::NotFound) => { next.report.already_missing = next.report.already_missing.saturating_add(1); } - DeleteOutcome::TokenMismatch => { + Ok(DeleteOutcome::TokenMismatch) => { return Err(Error::new( ErrorCode::PreconditionFailed, "GC candidate changed before exact deletion", @@ -3490,12 +3973,14 @@ impl Repository

{ next.phase = GcPhase::Cleanup; next.continuation = None; } - Ok(GcPage { + let page = GcPage { complete: false, cursor: next, processed, restarted_for_new_roots: false, - }) + }; + self.persist_gc_cursor(&page.cursor).await?; + Ok(page) } /// Start a restartable integrity check for one stable branch snapshot. @@ -3507,15 +3992,199 @@ impl Repository

{ self.locator.register(branch)?; self.require_branch_indexes_ready(branch).await?; let snapshot = self.head(branch).await?; - Ok(FsckCursor { + let snapshot_commit = self.load_commit_object(snapshot).await?.commit; + let job = self.options.ids.operation(); + let payloads = self.fsck_payload_engine(job)?.create(); + let cursor = FsckCursor { repository: self.format.repository_id, branch: branch.to_string(), snapshot, + snapshot_objects: snapshot_commit.state.objects, + snapshot_versions: snapshot_commit.state.versions, + job, + checkpoint_generation: 1, + payloads: RootManifest::from_tree(&payloads)?, closure: self.start_commit_closure(&[snapshot]).await?, phase: FsckPhase::DiscoverCommits, after: None, deep, report: FsckReport::default(), + }; + self.persist_fsck_cursor(&cursor).await?; + Ok(cursor) + } + + /// Resume one integrity-check job from its authoritative durable + /// checkpoint. Completed jobs remain resumable until explicitly forgotten. + pub async fn resume_fsck(&self, job: OperationId) -> Result> { + if job.is_nil() { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "fsck job identifier is nil", + )); + } + let Some(stored) = self + .plane + .load_mutable(&fsck_cursor_path(&self.options.repository_prefix, job)?) + .await? + else { + return Ok(None); + }; + let cursor: FsckCursor = decode_canonical(&stored.bytes)?; + self.validate_fsck_cursor(&cursor)?; + if cursor.job != job { + return Err(Error::new( + ErrorCode::CorruptCommit, + "durable fsck cursor belongs to another job", + )); + } + Ok(Some(cursor)) + } + + /// Start bounded cleanup of a completed integrity-check job. The durable + /// completed checkpoint remains present until all immutable work objects + /// and its older checkpoint versions have been removed. + pub async fn start_fsck_cleanup(&self, job: OperationId) -> Result { + let cursor = self + .resume_fsck(job) + .await? + .ok_or_else(|| Error::new(ErrorCode::InvalidRevision, "fsck checkpoint is missing"))?; + if cursor.phase != FsckPhase::Complete { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "in-progress fsck job cannot be cleaned up", + )); + } + Ok(FsckCleanupCursor { + repository: self.format.repository_id, + job, + closure_traversal: cursor.closure.traversal, + phase: FsckCleanupPhase::PayloadWork, + }) + } + + /// Exact-delete at most `max_objects` physical versions from a completed + /// fsck workspace. Restarting cleanup from `start_fsck_cleanup` is safe. + pub async fn advance_fsck_cleanup( + &self, + cursor: &FsckCleanupCursor, + max_objects: usize, + ) -> Result { + if !(1..=1_000).contains(&max_objects) { + return Err(Error::new( + ErrorCode::InvalidLimit, + "fsck cleanup page size must be between 1 and 1,000", + )); + } + if cursor.repository != self.format.repository_id + || cursor.job.is_nil() + || cursor.closure_traversal.is_nil() + { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "fsck cleanup cursor is malformed or belongs to another repository", + )); + } + if cursor.phase == FsckCleanupPhase::Complete { + return Ok(FsckCleanupPage { + cursor: cursor.clone(), + deleted: 0, + complete: true, + }); + } + let durable = self.resume_fsck(cursor.job).await?.ok_or_else(|| { + Error::new( + ErrorCode::InvalidContinuationToken, + "fsck checkpoint is missing", + ) + })?; + if durable.phase != FsckPhase::Complete + || durable.closure.traversal != cursor.closure_traversal + { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "fsck cleanup cursor does not match its completed checkpoint", + )); + } + let mut next = cursor.clone(); + let deleted = match next.phase { + FsckCleanupPhase::PayloadWork => { + let prefix = format!( + "{}/administration/fsck/{}/payloads/", + self.options.repository_prefix, next.job + ); + let deleted = self + .delete_fsck_cleanup_prefix(&prefix, max_objects) + .await?; + if deleted == 0 { + next.phase = FsckCleanupPhase::ClosureWork; + } + deleted + } + FsckCleanupPhase::ClosureWork => { + let prefix = format!( + "{}/administration/closure/{}/tree/", + self.options.repository_prefix, next.closure_traversal + ); + let deleted = self + .delete_fsck_cleanup_prefix(&prefix, max_objects) + .await?; + if deleted == 0 { + next.phase = FsckCleanupPhase::CheckpointHistory; + } + deleted + } + FsckCleanupPhase::CheckpointHistory => { + let path = fsck_cursor_path(&self.options.repository_prefix, next.job)?; + let page = self + .plane + .list(ListRequest { + prefix: path.as_str().to_string(), + continuation: None, + limit: 1_000, + include_versions: true, + }) + .await?; + let current = self.plane.load_mutable(&path).await?.ok_or_else(|| { + Error::new( + ErrorCode::InvalidContinuationToken, + "fsck checkpoint is missing", + ) + })?; + let targets = page + .entries + .into_iter() + .filter(|entry| { + entry.path == path && entry.metadata.token != current.metadata.token + }) + .take(max_objects) + .map(physical_list_entry_target) + .collect::>(); + let deleted = self.delete_fsck_cleanup_targets(targets).await?; + if deleted == 0 { + next.phase = FsckCleanupPhase::CheckpointCurrent; + } + deleted + } + FsckCleanupPhase::CheckpointCurrent => { + let path = fsck_cursor_path(&self.options.repository_prefix, next.job)?; + let current = self.plane.load_mutable(&path).await?.ok_or_else(|| { + Error::new( + ErrorCode::InvalidContinuationToken, + "fsck checkpoint is missing", + ) + })?; + let target = physical_stored_object_target(path, current); + let deleted = self.delete_fsck_cleanup_targets(vec![target]).await?; + next.phase = FsckCleanupPhase::Complete; + deleted + } + FsckCleanupPhase::Complete => 0, + }; + Ok(FsckCleanupPage { + complete: next.phase == FsckCleanupPhase::Complete, + cursor: next, + deleted, }) } @@ -3523,21 +4192,23 @@ impl Repository

{ /// current objects, or logical versions. Persist the returned cursor after /// every page. pub async fn advance_fsck(&self, cursor: &FsckCursor, max_steps: usize) -> Result { - if !(1..=1_000).contains(&max_steps) { + if !(1..=10_000).contains(&max_steps) { return Err(Error::new( ErrorCode::InvalidLimit, - "fsck page size must be between 1 and 1,000", + "fsck page size must be between 1 and 10,000", )); } - if cursor.repository != self.format.repository_id { - return Err(Error::new( - ErrorCode::InvalidContinuationToken, - "fsck cursor belongs to another repository", - )); + self.validate_fsck_cursor(cursor)?; + self.require_current_fsck_cursor(cursor).await?; + if cursor.phase == FsckPhase::Complete { + return Ok(FsckPage { + cursor: cursor.clone(), + processed: 0, + complete: true, + }); } - validate_branch(&cursor.branch)?; - self.validate_commit_closure_cursor(&cursor.closure)?; self.locator.register(&cursor.branch)?; + self.require_branch_indexes_ready(&cursor.branch).await?; let mut next = cursor.clone(); let mut processed = 0usize; match next.phase { @@ -3568,8 +4239,7 @@ impl Repository

{ } } FsckPhase::VerifyObjects => { - let commit = self.load_commit_object(next.snapshot).await?.commit; - let tree = self.tree_from_root(&commit.state.objects)?; + let tree = self.tree_from_root(&next.snapshot_objects)?; let engine = self.engine(self.node_store.clone()); let mut entries = match next.after.as_deref() { Some(after) => engine.range_after(&tree, after, None).await?, @@ -3584,66 +4254,105 @@ impl Repository

{ let (key, encoded) = entry?; let current: CurrentObject = decode_canonical(&encoded)?; current.version.validate()?; - let (payloads, bytes, deep_bytes) = self - .verify_payload_metadata(¤t.version, next.deep) - .await?; next.report.current_objects = checked_fsck_add(next.report.current_objects, 1, "current-object")?; - next.report.payloads_verified = - checked_fsck_add(next.report.payloads_verified, payloads, "payload")?; - next.report.payload_bytes_verified = checked_fsck_add( - next.report.payload_bytes_verified, - bytes, - "payload-byte", - )?; - next.report.deep_content_bytes_verified = checked_fsck_add( - next.report.deep_content_bytes_verified, - deep_bytes, - "deep-content-byte", - )?; + record_fsck_logical_payload(&mut next.report, ¤t.version, next.deep)?; next.after = Some(key); processed += 1; } } FsckPhase::VerifyVersions => { - let commit = self.load_commit_object(next.snapshot).await?.commit; - let tree = self.tree_from_root(&commit.state.versions)?; + let tree = self.tree_from_root(&next.snapshot_versions)?; let engine = self.engine(self.node_store.clone()); let mut entries = match next.after.as_deref() { Some(after) => engine.range_after(&tree, after, None).await?, None => engine.prefix(&tree, b"").await?, }; + let mut versions = Vec::with_capacity(max_steps); while processed < max_steps { let Some(entry) = entries.next().await else { - next.phase = FsckPhase::Complete; + next.phase = FsckPhase::VerifyPayloads; next.after = None; break; }; let (key, encoded) = entry?; let version: ObjectVersion = decode_canonical(&encoded)?; version.validate()?; - let (payloads, bytes, deep_bytes) = - self.verify_payload_metadata(&version, next.deep).await?; next.report.logical_versions = checked_fsck_add(next.report.logical_versions, 1, "logical-version")?; - next.report.payloads_verified = - checked_fsck_add(next.report.payloads_verified, payloads, "payload")?; - next.report.payload_bytes_verified = checked_fsck_add( - next.report.payload_bytes_verified, - bytes, - "payload-byte", - )?; - next.report.deep_content_bytes_verified = checked_fsck_add( - next.report.deep_content_bytes_verified, + record_fsck_logical_payload(&mut next.report, &version, next.deep)?; + versions.push(version); + next.after = Some(key); + processed += 1; + } + next.payloads = self + .stage_fsck_payloads(next.job, &next.payloads, &versions) + .await?; + if next.deep { + let deep_bytes = self.verify_fsck_logical_page(&versions).await?; + next.report.deep_physical_bytes_read = checked_fsck_add( + next.report.deep_physical_bytes_read, deep_bytes, - "deep-content-byte", + "deep-physical-byte", )?; + } + } + FsckPhase::VerifyPayloads => { + let payloads = self.tree_from_root(&next.payloads)?; + let engine = self.fsck_payload_engine(next.job)?; + let mut entries = match next.after.as_deref() { + Some(after) => engine.range_after(&payloads, after, None).await?, + None => engine.prefix(&payloads, b"").await?, + }; + let mut records = Vec::with_capacity(max_steps); + while processed < max_steps { + let Some(entry) = entries.next().await else { + next.phase = FsckPhase::Complete; + next.after = None; + break; + }; + let (key, encoded) = entry?; + records.push(decode_canonical::(&encoded)?); next.after = Some(key); processed += 1; } + let deep = next.deep; + let verified = stream::iter(records.into_iter().map(|record| async move { + if deep { + Ok((record.size, 0)) + } else { + self.verify_fsck_physical_payload(&record).await + } + })) + .buffered(32) + .collect::>() + .await; + for result in verified { + let (physical_bytes, deep_bytes) = result?; + next.report.physical_payloads_verified = checked_fsck_add( + next.report.physical_payloads_verified, + 1, + "physical-payload", + )?; + next.report.physical_payload_bytes_verified = checked_fsck_add( + next.report.physical_payload_bytes_verified, + physical_bytes, + "physical-payload-byte", + )?; + next.report.deep_physical_bytes_read = checked_fsck_add( + next.report.deep_physical_bytes_read, + deep_bytes, + "deep-physical-byte", + )?; + } } FsckPhase::Complete => {} } + next.checkpoint_generation = next + .checkpoint_generation + .checked_add(1) + .ok_or_else(|| Error::new(ErrorCode::InternalInvariant, "fsck checkpoint overflow"))?; + self.persist_fsck_cursor(&next).await?; Ok(FsckPage { complete: next.phase == FsckPhase::Complete, cursor: next, @@ -4010,6 +4719,7 @@ impl Repository

{ let limit = max_steps.min(remaining); let mut next = cursor.clone(); let mut mutations = checkpoint.mutations; + let checkpointed_mutations = mutations.len(); let processed; match cursor.phase { RepairPhase::CopySource => { @@ -4043,33 +4753,31 @@ impl Repository

{ { continue; } - let data = source - .get_object_at(&cursor.source_branch, cursor.source_snapshot, &object.key) - .await? - .ok_or_else(|| { - Error::new( - ErrorCode::CorruptCommit, - "repair listing points to a missing source object", - ) - })?; - let size = u64::try_from(data.bytes.len()).map_err(|_| { - Error::new(ErrorCode::EntityTooLarge, "repair payload exceeds u64") + object.version.validate()?; + let source_binding = object.version.binding.clone().ok_or_else(|| { + Error::new( + ErrorCode::CorruptCommit, + "repair source live object has no payload binding", + ) })?; - let binding = self.payloads.put(data.bytes).await?; let LogicalObjectVersionKind::Live { + size, logical_etag, headers, checksums, user_metadata, tags, - .. - } = data.version.body.kind + } = object.version.body.kind else { return Err(Error::new( ErrorCode::CorruptCommit, "repair source current object is a delete marker", )); }; + let binding = self + .payloads + .transfer_from(&source.payloads, &source_binding, size) + .await?; mutations.push(StagedMutation { body: StagedMutationBody::Put(Box::new(StagedPut { key: object.key, @@ -4140,7 +4848,12 @@ impl Repository

{ ) })?; let saved = self - .checkpoint_commit_session(&checkpoint.session, mutations, sequence) + .checkpoint_commit_session( + &checkpoint.session, + mutations[checkpointed_mutations..].to_vec(), + mutations.len(), + sequence, + ) .await?; next.checkpoint_sequence = saved.sequence; return Ok(RepairPage { @@ -4151,7 +4864,13 @@ impl Repository

{ }); } if mutations.is_empty() { - self.abort_commit_session(checkpoint.session, mutations, checkpoint.sequence) + let sequence = checkpoint.sequence.checked_add(1).ok_or_else(|| { + Error::new( + ErrorCode::InvalidLimit, + "repair abort checkpoint sequence overflow", + ) + })?; + self.abort_commit_session(checkpoint.session, Vec::new(), 0, sequence) .await?; return Ok(RepairPage { cursor: next, @@ -4345,6 +5064,14 @@ impl Repository

{ self.require_branch_indexes_ready(branch).await?; let from_commit = self.load_commit_metadata(from).await?; let to_commit = self.load_commit_metadata(to).await?; + if to_commit.parents.first() == Some(&from) && to_commit.delta.changes_root.is_none() { + return to_commit + .delta + .changes + .iter() + .map(object_diff_from_transition) + .collect(); + } let from_tree = self.tree_from_root(&from_commit.state.objects)?; let to_tree = self.tree_from_root(&to_commit.state.objects)?; self.engine(self.node_store.clone()) @@ -4388,15 +5115,51 @@ impl Repository

{ self.require_branch_indexes_ready(branch).await?; let from_commit = self.load_commit_metadata(from).await?; let to_commit = self.load_commit_metadata(to).await?; - let from_tree = self.tree_from_root(&from_commit.state.objects)?; - let to_tree = self.tree_from_root(&to_commit.state.objects)?; - let page = self - .engine(self.node_store.clone()) - .structural_diff_page( - &from_tree, - &to_tree, + // Direct children carry an exact logical transition stream, inline or + // in an immutable delta tree. Page that stream instead of comparing + // snapshot structure; tree boundary churn is irrelevant to this path. + if to_commit.parents.first() == Some(&from) + && !matches!( cursor.map(|cursor| &cursor.traversal), - limit, + Some(ObjectDiffTraversal::Structural(_)) + ) + { + let after = match cursor.map(|cursor| &cursor.traversal) { + Some(ObjectDiffTraversal::Delta { after }) => Some(after.as_slice()), + None => None, + Some(ObjectDiffTraversal::Structural(_)) => unreachable!("excluded above"), + }; + let (transitions, next_after) = + self.commit_delta_page(&to_commit, after, limit).await?; + let changes = transitions + .iter() + .map(object_diff_from_transition) + .collect::>>()?; + return Ok(ObjectDiffPage { + changes, + continuation: next_after.map(|after| ObjectDiffCursor { + repository: self.format.repository_id, + branch: branch.to_string(), + from, + to, + traversal: ObjectDiffTraversal::Delta { after }, + }), + compared_nodes: 0, + reused_subtrees: 0, + }); + } + let from_tree = self.tree_from_root(&from_commit.state.objects)?; + let to_tree = self.tree_from_root(&to_commit.state.objects)?; + let page = self + .engine(self.node_store.clone()) + .structural_diff_page( + &from_tree, + &to_tree, + cursor.map(|cursor| match &cursor.traversal { + ObjectDiffTraversal::Structural(traversal) => traversal, + ObjectDiffTraversal::Delta { .. } => unreachable!("handled direct delta"), + }), + limit, ) .await?; let changes = page @@ -4411,7 +5174,7 @@ impl Repository

{ branch: branch.to_string(), from, to, - traversal, + traversal: ObjectDiffTraversal::Structural(traversal), }), compared_nodes: page.stats.compared_nodes, reused_subtrees: page.stats.reused_subtrees, @@ -4639,6 +5402,7 @@ impl Repository

{ .await?; let processed = page.changes.len(); let mut mutations = checkpoint.mutations; + let checkpointed_mutations = mutations.len(); for change in page.changes { if change.to.is_none() { mutations.push(StagedMutation::delete(change.key)); @@ -4705,7 +5469,12 @@ impl Repository

{ ) })?; let checkpoint = self - .checkpoint_commit_session(&checkpoint.session, mutations, sequence) + .checkpoint_commit_session( + &checkpoint.session, + mutations[checkpointed_mutations..].to_vec(), + mutations.len(), + sequence, + ) .await?; next.checkpoint_sequence = checkpoint.sequence; return Ok(RestorePage { @@ -4716,7 +5485,13 @@ impl Repository

{ }); } if mutations.is_empty() { - self.abort_commit_session(checkpoint.session, mutations, checkpoint.sequence) + let sequence = checkpoint.sequence.checked_add(1).ok_or_else(|| { + Error::new( + ErrorCode::InvalidLimit, + "restore abort checkpoint sequence overflow", + ) + })?; + self.abort_commit_session(checkpoint.session, Vec::new(), 0, sequence) .await?; next.complete = true; return Ok(RestorePage { @@ -5024,6 +5799,7 @@ impl Repository

{ { return Ok(receipt); } + self.require_publication_admission_open().await?; let current = self.publisher.load(&cursor.target_branch).await?; if current.value.target != cursor.ours { return Err(Error::new( @@ -5212,6 +5988,7 @@ impl Repository

{ .remove(branch); let reference = self.publisher.load(branch).await?; self.record_branch_catalog(&reference).await?; + self.mark_local_index_head(branch, reference.value.target)?; Ok(report) } @@ -5703,6 +6480,15 @@ impl Repository

{ branch: &str, reference: &LoadedRef, ) -> Result<()> { + if self + .local_index_heads + .read() + .map_err(|_| Error::new(ErrorCode::InternalInvariant, "local-index lock poisoned"))? + .get(branch) + .is_some_and(|target| *target == reference.value.target) + { + return Ok(()); + } let health = self.branch_index_health_for(branch, reference).await?; Self::check_branch_index_health(health) } @@ -5815,10 +6601,10 @@ impl Repository

{ }; let object = CommitObject::new(commit.clone(), Some(prepared.pack().clone()))?; let encoded = object.encode_object()?; - let offset = CommitObject::node_payload_offset(&encoded)?.ok_or_else(|| { + let offset = CommitObject::node_region_offset(&encoded)?.ok_or_else(|| { Error::new( ErrorCode::InternalInvariant, - "prepared node pack has no payload offset", + "prepared node pack has no node-region offset", ) })?; self.node_store.commit_node_pack(id, prepared, offset).await @@ -5850,52 +6636,226 @@ impl Repository

{ }) } - async fn verify_payload_metadata( + async fn stage_fsck_payloads( &self, - version: &ObjectVersion, - deep: bool, - ) -> Result<(u64, u64, u64)> { - let LogicalObjectVersionKind::Live { size, .. } = &version.body.kind else { - return Ok((0, 0, 0)); - }; - let binding = version.binding.as_ref().ok_or_else(|| { - Error::new( - ErrorCode::CorruptCommit, - "live object version has no payload binding", - ) - })?; - if binding.path != self.payloads.expected_path(binding)? { - return Err(Error::new( - ErrorCode::CorruptContent, - "payload binding path does not match its content checksum", - )); + job: OperationId, + root: &RootManifest, + versions: &[ObjectVersion], + ) -> Result { + let engine = self.fsck_payload_engine(job)?; + let mut tree = self.tree_from_root(root)?; + let mut pending = BTreeMap::, FsckPhysicalPayload>::new(); + for version in versions { + let LogicalObjectVersionKind::Live { size, .. } = &version.body.kind else { + continue; + }; + let binding = version.binding.as_ref().ok_or_else(|| { + Error::new( + ErrorCode::CorruptCommit, + "live object version has no payload binding", + ) + })?; + if binding.path != self.payloads.expected_path(binding)? { + return Err(Error::new( + ErrorCode::CorruptContent, + "payload binding path does not match its content checksum", + )); + } + let record = pending + .entry(payload_physical_key(binding)) + .or_insert_with(|| FsckPhysicalPayload { + binding: binding.clone(), + size: *size, + }); + validate_same_fsck_physical_binding(&record.binding, binding)?; + if record.size != *size { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "whole-object payload bindings disagree about object size", + )); + } } + let keys = pending.keys().cloned().collect::>(); + let existing = engine.get_many(&tree, &keys).await?; + let mut mutations = Vec::with_capacity(pending.len()); + for ((key, record), existing) in pending.into_iter().zip(existing) { + if let Some(encoded) = existing { + let accumulated: FsckPhysicalPayload = decode_canonical(&encoded)?; + validate_same_fsck_physical_binding(&accumulated.binding, &record.binding)?; + if accumulated.size != record.size { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "payload records disagree about whole-object size", + )); + } + continue; + } + mutations.push(Mutation::Upsert { + key, + val: encode_canonical(&record)?, + }); + } + if !mutations.is_empty() { + tree = engine.batch(&tree, mutations).await?; + } + RootManifest::from_tree(&tree) + } + + async fn verify_fsck_physical_payload( + &self, + record: &FsckPhysicalPayload, + ) -> Result<(u64, u64)> { + let binding = &record.binding; let metadata = self.plane.head(&binding.path).await?.ok_or_else(|| { Error::new(ErrorCode::MissingClosure, "immutable payload is missing") })?; - let expected_physical_len = binding - .pack_range - .map(|(_, end)| end.saturating_add(1)) - .unwrap_or(*size); - if metadata.len < expected_physical_len - || (!binding.is_packed() && metadata.len != *size) - || metadata.sha256 != binding.physical_checksum_sha256() + if metadata.len != record.size + || metadata.sha256 != binding.checksum_sha256 || metadata.token.etag != binding.provider_etag || metadata.token.version_id != binding.provider_version_id { return Err(Error::new( ErrorCode::ChecksumMismatch, - "immutable payload metadata does not match its logical binding", + "immutable payload metadata does not match its physical binding", )); } - let deep_bytes = if deep { - u64::try_from(self.payloads.get(binding).await?.len()) - .map_err(|_| Error::new(ErrorCode::EntityTooLarge, "payload length exceeds u64"))? - } else { - 0 - }; - Ok((1, *size, deep_bytes)) + Ok((metadata.len, 0)) + } + + async fn verify_fsck_logical_page(&self, versions: &[ObjectVersion]) -> Result { + let mut pending = BTreeMap::, FsckPhysicalPayload>::new(); + for version in versions { + let LogicalObjectVersionKind::Live { size, .. } = &version.body.kind else { + continue; + }; + let binding = version.binding.as_ref().ok_or_else(|| { + Error::new( + ErrorCode::CorruptCommit, + "live object version has no payload binding", + ) + })?; + let record = pending + .entry(payload_physical_key(binding)) + .or_insert_with(|| FsckPhysicalPayload { + binding: binding.clone(), + size: *size, + }); + validate_same_fsck_physical_binding(&record.binding, binding)?; + if record.size != *size { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "whole-object payload bindings disagree about object size", + )); + } + } + let verified = stream::iter( + pending + .into_values() + .map(|record| async move { self.verify_fsck_page_payload(record).await }), + ) + .buffered(32) + .collect::>() + .await; + verified.into_iter().try_fold(0_u64, |total, result| { + total + .checked_add(result?) + .ok_or_else(|| Error::new(ErrorCode::EntityTooLarge, "deep fsck byte overflow")) + }) + } + + async fn verify_fsck_page_payload(&self, record: FsckPhysicalPayload) -> Result { + let binding = &record.binding; + let physical_version = + binding + .provider_version_id + .as_ref() + .map(|version_id| PhysicalVersion::Versioned { + version_id: version_id.clone(), + }); + let stored = self + .plane + .get(GetRequest { + path: binding.path.clone(), + range: None, + physical_version, + }) + .await? + .ok_or_else(|| Error::new(ErrorCode::MissingClosure, "immutable payload is missing"))?; + if u64::try_from(stored.bytes.len()).ok() != Some(record.size) + || crate::codec::sha256(&stored.bytes) != binding.checksum_sha256 + { + return Err(Error::new( + ErrorCode::CorruptContent, + "immutable whole-object payload checksum mismatch", + )); + } + u64::try_from(stored.bytes.len()) + .map_err(|_| Error::new(ErrorCode::EntityTooLarge, "payload length exceeds u64")) + } + + async fn gc_drain_publication_tickets( + &self, + continuation: Option, + max_steps: usize, + ) -> Result<(usize, Option, bool)> { + let page = self + .plane + .list(ListRequest { + prefix: gc_publication_ticket_prefix( + &self.options.repository_prefix, + self.format.repository_id, + ), + continuation: continuation.clone(), + limit: max_steps.min(1_000), + include_versions: false, + }) + .await?; + let now = self.options.clock.now_millis()?; + let mut processed = 0usize; + for entry in page.entries { + let Some(stored) = self + .plane + .get(GetRequest { + path: entry.path.clone(), + range: None, + physical_version: None, + }) + .await? + else { + continue; + }; + let ticket: GcPublicationTicket = decode_canonical(&stored.bytes)?; + if ticket.repository != self.format.repository_id { + return Err(Error::new( + ErrorCode::CorruptCommit, + "publication ticket belongs to another repository", + )); + } + if ticket.expires_at_millis > now { + return Ok((processed, continuation, false)); + } + let version = stored.metadata.token.version_id.clone().map_or_else( + || PhysicalVersion::Unversioned { + token: Some(stored.metadata.token.clone()), + }, + |version_id| PhysicalVersion::Versioned { version_id }, + ); + match self.plane.delete_exact(&entry.path, version).await? { + DeleteOutcome::Deleted | DeleteOutcome::NotFound => {} + DeleteOutcome::TokenMismatch => { + return Err(Error::new( + ErrorCode::PreconditionFailed, + "expired publication ticket changed during maintenance admission", + )); + } + } + processed = processed.saturating_add(1); + } + match page.continuation { + Some(next) => Ok((processed, Some(next), false)), + None => Ok((processed, None, true)), + } } fn gc_work_engine(&self, epoch: OperationId) -> Result>> { @@ -5905,15 +6865,163 @@ impl Repository

{ "GC epoch ID is nil", )); } - Ok(self.engine(ProllyObjectStore::new( + Ok(self.engine(ProllyObjectStore::new_cached_direct( self.plane.clone(), format!( "{}/administration/gc/{epoch}/tree", self.options.repository_prefix ), + self.format.repository_id, + tree_format_digest(&self.format.state_tree_format)?, + self.node_cache.clone(), ))) } + fn fsck_payload_engine(&self, job: OperationId) -> Result>> { + if job.is_nil() { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "fsck job ID is nil", + )); + } + Ok(self.engine(ProllyObjectStore::new_cached_direct( + self.plane.clone(), + format!( + "{}/administration/fsck/{job}/payloads", + self.options.repository_prefix + ), + self.format.repository_id, + tree_format_digest(&self.format.state_tree_format)?, + self.node_cache.clone(), + ))) + } + + fn validate_fsck_cursor(&self, cursor: &FsckCursor) -> Result<()> { + let format_digest = tree_format_digest(&self.format.state_tree_format)?; + if cursor.repository != self.format.repository_id + || cursor.job.is_nil() + || cursor.checkpoint_generation == 0 + || cursor.payloads.format_digest != format_digest + || cursor.snapshot_objects.format_digest != format_digest + || cursor.snapshot_versions.format_digest != format_digest + { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "fsck cursor is malformed or belongs to another repository", + )); + } + validate_branch(&cursor.branch)?; + self.validate_commit_closure_cursor(&cursor.closure)?; + self.tree_from_root(&cursor.payloads)?; + self.tree_from_root(&cursor.snapshot_objects)?; + self.tree_from_root(&cursor.snapshot_versions)?; + Ok(()) + } + + async fn require_current_fsck_cursor(&self, cursor: &FsckCursor) -> Result<()> { + let path = fsck_cursor_path(&self.options.repository_prefix, cursor.job)?; + let stored = self.plane.load_mutable(&path).await?.ok_or_else(|| { + Error::new( + ErrorCode::InvalidContinuationToken, + "durable fsck checkpoint is missing", + ) + })?; + let durable: FsckCursor = decode_canonical(&stored.bytes)?; + self.validate_fsck_cursor(&durable)?; + if durable != *cursor { + return Err(Error::new( + ErrorCode::RefConflict, + "fsck cursor is stale; resume its durable checkpoint", + )); + } + Ok(()) + } + + async fn persist_fsck_cursor(&self, cursor: &FsckCursor) -> Result<()> { + self.validate_fsck_cursor(cursor)?; + let path = fsck_cursor_path(&self.options.repository_prefix, cursor.job)?; + let current = self.plane.load_mutable(&path).await?; + if let Some(stored) = current.as_ref() { + let previous: FsckCursor = decode_canonical(&stored.bytes)?; + self.validate_fsck_cursor(&previous)?; + if previous.job != cursor.job { + return Err(Error::new( + ErrorCode::CorruptCommit, + "durable fsck cursor belongs to another job", + )); + } + if previous == *cursor { + return Ok(()); + } + if previous + .checkpoint_generation + .checked_add(1) + .filter(|generation| *generation == cursor.checkpoint_generation) + .is_none() + { + return Err(Error::new( + ErrorCode::RefConflict, + "durable fsck checkpoint generation changed concurrently", + )); + } + } else if cursor.checkpoint_generation != 1 { + return Err(Error::new( + ErrorCode::InvalidContinuationToken, + "durable fsck checkpoint is missing", + )); + } + let expected = current.map(|stored| stored.metadata.token); + match self + .maintenance_controls + .compare_exchange(CompareExchange { + path, + expected, + bytes: encode_canonical(cursor)?, + }) + .await? + { + CompareExchangeOutcome::Applied(_) => Ok(()), + CompareExchangeOutcome::Conflict(_) => Err(Error::new( + ErrorCode::RefConflict, + "durable fsck cursor changed concurrently", + )), + } + } + + async fn delete_fsck_cleanup_prefix(&self, prefix: &str, max_objects: usize) -> Result { + let page = self + .plane + .list(ListRequest { + prefix: prefix.to_string(), + continuation: None, + limit: max_objects, + include_versions: true, + }) + .await?; + let targets = page + .entries + .into_iter() + .map(physical_list_entry_target) + .collect::>(); + self.delete_fsck_cleanup_targets(targets).await + } + + async fn delete_fsck_cleanup_targets( + &self, + targets: Vec<(ObjectPath, PhysicalVersion)>, + ) -> Result { + let processed = targets.len(); + for outcome in self.plane.delete_exact_batch(targets).await? { + if outcome == DeleteOutcome::TokenMismatch { + return Err(Error::new( + ErrorCode::RefConflict, + "fsck workspace changed during exact cleanup", + )); + } + } + Ok(processed) + } + fn validate_gc_cursor(&self, cursor: &GcCursor) -> Result<()> { if cursor.repository != self.format.repository_id || cursor.epoch.is_nil() @@ -6062,84 +7170,106 @@ impl Repository

{ async fn gc_mark_nodes(&self, cursor: &mut GcCursor, max_steps: usize) -> Result { let engine = self.gc_work_engine(cursor.epoch)?; let mut tree = self.tree_from_root(&cursor.work)?; - let mut processed = 0usize; - while processed < max_steps { - let mut queue = engine.prefix(&tree, b"nq/").await?; + let mut queue = engine.prefix(&tree, b"nq/").await?; + let mut records = Vec::with_capacity(max_steps); + while records.len() < max_steps { let Some(entry) = queue.next().await else { - cursor.phase = if cursor.initial_scan_complete { - GcPhase::CatchUpDirtyRoots - } else { - GcPhase::ScanCandidates - }; break; }; let (queue_key, encoded) = entry?; - drop(queue); let work: GcNodeWork = decode_canonical(&encoded)?; - let mark_key = gc_node_mark_key(&work.cid, work.scan_versions); - let mut mutations = vec![Mutation::Delete { key: queue_key }]; - if engine.get(&tree, &mark_key).await?.is_none() { - mutations.push(Mutation::Upsert { - key: mark_key, - val: Vec::new(), - }); - mutations.push(Mutation::Upsert { - key: gc_path_mark_key(&self.node_store.direct_node_path(&work.cid)?), - val: Vec::new(), - }); - let bytes = self - .node_store - .get(work.cid.as_bytes()) - .await? - .ok_or_else(|| { - Error::new(ErrorCode::MissingClosure, "reachable node is missing") - })?; - let node = Node::from_bytes_with_format(&bytes, &self.format.state_tree_format) - .map_err(|error| { - Error::new( - ErrorCode::CorruptNode, - format!("reachable node could not be decoded: {error}"), - ) - })?; - if node.leaf { - if work.scan_versions { - for encoded in node.vals { - let version: ObjectVersion = decode_canonical(&encoded)?; - version.validate()?; - if let Some(binding) = version.binding { - mutations.push(Mutation::Upsert { - key: binding.provider_version_id.as_ref().map_or_else( - || gc_path_mark_key(&binding.path), - |version| gc_physical_mark_key(&binding.path, version), - ), - val: Vec::new(), - }); - } - cursor.report.logical_versions = - cursor.report.logical_versions.saturating_add(1); - } - } - } else { - for child in node.vals { - let cid = prolly::Cid(child.try_into().map_err(|_| { - Error::new(ErrorCode::CorruptNode, "internal node child CID is invalid") - })?); - mutations.push(Mutation::Upsert { - key: gc_node_queue_key(&cid, work.scan_versions), - val: encode_canonical(&GcNodeWork { - cid, - scan_versions: work.scan_versions, - })?, - }); + records.push((queue_key, work)); + } + drop(queue); + if records.is_empty() { + cursor.phase = if cursor.initial_scan_complete { + GcPhase::CatchUpDirtyRoots + } else { + GcPhase::ScanCandidates + }; + return Ok(0); + } + let mark_keys = records + .iter() + .map(|(_, work)| gc_node_mark_key(&work.cid, work.scan_versions)) + .collect::>(); + let marked = engine.get_many(&tree, &mark_keys).await?; + let pending = records + .iter() + .zip(marked) + .filter_map(|((_, work), marked)| marked.is_none().then_some(work.clone())) + .collect::>(); + let loaded = stream::iter(pending.into_iter().map(|work| async move { + let bytes = self + .node_store + .get(work.cid.as_bytes()) + .await? + .ok_or_else(|| { + Error::new(ErrorCode::MissingClosure, "reachable node is missing") + })?; + let node = Node::from_bytes_with_format(&bytes, &self.format.state_tree_format) + .map_err(|error| { + Error::new( + ErrorCode::CorruptNode, + format!("reachable node could not be decoded: {error}"), + ) + })?; + Ok::<_, Error>((work, node)) + })) + .buffered(32) + .collect::>() + .await; + let mut mutations = records + .into_iter() + .map(|(key, _)| Mutation::Delete { key }) + .collect::>(); + for loaded in loaded { + let (work, node) = loaded?; + mutations.push(Mutation::Upsert { + key: gc_node_mark_key(&work.cid, work.scan_versions), + val: Vec::new(), + }); + mutations.push(Mutation::Upsert { + key: gc_path_mark_key(&self.node_store.direct_node_path(&work.cid)?), + val: Vec::new(), + }); + if node.leaf && work.scan_versions { + let mut physical_marks = BTreeSet::new(); + for encoded in node.vals { + let version: ObjectVersion = decode_canonical(&encoded)?; + version.validate()?; + if let Some(binding) = version.binding { + physical_marks.insert(binding.provider_version_id.as_ref().map_or_else( + || gc_path_mark_key(&binding.path), + |version| gc_physical_mark_key(&binding.path, version), + )); } + cursor.report.logical_versions = + cursor.report.logical_versions.saturating_add(1); + } + mutations.extend(physical_marks.into_iter().map(|key| Mutation::Upsert { + key, + val: Vec::new(), + })); + } else if !node.leaf { + for child in node.vals { + let cid = prolly::Cid(child.try_into().map_err(|_| { + Error::new(ErrorCode::CorruptNode, "internal node child CID is invalid") + })?); + mutations.push(Mutation::Upsert { + key: gc_node_queue_key(&cid, work.scan_versions), + val: encode_canonical(&GcNodeWork { + cid, + scan_versions: work.scan_versions, + })?, + }); } - cursor.report.nodes = cursor.report.nodes.saturating_add(1); } - tree = engine.batch(&tree, mutations).await?; - processed += 1; + cursor.report.nodes = cursor.report.nodes.saturating_add(1); } + tree = engine.batch(&tree, mutations).await?; cursor.work = RootManifest::from_tree(&tree)?; - Ok(processed) + Ok(mark_keys.len()) } async fn gc_scan_candidates(&self, cursor: &mut GcCursor, max_steps: usize) -> Result { @@ -6154,26 +7284,40 @@ impl Repository

{ .await?; let engine = self.gc_work_engine(cursor.epoch)?; let mut tree = self.tree_from_root(&cursor.work)?; - for entry in &page.entries { - let Some(kind) = gc_managed_kind(&self.options.repository_prefix, &entry.path) else { - continue; - }; - if entry.metadata.last_modified_millis > cursor.cutoff_millis { - continue; - } - let path_retained = engine - .get(&tree, &gc_path_mark_key(&entry.path)) - .await? - .is_some(); - let physical_retained = match entry.metadata.token.version_id.as_deref() { - Some(version) => engine - .get(&tree, &gc_physical_mark_key(&entry.path, version)) - .await? - .is_some(), - None => false, - }; - let retained = path_retained || physical_retained; - if retained { + let eligible = page + .entries + .iter() + .filter_map(|entry| { + let kind = gc_managed_kind(&self.options.repository_prefix, &entry.path)?; + (entry.metadata.last_modified_millis <= cursor.cutoff_millis) + .then_some((entry, kind)) + }) + .collect::>(); + let path_keys = eligible + .iter() + .map(|(entry, _)| gc_path_mark_key(&entry.path)) + .collect::>(); + let physical_keys = eligible + .iter() + .map(|(entry, _)| { + entry + .metadata + .token + .version_id + .as_deref() + .map(|version| gc_physical_mark_key(&entry.path, version)) + .unwrap_or_else(|| gc_path_mark_key(&entry.path)) + }) + .collect::>(); + let (path_marks, physical_marks) = futures_util::try_join!( + engine.get_many(&tree, &path_keys), + engine.get_many(&tree, &physical_keys), + )?; + let mut mutations = Vec::new(); + for (((entry, kind), path_mark), physical_mark) in + eligible.into_iter().zip(path_marks).zip(physical_marks) + { + if path_mark.is_some() || physical_mark.is_some() { continue; } let physical_version = entry @@ -6193,19 +7337,21 @@ impl Repository

{ kind: kind.to_string(), }; let key = gc_candidate_key(&candidate)?; - if engine.get(&tree, &key).await?.is_none() { - tree = engine - .put(&tree, key, encode_canonical(&candidate)?) - .await?; - cursor.report.candidates = cursor.report.candidates.saturating_add(1); - cursor.report.candidate_bytes = - cursor.report.candidate_bytes.saturating_add(candidate.len); - *cursor - .report - .candidates_by_kind - .entry(kind.to_string()) - .or_default() += 1; - } + mutations.push(Mutation::Upsert { + key, + val: encode_canonical(&candidate)?, + }); + cursor.report.candidates = cursor.report.candidates.saturating_add(1); + cursor.report.candidate_bytes = + cursor.report.candidate_bytes.saturating_add(candidate.len); + *cursor + .report + .candidates_by_kind + .entry(kind.to_string()) + .or_default() += 1; + } + if !mutations.is_empty() { + tree = engine.batch(&tree, mutations).await?; } cursor.work = RootManifest::from_tree(&tree)?; cursor.continuation = page.continuation; @@ -6378,25 +7524,52 @@ impl Repository

{ repository: self.format.repository_id, generation, active_epoch, + admission_closed: active_epoch.is_some(), updated_at_millis: now_millis, }; - let controls = MutableControlStore::new( - self.plane.clone(), - self.options.repository_prefix.clone(), - self.options.mutable_control_versions_to_retain, - )?; - match controls + match self + .maintenance_controls + .compare_exchange(CompareExchange { + path, + expected, + bytes: encode_canonical(&value)?, + }) + .await? + { + CompareExchangeOutcome::Applied(_) => Ok(()), + CompareExchangeOutcome::Conflict(_) => Err(Error::new( + ErrorCode::RefConflict, + "GC coordinator changed concurrently", + )), + } + } + + async fn persist_gc_cursor(&self, cursor: &GcCursor) -> Result<()> { + let path = gc_cursor_path(&self.options.repository_prefix, cursor.epoch)?; + let current = self.plane.load_mutable(&path).await?; + if let Some(stored) = current.as_ref() { + let previous: GcCursor = decode_canonical(&stored.bytes)?; + if previous.repository != cursor.repository || previous.epoch != cursor.epoch { + return Err(Error::new( + ErrorCode::CorruptCommit, + "durable GC cursor belongs to another epoch", + )); + } + } + let expected = current.map(|stored| stored.metadata.token); + match self + .maintenance_controls .compare_exchange(CompareExchange { path, expected, - bytes: encode_canonical(&value)?, + bytes: encode_canonical(cursor)?, }) .await? { CompareExchangeOutcome::Applied(_) => Ok(()), CompareExchangeOutcome::Conflict(_) => Err(Error::new( ErrorCode::RefConflict, - "GC coordinator changed concurrently", + "durable GC cursor changed concurrently", )), } } @@ -6767,13 +7940,6 @@ impl Repository

{ "source live version has no payload binding", ) })?; - let bytes = source.payloads.get(source_binding).await?; - if bytes.len() as u64 != *size { - return Err(Error::new( - ErrorCode::ChecksumMismatch, - "source payload length disagrees with its logical version", - )); - } cursor.report.copied_payloads = checked_fsck_add( cursor.report.copied_payloads, 1, @@ -6784,7 +7950,11 @@ impl Repository

{ *size, "history-transfer-payload-byte", )?; - Some(self.payloads.put(bytes).await?) + Some( + self.payloads + .transfer_from(&source.payloads, source_binding, *size) + .await?, + ) } LogicalObjectVersionKind::DeleteMarker => None, }; @@ -7381,6 +8551,9 @@ impl Repository

{ let mut plan_tree = self.tree_from_merge_root(&cursor.plan_root)?; let mut processed = 0usize; let mut page_mutations = Vec::new(); + let direct_external_promotion = cursor.ours == base + && theirs_commit.parents.first() == Some(&base) + && theirs_commit.delta.changes_root.is_some(); let mut ours_buffer = VecDeque::new(); let mut theirs_buffer = VecDeque::new(); if let Some(pending) = cursor.ours_pending.take() { @@ -7389,24 +8562,83 @@ impl Repository

{ if let Some(pending) = cursor.theirs_pending.take() { theirs_buffer.push_back(pending); } + let mut theirs_delta_page = false; + if !cursor.ours_finished && cursor.ours_diff.is_none() { + if base == cursor.ours { + cursor.ours_finished = true; + } else if let Some(diffs) = self + .direct_parent_diffs(base, &base_commit, &ours_commit) + .await? + { + cursor.ours_diff = + structural_cursor_with_pending(None, &base_tree, &ours_tree, diffs) + .map(MergeDiffCursor::Structural); + cursor.ours_finished = cursor.ours_diff.is_none(); + } + } + if !cursor.theirs_finished && cursor.theirs_diff.is_none() { + if base == cursor.theirs { + cursor.theirs_finished = true; + } else if let Some(diffs) = self + .direct_parent_diffs(base, &base_commit, &theirs_commit) + .await? + { + cursor.theirs_diff = + structural_cursor_with_pending(None, &base_tree, &theirs_tree, diffs) + .map(MergeDiffCursor::Structural); + cursor.theirs_finished = cursor.theirs_diff.is_none(); + } + } + // The overwhelmingly common high-throughput merge has the target at + // the selected base and one direct source child. Its exact delta tree + // is already an ordered, restartable merge stream. + if cursor.ours_finished + && !cursor.theirs_finished + && theirs_commit.parents.first() == Some(&base) + && !matches!(cursor.theirs_diff, Some(MergeDiffCursor::Structural(_))) + { + let after = match cursor.theirs_diff.as_ref() { + Some(MergeDiffCursor::Delta { after }) => Some(after.as_slice()), + None => None, + Some(MergeDiffCursor::Structural(_)) => unreachable!("excluded above"), + }; + let (diffs, next_after) = self + .direct_parent_diff_page(&base_commit, &theirs_commit, after, max_steps) + .await?; + theirs_buffer.extend(diffs); + cursor.theirs_diff = next_after.map(|after| MergeDiffCursor::Delta { after }); + cursor.theirs_finished = cursor.theirs_diff.is_none(); + theirs_delta_page = true; + } if !cursor.ours_finished { let page = state_engine - .structural_diff_page(&base_tree, &ours_tree, cursor.ours_diff.as_ref(), max_steps) + .structural_diff_page( + &base_tree, + &ours_tree, + cursor.ours_diff.as_ref().map(|cursor| match cursor { + MergeDiffCursor::Structural(cursor) => cursor, + MergeDiffCursor::Delta { .. } => unreachable!("delta handled separately"), + }), + max_steps, + ) .await?; ours_buffer.extend(page.diffs); - cursor.ours_diff = page.next_cursor; + cursor.ours_diff = page.next_cursor.map(MergeDiffCursor::Structural); } - if !cursor.theirs_finished { + if !cursor.theirs_finished && !theirs_delta_page { let page = state_engine .structural_diff_page( &base_tree, &theirs_tree, - cursor.theirs_diff.as_ref(), + cursor.theirs_diff.as_ref().map(|cursor| match cursor { + MergeDiffCursor::Structural(cursor) => cursor, + MergeDiffCursor::Delta { .. } => unreachable!("delta handled separately"), + }), max_steps, ) .await?; theirs_buffer.extend(page.diffs); - cursor.theirs_diff = page.next_cursor; + cursor.theirs_diff = page.next_cursor.map(MergeDiffCursor::Structural); } while processed < max_steps { let key_order = match (ours_buffer.front(), theirs_buffer.front()) { @@ -7464,10 +8696,12 @@ impl Repository

{ conflict, }; if selected != ours_value { - page_mutations.push(Mutation::Upsert { - key: merge_change_key(&key), - val: encode_canonical(&record)?, - }); + if !direct_external_promotion { + page_mutations.push(Mutation::Upsert { + key: merge_change_key(&key), + val: encode_canonical(&record)?, + }); + } let change = merge_change_from_record(&record)?; emitted_changes.push(change); cursor.planned_changes = @@ -7492,17 +8726,34 @@ impl Repository

{ processed += 1; } cursor.ours_diff = structural_cursor_with_pending( - cursor.ours_diff.take(), + cursor.ours_diff.take().map(|cursor| match cursor { + MergeDiffCursor::Structural(cursor) => cursor, + MergeDiffCursor::Delta { .. } => unreachable!("delta handled separately"), + }), &base_tree, &ours_tree, ours_buffer.into(), - ); - cursor.theirs_diff = structural_cursor_with_pending( - cursor.theirs_diff.take(), - &base_tree, - &theirs_tree, - theirs_buffer.into(), - ); + ) + .map(MergeDiffCursor::Structural); + if theirs_delta_page { + if !theirs_buffer.is_empty() { + return Err(Error::new( + ErrorCode::InternalInvariant, + "direct-child delta merge page was not fully consumed", + )); + } + } else { + cursor.theirs_diff = structural_cursor_with_pending( + cursor.theirs_diff.take().map(|cursor| match cursor { + MergeDiffCursor::Structural(cursor) => cursor, + MergeDiffCursor::Delta { .. } => unreachable!("delta handled separately"), + }), + &base_tree, + &theirs_tree, + theirs_buffer.into(), + ) + .map(MergeDiffCursor::Structural); + } cursor.ours_finished = cursor.ours_diff.is_none(); cursor.theirs_finished = cursor.theirs_diff.is_none(); if !page_mutations.is_empty() { @@ -7516,6 +8767,13 @@ impl Repository

{ { if cursor.policy == MergePolicy::Fail && cursor.conflicts != 0 { cursor.phase = MergePhase::Conflicted; + } else if direct_external_promotion { + cursor.final_objects = Some(theirs_commit.state.objects.clone()); + cursor.final_versions = Some(theirs_commit.state.versions.clone()); + cursor.delta_root = theirs_commit.delta.changes_root.clone(); + cursor.built_changes = cursor.planned_changes; + cursor.version_diff_finished = true; + cursor.phase = MergePhase::ReadyToPublish; } else { cursor.final_objects = Some(ours_commit.state.objects.clone()); cursor.final_versions = Some(ours_commit.state.versions.clone()); @@ -7527,6 +8785,201 @@ impl Repository

{ Ok(processed) } + /// Materialize the exact inline delta of a direct first-parent child. + /// Only the changed keys are read, so merge planning remains proportional + /// to a sparse publication even when the surrounding snapshot is huge. + async fn direct_parent_diffs( + &self, + parent: CommitId, + parent_commit: &BucketCommit, + child_commit: &BucketCommit, + ) -> Result>> { + if child_commit.parents.first() != Some(&parent) + || child_commit.delta.changes_root.is_some() + { + return Ok(None); + } + if child_commit.delta.changes.is_empty() { + return Ok(Some(Vec::new())); + } + let keys = child_commit + .delta + .changes + .iter() + .map(|transition| transition.key.as_slice()) + .collect::>(); + let engine = self.engine(self.node_store.clone()); + let parent_tree = self.tree_from_root(&parent_commit.state.objects)?; + let child_tree = self.tree_from_root(&child_commit.state.objects)?; + let (parent_values, child_values) = futures_util::try_join!( + engine.get_many(&parent_tree, &keys), + engine.get_many(&child_tree, &keys), + )?; + let mut diffs = Vec::with_capacity(keys.len()); + for ((transition, before), after) in child_commit + .delta + .changes + .iter() + .zip(parent_values) + .zip(child_values) + { + if current_id(before.as_deref())? != transition.previous + || (!transition.delete_marker + && current_id(after.as_deref())? != Some(transition.next)) + || (transition.delete_marker && after.is_some()) + { + return Err(Error::new( + ErrorCode::CorruptCommit, + "inline commit delta disagrees with its state roots", + )); + } + let diff = match (before, after) { + (None, Some(val)) => Diff::Added { + key: transition.key.clone(), + val, + }, + (Some(val), None) => Diff::Removed { + key: transition.key.clone(), + val, + }, + (Some(old), Some(new)) => Diff::Changed { + key: transition.key.clone(), + old, + new, + }, + (None, None) => { + return Err(Error::new( + ErrorCode::CorruptCommit, + "inline commit delta records an absent-to-absent transition", + )); + } + }; + diffs.push(diff); + } + Ok(Some(diffs)) + } + + async fn direct_parent_diff_page( + &self, + parent_commit: &BucketCommit, + child_commit: &BucketCommit, + after: Option<&[u8]>, + limit: usize, + ) -> Result<(Vec, Option>)> { + let (transitions, next_after) = self.commit_delta_page(child_commit, after, limit).await?; + if transitions.is_empty() { + return Ok((Vec::new(), next_after)); + } + let keys = transitions + .iter() + .map(|transition| transition.key.as_slice()) + .collect::>(); + let engine = self.engine(self.node_store.clone()); + let parent_tree = self.tree_from_root(&parent_commit.state.objects)?; + let child_tree = self.tree_from_root(&child_commit.state.objects)?; + let (parent_values, child_values) = futures_util::try_join!( + engine.get_many(&parent_tree, &keys), + engine.get_many(&child_tree, &keys), + )?; + let mut diffs = Vec::with_capacity(keys.len()); + for ((transition, before), after) in + transitions.into_iter().zip(parent_values).zip(child_values) + { + if current_id(before.as_deref())? != transition.previous + || (!transition.delete_marker + && current_id(after.as_deref())? != Some(transition.next)) + || (transition.delete_marker && after.is_some()) + { + return Err(Error::new( + ErrorCode::CorruptCommit, + "commit delta page disagrees with its state roots", + )); + } + diffs.push(match (before, after) { + (None, Some(val)) => Diff::Added { + key: transition.key, + val, + }, + (Some(val), None) => Diff::Removed { + key: transition.key, + val, + }, + (Some(old), Some(new)) => Diff::Changed { + key: transition.key, + old, + new, + }, + (None, None) => { + return Err(Error::new( + ErrorCode::CorruptCommit, + "commit delta page records an absent-to-absent transition", + )); + } + }); + } + Ok((diffs, next_after)) + } + + async fn commit_delta_page( + &self, + commit: &BucketCommit, + after: Option<&[u8]>, + limit: usize, + ) -> Result<(Vec, Option>)> { + if let Some(root) = &commit.delta.changes_root { + let tree = self.tree_from_root(root)?; + let engine = self.engine(self.node_store.clone()); + let mut entries = match after { + Some(after) => engine.range_after(&tree, after, None).await?, + None => engine.range(&tree, b"", None).await?, + }; + let mut transitions = Vec::with_capacity(limit.saturating_add(1)); + while transitions.len() <= limit { + let Some(entry) = entries.next().await else { + break; + }; + let (key, encoded) = entry?; + let transition: ObjectTransition = decode_canonical(&encoded)?; + if transition.key != key { + return Err(Error::new( + ErrorCode::CorruptCommit, + "external commit delta key disagrees with its transition", + )); + } + transitions.push(transition); + } + let has_more = transitions.len() > limit; + if has_more { + transitions.truncate(limit); + } + let next = has_more.then(|| { + transitions + .last() + .expect("a full delta page has a last transition") + .key + .clone() + }); + return Ok((transitions, next)); + } + + let start = after.map_or(0, |after| { + commit + .delta + .changes + .partition_point(|transition| transition.key.as_slice() <= after) + }); + let end = start.saturating_add(limit).min(commit.delta.changes.len()); + let transitions = commit.delta.changes[start..end].to_vec(); + let next = (end < commit.delta.changes.len()).then(|| { + transitions + .last() + .expect("a truncated inline delta page has a last transition") + .key + .clone() + }); + Ok((transitions, next)) + } + async fn advance_merge_version_union( &self, cursor: &mut MergeCursor, @@ -7534,6 +8987,54 @@ impl Repository

{ ) -> Result { let ours_commit = self.load_commit_metadata(cursor.ours).await?; let theirs_commit = self.load_commit_metadata(cursor.theirs).await?; + let base = cursor.selected_base.ok_or_else(|| { + Error::new( + ErrorCode::InternalInvariant, + "merge plan has no selected base", + ) + })?; + if cursor.version_diff.is_none() && cursor.theirs == base { + cursor.version_diff_finished = true; + cursor.phase = MergePhase::BuildingObjects; + cursor.build_after = None; + return Ok(0); + } + if cursor.ours == base + && theirs_commit.parents.first() == Some(&base) + && !matches!(cursor.version_diff, Some(MergeDiffCursor::Structural(_))) + { + let after = match cursor.version_diff.as_ref() { + Some(MergeDiffCursor::Delta { after }) => Some(after.as_slice()), + None => None, + Some(MergeDiffCursor::Structural(_)) => unreachable!("excluded above"), + }; + let delta_is_safe = + after.is_some() || !self.commit_delta_contains_delete(&theirs_commit).await?; + if delta_is_safe { + let (mutations, next_after) = self + .direct_child_version_mutations_page(&theirs_commit, after, max_steps) + .await?; + let processed = mutations.len(); + if !mutations.is_empty() { + let versions = + self.tree_from_root(cursor.final_versions.as_ref().ok_or_else(|| { + Error::new(ErrorCode::InternalInvariant, "merge version root is absent") + })?)?; + let versions = self + .merge_state_engine() + .batch(&versions, mutations) + .await?; + cursor.final_versions = Some(RootManifest::from_tree(&versions)?); + } + cursor.version_diff = next_after.map(|after| MergeDiffCursor::Delta { after }); + if cursor.version_diff.is_none() { + cursor.version_diff_finished = true; + cursor.phase = MergePhase::BuildingObjects; + cursor.build_after = None; + } + return Ok(processed); + } + } let ours_versions = self.tree_from_root(&ours_commit.state.versions)?; let theirs_versions = self.tree_from_root(&theirs_commit.state.versions)?; let read_engine = self.engine(self.node_store.clone()); @@ -7541,7 +9042,10 @@ impl Repository

{ .structural_diff_page( &ours_versions, &theirs_versions, - cursor.version_diff.as_ref(), + cursor.version_diff.as_ref().map(|cursor| match cursor { + MergeDiffCursor::Structural(cursor) => cursor, + MergeDiffCursor::Delta { .. } => unreachable!("delta handled separately"), + }), max_steps, ) .await?; @@ -7568,7 +9072,7 @@ impl Repository

{ let versions = state_engine.batch(&versions, mutations).await?; cursor.final_versions = Some(RootManifest::from_tree(&versions)?); } - cursor.version_diff = page.next_cursor; + cursor.version_diff = page.next_cursor.map(MergeDiffCursor::Structural); if cursor.version_diff.is_none() { cursor.version_diff_finished = true; cursor.phase = MergePhase::BuildingObjects; @@ -7577,6 +9081,83 @@ impl Repository

{ Ok(processed) } + /// Build the immutable version-tree additions for an ordinary direct + /// child from its current-object records. Delete markers are not present + /// in the current-object tree, so those commits deliberately retain the + /// structural union fallback. + async fn direct_child_version_mutations_page( + &self, + child_commit: &BucketCommit, + after: Option<&[u8]>, + limit: usize, + ) -> Result<(Vec, Option>)> { + let (transitions, next_after) = self.commit_delta_page(child_commit, after, limit).await?; + let keys = transitions + .iter() + .map(|transition| transition.key.as_slice()) + .collect::>(); + let child_tree = self.tree_from_root(&child_commit.state.objects)?; + let values = self + .engine(self.node_store.clone()) + .get_many(&child_tree, &keys) + .await?; + let mut mutations = Vec::with_capacity(values.len()); + for (transition, value) in transitions.iter().zip(values) { + if transition.delete_marker { + return Err(Error::new( + ErrorCode::InternalInvariant, + "delete delta entered the direct version-union path", + )); + } + let value = value.ok_or_else(|| { + Error::new( + ErrorCode::CorruptCommit, + "non-delete delta is absent from the child state", + ) + })?; + let current: CurrentObject = decode_canonical(&value)?; + current.version.validate()?; + if current.version.id != transition.next { + return Err(Error::new( + ErrorCode::CorruptCommit, + "commit delta version disagrees with child state", + )); + } + mutations.push(Mutation::Upsert { + key: version_tree_key( + &transition.key, + current.version.body.order, + current.version.id, + ), + val: encode_canonical(¤t.version)?, + }); + } + Ok((mutations, next_after)) + } + + async fn commit_delta_contains_delete(&self, commit: &BucketCommit) -> Result { + if commit.delta.changes_root.is_none() { + return Ok(commit + .delta + .changes + .iter() + .any(|transition| transition.delete_marker)); + } + let mut after = None; + loop { + let (page, next) = self + .commit_delta_page(commit, after.as_deref(), 1_000) + .await?; + if page.iter().any(|transition| transition.delete_marker) { + return Ok(true); + } + let Some(next) = next else { + return Ok(false); + }; + after = Some(next); + } + } + async fn advance_merge_object_build( &self, cursor: &mut MergeCursor, @@ -7802,6 +9383,46 @@ impl Repository

{ "merge change cursor belongs to another plan", )); } + if cursor.selected_base == Some(cursor.ours) { + let theirs = self.load_commit_metadata(cursor.theirs).await?; + if theirs.parents.first() == Some(&cursor.ours) + && theirs.delta.changes_root.is_some() + && cursor.built_changes == cursor.planned_changes + { + let after = continuation + .map(|continuation| { + continuation + .after + .strip_prefix(MERGE_CHANGE_PREFIX) + .ok_or_else(|| { + Error::new( + ErrorCode::InvalidContinuationToken, + "direct merge change cursor has an invalid prefix", + ) + }) + }) + .transpose()?; + let (transitions, next_after) = + self.commit_delta_page(&theirs, after, limit).await?; + let changes = transitions + .into_iter() + .map(|transition| MergeChange { + key: transition.key, + from: transition.previous, + to: (!transition.delete_marker).then_some(transition.next), + }) + .collect(); + return Ok(MergeChangePage { + changes, + continuation: next_after.map(|after| MergeChangeCursor { + repository: cursor.repository, + job: cursor.job, + plan_root: cursor.plan_root.clone(), + after: merge_change_key(&after), + }), + }); + } + } let engine = self.merge_plan_engine(cursor.job)?; let tree = self.tree_from_merge_root(&cursor.plan_root)?; let end = prolly::prefix_range(MERGE_CHANGE_PREFIX).1; @@ -8069,6 +9690,63 @@ fn checked_fsck_add(current: u64, add: u64, counter: &str) -> Result { }) } +fn record_fsck_logical_payload( + report: &mut FsckReport, + version: &ObjectVersion, + deep: bool, +) -> Result<()> { + let LogicalObjectVersionKind::Live { size, .. } = &version.body.kind else { + return Ok(()); + }; + if version.binding.is_none() { + return Err(Error::new( + ErrorCode::CorruptCommit, + "live object version has no payload binding", + )); + } + report.payloads_verified = checked_fsck_add(report.payloads_verified, 1, "payload")?; + report.payload_bytes_verified = + checked_fsck_add(report.payload_bytes_verified, *size, "payload-byte")?; + if deep { + report.deep_content_bytes_verified = checked_fsck_add( + report.deep_content_bytes_verified, + *size, + "deep-content-byte", + )?; + } + Ok(()) +} + +fn validate_same_fsck_physical_binding( + existing: &crate::PayloadBinding, + candidate: &crate::PayloadBinding, +) -> Result<()> { + if existing.path != candidate.path + || existing.provider_etag != candidate.provider_etag + || existing.provider_version_id != candidate.provider_version_id + || existing.checksum_sha256 != candidate.checksum_sha256 + { + return Err(Error::new( + ErrorCode::ChecksumMismatch, + "logical payload bindings disagree about their physical object", + )); + } + Ok(()) +} + +fn payload_physical_key(binding: &crate::PayloadBinding) -> Vec { + let mut identity = binding.path.as_str().as_bytes().to_vec(); + identity.push(0); + if let Some(version) = binding.provider_version_id.as_deref() { + identity.extend_from_slice(version.as_bytes()); + } else { + identity.extend_from_slice(binding.provider_etag.as_bytes()); + } + let mut key = b"p/".to_vec(); + key.extend_from_slice(&crate::codec::sha256(&identity)); + key +} + fn history_transfer_version_operation( repository: crate::RepositoryId, source: ObjectVersionId, @@ -8184,6 +9862,14 @@ fn object_diff_from_prolly(diff: Diff) -> Result { }) } +fn object_diff_from_transition(transition: &ObjectTransition) -> Result { + Ok(ObjectDiff { + key: transition.key.clone(), + from: transition.previous, + to: (!transition.delete_marker).then_some(transition.next), + }) +} + fn current_id(value: Option<&[u8]>) -> Result> { value .map(|value| { @@ -8275,34 +9961,6 @@ fn decode_retention_pin_tag(tag: &str) -> Result> { }) } -fn staged_put( - key: Vec, - bytes: Vec, - headers: ObjectHeaders, - user_metadata: BTreeMap, - binding: crate::PayloadBinding, -) -> StagedMutation { - let size = bytes.len() as u64; - let checksum_md5: [u8; 16] = Md5::digest(&bytes).into(); - let checksum_sha256: [u8; 32] = Sha256::digest(&bytes).into(); - StagedMutation { - body: StagedMutationBody::Put(Box::new(StagedPut { - key, - size, - logical_etag: format!("\"{}\"", hex::encode(checksum_md5)), - checksums: Checksums { - md5: Some(checksum_md5), - sha256: Some(checksum_sha256), - algorithm_values: BTreeMap::new(), - }, - headers, - user_metadata, - tags: BTreeMap::new(), - binding, - })), - } -} - fn validate_options(options: &RepositoryOptions) -> Result<()> { crate::repository::validate_branch(&options.default_branch)?; options.idempotency_retention.validate()?; @@ -8357,23 +10015,73 @@ fn gc_coordinator_path(prefix: &str) -> Result { ObjectPath::new(format!("{prefix}/gc/coordinator.cbor")) } -fn gc_dirty_root_prefix(prefix: &str, epoch: OperationId) -> String { - format!("{prefix}/gc/dirty/{epoch}/") +fn gc_cursor_path(prefix: &str, epoch: OperationId) -> Result { + ObjectPath::new(format!("{prefix}/gc/epochs/{epoch}/cursor.cbor")) } -fn gc_dirty_root_sequence_prefix(prefix: &str, epoch: OperationId, sequence: u64) -> String { - format!("{}{sequence:020}/", gc_dirty_root_prefix(prefix, epoch)) +fn fsck_cursor_path(prefix: &str, job: OperationId) -> Result { + ObjectPath::new(format!("{prefix}/administration/fsck/{job}/cursor.cbor")) +} + +fn physical_list_entry_target(entry: crate::PhysicalListEntry) -> (ObjectPath, PhysicalVersion) { + let token = entry.metadata.token; + let physical = token + .version_id + .clone() + .map(|version_id| PhysicalVersion::Versioned { version_id }) + .unwrap_or_else(|| PhysicalVersion::Unversioned { token: Some(token) }); + (entry.path, physical) +} + +fn physical_stored_object_target( + path: ObjectPath, + stored: crate::StoredObject, +) -> (ObjectPath, PhysicalVersion) { + let token = stored.metadata.token; + let physical = token + .version_id + .clone() + .map(|version_id| PhysicalVersion::Versioned { version_id }) + .unwrap_or_else(|| PhysicalVersion::Unversioned { token: Some(token) }); + (path, physical) +} + +fn gc_publication_ticket_prefix(prefix: &str, repository: crate::RepositoryId) -> String { + format!( + "{prefix}/gc/publications/{}/", + hex::encode(repository.as_bytes()) + ) } -fn gc_dirty_root_path(prefix: &str, event: &GcDirtyRoot, digest: [u8; 32]) -> Result { +fn gc_publication_ticket_path( + prefix: &str, + repository: crate::RepositoryId, + instance: OperationId, + request_digest: [u8; 32], +) -> Result { ObjectPath::new(format!( - "{}{}/{:}", - gc_dirty_root_sequence_prefix(prefix, event.epoch, event.sequence), - event.namespace, - hex::encode(digest) + "{}{instance}/{}", + gc_publication_ticket_prefix(prefix, repository), + hex::encode(request_digest) )) } +fn publication_ticket_digest(request: &CompareExchange) -> [u8; 32] { + crate::model::derive_input_digest(&[ + b"publication-ticket", + request.path.as_str().as_bytes(), + &request.bytes, + ]) +} + +fn gc_dirty_root_prefix(prefix: &str, epoch: OperationId) -> String { + format!("{prefix}/gc/dirty/{epoch}/") +} + +fn gc_dirty_root_sequence_prefix(prefix: &str, epoch: OperationId, sequence: u64) -> String { + format!("{}{sequence:020}/", gc_dirty_root_prefix(prefix, epoch)) +} + fn commit_path(prefix: &str, id: CommitId) -> Result { let encoded = hex::encode(id.as_bytes()); ObjectPath::new(format!( @@ -8435,7 +10143,7 @@ fn gc_managed_kind(prefix: &str, path: &ObjectPath) -> Option<&'static str> { Some("commits") } else if relative.starts_with("nodes/sha256/") { Some("nodes") - } else if relative.starts_with("payloads/") || relative.starts_with("payload-packs/") { + } else if relative.starts_with("payloads/") { Some("payloads") } else { None diff --git a/extensions/s3/core/src/store.rs b/extensions/s3/core/src/store.rs index 27fff554..5fa5bf0c 100644 --- a/extensions/s3/core/src/store.rs +++ b/extensions/s3/core/src/store.rs @@ -535,17 +535,21 @@ impl ProllyObjectStore

{ &self, container: CommitId, prepared: PreparedNodePack, - payload_offset: u64, + node_region_offset: u64, ) -> Result<()> { - self.commit_node_pack_for(PackedCommitId::Native(container), prepared, payload_offset) - .await + self.commit_node_pack_for( + PackedCommitId::Native(container), + prepared, + node_region_offset, + ) + .await } async fn commit_node_pack_for( &self, container: PackedCommitId, prepared: PreparedNodePack, - payload_offset: u64, + node_region_offset: u64, ) -> Result<()> { let Some(state) = &self.packed else { return Err(Error::new( @@ -568,7 +572,7 @@ impl ProllyObjectStore

{ PackedNodeLocation { container, pack: reference.id, - absolute_offset: payload_offset + entry.offset, + absolute_offset: node_region_offset + entry.offset, len: entry.len, sha256: entry.sha256, }, @@ -599,11 +603,11 @@ impl ProllyObjectStore

{ object: &CommitObject, encoded: &[u8], ) -> Result<()> { - let payload_offset = CommitObject::node_payload_offset(encoded)?; + let node_region_offset = CommitObject::node_region_offset(encoded)?; self.register_node_pack( PackedCommitId::Native(container), object.node_pack.as_ref(), - payload_offset, + node_region_offset, ) } @@ -612,7 +616,7 @@ impl ProllyObjectStore

{ &self, container: PackedCommitId, pack: Option<&NodePack>, - payload_offset: Option, + node_region_offset: Option, ) -> Result<()> { let Some(pack) = pack else { return Ok(()); @@ -620,10 +624,10 @@ impl ProllyObjectStore

{ let Some(state) = &self.packed else { return Ok(()); }; - let payload_offset = payload_offset.ok_or_else(|| { + let node_region_offset = node_region_offset.ok_or_else(|| { Error::new( ErrorCode::CorruptCommit, - "commit node-pack payload is absent", + "commit node-pack node region is absent", ) })?; let reference = pack.reference()?; @@ -637,7 +641,7 @@ impl ProllyObjectStore

{ PackedNodeLocation { container, pack: reference.id, - absolute_offset: payload_offset + entry.offset, + absolute_offset: node_region_offset + entry.offset, len: entry.len, sha256: entry.sha256, }, diff --git a/extensions/s3/core/src/tag.rs b/extensions/s3/core/src/tag.rs index da807d7e..ced69d37 100644 --- a/extensions/s3/core/src/tag.rs +++ b/extensions/s3/core/src/tag.rs @@ -254,6 +254,7 @@ impl TagStore

{ ErrorCode::RefConflict, "tag changed concurrently", )), + Err(error) if error.code == ErrorCode::PreconditionFailed => Err(error), Err(error) => { if let Some(current) = self.plane.load_mutable(&path).await? { if current.bytes == bytes { diff --git a/extensions/s3/core/tests/control_version_compaction.rs b/extensions/s3/core/tests/control_version_compaction.rs index 51af26fe..6697dbf3 100644 --- a/extensions/s3/core/tests/control_version_compaction.rs +++ b/extensions/s3/core/tests/control_version_compaction.rs @@ -67,6 +67,15 @@ fn every_mutable_control_family_has_one_canonical_classification() { "journal-index/heads/6d61696e.cbor", MutableControlKind::JournalDerivedIndexHead, ), + ("gc/coordinator.cbor", MutableControlKind::GcCoordinator), + ( + "gc/epochs/00000000-0000-0000-0000-000000000001/cursor.cbor", + MutableControlKind::GcCursor, + ), + ( + "administration/fsck/00000000-0000-0000-0000-000000000002/cursor.cbor", + MutableControlKind::FsckCursor, + ), ]; for (relative, expected) in cases { let path = ObjectPath::new(format!("{prefix}/{relative}")).unwrap(); diff --git a/extensions/s3/core/tests/gc.rs b/extensions/s3/core/tests/gc.rs index fb6f69c4..76f91fc4 100644 --- a/extensions/s3/core/tests/gc.rs +++ b/extensions/s3/core/tests/gc.rs @@ -1,13 +1,13 @@ use std::{collections::BTreeMap, sync::Arc, time::Duration}; use prolly_s3_core::{ - decode_canonical, encode_canonical, GcCursor, GcPhase, ImmutablePut, MemoryObjectPlane, - ObjectHeaders, ObjectPath, ObjectPlane, Repository, RepositoryOptions, + GcPhase, ImmutablePut, MemoryObjectPlane, ObjectHeaders, ObjectPath, ObjectPlane, Repository, + RepositoryOptions, }; use sha2::{Digest, Sha256}; #[tokio::test] -async fn concurrent_gc_retains_dirty_and_pinned_roots_and_deletes_exact_orphans() { +async fn gc_fences_cross_handle_publications_and_deletes_exact_orphans() { let plane = Arc::new(MemoryObjectPlane::new(true)); let options = RepositoryOptions { repository_prefix: ".tests/gc".to_string(), @@ -17,6 +17,44 @@ async fn concurrent_gc_retains_dirty_and_pinned_roots_and_deletes_exact_orphans( let repository = Repository::initialize(plane.clone(), options.clone()) .await .unwrap(); + let session = repository + .begin_commit_session("main", "independent live objects", 60_000) + .await + .unwrap(); + let staged = repository + .stage_commit_session_put_batch( + &session, + vec![ + ( + b"packed/a".to_vec(), + b"alpha".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ), + ( + b"packed/b".to_vec(), + b"bravo".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ), + ], + 2, + ) + .await + .unwrap(); + let batch_commit = repository + .publish_commit_session(session, staged) + .await + .unwrap(); + let live_payload = repository + .head_object_at("main", batch_commit.id, b"packed/a") + .await + .unwrap() + .unwrap() + .version + .binding + .unwrap() + .path; let initial = repository.head("main").await.unwrap(); repository.create_branch("scratch", initial).await.unwrap(); let pinned = repository @@ -50,19 +88,41 @@ async fn concurrent_gc_retains_dirty_and_pinned_roots_and_deletes_exact_orphans( }) .await .unwrap(); + let protected_path = ObjectPath::new(format!( + ".tests/gc/payloads/sha256/dd/dd/{}", + "dd".repeat(32) + )) + .unwrap(); + let protected = b"provider-retained orphan".to_vec(); + plane + .put_immutable(ImmutablePut { + path: protected_path.clone(), + expected_sha256: Sha256::digest(&protected).into(), + bytes: protected.clone(), + }) + .await + .unwrap(); + plane.protect_exact_deletes(protected_path.clone()); tokio::time::sleep(Duration::from_millis(5)).await; + let external_writer = Repository::open(plane.clone(), options.clone()) + .await + .unwrap(); let mut gc = repository.start_gc(1).await.unwrap(); - repository + let during_gc = external_writer .put_object( "main", b"during-gc.txt".to_vec(), - b"published while marking".to_vec(), + b"must be fenced".to_vec(), ObjectHeaders::default(), BTreeMap::new(), ) .await - .unwrap(); + .unwrap_err(); + assert_eq!( + during_gc.code, + prolly_s3_core::ErrorCode::PreconditionFailed + ); for _ in 0..10_000 { if gc.phase == GcPhase::Ready { break; @@ -70,27 +130,52 @@ async fn concurrent_gc_retains_dirty_and_pinned_roots_and_deletes_exact_orphans( gc = repository.advance_gc(&gc, 1).await.unwrap().cursor; } assert_eq!(gc.phase, GcPhase::Ready); - assert!(gc.report.dirty_roots >= 1); + assert_eq!(gc.report.dirty_roots, 0); assert!(gc.report.candidates >= 1); - repository + let before_sweep = external_writer .put_object( "main", b"before-sweep.txt".to_vec(), - b"forces dirty-root catch-up".to_vec(), + b"must also be fenced".to_vec(), ObjectHeaders::default(), BTreeMap::new(), ) .await - .unwrap(); - let restarted = repository.sweep_gc(&gc, 1).await.unwrap(); - assert!(restarted.restarted_for_new_roots); - assert_eq!(restarted.cursor.phase, GcPhase::CatchUpDirtyRoots); - gc = restarted.cursor; - let persisted = encode_canonical(&gc).unwrap(); + .unwrap_err(); + assert_eq!( + before_sweep.code, + prolly_s3_core::ErrorCode::PreconditionFailed + ); drop(repository); let repository = Repository::open(plane.clone(), options).await.unwrap(); - gc = decode_canonical::(&persisted).unwrap(); + gc = repository.resume_gc().await.unwrap().unwrap(); + let after_restart_writer = Repository::open( + plane.clone(), + RepositoryOptions { + repository_prefix: ".tests/gc".to_string(), + provider_per_key_version_limit: prolly_s3_core::ProviderPerKeyVersionLimit::Finite( + 10_000, + ), + ..RepositoryOptions::default() + }, + ) + .await + .unwrap(); + let restart_fence = after_restart_writer + .put_object( + "main", + b"restart-race.txt".to_vec(), + b"must remain fenced".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap_err(); + assert_eq!( + restart_fence.code, + prolly_s3_core::ErrorCode::PreconditionFailed + ); for _ in 0..10_000 { gc = match gc.phase { @@ -101,16 +186,39 @@ async fn concurrent_gc_retains_dirty_and_pinned_roots_and_deletes_exact_orphans( } assert_eq!(gc.phase, GcPhase::Complete); assert!(gc.report.deleted_versions >= 1); + assert_eq!(gc.report.protected_versions, 1); + assert_eq!(gc.report.protected_bytes, protected.len() as u64); assert!(plane.head(&orphan_path).await.unwrap().is_none()); + assert!(plane.head(&protected_path).await.unwrap().is_some()); + assert!(plane.head(&live_payload).await.unwrap().is_some()); + assert_eq!( + repository + .get_object("main", b"packed/b") + .await + .unwrap() + .unwrap() + .bytes, + b"bravo" + ); + external_writer + .put_object( + "main", + b"after-gc.txt".to_vec(), + b"publication admission reopened".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap(); repository.advance_branch_indexes("main").await.unwrap(); assert_eq!( repository - .get_object("main", b"during-gc.txt") + .get_object("main", b"after-gc.txt") .await .unwrap() .unwrap() .bytes, - b"published while marking" + b"publication admission reopened" ); repository.commit(pinned).await.unwrap(); } diff --git a/extensions/s3/core/tests/history_recovery.rs b/extensions/s3/core/tests/history_recovery.rs index 11f69ef2..f342e7c2 100644 --- a/extensions/s3/core/tests/history_recovery.rs +++ b/extensions/s3/core/tests/history_recovery.rs @@ -1,10 +1,158 @@ use std::{collections::BTreeMap, sync::Arc, time::Duration}; use prolly_s3_core::{ - FixedClock, MemoryObjectPlane, ObjectHeaders, ProviderPerKeyVersionLimit, Repository, - RepositoryOptions, SequenceIdSource, TraversalBudget, + ErrorCode, FixedClock, FsckPhase, ListRequest, MemoryObjectPlane, ObjectHeaders, ObjectPlane, + ProviderPerKeyVersionLimit, Repository, RepositoryOptions, SequenceIdSource, TraversalBudget, }; +#[tokio::test] +async fn fsck_checkpoint_resumes_across_processes_and_fences_stale_workers() { + let plane = Arc::new(MemoryObjectPlane::new(true)); + let options = RepositoryOptions { + repository_prefix: ".tests/fsck-durable-resume".to_string(), + writer: "fsck-writer".to_string(), + mutable_control_versions_to_retain: 4, + provider_per_key_version_limit: ProviderPerKeyVersionLimit::Finite(10_000), + ..RepositoryOptions::default() + }; + let repository = Repository::initialize(plane.clone(), options.clone()) + .await + .unwrap(); + for (key, body) in [ + (b"a".to_vec(), b"alpha".to_vec()), + (b"b".to_vec(), b"bravo".to_vec()), + ] { + repository + .put_object("main", key, body, ObjectHeaders::default(), BTreeMap::new()) + .await + .unwrap(); + } + + let initial = repository.start_fsck("main", true).await.unwrap(); + assert_eq!(initial.checkpoint_generation, 1); + let first = repository.advance_fsck(&initial, 1).await.unwrap().cursor; + assert_eq!(first.checkpoint_generation, 2); + + let reopened = Repository::open(plane.clone(), options.clone()) + .await + .unwrap(); + assert_eq!( + reopened.resume_fsck(initial.job).await.unwrap(), + Some(first.clone()) + ); + assert_eq!( + reopened + .start_fsck_cleanup(initial.job) + .await + .unwrap_err() + .code, + ErrorCode::PreconditionFailed + ); + assert_eq!( + repository.advance_fsck(&initial, 1).await.unwrap_err().code, + ErrorCode::RefConflict + ); + + assert_eq!( + reopened.advance_fsck(&first, 1).await.unwrap_err().code, + ErrorCode::MissingClosure + ); + reopened.advance_branch_indexes("main").await.unwrap(); + let mut cursor = reopened.advance_fsck(&first, 1).await.unwrap().cursor; + drop(repository); + drop(reopened); + let resumed_process = Repository::open(plane.clone(), options).await.unwrap(); + assert_eq!( + resumed_process.resume_fsck(initial.job).await.unwrap(), + Some(cursor.clone()) + ); + while cursor.phase != FsckPhase::Complete { + cursor = resumed_process + .advance_fsck(&cursor, 1) + .await + .unwrap() + .cursor; + } + assert_eq!(cursor.report.physical_payloads_verified, 2); + assert_eq!(cursor.report.deep_physical_bytes_read, 10); + assert_eq!( + resumed_process.resume_fsck(initial.job).await.unwrap(), + Some(cursor) + ); + let checkpoint_prefix = format!( + ".tests/fsck-durable-resume/administration/fsck/{}/cursor.cbor", + initial.job + ); + let retained = plane + .list(ListRequest { + prefix: checkpoint_prefix.clone(), + continuation: None, + limit: 1_000, + include_versions: true, + }) + .await + .unwrap(); + assert!(retained.entries.len() <= 4, "{retained:?}"); + let cleanup = resumed_process + .start_fsck_cleanup(initial.job) + .await + .unwrap(); + let interrupted = resumed_process + .advance_fsck_cleanup(&cleanup, 1) + .await + .unwrap(); + assert!(!interrupted.complete); + let mut cleanup = resumed_process + .start_fsck_cleanup(initial.job) + .await + .unwrap(); + while cleanup.phase != prolly_s3_core::FsckCleanupPhase::Complete { + cleanup = resumed_process + .advance_fsck_cleanup(&cleanup, 1) + .await + .unwrap() + .cursor; + } + assert!(resumed_process + .resume_fsck(initial.job) + .await + .unwrap() + .is_none()); + assert!(plane + .list(ListRequest { + prefix: checkpoint_prefix, + continuation: None, + limit: 1_000, + include_versions: true, + }) + .await + .unwrap() + .entries + .is_empty()); + for prefix in [ + format!( + ".tests/fsck-durable-resume/administration/fsck/{}/payloads/", + initial.job + ), + format!( + ".tests/fsck-durable-resume/administration/closure/{}/tree/", + initial.closure.traversal + ), + ] { + assert!(plane + .list(ListRequest { + prefix, + continuation: None, + limit: 1_000, + include_versions: true, + }) + .await + .unwrap() + .entries + .is_empty()); + } +} + #[tokio::test] async fn history_diff_reflog_reset_and_recovery_are_bounded_and_audited() { let plane = Arc::new(MemoryObjectPlane::new(true)); @@ -265,8 +413,9 @@ async fn history_diff_reflog_reset_and_recovery_are_bounded_and_audited() { #[tokio::test] async fn logical_repair_rebinds_across_repositories_and_removes_destination_only_keys() { + let source_plane = Arc::new(MemoryObjectPlane::new(true)); let source = Repository::initialize( - Arc::new(MemoryObjectPlane::new(true)), + source_plane.clone(), RepositoryOptions { repository_prefix: ".tests/repair-source".to_string(), writer: "source".to_string(), @@ -277,8 +426,9 @@ async fn logical_repair_rebinds_across_repositories_and_removes_destination_only ) .await .unwrap(); + let destination_plane = Arc::new(MemoryObjectPlane::new(false)); let destination = Repository::initialize( - Arc::new(MemoryObjectPlane::new(false)), + destination_plane.clone(), RepositoryOptions { repository_prefix: ".tests/repair-destination".to_string(), writer: "destination".to_string(), @@ -343,6 +493,8 @@ async fn logical_repair_rebinds_across_repositories_and_removes_destination_only ) .await .unwrap(); + source_plane.reset_request_counts(); + destination_plane.reset_request_counts(); loop { let page = destination .advance_repair_from(&source, &repair, 1) @@ -355,6 +507,11 @@ async fn logical_repair_rebinds_across_repositories_and_removes_destination_only } assert_eq!(repair.report.copied_objects, 2); assert_eq!(repair.report.deleted_objects, 1); + assert_eq!( + destination_plane.request_snapshot().immutable_transfer, + repair.report.copied_objects, + "repair must delegate each complete object transfer to the provider boundary" + ); assert_eq!( destination .get_object("main", b"same.txt") diff --git a/extensions/s3/core/tests/history_transfer.rs b/extensions/s3/core/tests/history_transfer.rs index c5205d1e..07a6a65e 100644 --- a/extensions/s3/core/tests/history_transfer.rs +++ b/extensions/s3/core/tests/history_transfer.rs @@ -34,8 +34,9 @@ async fn put(repository: &Repository, branch: &str, key: &str async fn history_transfer_rebinds_payloads_and_preserves_merge_topology() { let source_clock = Arc::new(FixedClock::new(100_000)); let destination_clock = Arc::new(FixedClock::new(200_000)); + let source_plane = Arc::new(MemoryObjectPlane::new(true)); let source = Repository::initialize( - Arc::new(MemoryObjectPlane::new(true)), + source_plane.clone(), options( ".tests/history-transfer-source", "source-writer", @@ -45,8 +46,9 @@ async fn history_transfer_rebinds_payloads_and_preserves_merge_topology() { ) .await .unwrap(); + let destination_plane = Arc::new(MemoryObjectPlane::new(false)); let destination = Repository::initialize( - Arc::new(MemoryObjectPlane::new(false)), + destination_plane.clone(), options( ".tests/history-transfer-destination", "destination-writer", @@ -81,6 +83,8 @@ async fn history_transfer_rebinds_payloads_and_preserves_merge_topology() { .start_history_transfer_from(&source, "main", source_head, "main", destination_root) .await .unwrap(); + source_plane.reset_request_counts(); + destination_plane.reset_request_counts(); while !transfer.complete { let persisted = encode_canonical(&transfer).unwrap(); let restored: HistoryTransferCursor = decode_canonical(&persisted).unwrap(); @@ -94,6 +98,11 @@ async fn history_transfer_rebinds_payloads_and_preserves_merge_topology() { } assert!(transfer.report.imported_commits >= 5); assert!(transfer.report.copied_payloads >= 3); + assert_eq!( + destination_plane.request_snapshot().immutable_transfer, + transfer.report.copied_payloads, + "history import must delegate complete-object transfer to the provider boundary" + ); let mapped_head = transfer.mapped_head.unwrap(); assert_ne!(mapped_head, source_head); destination diff --git a/extensions/s3/core/tests/merge.rs b/extensions/s3/core/tests/merge.rs index ac0759da..2106599a 100644 --- a/extensions/s3/core/tests/merge.rs +++ b/extensions/s3/core/tests/merge.rs @@ -1,4 +1,4 @@ -use std::{collections::BTreeMap, sync::Arc}; +use std::{collections::BTreeMap, sync::Arc, time::Instant}; use prolly_s3_core::{ decode_canonical, encode_canonical, FixedClock, LogicalObjectVersionKind, MemoryObjectPlane, @@ -382,6 +382,105 @@ async fn repository_sparse_merge_prunes_unchanged_10k_snapshot() { ); } +#[tokio::test] +async fn repository_direct_external_delta_merge_is_paged_without_state_rebuild() { + let plane = Arc::new(MemoryObjectPlane::new(true)); + let clock = Arc::new(FixedClock::new(55_000)); + let repository_options = RepositoryOptions { + repository_prefix: ".tests/repository-direct-external-merge".to_string(), + ..options(clock) + }; + let repository = Repository::initialize(plane.clone(), repository_options.clone()) + .await + .unwrap(); + let base = repository.head("main").await.unwrap(); + repository.create_branch("feature", base).await.unwrap(); + let session = repository + .begin_commit_session("feature", "external source delta", 60_000) + .await + .unwrap(); + let mut mutations = Vec::new(); + for index in 0..129usize { + mutations.push( + repository + .stage_commit_session_put( + &session, + format!("promote/{index:03}").into_bytes(), + vec![index as u8], + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap(), + ); + } + let feature_head = repository + .publish_commit_session(session, mutations) + .await + .unwrap() + .id; + + let mut cursor = repository + .start_merge( + "main", + "feature", + None, + MergePolicy::Fail, + "promote external delta", + ) + .await + .unwrap(); + let mut processed = 0usize; + while cursor.phase != MergePhase::ReadyToPublish { + let page = repository.advance_merge(&cursor, 32).await.unwrap(); + processed += page.processed; + cursor = page.cursor; + } + assert_eq!(processed, 129); + assert_eq!(cursor.planned_changes, 129); + assert_eq!(cursor.built_changes, 129); + + let mut continuation = None; + let mut reported = 0usize; + loop { + let page = repository + .merge_changes_page(&cursor, continuation.as_ref(), 31) + .await + .unwrap(); + reported += page.changes.len(); + continuation = page.continuation; + if continuation.is_none() { + break; + } + } + assert_eq!(reported, 129); + repository.publish_merge(&cursor).await.unwrap(); + repository.advance_branch_indexes("main").await.unwrap(); + repository + .delete_branch("feature", feature_head) + .await + .unwrap(); + drop(repository); + let reopened = Repository::open( + plane, + RepositoryOptions { + read_only: true, + ..repository_options + }, + ) + .await + .unwrap(); + assert_eq!( + reopened + .get_object("main", b"promote/128") + .await + .unwrap() + .unwrap() + .bytes, + vec![128] + ); +} + #[tokio::test] #[ignore = "4K commit-graph skip-pointer gate"] async fn repository_merge_base_skips_deep_first_parent_history() { @@ -393,7 +492,9 @@ async fn repository_merge_base_skips_deep_first_parent_history() { operation_index_max_unindexed_events: 8_192, ..options(clock.clone()) }; - let repository = Repository::initialize(plane, options).await.unwrap(); + let repository = Repository::initialize(plane.clone(), options) + .await + .unwrap(); put(&repository, "main", "counter.txt", "0").await; let base = repository.head("main").await.unwrap(); repository.create_branch("stale", base).await.unwrap(); @@ -401,6 +502,8 @@ async fn repository_merge_base_skips_deep_first_parent_history() { put(&repository, "main", "counter.txt", &generation.to_string()).await; } repository.advance_branch_indexes("main").await.unwrap(); + plane.reset_request_counts(); + let started = Instant::now(); let cursor = repository .start_merge( "stale", @@ -411,7 +514,21 @@ async fn repository_merge_base_skips_deep_first_parent_history() { ) .await .unwrap(); + let elapsed = started.elapsed(); + let requests = plane.request_snapshot(); + eprintln!( + "DEEP_HISTORY_MERGE_BASE commits=4096 wall_us={} object_plane_calls={} get={} head={} list={}", + elapsed.as_micros(), + requests.total(), + requests.get, + requests.head, + requests.list, + ); assert_eq!(cursor.phase, MergePhase::Planning); assert_eq!(cursor.selected_base, Some(base)); assert_eq!(cursor.visited_commits, 0); + assert!( + requests.total() < 64, + "skip-pointer merge-base lookup used too many object-plane calls: {requests:?}" + ); } diff --git a/extensions/s3/core/tests/operation_index.rs b/extensions/s3/core/tests/operation_index.rs index a58ae947..49c80d11 100644 --- a/extensions/s3/core/tests/operation_index.rs +++ b/extensions/s3/core/tests/operation_index.rs @@ -114,6 +114,21 @@ async fn branch_local_lsm_index_catches_up_tail_merges_segments_and_prunes_reten ) .await .unwrap(); + plane.reset_request_counts(); + assert!(index + .lookup( + &publisher, + "main", + operation(10_000 + u128::from(generation)), + 1_000 + generation, + ) + .await + .unwrap() + .is_none()); + assert!( + plane.request_snapshot().get <= 6, + "one new journal event should require bounded lookup I/O" + ); if generation == 2 { let tail = index .lookup(&publisher, "main", operation(101), 1_002) diff --git a/extensions/s3/core/tests/protocol.rs b/extensions/s3/core/tests/protocol.rs index 880433f2..eb5a94e8 100644 --- a/extensions/s3/core/tests/protocol.rs +++ b/extensions/s3/core/tests/protocol.rs @@ -179,7 +179,7 @@ fn commit_envelope_is_range_readable_and_rejects_invalid_magic() { assert_eq!(decoded.commit, commit); assert_eq!(decoded.node_pack, Some(pack)); - assert!(CommitObject::node_payload_offset(&encoded) + assert!(CommitObject::node_region_offset(&encoded) .unwrap() .is_some()); let mut invalid_magic = encoded.clone(); diff --git a/extensions/s3/core/tests/repository.rs b/extensions/s3/core/tests/repository.rs index 637210d2..8b7ddb33 100644 --- a/extensions/s3/core/tests/repository.rs +++ b/extensions/s3/core/tests/repository.rs @@ -2,9 +2,9 @@ use std::{collections::BTreeMap, io::Write as _, sync::Arc}; use md5::{Digest as _, Md5}; use prolly_s3_core::{ - FixedClock, GetRequest, ListRequest, LogicalObjectVersionKind, MemoryNodeCache, - MemoryObjectPlane, ObjectHeaders, ObjectPath, ObjectPlane, ProviderPerKeyVersionLimit, - Repository, RepositoryOptions, SequenceIdSource, + decode_canonical, encode_canonical, FixedClock, GetRequest, ListRequest, + LogicalObjectVersionKind, MemoryNodeCache, MemoryObjectPlane, ObjectHeaders, ObjectPath, + ObjectPlane, ProviderPerKeyVersionLimit, Repository, RepositoryOptions, SequenceIdSource, }; use sha2::Sha256; @@ -87,6 +87,20 @@ async fn repository_put_read_replay_and_reopen_use_only_authority() { .await .unwrap(); assert!(!first.idempotent_replay); + let upper = repository + .prewarm_node_cache_levels("main", first.id, 1) + .await + .unwrap(); + assert_eq!(upper.object_nodes, 1); + assert_eq!(upper.version_nodes, 1); + assert_eq!( + repository + .prewarm_node_cache_levels("main", first.id, 0) + .await + .unwrap_err() + .code, + prolly_s3_core::ErrorCode::InvalidLimit + ); let replay = repository .put_object_with_operation( "main", @@ -117,6 +131,22 @@ async fn repository_put_read_replay_and_reopen_use_only_authority() { ))); assert_ne!(binding.path.as_str(), "docs/readme.txt"); + plane.reset_request_counts(); + assert_eq!( + repository + .get_object("main", b"docs/readme.txt") + .await + .unwrap() + .unwrap() + .bytes, + b"repository" + ); + assert_eq!( + plane.request_snapshot().get, + 2, + "a locally indexed exact branch target needs only the ref and whole payload GETs" + ); + repository.advance_branch_indexes("main").await.unwrap(); let read_only = Repository::open( plane.clone(), @@ -140,6 +170,57 @@ async fn repository_put_read_replay_and_reopen_use_only_authority() { assert_eq!(plane.request_snapshot().list, 0); } +#[tokio::test(flavor = "multi_thread", worker_threads = 8)] +async fn concurrent_puts_prepare_payloads_before_the_ordered_publication_lane() { + let plane = Arc::new(MemoryObjectPlane::new(true)); + let repository = Arc::new( + Repository::initialize( + plane.clone(), + RepositoryOptions { + repository_prefix: ".tests/concurrent-payload-preparation".to_string(), + writer: "concurrent-payload-writer".to_string(), + provider_per_key_version_limit: ProviderPerKeyVersionLimit::Finite(10_000), + ..RepositoryOptions::default() + }, + ) + .await + .unwrap(), + ); + plane.set_immutable_put_delay_millis(25); + plane.reset_immutable_put_concurrency(); + + let mut writes = Vec::new(); + for index in 0..8_u8 { + let repository = repository.clone(); + writes.push(tokio::spawn(async move { + repository + .put_object( + "main", + format!("parallel/{index}.bin").into_bytes(), + vec![index; 32], + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + })); + } + for write in writes { + write.await.unwrap().unwrap(); + } + + assert!( + plane.max_immutable_puts_in_flight() >= 4, + "payload preparation remained serialized: max_in_flight={}", + plane.max_immutable_puts_in_flight() + ); + let (_, objects, truncated) = repository + .list_objects("main", b"parallel/", None, 16) + .await + .unwrap(); + assert!(!truncated); + assert_eq!(objects.len(), 8); +} + #[tokio::test] async fn repository_history_listing_delete_markers_and_payload_dedup_are_stable() { let plane = Arc::new(MemoryObjectPlane::new(true)); @@ -596,11 +677,12 @@ async fn durable_commit_session_survives_repeated_authority_renewal_and_restart( ) .await .unwrap(); - staged.push(mutation); + staged.push(mutation.clone()); repository .checkpoint_commit_session( &checkpoint.session, - staged.clone(), + vec![mutation], + staged.len(), u64::try_from(index + 1).unwrap(), ) .await @@ -624,6 +706,127 @@ async fn durable_commit_session_survives_repeated_authority_renewal_and_restart( assert_eq!(receipt.changed_keys, 3); } +#[tokio::test] +async fn durable_checkpoint_windows_are_append_only_and_resume_last_write_per_key() { + let plane = Arc::new(MemoryObjectPlane::new(true)); + let repository = Repository::initialize( + plane.clone(), + RepositoryOptions { + repository_prefix: ".tests/checkpoint-windows".to_string(), + writer: "checkpoint-window-writer".to_string(), + provider_per_key_version_limit: ProviderPerKeyVersionLimit::Finite(10_000), + ..RepositoryOptions::default() + }, + ) + .await + .unwrap(); + let checkpoint = repository + .begin_durable_commit_session("main", "append-only checkpoints", 60_000) + .await + .unwrap(); + let first = repository + .stage_commit_session_put( + &checkpoint.session, + b"a".to_vec(), + b"first".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap(); + repository + .checkpoint_commit_session(&checkpoint.session, vec![first], 1, 1) + .await + .unwrap(); + let second = repository + .stage_commit_session_put( + &checkpoint.session, + b"b".to_vec(), + b"second".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap(); + repository + .checkpoint_commit_session(&checkpoint.session, vec![second], 2, 2) + .await + .unwrap(); + let replacement = repository + .stage_commit_session_put( + &checkpoint.session, + b"a".to_vec(), + b"replacement".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ) + .await + .unwrap(); + repository + .checkpoint_commit_session(&checkpoint.session, vec![replacement], 2, 3) + .await + .unwrap(); + + let page = plane + .list(ListRequest { + prefix: ".tests/checkpoint-windows/staging/".to_string(), + continuation: None, + limit: 100, + include_versions: false, + }) + .await + .unwrap(); + let mut windows = Vec::new(); + for entry in page.entries { + let stored = plane + .get(GetRequest { + path: entry.path, + range: None, + physical_version: None, + }) + .await + .unwrap() + .unwrap(); + windows.push( + decode_canonical::(&stored.bytes).unwrap(), + ); + } + windows.sort_by_key(|window| window.sequence); + assert_eq!(windows.len(), 4); + assert_eq!( + windows + .iter() + .map(|window| ( + window.sequence, + window.total_mutations, + window.mutations.len() + )) + .collect::>(), + vec![(0, 0, 0), (1, 1, 1), (2, 2, 1), (3, 2, 1)] + ); + + let resumed = repository + .resume_commit_session(checkpoint.session.id) + .await + .unwrap(); + assert_eq!(resumed.sequence, 3); + assert_eq!(resumed.total_mutations, 2); + assert_eq!(resumed.mutations.len(), 2); + repository + .publish_commit_session(resumed.session, resumed.mutations) + .await + .unwrap(); + assert_eq!( + repository + .get_object("main", b"a") + .await + .unwrap() + .unwrap() + .bytes, + b"replacement" + ); +} + #[tokio::test] async fn concurrent_session_staging_and_authority_renewal_do_not_self_fence() { let plane = Arc::new(MemoryObjectPlane::new(true)); @@ -711,7 +914,7 @@ async fn real_takeover_fences_an_open_commit_session() { ) .await .unwrap(); - old.checkpoint_commit_session(&checkpoint.session, vec![staged.clone()], 1) + old.checkpoint_commit_session(&checkpoint.session, vec![staged.clone()], 1, 1) .await .unwrap(); @@ -842,13 +1045,13 @@ async fn repository_commit_session_batches_payloads_into_one_replayable_publicat } #[tokio::test] -async fn repository_small_object_pack_deduplicates_and_bounds_logical_ranges() { +async fn repository_batch_stages_independent_whole_objects_with_whole_object_deduplication() { let plane = Arc::new(MemoryObjectPlane::new(true)); let repository = Repository::initialize( - plane, + plane.clone(), RepositoryOptions { - repository_prefix: ".tests/repository-payload-pack".to_string(), - writer: "pack-writer".to_string(), + repository_prefix: ".tests/repository-whole-payload".to_string(), + writer: "whole-object-writer".to_string(), provider_per_key_version_limit: ProviderPerKeyVersionLimit::Finite(10_000), ..RepositoryOptions::default() }, @@ -856,7 +1059,7 @@ async fn repository_small_object_pack_deduplicates_and_bounds_logical_ranges() { .await .unwrap(); let session = repository - .begin_commit_session("main", "packed import", 60_000) + .begin_commit_session("main", "whole-object import", 60_000) .await .unwrap(); let staged = repository @@ -864,19 +1067,19 @@ async fn repository_small_object_pack_deduplicates_and_bounds_logical_ranges() { &session, vec![ ( - b"pack/a".to_vec(), + b"objects/a".to_vec(), b"same".to_vec(), ObjectHeaders::default(), BTreeMap::new(), ), ( - b"pack/b".to_vec(), + b"objects/b".to_vec(), b"same".to_vec(), ObjectHeaders::default(), BTreeMap::new(), ), ( - b"pack/c".to_vec(), + b"objects/c".to_vec(), b"next".to_vec(), ObjectHeaders::default(), BTreeMap::new(), @@ -891,7 +1094,7 @@ async fn repository_small_object_pack_deduplicates_and_bounds_logical_ranges() { .await .unwrap(); let mut bindings = Vec::new(); - for key in [b"pack/a".as_slice(), b"pack/b", b"pack/c"] { + for key in [b"objects/a".as_slice(), b"objects/b", b"objects/c"] { bindings.push( repository .head_object_at("main", receipt.id, key) @@ -903,18 +1106,109 @@ async fn repository_small_object_pack_deduplicates_and_bounds_logical_ranges() { .unwrap(), ); } - assert!(bindings.iter().all(|binding| binding.is_packed())); assert_eq!(bindings[0].path, bindings[1].path); - assert_eq!(bindings[0].pack_range, bindings[1].pack_range); - assert_ne!(bindings[1].pack_range, bindings[2].pack_range); + assert_ne!(bindings[1].path, bindings[2].path); + assert_eq!(bindings[0].checksum_sha256, bindings[1].checksum_sha256); + assert_ne!(bindings[1].checksum_sha256, bindings[2].checksum_sha256); + + let stored = plane + .list(ListRequest { + prefix: ".tests/repository-whole-payload/".to_string(), + continuation: None, + limit: 100, + include_versions: false, + }) + .await + .unwrap(); + assert_eq!( + stored + .entries + .iter() + .filter(|entry| entry.path.as_str().contains("/payloads/")) + .count(), + 2 + ); + assert!(stored + .entries + .iter() + .all(|entry| !entry.path.as_str().contains("payload-packs"))); let range = repository - .get_object_range("main", receipt.id, b"pack/a", 0..=u64::MAX) + .get_object_range("main", receipt.id, b"objects/a", 1..=2) .await .unwrap() .unwrap(); - assert_eq!(range.bytes, b"same"); - assert_eq!(range.range, 0..=3); + assert_eq!(range.bytes, b"am"); + assert_eq!(range.range, 1..=2); + + let mut fsck = repository.start_fsck("main", true).await.unwrap(); + while fsck.phase != prolly_s3_core::FsckPhase::Complete { + let page = repository.advance_fsck(&fsck, 100).await.unwrap(); + fsck = decode_canonical(&encode_canonical(&page.cursor).unwrap()).unwrap(); + } + assert_eq!(fsck.report.physical_payloads_verified, 2); + assert_eq!(fsck.report.physical_payload_bytes_verified, 8); + assert_eq!(fsck.report.deep_physical_bytes_read, 8); +} + +#[tokio::test] +async fn repository_batch_results_isolates_invalid_objects_after_one_session_validation() { + let plane = Arc::new(MemoryObjectPlane::new(true)); + let repository = Repository::initialize( + plane.clone(), + RepositoryOptions { + repository_prefix: ".tests/repository-batch-results".to_string(), + writer: "batch-results-writer".to_string(), + provider_per_key_version_limit: ProviderPerKeyVersionLimit::Finite(10_000), + ..RepositoryOptions::default() + }, + ) + .await + .unwrap(); + let session = repository + .begin_commit_session("main", "per-object batch results", 60_000) + .await + .unwrap(); + plane.reset_request_counts(); + let results = repository + .stage_commit_session_put_batch_results( + &session, + vec![ + ( + b"batch-results/a".to_vec(), + b"first".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ), + ( + Vec::new(), + b"invalid".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ), + ( + b"batch-results/b".to_vec(), + b"second".to_vec(), + ObjectHeaders::default(), + BTreeMap::new(), + ), + ], + 3, + ) + .await + .unwrap(); + assert_eq!(results.len(), 3); + assert_eq!( + results[1].as_ref().unwrap_err().code, + prolly_s3_core::ErrorCode::InvalidKey + ); + assert_eq!(plane.request_snapshot().immutable_put, 2); + let staged = results.into_iter().filter_map(Result::ok).collect(); + let receipt = repository + .publish_commit_session(session, staged) + .await + .unwrap(); + assert_eq!(receipt.changed_keys, 2); } #[tokio::test] @@ -972,12 +1266,23 @@ async fn large_commit_delta_is_external_and_survives_toc_only_reopen() { ) .await .unwrap(); - let page = reopened - .diff_page_bounded("main", base, receipt.id, None, 1_000) - .await - .unwrap(); - assert_eq!(page.changes.len(), 129); - assert!(page.continuation.is_none()); + let mut continuation = None; + let mut changes = Vec::new(); + loop { + let page = reopened + .diff_page_bounded("main", base, receipt.id, continuation.as_ref(), 32) + .await + .unwrap(); + assert_eq!(page.compared_nodes, 0); + changes.extend(page.changes); + continuation = page.continuation; + if continuation.is_none() { + break; + } + } + assert_eq!(changes.len(), 129); + assert_eq!(changes.first().unwrap().key, b"external/000"); + assert_eq!(changes.last().unwrap().key, b"external/128"); assert_eq!( reopened .get_object("main", b"external/128") @@ -1020,7 +1325,7 @@ async fn repository_durable_session_resumes_after_process_authority_reacquisitio .await .unwrap(); original - .checkpoint_commit_session(&checkpoint.session, vec![staged], 1) + .checkpoint_commit_session(&checkpoint.session, vec![staged], 1, 1) .await .unwrap(); let payload_puts = plane.request_snapshot().immutable_put; diff --git a/extensions/s3/diagram/prolly-s3-architecture.svg b/extensions/s3/diagram/prolly-s3-architecture.svg index 29e5ed25..e00bfb60 100644 --- a/extensions/s3/diagram/prolly-s3-architecture.svg +++ b/extensions/s3/diagram/prolly-s3-architecture.svg @@ -1,18 +1,18 @@ Prolly S3 architecture - An application uses the Prolly S3 client. Whole files are stored at original keys as S3 versions, while Prolly metadata records exact version identifiers and publishes commits through a branch reference. + An application uses the Prolly S3 client. Each logical payload is stored as one complete immutable provider object, while Prolly metadata maps logical keys to exact object identities and publishes commits through a branch reference. - Prolly is the history; versioned S3 holds whole files + Prolly is the history; S3 holds whole immutable payloads One authoritative writer · exact VersionId reads · no repository chunks Application AWS-shaped Rust API - put · get · list · multipart + put · get · list · external handoff branch · diff · merge · restore Prolly S3 client @@ -22,11 +22,11 @@ Prolly engine state · versions · operations - Managed object keys - reports/q3.parquet - VersionId A · VersionId B · … - photos/launch.jpg - whole S3 object versions + Content-addressed payload keys + .prolly/payloads/…/sha256/abcd… + one complete immutable provider object + logical: reports/q3.parquet + binding: path · VersionId · checksum Repository metadata .prolly/ @@ -41,5 +41,5 @@ Canonical read 1. Pin a Prolly commit and resolve the logical key. 2. Load the recorded physical binding. - 3. GetObject(key, version_id) returns the exact historical bytes. + 3. GetObject(binding.path, version_id) returns the complete historical bytes. diff --git a/extensions/s3/docs/adr/0008-production-scaling-and-maintenance-admission.md b/extensions/s3/docs/adr/0008-production-scaling-and-maintenance-admission.md new file mode 100644 index 00000000..1a64c554 --- /dev/null +++ b/extensions/s3/docs/adr/0008-production-scaling-and-maintenance-admission.md @@ -0,0 +1,84 @@ +# ADR 0008: Production scaling and maintenance admission + +Status: accepted + +## Context + +The 20K RustFS qualification demonstrated efficient snapshot reads, listing, +branching, sparse diff, and merge after the node-index and traversal-cursor +work. It did not establish a production envelope for deep hot-branch history, +large streamed files, whole-object lifecycle, or garbage collection while +other operating-system processes were publishing. + +Profiling the 4K first-parent test found that every ordinary write walked the +complete unindexed operation-journal suffix to prove that a freshly allocated +operation ID was absent. That made history construction quadratic. Large +streamed files still used one `PutObject`, and GC's publication barrier and +dirty-root sequence were process-local. + +## Decision + +1. Immutable payload hashing/upload happens before the branch publication + lane. The lane orders tree mutation and the ref CAS only. Commit sessions + remain the group-commit mechanism for bulk ingestion. +2. Internally allocated operation IDs skip retry lookup. Caller-stable IDs use + an incremental in-memory view of the unindexed journal suffix; the durable + segmented index remains authoritative across restart. +3. Journal-derived node and commit-graph indexes load commit descriptors with + bounded concurrency and apply one deterministic tree batch per journal + page. First-parent graph entries retain binary-lifting skip pointers. +4. Every distinct logical payload is represented by one complete immutable S3 + object. Built-in uploads use one conditional `PutObject`; multipart, + resumable parts, retry, buffering, and abort belong to an external provider + transfer manager. Prolly verifies and publishes only the completed object. +5. Prolly never packs multiple user bodies together and never splits one body + into repository-managed chunks. Payload bindings contain a whole-object + path, provider version, ETag, and checksum. Metadata node packs remain an + internal encoding for Prolly index nodes and never contain user payloads. +6. GC closes durable repository-wide publication admission before discovering + roots. Every branch/tag CAS owns an expiring, instance-scoped ticket from + immediately before provider CAS through outcome resolution. GC drains or + expires all tickets before scanning refs. This favors correctness over + write availability during destructive maintenance and works across + processes that do not share memory. +7. Repair and history import delegate complete-object copying to the object + provider boundary. The repository supplies source identity, destination, + size, and whole-object checksum only; transfer-part state is never part of + repository state. +8. Independent same-branch submissions may enter a bounded ordered publication + queue. The queue coalesces unique keys, prepares whole objects concurrently, + appends bounded recovery windows, and performs one ref CAS per group. + Duplicate-key submissions are separated by a commit boundary, and callers + are acknowledged only after their group publishes. + +## Consequences + +- Same-branch publication remains intentionally serialized. Use commit-session + group commits for bulk loads and independent branches when independent + histories are acceptable; merge them structurally afterward. +- Maintenance admission adds one immutable ticket create, one coordinator + read, and one exact ticket delete to branch/tag publication. This cost must be + included in AWS request budgets. Independent branches avoid ref-CAS + contention but not provider account/prefix quotas. +- Small-object request and object-count costs are not hidden by a Prolly-owned + blob layer. Throughput comes from bounded whole-object concurrency and + grouped metadata publication; exact duplicate bodies may reuse one complete + content-addressed object. +- Externally uploaded objects are immutable by repository convention and exact + version binding. Repository IAM must deny writes by other principals under + the reserved prefix, while the transfer manager owns incomplete uploads. +- RustFS is a compatibility and regression substrate, not evidence of AWS + latency, throttling, cost, lifecycle, replication, or regional behavior. + +## Release evidence + +- The release-mode 4K first-parent merge-base gate completes in 11.56 seconds + after the operation-journal fix; the pre-fix release run remained CPU-bound + after five minutes. +- Deterministic core tests cover concurrent whole-object preparation, + whole-object deduplication, bounded operation-suffix lookup, batched journal + indexing, and cross-handle GC fencing. They also assert that payload + bindings never contain byte extents. +- External transfer-manager handoff, 10K hot-commit, and 100K–1M + multi-operation provider results remain release evidence requirements; they + are not inferred from unit tests. diff --git a/extensions/s3/spec/prolly-s3/paths.md b/extensions/s3/spec/prolly-s3/paths.md index a5bb7084..8b7700b2 100644 --- a/extensions/s3/spec/prolly-s3/paths.md +++ b/extensions/s3/spec/prolly-s3/paths.md @@ -23,6 +23,13 @@ repository format. | ref-catalog tree | `P/ref-catalog/tree/...` | | index-rebuild chunk | `P/administration/index-rebuild/N/E/chunks/sha256/...` | | merge plan | `P/administration/merge/E/plan/...` | +| commit-closure work tree | `P/administration/closure/E/tree/nodes/sha256/...` | +| fsck cursor | `P/administration/fsck/E/cursor.cbor` | +| fsck distinct-payload work tree | `P/administration/fsck/E/payloads/nodes/sha256/...` | +| GC coordinator | `P/gc/coordinator.cbor` | +| GC epoch cursor | `P/gc/epochs/E/cursor.cbor` | +| GC reachability work tree | `P/administration/gc/E/tree/nodes/sha256/...` | +| publication admission ticket | `P/gc/publications/R/E/H` | `R` is the repository identity. `N` is the canonical encoded ref name. `S` is an encoded authority scope. `B` is a commit-session ID. `E` is an administration @@ -37,10 +44,17 @@ byte pairs. `SS` is a two-digit ref-catalog shard. - Branch refs point to immutable publication events; the event points to the commit and previous event. - Payload paths derive from repository identity and content hash. Identical - bytes in one repository may reuse a payload object. + bytes in one repository may reuse a complete payload object. A payload path + never stores multiple logical bodies or a chunk of one logical body. - Delete markers do not have payload paths. - Ordinary reads and index maintenance must not discover nodes by namespace listing. - Mutable control records retain a bounded number of provider versions. +- Every fsck page advances a CAS-protected checkpoint generation. Only the + durable generation may advance. Completed jobs exact-delete payload and + closure work trees before deleting older and then current checkpoint + versions in bounded cleanup pages. +- Branch/tag publication tickets are immutable, instance-scoped, and removed + by exact version after the ref CAS resolves. GC may remove expired tickets. - There are no alternative versioned path families and no format migration namespace.