Make the S3 extension production-ready at million-object scale - #105
Merged
Conversation
Remove Prolly-facing multipart upload IDs, part geometry, ListParts reconciliation, and chunk lifecycle state. Hand externally uploaded final objects back by whole-object identity and verify them before publication. Also add exact sparse-pack candidate accounting and snapshot-safe repack publication.
Combine durable object age with scheduler-observed temperature when repacking tiny objects. Keep recent or hot payloads in smaller packs, promote only old cold payloads to larger targets, and make maintenance sessions ephemeral.
Check the repository-wide GC coordinator before preparing new publications, preserve idempotent replay, and reconcile only genuinely unknown publication outcomes. Verify an independent RustFS writer remains retryably fenced across GC owner restart and becomes usable after completion.
forhappy
marked this pull request as ready for review
August 17, 2026 01:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
The original benchmark work exposed correctness failures under authority renewal and severe request/byte amplification at bucket scale. Follow-up large-object work also crossed an architectural boundary by putting transfer-part lifecycle into Prolly. This PR keeps versioned repository metadata in Prolly, tiny-object packing as an explicit repository optimization, and transport multipart state entirely with the provider uploader.
Validation
prolly-s3-coreall-target/all-feature test suiteprolly-s3-clientall-target/all-feature test suitePutObjectstream gateThe ignored AWS qualification and scale gates still require an operator-owned real bucket and explicit SLO configuration.