Add the S3 production cache profile and GA contract

extensions/s3/API.md

@@ -1,5 +1,9 @@
 # Prolly S3 API guide
 
+Stable-release compatibility, upgrade/downgrade recovery, production cache,
+OpenTelemetry, and provider/cardinality support rules are defined in
+[`GA-CONTRACT.md`](GA-CONTRACT.md).
+
 The application-facing type is `prolly_s3_client::Client`. This guide describes
 the public client surface in version 0.1.0 and separates ordinary application
 operations from administrative maintenance.

extensions/s3/CACHE-AND-SCALE-DESIGN.md

@@ -24,8 +24,10 @@ content-hash verified before use.
 The client supports:
 
 - bounded in-process memory caching for hot nodes and pack locations;
-- optional Foyer memory/disk caching behind the `foyer-cache` feature;
+- Foyer memory/disk caching enabled by the default client feature and required
+  by `ProductionCacheProfile`;
 - safe cache persistence across restarts;
+- byte-bounded root/upper-level pinning and cardinality-derived sizing;
 - provider reads as the authoritative fallback.
 
 Cache corruption becomes a miss or a validation error; it cannot silently
@@ -53,9 +55,10 @@ A process should:
 
 1. open the repository and validate provider attestation;
 2. load the branch ref and index heads;
-3. open the persistent Foyer cache, if configured;
+3. open the persistent Foyer cache (required by the production profile);
 4. catch indexes up to the current journal generation;
-5. optionally prewarm upper tree levels for known hot branches.
+5. prewarm and pin a bounded number of upper tree levels for the attached
+   branch, subject to the configured startup timeout.
 
 Persisting immutable nodes removes the 421+ request cold-traversal pattern seen
 in early prototypes. A fully cold cache still performs bounded tree-depth and
@@ -118,10 +121,11 @@ block-index and entry overhead.
 Prewarming is advisory and cancellable. A reader remains correct if prewarming
 never runs or the cache directory is deleted.
 
-## Remaining production gaps
+## Remaining qualification gaps
 
 - production garbage collection for unreachable immutable data;
 - published AWS qualification at customer-specific scale and traffic;
-- automatic cache sizing from observed working set;
+- feedback-driven cache resizing from observed working set (the initial size is
+  derived from expected cardinality);
 - operational SLOs for index lag and rebuild completion;
 - cross-region and disaster-recovery workflows.

extensions/s3/GA-CONTRACT.md

@@ -0,0 +1,137 @@
+# Prolly S3 general-availability contract
+
+Status: proposed contract for the first stable release. The `0.1.x` crates are
+still a preview and do not yet carry the GA compatibility promise below.
+
+## Storage boundary
+
+One logical live file is one complete immutable provider object. Prolly S3
+does not pack payloads, split payloads into chunks, or own multipart-upload
+state. Prolly metadata nodes may share a range-addressable commit object;
+logical payloads never do.
+
+## Repository-format compatibility
+
+The create-once `format/repository.cbor` marker is the repository's canonical
+format identity. Its tree descriptor, canonical limits, idempotency retention,
+and provider version-limit profile are immutable for the lifetime of the
+repository prefix.
+
+The first stable release will apply these rules:
+
+- patch and minor client upgrades must open and write every repository created
+  by an earlier release in the same major series;
+- new readers may supply documented defaults for fields absent from older
+  objects, while canonical decoding rejects unknown fields, object magic, and
+  version values rather than silently changing their meaning;
+- a writer must fail closed with `RepositoryFormatConflict` or
+  `UnsupportedRepositoryFormat` before publishing when it cannot reproduce the
+  stored canonical format exactly;
+- caches, journal-derived indexes, rightmost-path hints, and prewarm state are
+  advisory and may be deleted/rebuilt by any compatible release;
+- commit IDs, object-version IDs, payload bindings, authority epochs, branch
+  generations, and published state roots are durable protocol state and may
+  not be rewritten during an upgrade;
+- a future incompatible format uses a new major protocol generation and a new
+  repository prefix. It is transferred through the verified history-transfer
+  API, never migrated in place.
+
+The repository core already enforces exact create-once canonical settings on
+open. Commit metadata and node packs use explicit versioned wire magic and
+content hashes; unsupported or corrupt encodings fail before becoming visible.
+
+## Upgrade, downgrade, and recovery guarantees
+
+Before an upgrade:
+
+1. complete or cancel active merges, restores, transfers, fsck, and GC jobs;
+2. retain a provider-versioned backup or verified replica and the provider
+   attestation;
+3. record every branch/tag/pin target and run shallow fsck;
+4. open the repository read-only with the candidate release and rebuild
+   advisory indexes/cache from an empty local directory;
+5. run the provider- and cardinality-specific qualification gate before
+   enabling writers.
+
+A downgrade is supported only when the older binary declares support for the
+exact stored repository format and every required feature. Canonical derived
+indexes written by a newer binary can intentionally fail closed in an older
+binary; downgrade preparation must reset those advisory index heads with the
+newer release before the older release rebuilds them. Otherwise restore the
+pre-upgrade provider snapshot or transfer verified history into a new prefix.
+Downgrade code must never delete newer immutable objects or rewrite the format
+marker. If a release does not publish and test that reset procedure, downgrade
+from that release is explicitly unsupported.
+
+Recovery guarantees are based on immutable closure and fenced mutable
+controls: branch publication is CAS-protected, retry outcomes are reconciled,
+authority takeover fences old epochs, commit sessions and maintenance cursors
+are restartable, and exact physical versions are deleted only after the GC
+safety protocol. Recovery does not guarantee retention of unreachable payloads
+outside the configured GC policy.
+
+## Production cache profile
+
+`ProductionCacheProfile` is the supported production path. It requires a
+single-owner persistent Foyer directory, chooses memory/disk/location bounds
+from expected repository cardinality, enables bounded sibling prefetch, and
+prewarms and pins root/upper-level metadata nodes. New repositories initialized
+with the profile use encoded-byte-bounded metadata nodes; existing repositories
+retain their create-once tree format and still use exact node-pack range reads.
+
+The profile is a latency requirement, never a correctness dependency. Removing
+the cache must leave every repository operation correct, although cold SLOs may
+fail until prewarming completes.
+
+## OpenTelemetry and reference alerts
+
+Build `prolly-s3-client` with `opentelemetry` and attach an
+`OpenTelemetryClientMetrics` sink through `ClientBuilder::telemetry`. The
+application owns the `MeterProvider`, exporter, resource attributes, sampling,
+and shutdown. The client exports bounded-dimension metrics for:
+
+- cache hits/misses, admissions, errors, corruptions, and singleflight waits;
+- requested, provider-fetched, and cache-avoided metadata bytes;
+- predictive prefetch batches/nodes;
+- S3 operations and transferred bytes;
+- total open, index catch-up, and prewarm duration/failure.
+
+Recommended initial alerts, tuned after a representative baseline:
+
+| Condition | Initial threshold | Window |
+| --- | ---: | ---: |
+| Metadata cache hit ratio | below 90% after warmup | 15 minutes |
+| Provider-fetched / requested metadata bytes | above 4x | 10 minutes |
+| Cache admission rejects | above 1% of node requests | 10 minutes |
+| Cache corruption | any sustained nonzero value | 5 minutes |
+| Startup prewarm timeout/failure | any occurrence | immediate |
+| Branch-index lag | above 100 generations or not ready | 5 minutes |
+| S3 429/503 wire attempts | above provider-specific error budget | 5 minutes |
+| GC/fsck checkpoint progress | no progress during an active job | 15 minutes |
+
+SDK operation counters do not include SDK-internal retries. Attach
+`S3WireAttemptInterceptor` and provider-side request metrics for retry/error
+alerts.
+
+## Explicit support envelope
+
+`SupportedEnvelope::for_deployment` exposes the same policy to applications.
+The current evidence supports only a controlled RustFS/local pilot through
+100K objects after its release gates pass. AWS always requires workload-specific
+qualification. Repositories above 100K require cardinality-matched maintenance
+and performance qualification, and one-million-object production support is
+not claimed while the published cold and ingestion gates remain below target.
+
+Promotion requires all of the following at the intended provider, region,
+cardinality, key distribution, concurrency, retention, and cache size:
+
+- provider capability and lifecycle/Object Lock/replication attestation;
+- cold, prewarmed, steady-state, and cache-loss read/list SLOs;
+- ingest throughput and request/byte amplification budgets;
+- branch, arbitrary-snapshot diff, and merge SLOs;
+- authority expiry/takeover and process-loss fault injection;
+- fsck, journal-driven GC, restart, backup, and restore drills;
+- operator dashboards, alerts, runbooks, IAM/KMS review, and cost approval.
+
+RustFS conformance proves protocol behavior, not AWS latency, throttling, cost,
+or operational readiness.

extensions/s3/OPERATIONS.md

@@ -1,5 +1,8 @@
 # Prolly S3 operations
 
+The stable upgrade/recovery contract, exported OpenTelemetry instruments, and
+reference production alerts are defined in [GA-CONTRACT.md](GA-CONTRACT.md).
+
 ## Provisioning
 
 1. Create a dedicated S3 or S3-compatible bucket.

extensions/s3/README.md

@@ -1,5 +1,10 @@
 # Prolly S3
 
+The proposed stable compatibility, cache, telemetry, recovery, and provider
+support promises are defined in [GA-CONTRACT.md](GA-CONTRACT.md). The current
+`0.1.x` crates remain suitable for controlled pilots after the qualification
+gates in that contract pass; they are not yet a universal GA claim.
+
 Prolly S3 adds repository history to a versioned S3 bucket. It stores each file
 as one immutable, content-addressed S3 object and stores directory state,
 commits, branches, tags, and indexes as Prolly trees.

extensions/s3/client/Cargo.toml

@@ -8,8 +8,9 @@ license = "MIT OR Apache-2.0"
 repository = "https://github.com/crabbuild/prolly"
 
 [features]
-default = []
+default = ["foyer-cache"]
 foyer-cache = ["dep:foyer"]
+opentelemetry = ["dep:opentelemetry"]
 
 [dependencies]
 async-trait = "0.1"
@@ -24,6 +25,7 @@ hmac = "0.12"
 http-body = "1"
 md-5 = "0.10"
 hex = "0.4"
+opentelemetry = { version = "=0.32.0", default-features = false, features = ["metrics"], optional = true }
 prolly-s3-core = { path = "../core", version = "0.1.0" }
 serde = { version = "1.0", features = ["derive"] }
 sha2 = "0.10"

extensions/s3/client/README.md

@@ -520,8 +520,44 @@ are discovered and journaled by the same protocol.
 
 ## Cache immutable nodes
 
-The in-memory cache is enabled through repository limits. For a persistent
-Foyer cache, enable the `foyer-cache` feature:
+The `foyer-cache` feature is enabled by default. Production deployments should
+use the cardinality-aware profile so persistent storage, metadata-node bounds,
+startup prewarming, and upper-level pinning are configured together:
+
+```rust
+use prolly_s3_client::{Client, ProductionCacheProfile};
+
+let profile = ProductionCacheProfile::new(
+    "./prolly-node-cache",
+    1_000_000, // expected live logical objects
+);
+let client = Client::builder()
+    // supply the required AWS client, provider identity, signer, bucket, etc.
+    .production_cache_profile(profile)
+    .open()
+    .await?;
+
+let startup = client.startup_metrics();
+let performance = client.performance_snapshot();
+println!(
+    "startup={}ms hit_ratio={:.3} metadata_amplification={:.3}x",
+    startup.total_open_millis,
+    performance.cache.hit_ratio(),
+    performance.metadata_download_amplification(),
+);
+```
+
+`CacheSizingRecommendation::for_object_count` exposes the selected memory,
+disk, location, and prewarm bounds without opening a client. Supplying the
+profile always opens persistent Foyer storage; `--no-default-features` callers
+must enable `foyer-cache` explicitly.
+
+After stopping request traffic and dropping other client clones, call
+`client.close_production_cache().await?` to flush and close the cache opened by
+the production profile. This also stops the shared authority, branch-index,
+and telemetry maintenance tasks for that client.
+
+For custom deployments, construct the cache directly:
 
 ```rust
 use std::path::PathBuf;
@@ -559,8 +595,34 @@ Use `prewarm_node_cache(snapshot)` during startup to traverse both state trees.
 Use `prewarm_node_cache_levels(snapshot, levels)` when startup should load only
 the roots and shared upper paths instead of scanning every leaf.
 Use `node_cache_snapshot()` before and after to observe hits, misses,
-insertions, corruptions, coalesced waits, ranged fetches, fetched and avoided
-bytes, and admission rejections.
+insertions, corruptions, coalesced waits, ranged fetches, requested/fetched/
+avoided bytes, byte amplification, predictive prefetches, pinned nodes, and
+admission rejections.
+Take `performance_snapshot()` before and after a metadata-only operation and
+use `delta_since` plus `metadata_download_amplification()` to include commit,
+index, and control-object response bytes in the measured amplification.
+
+## OpenTelemetry
+
+Enable the `opentelemetry` feature and supply the application-owned meter. The
+client records deltas on a bounded maintenance interval; the embedding service
+continues to own the SDK, exporter, resource attributes, and shutdown:
+
+```rust
+use std::{sync::Arc, time::Duration};
+use opentelemetry::global;
+use prolly_s3_client::{Client, OpenTelemetryClientMetrics};
+
+let telemetry = OpenTelemetryClientMetrics::new(global::meter("prolly-s3"));
+let client = Client::builder()
+    // supply the remaining required settings
+    .telemetry(telemetry, Duration::from_secs(15))
+    .open()
+    .await?;
+```
+
+Metric names and initial alert thresholds are defined in
+[`GA-CONTRACT.md`](../GA-CONTRACT.md).
 
 ## Performance model
 

extensions/s3/client/examples/rustfs_small_files_benchmark.rs

@@ -734,7 +734,7 @@ async fn main() -> BenchResult {
             let page = client.advance_fsck(&cursor, 10_000).await?;
             pages += 1;
             cursor = page.cursor;
-            if pages % 10 == 0 {
+            if pages.is_multiple_of(10) {
                 println!(
                     "FSCK_PROGRESS mode={fsck_mode} pages={pages} phase={:?} commits={} current_objects={} logical_versions={}",
                     cursor.phase,
@@ -793,7 +793,7 @@ async fn main() -> BenchResult {
             };
             pages += 1;
             cursor = page.cursor;
-            if pages % 100 == 0 {
+            if pages.is_multiple_of(100) {
                 println!(
                     "GC_PROGRESS pages={pages} phase={:?} commits={} nodes={} logical_versions={} candidates={} deleted_versions={}",
                     cursor.phase,

extensions/s3/client/src/aws_object.rs

@@ -49,6 +49,26 @@ impl S3OperationMetrics {
             + self.delete_object
             + self.delete_objects
     }
+
+    pub fn delta_since(self, earlier: Self) -> Self {
+        Self {
+            get_object: self.get_object.saturating_sub(earlier.get_object),
+            head_object: self.head_object.saturating_sub(earlier.head_object),
+            put_object: self.put_object.saturating_sub(earlier.put_object),
+            list_objects_v2: self.list_objects_v2.saturating_sub(earlier.list_objects_v2),
+            list_object_versions: self
+                .list_object_versions
+                .saturating_sub(earlier.list_object_versions),
+            delete_object: self.delete_object.saturating_sub(earlier.delete_object),
+            delete_objects: self.delete_objects.saturating_sub(earlier.delete_objects),
+            uploaded_body_bytes: self
+                .uploaded_body_bytes
+                .saturating_sub(earlier.uploaded_body_bytes),
+            downloaded_body_bytes: self
+                .downloaded_body_bytes
+                .saturating_sub(earlier.downloaded_body_bytes),
+        }
+    }
 }
 
 #[derive(Default)]