feat(storage): adopt staged files as contiguous representations

[joshuajbouw]

Linked Issue

Closes #1467.

Summary

Publish sealed native staging generations as verified contiguous physical
representations while preserving canonical logical objects, ordinary
principal-root publication, and materialized archival export.

Changes

• retain sealed staging generations as durable retry witnesses and publish
  APFS/Linux copy-on-write clones with a verified copy fallback
• serve canonical Chunk objects from authenticated contiguous slices while
  keeping File, ChunkTree, Evidence, and structural records in the arena
• pin the store, representation, staging, generation, and quarantine directory
  capabilities before startup migration or recovery mutates private state
• bind publication to the still-open verified source and temporary file
  identities, then verify the installed blob and metadata handles before CAS
• rebuild represented slices with positional reads so duplicate chunks cannot
  move the streaming recovery cursor
• use one current sealed-footer format with mandatory source identity, while
  retaining the separate legacy staging-intent migration
• model private volume/file identity with domain newtypes and use a named
  private-file mode constant
• recover and publish all representation metadata, journals, blobs, compaction
  arenas, and staging state through retained capabilities rather than ambient
  paths
• keep proof-audited compaction closed over represented objects, materialize
  every live object into the replacement arena, publish replacement authority,
  and only then retire loose blobs
• add a manually dispatched, content-bound cross-platform benchmark workflow
  for Linux x86_64, macOS arm64, and Windows x86_64 without hard-coding local
  device results as portable product claims

Verification

• cargo fmt --all
• cargo clippy --workspace --all-features -- -D warnings
• cargo clippy -p astrid-storage-engine -p astrid-storage --all-features --all-targets -- -D warnings
• cargo test -p astrid-storage-engine --lib -- --quiet (208 passed, 1 ignored)
• cargo test -p astrid-storage --lib -- --quiet (279 passed, 6 ignored)
• cargo test -p astrid-storage retained_directory_capability_has_a_durable_sync_handle
• cargo test -p astrid-integration-tests --test gateway_e2e kernel_and_gateway_boot_against_shared_home -- --exact
• focused capability-replacement, publication-identity, and recovery-cursor
  regressions
• actionlint .github/workflows/storage-benchmark.yml
• release-mode 1 MiB benchmark smoke run with parsed content-bound JSON output

The full workspace test run compiled and completed its preceding suites, but an
existing macOS-only MCP fat-binary fixture hung after copying /usr/bin/false
and spawning the copy (engine::mcp_tests::tests::test_fat_binary_resolution).
That fixture and code path are not changed by this pull request; the complete
storage crate suites above were run independently.

Security and claim boundary

Publication never trusts a pathname after verification: the installed
hard-link must match the still-open verified temporary's filesystem identity
and length. Startup recovery pins its authority roots before opening journals,
migrating legacy intents, or moving quarantine entries. Recovery validates
repeated chunks with positional reads, leaving the sequential cursor attached
to the physical stream order. On Unix, retained routing capabilities and their
identity-bound, fsync-capable directory handles are distinct so Linux O_PATH
authority is never used as a durability descriptor.

The benchmark workflow produces reproducible evidence artifacts for each
supported runner. Device-local historical numbers remain evidence for those
specific runs, not universal performance guarantees.

AI / Tool Assistance

Codex implemented and tested the change under Joshua's direction. GitHub
Copilot reviewed the pull request. Joshua owns the design decisions, validation
claims, and final review responses.

[Copilot]

Pull request overview

Adopts sealed staging files as contiguous physical representations while retaining canonical logical objects and archival behavior.

Changes:

• Adds contiguous-blob publication, recovery, admission evidence, and verified slice reads.
• Integrates staged publication, closure validation, snapshots, and fault testing.
• Adds benchmark evidence and updated storage documentation.

Reviewed changes

Copilot reviewed 39 out of 40 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
docs/benchmarks/storage-io/SHA256SUMS	Registers benchmark evidence checksum.
docs/benchmarks/storage-io/README.md	Summarizes benchmark results.
docs/benchmarks/storage-io/astrid-storage-contiguous-da8e3cd.json	Adds benchmark evidence.
docs/astrid-storage-performance.md	Documents performance results.
docs/astrid-contiguous-staged-file-adoption.md	Revises adoption protocol.
crates/astrid-storage/src/principal_state/staging.rs	Publishes staged files contiguously.
crates/astrid-storage/src/principal_state/runtime_tests.rs	Tests snapshot materialization.
crates/astrid-storage/src/content/store/bulk.rs	Publishes verified batches.
crates/astrid-storage/src/content/store.rs	Exposes verified publication internally.
crates/astrid-storage/README.md	Updates performance documentation.
crates/astrid-storage/benches/storage_io/workloads.rs	Updates benchmark accounting.
crates/astrid-storage-model/src/physical/representation.rs	Normalizes admission subjects.
crates/astrid-storage-model/src/physical/mod.rs	Exports admission types.
crates/astrid-storage-model/src/physical/admission.rs	Defines canonical admission evidence.
crates/astrid-storage-model/src/lib.rs	Re-exports admission APIs.
crates/astrid-storage-engine/src/lib.rs	Exports contiguous publication types.
crates/astrid-storage-engine/src/durable/validation.rs	Loads represented closure objects.
crates/astrid-storage-engine/src/durable/tests.rs	Includes contiguous tests.
crates/astrid-storage-engine/src/durable/test_cases/contiguous.rs	Tests publication and recovery.
crates/astrid-storage-engine/src/durable/roots.rs	Recovers represented root closures.
crates/astrid-storage-engine/src/durable/restore.rs	Updates recovery invocation.
crates/astrid-storage-engine/src/durable/representations/mod.rs	Tracks contiguous locations.
crates/astrid-storage-engine/src/durable/representations/format.rs	Broadens identity visibility.
crates/astrid-storage-engine/src/durable/representations/contiguous/recovery.rs	Rebuilds contiguous indexes.
crates/astrid-storage-engine/src/durable/representations/contiguous/platform.rs	Implements reflink primitives.
crates/astrid-storage-engine/src/durable/representations/contiguous.rs	Implements loose-blob storage.
crates/astrid-storage-engine/src/durable/representation_engine.rs	Generalizes pending updates.
crates/astrid-storage-engine/src/durable/recovery.rs	Rebuilds represented objects on open.
crates/astrid-storage-engine/src/durable/mod.rs	Adds contiguous modules and errors.
crates/astrid-storage-engine/src/durable/group.rs	Generalizes grouped representation updates.
crates/astrid-storage-engine/src/durable/faults.rs	Adds contiguous fault points.
crates/astrid-storage-engine/src/durable/engine.rs	Serves represented objects and snapshots.
crates/astrid-storage-engine/src/durable/crash_replay_tests.rs	Recognizes new fault points.
crates/astrid-storage-engine/src/durable/contiguous.rs	Implements preparation and publication.
crates/astrid-storage-engine/src/durable/compaction/recovery.rs	Updates root recovery call.
crates/astrid-storage-engine/src/durable/compaction/mod.rs	Makes closure discovery representation-aware.
crates/astrid-storage-engine/Cargo.toml	Adds platform syscall dependency.
crates/astrid-storage-content/src/read.rs	Uses public content-root accessor.
crates/astrid-storage-content/src/lib.rs	Exposes the content root.
Cargo.lock	Locks the added dependency.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 4 comments.

Suppressed comments (3)

crates/astrid-storage-engine/src/durable/contiguous.rs:313

• append_contiguous_update mutates the in-memory physical maps before appending metadata, but this ? is outside the error path that calls mark_requires_recovery. An append/encoding failure leaves the engine usable with mutated maps; a retry can then see changed == false, install slice indexes, and publish a root although the durable state never named this representation. Include this mutation in the poisoned publication transaction or make the update rollback-safe.

                representations.append_contiguous_update(
                    &payload.profile,
                    &payload.representation,
                    &payload.slices,
                )?

crates/astrid-storage-engine/src/durable/representations/contiguous.rs:104

• Returning only a path does not pin the selected representation across compaction. A reader releases the engine lock before opening this path, while compaction can rebase authority and delete blobs under that lock; the reader then gets a spurious missing-blob error instead of retrying from the replacement arena. Return an opened handle/lease, or recheck the generation and retry when the representation changes.

    pub(in crate::durable) fn open_contiguous_read(
        &self,
        object: ObjectId,
    ) -> Result<Option<(File, ContiguousLocation)>, DurableError> {
        let Some(location) = self.contiguous.get(&object).copied() else {

crates/astrid-storage-model/src/physical/admission.rs:195

• observed_encoded_bytes is accepted without validation; only output bytes and the primary blob are checked below. Consequently this public checked constructor can issue canonical evidence claiming zero or over-limit encoded input, despite its contract saying byte observations are checked against reconstruction bounds. Pass the profile/expected encoded length into admission and reject inconsistent observations before deriving the transcript.

[Copilot]

Pull request overview

Copilot reviewed 47 out of 48 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

crates/astrid-storage/src/principal_state/staging.rs:391

• This revalidation does not enforce the claimed source-identity binding. StagingIntent records sequence/id/owner/name/profile/length only, and validate_generation compares only the filename and decoded footer, so replacing the sealed file with another regular file carrying the same footer is accepted. That contradicts #1467's changed-source-identity rejection gate and the updated protocol's statement that the existing intent binds source identity; persist and compare the opened file identity before adopting it.
  crates/astrid-storage-engine/src/durable/representations/contiguous.rs:414
• Any verification, occupied-preimage comparison, or hard-link error after the temporary reaches this function returns without removing it. A rejected wrong-source publication therefore leaves a full-size .blob.*.tmp; repeated retries can consume unbounded disk space. Ensure the temporary is removed on every error path (and apply the same cleanup guard around earlier copy/flush failures).

    verify_blob_bytes(temporary, profile, blob, logical_bytes)?;
    match std::fs::hard_link(temporary, path) {

[Copilot]

Pull request overview

Copilot reviewed 58 out of 59 changed files in this pull request and generated 4 comments.

Suppressed comments (2)

crates/astrid-storage-engine/src/durable/representations/contiguous/platform.rs:135

• Treat ENOTTY as an unsupported clone operation. Linux filesystems/drivers that do not implement FICLONE may report an inappropriate ioctl this way; currently those hosts fail publication instead of taking the required verified copy fallback.

pub(super) fn clone_is_unsupported(error: &std::io::Error) -> bool {
    error.kind() == std::io::ErrorKind::Unsupported
        || matches!(
            error.raw_os_error(),
            Some(libc::ENOTSUP | libc::EXDEV | libc::EINVAL)
        )

CHANGELOG.md:35

• These figures are from the earlier da8e3cd0 checkpoint, not the final clean 6384aaec benchmark documented for the current implementation. The updated README reports 1,212.1 MiB/s for eight-worker ingest and 1,774.6 MiB/s for warm verified reads, so the changelog currently presents stale performance results as the PR's clean benchmark.

  protocol. On the clean 512 MiB APFS benchmark, staging remains at native
  speed, eight-worker publication reaches 1,556.4 MiB/s, warm verified reads
  reach 1,754.4 MiB/s, and duplicate publication appends 8,387 authoritative

[Copilot]

Pull request overview

Copilot reviewed 58 out of 59 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 59 out of 60 changed files in this pull request and generated 4 comments.

Suppressed comments (1)

crates/astrid-storage-engine/src/durable/compaction/recovery.rs:165

• Compaction recovery discards the engine's retained store capability and reopens directory with ambient authority. During in-process recovery, replacing that pathname can make the rebase read/write substituted representation state while retiring blobs under a different tree. Thread the existing store_root through interrupted-compaction recovery instead of reopening it here.

    let store_root = super::super::representations::open_store_root(directory)?;
    let Some(mut representations) = RepresentationStore::open(directory, &store_root, limits)?

[Copilot]

Pull request overview

Copilot reviewed 59 out of 60 changed files in this pull request and generated no new comments.

Suppressed comments (1)

crates/astrid-storage-engine/src/durable/representations/mod.rs:158

• The retained store capability does not actually anchor representation recovery: current, metadata, and journal are still resolved through the ambient store/root paths above and below this line, while only the loose-blob directory is derived from store_root. If the configured store path is renamed and replaced before an in-process reopen, recovery can load physical authority from the replacement tree but read or retire blobs through the original pinned tree. Open the representation root and all authority files relative to store_root (using ambient paths only for diagnostics) so metadata and payloads cannot come from different stores.

        let root_directory = contiguous::open_representation_root(store_root)?;

[Copilot]

Pull request overview

Copilot reviewed 61 out of 62 changed files in this pull request and generated 4 comments.

Suppressed comments (1)

crates/astrid-storage/src/principal_state/staging.rs:469

• Duplicate names are not rejected until publish_verified_batch after this loop, so an invalid batch can clone/copy every staged file and publish orphan physical representations before returning DuplicateBatchName. Prevalidate names before beginning physical publication, preserving the prior batch behavior of rejecting duplicates before consuming work or disk.

[joshuajbouw]

Also addressed the latest suppressed review finding in 781f6e7: publish_batch now rejects duplicate content names before spawning physical publication, so invalid batches cannot create loose blobs or append representation metadata. native_staging_batch_rejects_duplicate_names_before_physical_publication proves metadata length, ready generations, and the logical catalogue remain unchanged.

[Copilot]

Pull request overview

Copilot reviewed 61 out of 62 changed files in this pull request and generated 3 comments.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.