diff --git a/README.md b/README.md index dd2599bce..b1378c2f2 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,10 @@ backend. ## Validate your first scenario -You need Python 3.11 or newer. +You need Python 3.11 or newer. OCI module extraction additionally requires +Python 3.11.4 or newer because earlier 3.11 patch releases lack the mandatory +safe tar extraction filter. Those releases fail closed instead of extracting +without it. ```console python -m venv .venv diff --git a/docs/decisions/issue-1096-oci-module-registry-reliability-remediation.md b/docs/decisions/issue-1096-oci-module-registry-reliability-remediation.md new file mode 100644 index 000000000..8c4afa350 --- /dev/null +++ b/docs/decisions/issue-1096-oci-module-registry-reliability-remediation.md @@ -0,0 +1,109 @@ +# Issue 1096 OCI Module Registry Reliability Remediation + +Date: 2026-08-11 + +Issue: [#1096](https://github.com/OpenRAE/rae/issues/1096). + +Requirement: GOV-913. + +This note records the implementation defense for deterministic OCI module +publication, transactional extraction caching, and stable registry failures. It +changes implementation behavior and operator documentation only. It adds no SDL +field, portable contract, schema, registry service, key-distribution mechanism, +or backend protocol. + +Issue [#1107](https://github.com/OpenRAE/rae/issues/1107) subsequently found +that the digest-only cache marker, two-directory replacement, and post-parse +tar limits below were insufficient. The current implementation strengthens +them with pre-parse whole-stream gzip bounds, a canonical full-tree cache +manifest, immutable version directories, and one atomic pointer. The issue +#1107 preflight is the controlling implementation defense where this historical +note describes the earlier marker or backup/rollback protocol. + +## Gap claim and ownership + +The existing architecture already had the correct owners: the module registry +owns bundle bytes, OCI descriptors, signatures, registry reads, and extraction. +The defect was that several implementation details were not canonical or +transactional. The remediation therefore hardens those incumbents rather than +introducing a second registry, cache, trust model, exception hierarchy, or +runtime adapter. + +| Root cause | Consequence | Owning requirement | Remediation | +| --- | --- | --- | --- | +| `tarfile` and gzip inherited creation time and host metadata | Identical modules could receive different bundle, manifest, and lock digests | GOV-913 | Sort POSIX names and normalize gzip filename/mtime/compression plus tar uid, gid, owner names, mode, and mtime. | +| Relative and symlink entrypoints were used before one strict canonical resolution | Bundle membership and `root_file` could depend on invocation spelling | GOV-913 | Resolve the entrypoint once, derive one canonical publishing root, and contain every local import within it. | +| Cache identity used only the manifest directory and wrote extraction directly there | A stale, mismatched, concurrent, interrupted, or locally modified entry could be reused | GOV-913 | Bind a canonical complete-tree manifest to the verified layer digest, serialize with per-entry thread and OS locks, validate in a sibling stage, install an immutable version, and atomically replace one pointer. | +| Partial signing options silently produced unsigned output | Operator intent could be downgraded without a failure | GOV-913 | Require both signer id and readable valid Ed25519 PEM key, or neither; reject whitespace-confused ids and sanitize key errors. | +| OCI layouts were updated in place | Stale blobs/files survived and failed writes exposed partial inventory | GOV-913 | Build and validate the exact inventory in a sibling stage, install an immutable version, then atomically replace one pointer without moving the prior version. | +| Network helpers used a separate timeout constant | Configured OCI limits did not govern actual reads | GOV-913 | Use `_OCI_LIMITS.timeout_seconds` for metadata and blob requests. | +| UTF-8/JSON decoder exceptions escaped from three registry surfaces | Public errors varied by payload and could reflect decoder detail | GOV-913 | Decode tag metadata, manifests, and configs through one object-only helper with stable `SDLParseError` messages. | + +## Incumbents and lineage + +- Issues #12, #13, and #14 established capped reads, safe prevalidated tar + extraction, config-blob verification, and `root_file` signature binding. +- Issue #115, ADR-071, and the reusable-asset trust specification keep identity, + integrity, and authenticity distinct. A cache directory or module id is never + proof of payload integrity. +- `SDLParseError` and `RegistryTrustPolicy` remain the relevant public error and + policy seams. + +## Standards basis + +- The [OCI image-layout specification](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) + requires `oci-layout`, `index.json`, and content-addressed blobs whose bytes + match their descriptors. +- OCI [image configuration guidance](https://github.com/opencontainers/image-spec/blob/main/config.md) + recommends reproducible packing and unpacking so content identities do not + drift. +- Python's [gzip documentation](https://docs.python.org/3/library/gzip.html) + defines `mtime=0` as creation-time-independent output, while + [tarfile](https://docs.python.org/3/library/tarfile.html) exposes the member + metadata and extraction filters normalized here. +- [TUF](https://theupdateframework.github.io/specification/latest/) and + [in-toto](https://github.com/in-toto/specification/blob/master/in-toto-spec.md) + provide the precedent for hash-bound payloads, authenticated metadata, and + consistent repository views. + +## Transaction and failure invariants + +1. Publication and extraction never stream files into their public final + directory. A complete sibling stage is validated before commit. +2. If replacement fails after moving a prior directory aside, the prior + directory is restored. Temporary stages are removed on rejection. +3. Every cache reader takes the same manifest-keyed lock and revalidates the + canonical full-tree manifest: exact layer digest, paths, types, safe modes, + sizes, file digests, declared regular root, and containment. +4. The complete decoded tar stream, including PAX/GNU metadata and padding, is + bounded by absolute and expansion-ratio limits before `tarfile` parses it. + The PEP 706 `data` filter is mandatory; early Python 3.11 patch releases fail + closed instead of using unfiltered extraction. +5. Registry response sizes stay bounded before JSON decoding. Invalid UTF-8, + syntax, or top-level JSON shape never includes body or decoder detail in the + public error. +6. Private key bytes and registry response bodies never enter public errors or + portable results. + +## Verification evidence + +`test_sdl_module_registry.py` covers byte-identical publication after host +metadata changes, normalized headers, relative/symlink identity, escape +rejection, strict signing, exact immutable layout inventory, atomic-pointer +failure and recovery, bounded version retention, configured timeout +propagation, stable metadata/manifest/config JSON errors, +PAX/GNU metadata bombs, pre-parse absolute/ratio bounds, full-tree cache +binding, content/mode/type/symlink/extra/missing tamper repair, concurrent +single extraction, old-reader continuity, and partial-extraction cleanup. + +Repository policy, requirement governance, Ruff, focused pytest, and the full +`tools/verify_all.py` graph remain required before merge. + +## Non-goals + +- No hosted registry, registry authentication, key rotation, transparency log, + signer distribution, or multi-signature threshold change. +- No cache eviction policy, cross-host shared cache, or general artifact store. +- No new OCI media type, SDL syntax, contract schema, or portable exception. +- No relaxation of the existing download, member-count, per-member, aggregate, + traversal, link, special-file, duplicate-path, or root-containment checks. diff --git a/docs/decisions/issue-1107-oci-archive-cache-publication-preflight.md b/docs/decisions/issue-1107-oci-archive-cache-publication-preflight.md new file mode 100644 index 000000000..a80084d63 --- /dev/null +++ b/docs/decisions/issue-1107-oci-archive-cache-publication-preflight.md @@ -0,0 +1,210 @@ +# Issue 1107 OCI Archive, Cache, and Publication Preflight + +Date: 2026-08-11 + +Issue: [#1107](https://github.com/OpenRAE/rae/issues/1107). + +Requirement: GOV-913. + +This note records the implementation defense before changes to the OCI module +archive, extraction cache, or publication transaction. It is a correction to +the issue #1096 implementation, not a new SDL feature, portable contract, OCI +media type, hosted registry, or experiment surface. + +## Gap claim and ownership + +The incumbent module-registry boundaries remain correct, but three mechanisms +inside them are insufficient: + +| Gap | Consequence | Incumbent owner | +| --- | --- | --- | +| `tarfile` receives the gzip stream before the complete decoded tar stream is bounded | PAX and GNU extension metadata can consume unbounded expansion and parse work before member limits run | `raes.module_registry` archive admission | +| Cache completion binds only the layer digest and declared root | An imported SDL, permission, type, or extra entry can change while a retained marker still produces a hit | `raes.module_registry` cache admission | +| Directory replacement moves the active target aside before installing the stage | Readers can observe absence, and process death can strand the only valid tree under a backup name | `raes.module_registry._filesystem` and `.publishing` | +| The completion manifest is derived only from the writable cache tree | Coordinated edits to both a cached file and its local manifest can falsely retain the verified layer-digest label | `raes.module_registry` cache admission | +| A validated cache path is reopened after its lock is released | A same-user writer can replace the root or a nested local import after inventory validation, while resolution still reports the verified layer digest and signer | `raes.module_registry` source admission and composition | +| Version pruning follows pointer replacement | A pruning error can report failure after the new version has already become current | `raes.module_registry._filesystem` transaction ordering | +| Legacy root layouts, linked cache/lock paths, recursive walks, and unsynced renames are admitted | Upgrades can expose stale alternate content, special lock nodes can block or escape the cache, deep trees can escape stable errors, and power loss can discard an apparently committed directory entry | `raes.module_registry._filesystem` and `.publishing` | + +`SDLParseError`, the existing per-key OS/thread lock, deterministic archive +builder, content-addressed OCI blobs, and `test_sdl_module_registry.py` remain +the error, concurrency, artifact, and regression surfaces. No second cache, +database, archive library, or transaction service is justified. + +## Lineage and standards basis + +- Issues #12 and #13 introduced bounded OCI reads and prevalidated extraction; + issue #14 bound config integrity and `root_file` into signatures. +- Issue #115, ADR-071, and GOV-913 establish separate identity, integrity, + authenticity, bounded-input, and redaction duties for reusable assets. +- Issue #1096 added deterministic bundles, digest markers, locks, staging, and + rollback. Issue #1107 closes the remaining P1 composition gaps. +- Python's + [tarfile documentation](https://docs.python.org/3/library/tarfile.html) and + [PEP 706](https://peps.python.org/pep-0706/) establish safe extraction filters + but explicitly do not make archive resource use safe. +- [RFC 1952](https://www.rfc-editor.org/rfc/rfc1952) defines gzip framing. The + whole decoded stream must be admitted before a tar parser sees it. +- The [OCI image-layout specification](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) + uses immutable content-addressed blobs. +- Python [`os.replace`](https://docs.python.org/3/library/os.html#os.replace) + provides the one-directory-entry atomic commit needed for a small pointer. + +## Primary implementation practice + +### Archive admission + +Decode gzip into a bounded binary spool while counting every output byte. Apply +an absolute decoded-tar cap and a compressed-to-decoded expansion-ratio cap, +validate a complete gzip stream, rewind, and only then construct `TarFile` in +uncompressed mode. This bounds regular payload, headers, padding, and PAX/GNU +metadata before parsing. Existing member-count, per-file, aggregate-file, +path-depth, traversal, duplicate, link, special-file, and mode checks remain +defense in depth. `extractall(filter="data")` is mandatory; runtimes without it +fail closed. Cache inventory uses an explicit bounded work stack rather than +Python recursion, so admitted depth cannot produce `RecursionError` residue. + +### Cache integrity + +Before any extracted-tree write, stream every validated regular member from the +bounded, already SHA-256-verified tar and build the trusted expected inventory. +It contains normalized paths, synthesized implicit directories, entry types, +safe modes in the permission classes representable by the host platform, sizes, +and file digests. POSIX directory modes are normalized to owner-only access; +Windows file and directory modes use CPython's writable `0666`/`0777` +representation. A hit parses the local +completion record under a size cap, reconstructs the exact on-disk inventory +without following links, and compares its bundle-derived projection to that +trusted inventory. The writable cache manifest is never accepted as the +authority for its own tree. Only a miss allocates a version stage and extracts +files. Hit validation remains linear in bundle bytes plus cached bytes, but it +performs no extracted-tree writes, version rename, or pointer commit; only the +bounded decoded-tar spool may spill after its memory threshold. Missing, extra, +replaced, linked, mode-changed, byte-changed, or coordinated tree-plus-manifest +edits invalidate the version and trigger a clean rebuild. + +Validation and consumption are one boundary. While the per-entry cache lock is +still held, every reachable SDL document in the root module's local-import graph +is opened through a no-follow descriptor, bound to the inventory's regular-file +identity, size, mode, and SHA-256 digest, and decoded into an immutable in-memory +source document. Composition consumes that verified source map; the retained +cache paths are metadata and cycle identities only. Nested local imports inherit +the consumer-admitted lockfile and trust policy instead of discovering mutable +policy or lock files inside the extracted cache after validation. Ordinary local +filesystem imports outside an OCI source keep their existing read and policy +behavior. An uncooperative same-user writer can still mutate cache paths, but +cannot change the bytes composed under the already-reported bundle digest. + +First-use lock creation is also a concurrent admission path. If a peer creates +the regular lock file after the missing-path check but before exclusive create, +the loser must re-enter the same no-follow type, identity, and anchored-parent +validation before opening the peer-created file. Ordinary concurrent first use +must not become an integrity failure, while linked or special files still fail +closed. + +### Gapless publication + +Each logical cache or OCI-layout slot contains immutable `versions/` trees +and one small pointer file. A writer builds and validates a sibling stage, moves +it once into a new immutable version name, and atomically replaces the pointer +without first removing the previous pointer. Existing readers retain paths into +the old version. Startup repair removes abandoned stages and restores a missing +or invalid pointer only from a fully validated immutable version. Publication +never mutates or moves the version currently addressed by the pointer. +Version names use a bounded digest lookup prefix plus a nonce so internal path +growth remains portable on Windows; the full digest is still verified from the +domain's trusted manifest or expected OCI bytes and is not truncated as an +integrity claim. + +All fallible bounded-retention pruning occurs before pointer replacement while +retaining both the selected version and the previously selected version. Pointer +replacement is therefore the publication commit, rather than an intermediate +step followed by a failure-capable cleanup. Before renaming a complete version, +regular files and directories are synced; the version-store directory is synced +after the rename; the pointer file is synced before replacement; and the slot +directory is synced after replacement on hosts with directory-fsync support. +Cache roots, lock parents, and lock files reject links and non-regular nodes +before use, so a FIFO/device cannot block before the configured lock timeout and +an unsafe link cannot redirect creation outside the intended store. Where +directory-relative opens are available, lock-file lookup and creation are +anchored to a no-follow parent descriptor and the public parent identity is +rechecked afterward. Other hosts bind and recheck the parent identity around +the open. + +The public publication result deliberately reports the selected usable +`/versions/` OCI layout, not the logical `-.oci` slot +container. The returned path therefore retains the prior contract that +`layout_dir` itself contains `oci-layout`, `index.json`, and `blobs/`; callers do +not need to parse the private pointer. The slot spelling remains stable for +locking and recovery but is not itself an OCI image-layout directory. + +A pre-versioned output already containing root-level `oci-layout`, `index.json`, +or `blobs/` is not silently converted in place. Publication fails with an +actionable instruction to move or remove that legacy output. This prevents a +dual layout in which updated callers consume `versions/` while older callers +continue consuming permanently stale root files. + +Version retention is bounded to eight complete versions per logical slot: the +selected current version, the pointer value observed immediately before the +update, and the newest remaining versions that fit the bound. This guarantees +that an immediate prior reader keeps a complete immutable path across a writer +transition while repeated changed publications cannot grow storage without +bound. A returned version is not a permanent artifact-retention service; +callers needing longer retention must copy or push the OCI layout. + +## Alternatives rejected + +1. Raise only tar member or extracted-payload limits. PAX/GNU metadata is parsed + before those counters run. +2. Hash only `root_file` or retain the layer-digest text marker. Imported files + and tree structure affect resolved behavior. +3. Add exception handling to the backup/rollback protocol. Process death skips + rollback and cannot remove the observable absence interval. +4. Rebuild the active directory in place. Concurrent readers can observe partial + mutation, and a failed writer destroys the last known-good view. +5. Keep the early Python 3.11 unfiltered fallback. A security boundary must fail + closed when its mandatory runtime primitive is unavailable. + +## Documentation and compatibility defense + +The behavior is internal and operator-visible only on rejection: adversarial +archives, corrupted cache trees, unsupported early Python 3.11 patch releases, +and failed publication produce stable `SDLParseError` failures. Artifact bytes, +OCI media types, SDL syntax, lockfile schema, and the successful CLI JSON field +schema stay unchanged. The `layout_dir` value identifies an immutable complete +version below the logical slot; callers must not assume it is a mutable +stable-name directory. + +## Verification plan + +- Adversarial PAX and GNU metadata bombs prove decoding is rejected before any + tar member is parsed; exact absolute, ratio, and path-depth boundaries are + covered. +- Simulated absence of the PEP 706 `data` filter proves fail-closed behavior and + no extracted file. +- Full-tree cache tests cover content, digest, size, mode, type, symlink, missing, + extra, malformed/noncanonical manifest, coordinated tree-plus-manifest + forgery, verified-bundle-derived expectation, retained completion record, + clean rebuild, and a hit that performs no extraction or version staging. +- Descriptor-bound source tests replace the root and a nested local document + immediately after tree validation, inject cache-local policy and lock files, + and prove that composition and provenance retain the exact verified bytes and + digests on both cache misses and hits. +- Transaction tests cover pointer replacement failure, process-death residue, + prune-before-pointer ordering, sync ordering, startup repair, immutable prior + readers, linked/special lock rejection, and concurrent writers/readers. +- Publication tests cover exact OCI inventory, deterministic bytes, atomic + pointer transitions, failed-commit preservation, legacy-layout fail-closed + upgrade behavior, and repair. +- Run focused line and branch coverage, focused tests on Python 3.11 through + 3.14, integration tests, Ruff, policy, requirement governance, and the full + verification graph. + +## Non-goals + +- No cache eviction policy, cross-host cache, general artifact store, hosted + registry, authentication, signing, or trust-policy change. +- No new SDL, schema, contract, media type, backend, experiment, or portable + exception. +- No relaxation of digest, signature, path, link, type, size, concurrency, + deterministic-byte, or redaction guarantees. diff --git a/docs/explain/sdl/runtime-architecture.md b/docs/explain/sdl/runtime-architecture.md index d721b74ba..050a5ed7d 100644 --- a/docs/explain/sdl/runtime-architecture.md +++ b/docs/explain/sdl/runtime-architecture.md @@ -416,6 +416,42 @@ Composition is registry-ready as well: - `raes sdl verify-imports` verifies lockfile, trust, digests, and signatures - `raes sdl publish` packages a publishable SDL module as an OCI image layout +OCI acquisition bounds the compressed response and the complete decoded tar +stream before parsing any tar header. The decoded-stream limit includes PAX/GNU +metadata and padding and has both absolute and expansion-ratio ceilings. Safe +extraction requires the standard-library `data` filter; Python 3.11.0 through +3.11.3 fail closed for OCI extraction rather than using unfiltered extraction. + +Resolved module caches and published layouts use immutable version directories +under a logical slot plus one atomically replaced `.raes-current` pointer. A +cache hit streams a trusted expected inventory directly from the verified, +bounded tar, then re-hashes the exact extracted tree against its canonical +completion manifest. The comparison covers every path and entry type plus each +file's filtered mode, size, and digest. The cache's own writable manifest is not +its authority. A hit performs no extraction or version staging; only the +bounded decoded-tar spool may spill beyond its memory threshold. Missing, +extra, linked, mode-changed, byte-changed, or coordinated tree-plus-manifest +edits rebuild cleanly. +Publication returns the selected immutable OCI-layout version path; a failed +writer leaves the prior pointer and every extant reader path intact. Startup +repair removes incomplete stages and can repoint to a complete validated orphan +version left by process death. + +For publication, the stable `-.oci` name is the private logical +slot, not a directly consumable OCI layout. CLI/API `layout_dir` points to the +selected immutable `versions/` child, which contains the standard +`oci-layout`, `index.json`, and `blobs/` inventory. Consumers should use that +reported path rather than infer the slot. Each slot retains at most eight +complete versions: current, the immediately prior pointer target, and the +newest remaining versions. This window protects an in-flight prior reader but +is not archival retention; push or copy layouts that must outlive later +publications. + +An older root-level OCI layout at the stable slot name is not silently upgraded +into a dual layout. Publication fails with instructions to move or remove that +legacy output first, preventing older consumers from continuing to read stale +root `index.json` and `blobs/` while newer consumers use a version child. + Resolution and trust happen before instantiation and semantic validation. Planner/runtime semantics see one admitted concrete scenario; replay-relevant resolution and binding facts remain under its typed provenance rather than in diff --git a/docs/public/guides/cli.md b/docs/public/guides/cli.md index dc630ad18..02cdd3055 100644 --- a/docs/public/guides/cli.md +++ b/docs/public/guides/cli.md @@ -61,8 +61,22 @@ conformance report. Use `raes sdl format --check` to check the source format. The old `raes sdl resolve`, `verify-imports`, and `publish` commands maintain module packages. `sdl resolve` writes a lockfile. `verify-imports` checks locked imports. -`publish` writes an OCI layout. These commands are not part of the offline -`raes semantic` contract. Pack-aware workflows belong in env-packs. +`publish` writes an OCI layout. Publication resolves relative and symlinked +entrypoints to one canonical module root and produces deterministic tar+gzip +bytes: identical module bytes produce identical bundle and manifest digests. +The completed layout replaces an older layout transactionally, so stale blobs +or files are not retained and a failed commit preserves the prior layout. + +Signing is explicit. Pass `--signer-id` and `--private-key` together to produce +an Ed25519 signature, or omit both for unsigned publication. Supplying only one, +an unreadable/invalid key, or a signer id with surrounding whitespace fails +closed rather than silently publishing unsigned content. + +These commands are not part of the offline `raes semantic` contract. Pack-aware +workflows belong in env-packs. OCI resolution uses bounded configured timeouts; +its digest-bound extraction cache is committed only after complete validation, +and malformed registry JSON is reported as a stable `SDLParseError` without +including response bodies or decoder internals. Use `raes processor --help` and `raes conformance --help` for the processor and backend-contract surfaces. The [CLI API reference](../api/cli.rst) lists the diff --git a/docs/requirements/GOV-913/requirement.md b/docs/requirements/GOV-913/requirement.md index a8e4454d8..4b4a30cd3 100644 --- a/docs/requirements/GOV-913/requirement.md +++ b/docs/requirements/GOV-913/requirement.md @@ -38,3 +38,16 @@ Requirement inventory expansion. Reusable ecosystem assets need explicit trust a - DOCUMENTS → DOCUMENTATION `docs/decisions/issue-1106-gov-913-osv-cache-integrity.md` (Repository pin and atomic cache decision) - IMPLEMENTS → CODE_FILE `tools/osv_scanner_tool.py` (Per-use cache type, mode, and digest validation) - TESTS → TEST `implementations/python/tests/test_repo_policy_tools.py` (Tampered, symlinked, and atomic OSV cache regressions) +- DOCUMENTS → GITHUB_ISSUE `1096` (OCI module-registry reliability remediation) +- DOCUMENTS → DOCUMENTATION `docs/decisions/issue-1096-oci-module-registry-reliability-remediation.md` (OCI module-registry reliability decision) +- IMPLEMENTS → GITHUB_ISSUE `1107` (Bound OCI archive parsing and make cache/layout publication gapless) +- DOCUMENTS → DOCUMENTATION `docs/decisions/issue-1107-oci-archive-cache-publication-preflight.md` (Verified-bundle cache admission, bounded archive trees, and durable versioned publication preflight) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/_archive.py` (Bounded gzip admission and safe tar extraction) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/_verified_sources.py` (Descriptor-bound immutable OCI source graph) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/_cache.py` (Anchored cache locking, bounded archive admission, and concurrent first-use recovery) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/_filesystem.py` (Immutable version directories and atomic pointer repair) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/publishing.py` (Deterministic bundles and durable OCI layout publication) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/module_registry/resolution.py` (Stable registry parsing and verified cache admission) +- IMPLEMENTS → CODE_FILE `implementations/python/packages/raes/composition/_expand.py` (Verified nested-source and admitted policy context propagation) +- TESTS → TEST `implementations/python/tests/test_sdl_module_registry.py` (Bundle-bound cache integrity, safe lock/depth handling, legacy-layout rejection, and atomic-pointer regressions) +- TESTS → TEST `implementations/python/tests/test_issue_1107_oci_source_snapshot.py` (Descriptor-bound source snapshot regressions) diff --git a/implementations/python/packages/raes/composition/_expand.py b/implementations/python/packages/raes/composition/_expand.py index e0b2b087a..2dcde6765 100644 --- a/implementations/python/packages/raes/composition/_expand.py +++ b/implementations/python/packages/raes/composition/_expand.py @@ -9,7 +9,7 @@ from dataclasses import dataclass from pathlib import Path -from typing import Any +from typing import Any, TypedDict, Unpack from pydantic import ValidationError @@ -49,6 +49,7 @@ from ..module_registry import ( Lockfile, TrustPolicy, + _VerifiedSourceBundle, load_lockfile, load_trust_policy, resolve_import, @@ -199,6 +200,43 @@ class _ImportContext: migration_policy: SDLMigrationPolicy | str limits: SDLParserLimits source_diagnostics: list[SDLParseDiagnostic] | None + verified_sources: _VerifiedSourceBundle | None + registry_base_dir: Path + + +@dataclass(frozen=True) +class _ExpansionContext: + """Private traversal and trust state carried across recursive expansion.""" + + traversal: CompositionTraversal | None = None + verified_sources: _VerifiedSourceBundle | None = None + inherited_lockfile: Lockfile | None = None + inherited_trust_policy: TrustPolicy | None = None + registry_base_dir: Path | None = None + + +class _ExpansionPrivateOptions(TypedDict, total=False): + """Typed compatibility keywords for private recursive expansion state.""" + + _context: _ExpansionContext + _traversal: CompositionTraversal | None + _verified_sources: _VerifiedSourceBundle | None + _inherited_lockfile: Lockfile | None + _inherited_trust_policy: TrustPolicy | None + _registry_base_dir: Path | None + + +def _expansion_context(options: _ExpansionPrivateOptions) -> _ExpansionContext: + current = options.get("_context") + if current is not None: + return current + return _ExpansionContext( + traversal=options.get("_traversal"), + verified_sources=options.get("_verified_sources"), + inherited_lockfile=options.get("_inherited_lockfile"), + inherited_trust_policy=options.get("_inherited_trust_policy"), + registry_base_dir=options.get("_registry_base_dir"), + ) def _expand_one_import( @@ -233,6 +271,8 @@ def _expand_one_import( limits=context.limits, ), source_diagnostics=context.source_diagnostics, + verified_sources=context.verified_sources, + _registry_base_dir=context.registry_base_dir, ) import_path = resolved_import.root_file imported_raw = _load_normalized_data( @@ -250,7 +290,13 @@ def _expand_one_import( migration_policy=context.migration_policy, limits=context.limits, source_diagnostics=context.source_diagnostics, - _traversal=context.child_traversal, + _context=_ExpansionContext( + traversal=context.child_traversal, + verified_sources=resolved_import.verified_sources, + inherited_lockfile=context.lockfile, + inherited_trust_policy=context.trust_policy, + registry_base_dir=context.registry_base_dir if resolved_import.verified_sources is not None else None, + ), ) try: imported_scenario = ExpandedScenario.model_validate(imported_expanded) @@ -320,11 +366,12 @@ def expand_sdl_modules( migration_policy: SDLMigrationPolicy | str = SDLMigrationPolicy.REJECT, limits: SDLParserLimits = DEFAULT_PARSER_LIMITS, source_diagnostics: list[SDLParseDiagnostic] | None = None, - _traversal: CompositionTraversal | None = None, + **private_options: Unpack[_ExpansionPrivateOptions], ) -> tuple[dict[str, Any], ExpansionProvenance]: """Expand trusted imports into executable content and portable evidence.""" - traversal = _traversal or CompositionTraversal( + expansion_context = _expansion_context(private_options) + traversal = expansion_context.traversal or CompositionTraversal( seen=frozenset(), budget=CompositionBudget(limits), depth=0, @@ -332,10 +379,17 @@ def expand_sdl_modules( budget = traversal.budget budget.check_depth(traversal.depth, path=path) budget.add_document(data, path=path) - resolved_path = path.resolve() + resolved_path = ( + expansion_context.verified_sources.identity_path(path) + if expansion_context.verified_sources is not None + else path.resolve() + ) if resolved_path in traversal.seen: raise SDLParseError(f"Import cycle detected at {resolved_path}", path=path) child_traversal = traversal.descend_from(resolved_path) + registry_base_dir = ( + resolved_path.parent if expansion_context.registry_base_dir is None else expansion_context.registry_base_dir + ) merged = dict(data) merged.setdefault("imports", []) @@ -353,8 +407,16 @@ def expand_sdl_modules( realization_constraints.extend(constraint_records(typed_designation)) except ValidationError as exc: raise SDLParseError("Realization designation is structurally invalid", path=path) from exc - lockfile = load_lockfile(resolved_path.parent) - trust_policy = load_trust_policy(resolved_path.parent) + if expansion_context.verified_sources is None: + lockfile = load_lockfile(resolved_path.parent) + trust_policy = load_trust_policy(resolved_path.parent) + else: + lockfile = expansion_context.inherited_lockfile + trust_policy = ( + expansion_context.inherited_trust_policy + if expansion_context.inherited_trust_policy is not None + else TrustPolicy() + ) context = _ImportContext( path=path, resolved_path=resolved_path, @@ -366,6 +428,8 @@ def expand_sdl_modules( migration_policy=migration_policy, limits=limits, source_diagnostics=source_diagnostics, + verified_sources=expansion_context.verified_sources, + registry_base_dir=registry_base_dir, ) for raw_import in merged.get("imports", []): diff --git a/implementations/python/packages/raes/module_registry/__init__.py b/implementations/python/packages/raes/module_registry/__init__.py index 5229ab834..0bfd92f24 100644 --- a/implementations/python/packages/raes/module_registry/__init__.py +++ b/implementations/python/packages/raes/module_registry/__init__.py @@ -1,43 +1,59 @@ """Registry-aware SDL module resolution and publishing. -This package is a thin facade over cohesive subdomains: - -* :mod:`._constants` - lockfile / trust-policy / OCI-layout names and schema versions. -* :mod:`._digests` - digest and version-matching helpers. -* :mod:`.models` - Pydantic policy/lock models, the resolved-module DTO, and lockfile persistence. -* :mod:`.signing` - Ed25519 signature payloads and trusted-signer verification. -* :mod:`.resolution` - local/locked/OCI import resolution orchestration. -* :mod:`.publishing` - OCI-layout publishing. - -The OCI transport and archive-safety security boundary (URL fetch with an explicit -timeout, capped reads, and tar-member validation before extraction) is defined in -this module rather than a submodule on purpose. ``test_sdl_module_registry.py`` -patches ``raes.module_registry.urlopen`` and ``raes.module_registry._OCI_LIMITS`` on -the package object, and a Python function resolves such globals from the module -where it is *defined*. Keeping these seams defined here preserves that patch -behavior for the request and archive paths without modifying the tests; the -resolution orchestrator reaches them through a function-local ``from . import``. +This package is a thin facade over cohesive cache, extraction, filesystem, +model, resolution, signing, and publishing subdomains. + +The package facade remains the compatibility and injection surface for OCI +transport and archive limits. The cache/extraction helpers dynamically use the +facade's patchable seams, preserving historical test behavior while keeping each +source module below the repository size cap. """ from __future__ import annotations -import io import json +import os import tarfile +import time +import zlib from dataclasses import dataclass -from pathlib import Path +from pathlib import Path, PurePosixPath from typing import Any, Protocol from urllib.error import URLError from urllib.request import Request, urlopen -# The submodule imports below are this package's public re-export surface, plus -# the private ``_sha256_digest`` / ``_signable_payload`` / ``_verify_signatures`` -# seams the pre-split module exposed for tests. They are deliberately NOT narrowed -# by an ``__all__``: the single-file module had none, so adding one would change -# the legacy ``from raes.module_registry import *`` semantics. F401 is ignored for -# this facade in pyproject.toml - the "unused import" claim is false for re-exports. +# The submodule imports below retain the package's documented domain re-exports +# plus the private ``_sha256_digest`` / ``_signable_payload`` / +# ``_verify_signatures`` seams used by existing tests. Incidental implementation +# imports are not an API; F401 is ignored for this facade because these deliberate +# domain re-exports otherwise look unused inside this file. from .._errors import SDLParseError +from .._source_profile import SDLSourceParseOptions from ..scenario import ImportDecl, ModuleDescriptor, Scenario +from ._archive import _expected_cache_tree_manifest +from ._cache import ( + _CACHE_THREAD_LOCKS, + _CACHE_THREAD_LOCKS_GUARD, + _SPOOL_MEMORY_BYTES, + _acquire_file_lock, + _bounded_gzip_tar_stream, + _cache_entry_lock, + _open_cache_lock, + _recover_cache_root, + _release_file_lock, +) +from ._cache_integrity import ( + _CACHE_TREE_SCHEMA, + _DECOMPRESSION_CHUNK_BYTES, + _cache_tree_entries, + _cache_tree_manifest, + _canonical_json_bytes, + _hash_cache_file, + _read_cache_manifest_bytes, + _trusted_entry_projection, + _validated_cache_root, + _write_cache_tree_manifest, +) from ._constants import ( LOCKFILE_NAME, LOCKFILE_SCHEMA_VERSION, @@ -49,6 +65,27 @@ TRUST_POLICY_SCHEMA_VERSION, ) from ._digests import _sha256_digest +from ._extraction import ( + _HTTP_TIMEOUT_SECONDS, + _extract_tar_to_stage, + _OCIResourceLimits, + _safe_tar_members_with_limits, + _validate_tar_member_shape, +) +from ._filesystem import ( + _install_version_directory, + _iter_version_directories, + _new_digest_version_name, + _new_version_stage, + _prepare_versioned_slot, + _prune_version_directories, + _read_version_pointer, + _remove_path, + _require_directory, + _same_file_identity, + _write_version_pointer, +) +from ._verified_sources import _cache_source_result, _VerifiedSourceBundle from .models import ( Lockfile, LockRecord, @@ -63,31 +100,6 @@ from .resolution import resolve_import, resolve_lock_records from .signing import _signable_payload, _verify_signatures -_HTTP_TIMEOUT_SECONDS = 30 - - -@dataclass(frozen=True) -class _OCIResourceLimits: - """Bounds for remote OCI fetches and bundle extraction (issue #12). - - The OCI import path pulls attacker-influenceable bytes from allowlisted - registries; without caps a compromised registry, mirror, or oversized module - can exhaust process memory (buffering an unbounded response) or disk/CPU - (extracting an unbounded bundle). Compressed-download limits are kept separate - from extracted-archive limits because a small gzip can expand into a large tar - payload. This is the single extensibility seam: operator-tunable overrides - should later extend ``RegistryTrustPolicy`` and merge with these defaults, - rather than threading limit arguments through parser/compiler/runtime/CLI. - """ - - timeout_seconds: int = _HTTP_TIMEOUT_SECONDS - max_metadata_bytes: int = 8 * 1024 * 1024 - max_bundle_bytes: int = 128 * 1024 * 1024 - max_bundle_members: int = 8192 - max_member_bytes: int = 64 * 1024 * 1024 - max_total_bytes: int = 256 * 1024 * 1024 - - _OCI_LIMITS = _OCIResourceLimits() @@ -145,140 +157,332 @@ def _json_request(url: str, *, headers: dict[str, str] | None = None, max_bytes: request = Request(url, headers=headers or {}) limit = _OCI_LIMITS.max_metadata_bytes if max_bytes is None else max_bytes try: - with urlopen(request, timeout=_HTTP_TIMEOUT_SECONDS) as response: - return json.loads(_read_capped(response, url=url, max_bytes=limit).decode("utf-8")) - except (URLError, json.JSONDecodeError) as exc: - raise SDLParseError(f"Failed to fetch OCI metadata from {url}: {exc}") from exc + with urlopen(request, timeout=_OCI_LIMITS.timeout_seconds) as response: + payload = _read_capped(response, url=url, max_bytes=limit) + except URLError as exc: + raise SDLParseError(f"Failed to fetch OCI metadata from {url}") from exc + return _decode_json_object(payload, context=f"OCI metadata from {url}") + + +def _decode_json_object(payload: bytes, *, context: str) -> dict[str, Any]: + """Decode attacker-controlled JSON behind one stable, bounded error surface.""" + + try: + decoded = json.loads(payload.decode("utf-8")) + except (RecursionError, ValueError) as exc: + raise SDLParseError(f"{context} is not valid UTF-8 JSON") from exc + if not isinstance(decoded, dict): + raise SDLParseError(f"{context} must be a JSON object") + return decoded def _bytes_request(url: str, *, headers: dict[str, str] | None = None, max_bytes: int | None = None) -> bytes: request = Request(url, headers=headers or {}) limit = _OCI_LIMITS.max_metadata_bytes if max_bytes is None else max_bytes try: - with urlopen(request, timeout=_HTTP_TIMEOUT_SECONDS) as response: + with urlopen(request, timeout=_OCI_LIMITS.timeout_seconds) as response: return _read_capped(response, url=url, max_bytes=limit) except URLError as exc: - raise SDLParseError(f"Failed to fetch OCI blob from {url}: {exc}") from exc + raise SDLParseError(f"Failed to fetch OCI blob from {url}") from exc def _oci_cache_dir(base_dir: Path) -> Path: return base_dir / ".raes" / "module-cache" -def _validate_tar_member_shape( - member: tarfile.TarInfo, - *, +def _safe_tar_members( + tar: tarfile.TarFile, dest: Path, - resolved_dest: Path, - seen_paths: set[str], - limits: _OCIResourceLimits, -) -> None: - """Fail closed on an unsafe or oversized single tar member (issues #12/#13). - - Rejects path traversal, symlinks, hard links, special files, and duplicate - normalized paths, and enforces the per-member extracted-size cap. Records the - member's normalized path in ``seen_paths`` so a later duplicate is caught. - """ - member_path = (dest / member.name).resolve() - if not member_path.is_relative_to(resolved_dest): - raise SDLParseError(f"Path traversal detected in OCI bundle tar member: {member.name!r}") - if member.issym() or member.islnk(): - raise SDLParseError(f"Links are not allowed in OCI bundle tar: {member.name!r}") - if not (member.isfile() or member.isdir()): - raise SDLParseError(f"Unsupported tar member type in OCI bundle: {member.name!r}") - normalized = member_path.as_posix() - if normalized in seen_paths: - raise SDLParseError(f"Duplicate tar member path in OCI bundle: {member.name!r}") - seen_paths.add(normalized) - # Account by the logical member size so a sparse or padded member cannot - # understate the bytes it will extract. - if member.isfile() and member.size > limits.max_member_bytes: - raise SDLParseError( - f"OCI bundle member {member.name!r} exceeds the {limits.max_member_bytes}-byte per-member limit" +) -> list[tarfile.TarInfo]: + """Validate every tar member under the facade's patchable limits seam.""" + + return _safe_tar_members_with_limits(tar, dest, limits=_OCI_LIMITS) + + +@dataclass(frozen=True) +class _CacheExtraction: + """Validated immutable inputs for one cache transaction.""" + + bundle_bytes: bytes + content_digest: str + root_file: str + root_relative: PurePosixPath + cache_slot: Path + lock_path: Path + source_options: SDLSourceParseOptions | None + + +def _validated_cache_key(manifest_digest: str) -> str: + invalid = any( + ( + not manifest_digest, + manifest_digest in {".", ".."}, + "/" in manifest_digest, + "\\" in manifest_digest, + "\x00" in manifest_digest, ) + ) + if invalid: + raise SDLParseError("Invalid OCI manifest digest cache key") + return manifest_digest + + +def _is_windows_absolute_path(path: PurePosixPath) -> bool: + return bool(path.parts) and len(path.parts[0]) == 2 and path.parts[0][1] == ":" + + +def _validated_root_file(root_file: str) -> PurePosixPath: + root_relative = PurePosixPath(root_file) + invalid = any( + ( + not root_file, + "\\" in root_file, + "\x00" in root_file, + ".." in root_relative.parts, + root_relative.is_absolute(), + root_relative == PurePosixPath("."), + root_file != root_relative.as_posix(), + _is_windows_absolute_path(root_relative), + ) + ) + if invalid: + raise SDLParseError(f"Invalid OCI root_file path: {root_file!r}") + return root_relative -def _safe_tar_members( +def _validated_content_digest(bundle_bytes: bytes, expected: str | None) -> str: + actual = f"sha256:{_sha256_digest(bundle_bytes)}" + expected_digest = expected or actual + if expected_digest != actual: + raise SDLParseError("OCI module bundle does not match its expected content digest") + return expected_digest + + +def _cache_extraction( + *, + bundle_bytes: bytes, + manifest_digest: str, + content_digest: str | None = None, + root_file: str, + base_dir: Path, + source_options: SDLSourceParseOptions | None = None, +) -> _CacheExtraction: + cache_key = _validated_cache_key(manifest_digest) + root_relative = _validated_root_file(root_file) + expected_content_digest = _validated_content_digest(bundle_bytes, content_digest) + cache_root = _oci_cache_dir(base_dir) + cache_error = "Unable to create the OCI module cache" + _require_directory(cache_root.parent, error_message=cache_error) + _require_directory(cache_root, error_message=cache_error) + return _CacheExtraction( + bundle_bytes=bundle_bytes, + content_digest=expected_content_digest, + root_file=root_file, + root_relative=root_relative, + cache_slot=cache_root / cache_key, + lock_path=cache_root / ".locks" / f"{cache_key}.lock", + source_options=source_options, + ) + + +def _expected_extraction_manifest( + extraction: _CacheExtraction, + *, tar: tarfile.TarFile, - dest: Path, -) -> list[tarfile.TarInfo]: - """Validate every tar member before extraction (fail closed). - - The OCI bundle bytes are attacker-controlled even after registry allowlisting, - digest pinning, and signature verification, so this validation is the - filesystem-write boundary for module import resolution. It must hold on every - supported runtime, not just on Python 3.12+ where ``extractall(filter="data")`` - is available, because the PEP 706 ``filter`` keyword was backported only in - Python 3.11.4 while the project supports ``>=3.11``. Validation therefore - matches the ``data`` filter's guarantees: reject path traversal, symlinks, - hard links, and special files, and strip setuid/setgid/sticky bits. - - It is also the resource-exhaustion boundary (issue #12): the archive member - count, per-member extracted size, and total extracted bytes are bounded by - ``_OCI_LIMITS`` and duplicate normalized paths are rejected, so a malicious or - oversized bundle cannot exhaust disk or CPU during extraction. - """ - limits = _OCI_LIMITS - safe: list[tarfile.TarInfo] = [] - resolved_dest = dest.resolve() - seen_paths: set[str] = set() - total_bytes = 0 - # Iterate lazily rather than materialising ``tar.getmembers()`` so a bundle that - # declares an unbounded member list, or expands into an unbounded extraction, is - # rejected as soon as a cap is crossed - before the remainder of the archive is - # decompressed (issue #12). - for member_count, member in enumerate(tar, start=1): - if member_count > limits.max_bundle_members: - raise SDLParseError(f"OCI bundle exceeds the maximum of {limits.max_bundle_members} archive members") - _validate_tar_member_shape( - member, - dest=dest, - resolved_dest=resolved_dest, - seen_paths=seen_paths, - limits=limits, + safe_members: list[tarfile.TarInfo], +) -> dict[str, Any]: + expected_manifest = _expected_cache_tree_manifest( + tar=tar, + members=safe_members, + content_digest=extraction.content_digest, + root_file=extraction.root_file, + ) + expected_root = next( + (entry for entry in expected_manifest["entries"] if entry["path"] == extraction.root_relative.as_posix()), + None, + ) + if expected_root is None or expected_root["type"] != "file": + raise SDLParseError(f"Resolved OCI module bundle is missing declared root file '{extraction.root_file}'") + return expected_manifest + + +def _source_result( + extraction: _CacheExtraction, + root: Path, + expected_manifest: dict[str, Any], +) -> Path | _VerifiedSourceBundle: + return _cache_source_result( + root, + expected_manifest=expected_manifest, + root_relative=extraction.root_relative, + source_options=extraction.source_options, + ) + + +def _recover_extracted_source( + extraction: _CacheExtraction, + *, + versions: Path, + expected_manifest: dict[str, Any], +) -> Path | _VerifiedSourceBundle | None: + hit = _recover_cache_root( + slot=extraction.cache_slot, + versions=versions, + expected_content_digest=extraction.content_digest, + expected_manifest=expected_manifest, + root_relative=extraction.root_relative, + ) + if hit is None: + return None + return _source_result(extraction, hit, expected_manifest) + + +def _valid_staged_root(staged_root: Path, *, staging: Path) -> bool: + if staged_root.is_symlink() or not staged_root.is_file(): + return False + return staged_root.resolve(strict=True).is_relative_to(staging.resolve(strict=True)) + + +def _commit_extracted_stage( + extraction: _CacheExtraction, + *, + staging: Path, + versions: Path, + prior_version: str | None, + expected_manifest: dict[str, Any], +) -> Path | _VerifiedSourceBundle: + staged_validation = _validated_cache_root( + version=staging, + expected_manifest=expected_manifest, + root_relative=extraction.root_relative, + ) + if staged_validation is None: + raise SDLParseError("Staged OCI module cache entry failed validation") + version_name = _new_digest_version_name(extraction.content_digest) + installed = _install_version_directory( + staged=staging, + versions=versions, + version_name=version_name, + error_message="Unable to commit the OCI module cache entry atomically", + ) + committed = _validated_cache_root( + version=installed, + expected_manifest=expected_manifest, + root_relative=extraction.root_relative, + ) + if committed is None: + raise SDLParseError("Committed OCI module cache entry failed validation") + _prune_version_directories( + versions=versions, + retain_names={version_name, *(() if prior_version is None else (prior_version,))}, + error_message="Unable to prune stale OCI module cache versions", + ) + _write_version_pointer( + slot=extraction.cache_slot, + version_name=version_name, + error_message="Unable to commit the OCI module cache pointer atomically", + ) + return _source_result(extraction, committed, expected_manifest) + + +def _install_extracted_source( + extraction: _CacheExtraction, + *, + tar: tarfile.TarFile, + safe_members: list[tarfile.TarInfo], + versions: Path, + expected_manifest: dict[str, Any], +) -> Path | _VerifiedSourceBundle: + staging = _new_version_stage( + versions=versions, + error_message="Unable to stage the OCI module cache entry", + ) + try: + _extract_tar_to_stage( + tar=tar, + members=safe_members, + staging=staging, + expected_entries=expected_manifest["entries"], + ) + staged_root = staging.joinpath(*extraction.root_relative.parts) + if not _valid_staged_root(staged_root, staging=staging): + raise SDLParseError(f"Resolved OCI module bundle is missing declared root file '{extraction.root_file}'") + prior_version = _read_version_pointer(slot=extraction.cache_slot) + _write_cache_tree_manifest( + root=staging, + content_digest=extraction.content_digest, + root_file=extraction.root_file, ) - if member.isfile(): - total_bytes += member.size - if total_bytes > limits.max_total_bytes: - raise SDLParseError(f"OCI bundle exceeds the {limits.max_total_bytes}-byte total extraction limit") - # Drop setuid/setgid/sticky bits. - member.mode &= 0o777 - safe.append(member) - return safe + return _commit_extracted_stage( + extraction, + staging=staging, + versions=versions, + prior_version=prior_version, + expected_manifest=expected_manifest, + ) + finally: + _remove_path(staging) + + +def _read_or_install_extracted_source( + extraction: _CacheExtraction, + *, + tar: tarfile.TarFile, + versions: Path, +) -> Path | _VerifiedSourceBundle: + # The complete uncompressed stream is admitted before ``tarfile`` parses its + # first header. Member validation and the standard data filter then provide + # independent filesystem-write defenses. + safe_members = _safe_tar_members(tar, versions / ".inventory") + expected_manifest = _expected_extraction_manifest( + extraction, + tar=tar, + safe_members=safe_members, + ) + recovered = _recover_extracted_source( + extraction, + versions=versions, + expected_manifest=expected_manifest, + ) + if recovered is not None: + return recovered + return _install_extracted_source( + extraction, + tar=tar, + safe_members=safe_members, + versions=versions, + expected_manifest=expected_manifest, + ) def _extract_bundle_to_cache( *, bundle_bytes: bytes, manifest_digest: str, + content_digest: str | None = None, root_file: str, base_dir: Path, -) -> Path: - cache_dir = _oci_cache_dir(base_dir) / manifest_digest - if ".." in Path(root_file).parts or Path(root_file).is_absolute(): - raise SDLParseError(f"Invalid OCI root_file path: {root_file!r}") - resolved_cache = cache_dir.resolve() - root_path = cache_dir / root_file - if not root_path.exists(): - cache_dir.mkdir(parents=True, exist_ok=True) - with tarfile.open(fileobj=io.BytesIO(bundle_bytes), mode="r:gz") as tar: - # Validate every member up front so the security property is identical on - # all supported runtimes and never depends on the runtime's tarfile filter - # support. ``filter="data"`` is applied as defense in depth where available - # (Python 3.11.4+/3.12+); on 3.11.0–3.11.3 the keyword is absent and the - # already-validated members are the guarantee. No path falls back to an - # unfiltered ``tar.extractall(cache_dir)``. - safe_members = _safe_tar_members(tar, cache_dir) - try: - tar.extractall(cache_dir, members=safe_members, filter="data") - # Python 3.11.0–3.11.3 lack the PEP 706 filter keyword. - except TypeError: - tar.extractall(cache_dir, members=safe_members) - # Enforce the root-file containment contract on EVERY return path, including - # the cache-hit fast path: a stale cache (e.g. one populated by an earlier - # unsafe extractor) could hold a symlink or a non-regular file at root_file - # that resolves outside the digest cache. Validating here fails closed - # regardless of whether extraction ran this call. - if not root_path.is_file() or not root_path.resolve().is_relative_to(resolved_cache): - raise SDLParseError(f"Resolved OCI module bundle is missing declared root file '{root_file}'") - return root_path + source_options: SDLSourceParseOptions | None = None, +) -> Path | _VerifiedSourceBundle: + extraction = _cache_extraction( + bundle_bytes=bundle_bytes, + manifest_digest=manifest_digest, + content_digest=content_digest, + root_file=root_file, + base_dir=base_dir, + source_options=source_options, + ) + + with _cache_entry_lock(extraction.lock_path): + versions = _prepare_versioned_slot( + slot=extraction.cache_slot, + error_message="Unable to prepare the OCI module cache entry", + ) + try: + with ( + _bounded_gzip_tar_stream(extraction.bundle_bytes) as tar_stream, + tarfile.open(fileobj=tar_stream, mode="r:") as tar, + ): + return _read_or_install_extracted_source(extraction, tar=tar, versions=versions) + except (EOFError, OSError, tarfile.TarError, zlib.error) as exc: + raise SDLParseError("OCI module bundle is not a valid gzip-compressed tar archive") from exc diff --git a/implementations/python/packages/raes/module_registry/_archive.py b/implementations/python/packages/raes/module_registry/_archive.py new file mode 100644 index 000000000..a0608c028 --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_archive.py @@ -0,0 +1,137 @@ +"""Trusted OCI archive inventory derived without extracting filesystem content.""" + +from __future__ import annotations + +import hashlib +import os +import tarfile +from pathlib import PurePosixPath +from typing import TYPE_CHECKING, Any + +from .._errors import SDLParseError +from ._cache_integrity import ( + _CACHE_TREE_SCHEMA, + _DECOMPRESSION_CHUNK_BYTES, + _canonical_json_bytes, + _representable_directory_mode, +) + +if TYPE_CHECKING: + from . import _OCIResourceLimits + +_ARCHIVE_PATH_CONFLICT_ERROR = "OCI bundle contains conflicting file and directory paths" + + +def _limits() -> _OCIResourceLimits: + from . import _OCI_LIMITS + + return _OCI_LIMITS + + +def _filtered_file_mode(mode: int) -> int: + """Return the host representation of the PEP 706-filtered regular-file mode.""" + + filtered = mode & 0o755 + if not filtered & 0o100: + filtered &= ~0o111 + return {"nt": 0o666}.get(os.name, filtered | 0o600) + + +def _add_directory_node(nodes: dict[str, dict[str, Any]], relative: PurePosixPath) -> None: + relative_name = relative.as_posix() + existing = nodes.get(relative_name) + if existing is not None and existing["type"] != "directory": + raise SDLParseError(_ARCHIVE_PATH_CONFLICT_ERROR) + nodes[relative_name] = { + "mode": _representable_directory_mode(), + "path": relative_name, + "type": "directory", + } + + +def _add_parent_directories(nodes: dict[str, dict[str, Any]], relative: PurePosixPath) -> None: + for parent in reversed(relative.parents): + if parent != PurePosixPath("."): + _add_directory_node(nodes, parent) + + +def _hash_archive_member(tar: tarfile.TarFile, member: tarfile.TarInfo) -> tuple[int, str]: + extracted = tar.extractfile(member) + if extracted is None: + raise SDLParseError("Unable to read a regular file from the OCI module bundle") + digest = hashlib.sha256() + size = 0 + with extracted: + while chunk := extracted.read(_DECOMPRESSION_CHUNK_BYTES): + size += len(chunk) + if size > member.size: + raise SDLParseError("OCI bundle member payload exceeds its declared size") + digest.update(chunk) + if size != member.size: + raise SDLParseError("OCI bundle member payload is shorter than its declared size") + return size, f"sha256:{digest.hexdigest()}" + + +def _add_archive_member( + nodes: dict[str, dict[str, Any]], + *, + tar: tarfile.TarFile, + member: tarfile.TarInfo, + relative: PurePosixPath, +) -> None: + if member.isdir(): + _add_directory_node(nodes, relative) + return + relative_name = relative.as_posix() + if relative_name in nodes: + raise SDLParseError(_ARCHIVE_PATH_CONFLICT_ERROR) + size, digest = _hash_archive_member(tar, member) + nodes[relative_name] = { + "digest": digest, + "mode": _filtered_file_mode(member.mode), + "path": relative_name, + "size": size, + "type": "file", + } + + +def _is_archive_root(member: tarfile.TarInfo, relative: PurePosixPath) -> bool: + if relative != PurePosixPath("."): + return False + if not member.isdir(): + raise SDLParseError("OCI bundle cannot replace the cache tree root") + return True + + +def _expected_cache_tree_manifest( + *, + tar: tarfile.TarFile, + members: list[tarfile.TarInfo], + content_digest: str, + root_file: str, +) -> dict[str, Any]: + """Hash a verified tar into the platform-neutral cache integrity inventory.""" + + nodes: dict[str, dict[str, Any]] = { + ".": {"mode": _representable_directory_mode(), "path": ".", "type": "directory"} + } + entry_limit = _limits().max_bundle_members + 1 + for member in members: + relative = PurePosixPath(member.name) + if _is_archive_root(member, relative): + continue + _add_parent_directories(nodes, relative) + _add_archive_member(nodes, tar=tar, member=member, relative=relative) + if len(nodes) > entry_limit: + raise SDLParseError("OCI bundle exceeds the bounded extracted-tree entry limit") + entries = [nodes[name] for name in sorted(nodes, key=lambda name: PurePosixPath(name).parts)] + manifest = { + "content_digest": content_digest, + "entries": entries, + "root_file": root_file, + "schema": _CACHE_TREE_SCHEMA, + "tree_digest": f"sha256:{hashlib.sha256(_canonical_json_bytes(entries)).hexdigest()}", + } + if len(_canonical_json_bytes(manifest)) > _limits().max_metadata_bytes: + raise SDLParseError("OCI module cache integrity manifest exceeds the metadata limit") + return manifest diff --git a/implementations/python/packages/raes/module_registry/_cache.py b/implementations/python/packages/raes/module_registry/_cache.py new file mode 100644 index 000000000..cfd895193 --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_cache.py @@ -0,0 +1,318 @@ +"""Bounded OCI cache admission, locking, inventory, and recovery helpers.""" + +from __future__ import annotations + +import contextlib +import gzip +import io +import os +import stat +import tempfile +import threading +import time +from collections.abc import Iterator +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Any, BinaryIO + +from .._errors import SDLParseError +from ._cache_integrity import _DECOMPRESSION_CHUNK_BYTES, _validated_cache_root +from ._filesystem import ( + _O_BINARY, + _O_DIRECTORY, + _O_NOFOLLOW, + _iter_version_directories, + _prune_version_directories, + _read_version_pointer, + _require_directory, + _same_file_identity, + _version_digest_prefix, + _write_version_pointer, +) + +if TYPE_CHECKING: + from . import _OCIResourceLimits + +_CACHE_THREAD_LOCKS: dict[str, threading.Lock] = {} +_CACHE_THREAD_LOCKS_GUARD = threading.Lock() +_SPOOL_MEMORY_BYTES = 8 * 1024 * 1024 +_WINDOWS_LOCKING = os.name == "nt" +_LOCK_DIR_FD_SUPPORTED = os.open in os.supports_dir_fd + + +def _limits() -> _OCIResourceLimits: + # Keep the historical package-facade test/operator seam authoritative. + from . import _OCI_LIMITS + + return _OCI_LIMITS + + +def _acquire_file_lock(handle: BinaryIO) -> None: + """Acquire an OS-backed exclusive lock within the configured OCI timeout.""" + + deadline = time.monotonic() + _limits().timeout_seconds + while True: + try: + if _WINDOWS_LOCKING: + import msvcrt + + handle.seek(0) + if handle.read(1) == b"": + handle.write(b"\0") + handle.flush() + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + return + except OSError as exc: + if time.monotonic() >= deadline: + raise SDLParseError("Timed out waiting for the OCI module cache lock") from exc + time.sleep(0.01) + + +def _release_file_lock(handle: BinaryIO) -> None: + if _WINDOWS_LOCKING: + import msvcrt + + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + + +def _anchored_lock_parent( + lock_path: Path, + *, + error_message: str, +) -> tuple[int, os.stat_result, str | Path, dict[str, int]]: + parent_expected = lock_path.parent.lstat() + if not stat.S_ISDIR(parent_expected.st_mode): + raise SDLParseError(error_message) + if not _LOCK_DIR_FD_SUPPORTED: + return -1, parent_expected, lock_path, {} + + parent_flags = os.O_RDONLY | _O_DIRECTORY | _O_NOFOLLOW + parent_descriptor = os.open(lock_path.parent, parent_flags) + try: + parent_actual = os.fstat(parent_descriptor) + if not stat.S_ISDIR(parent_actual.st_mode) or not _same_file_identity(parent_expected, parent_actual): + raise SDLParseError(error_message) + except (OSError, SDLParseError): + os.close(parent_descriptor) + raise + return parent_descriptor, parent_actual, lock_path.name, {"dir_fd": parent_descriptor} + + +def _lock_file_stat(lock_path: Path, parent_descriptor: int) -> os.stat_result: + if parent_descriptor >= 0: + return os.stat(lock_path.name, dir_fd=parent_descriptor, follow_symlinks=False) + return lock_path.lstat() + + +def _open_existing_lock_file( + open_path: str | Path, + *, + expected: os.stat_result, + common_flags: int, + open_options: dict[str, int], + error_message: str, +) -> int: + if not stat.S_ISREG(expected.st_mode): + raise SDLParseError(error_message) + return os.open(open_path, common_flags, **open_options) + + +def _open_lock_file( + lock_path: Path, + *, + parent_descriptor: int, + open_path: str | Path, + common_flags: int, + open_options: dict[str, int], + error_message: str, +) -> tuple[int, os.stat_result | None]: + try: + expected = _lock_file_stat(lock_path, parent_descriptor) + except FileNotFoundError: + try: + descriptor = os.open(open_path, common_flags | os.O_CREAT | os.O_EXCL, 0o600, **open_options) + except FileExistsError: + # A concurrent first user may create the shared lock after our + # missing-path check. Re-enter the same no-follow admission path. + expected = _lock_file_stat(lock_path, parent_descriptor) + descriptor = _open_existing_lock_file( + open_path, + expected=expected, + common_flags=common_flags, + open_options=open_options, + error_message=error_message, + ) + else: + expected = None + else: + descriptor = _open_existing_lock_file( + open_path, + expected=expected, + common_flags=common_flags, + open_options=open_options, + error_message=error_message, + ) + return descriptor, expected + + +def _validate_open_lock( + lock_path: Path, + *, + descriptor: int, + expected: os.stat_result | None, + parent_actual: os.stat_result, + error_message: str, +) -> None: + actual = os.fstat(descriptor) + if not stat.S_ISREG(actual.st_mode) or (expected is not None and not _same_file_identity(expected, actual)): + raise SDLParseError(error_message) + parent_after = lock_path.parent.lstat() + if not stat.S_ISDIR(parent_after.st_mode) or not _same_file_identity(parent_actual, parent_after): + raise SDLParseError(error_message) + + +def _open_cache_lock(lock_path: Path) -> BinaryIO: + """Open one regular lock file, anchored to a validated parent directory.""" + + error_message = "Unable to open the OCI module cache lock" + _require_directory(lock_path.parent, error_message=error_message) + common_flags = os.O_RDWR | _O_BINARY | _O_NOFOLLOW + descriptor = -1 + parent_descriptor = -1 + try: + parent_descriptor, parent_actual, open_path, open_options = _anchored_lock_parent( + lock_path, + error_message=error_message, + ) + descriptor, expected = _open_lock_file( + lock_path, + parent_descriptor=parent_descriptor, + open_path=open_path, + common_flags=common_flags, + open_options=open_options, + error_message=error_message, + ) + _validate_open_lock( + lock_path, + descriptor=descriptor, + expected=expected, + parent_actual=parent_actual, + error_message=error_message, + ) + handle = os.fdopen(descriptor, "r+b") + descriptor = -1 + return handle + except SDLParseError: + if descriptor >= 0: + os.close(descriptor) + raise + except OSError as exc: + if descriptor >= 0: + os.close(descriptor) + raise SDLParseError(error_message) from exc + finally: + if parent_descriptor >= 0: + os.close(parent_descriptor) + + +@contextlib.contextmanager +def _cache_entry_lock(lock_path: Path) -> Iterator[None]: + from . import _acquire_file_lock, _open_cache_lock, _release_file_lock + + lock_key = str(lock_path.absolute()) + with _CACHE_THREAD_LOCKS_GUARD: + thread_lock = _CACHE_THREAD_LOCKS.setdefault(lock_key, threading.Lock()) + with thread_lock: + handle = _open_cache_lock(lock_path) + acquired = False + try: + _acquire_file_lock(handle) + acquired = True + yield + finally: + if acquired: + _release_file_lock(handle) + handle.close() + + +@contextlib.contextmanager +def _bounded_gzip_tar_stream(bundle_bytes: bytes) -> Iterator[BinaryIO]: + """Decode a complete gzip stream behind absolute and ratio bounds.""" + + limits = _limits() + absolute_limit = limits.max_tar_stream_bytes + ratio_limit = len(bundle_bytes) * limits.max_gzip_expansion_ratio + if absolute_limit < 0 or limits.max_gzip_expansion_ratio < 0: + raise SDLParseError("OCI gzip expansion limits must be non-negative") + total = 0 + with ( + tempfile.SpooledTemporaryFile(max_size=_SPOOL_MEMORY_BYTES, mode="w+b") as stream, + gzip.GzipFile(fileobj=io.BytesIO(bundle_bytes), mode="rb") as decoded, + ): + while True: + remaining = min(absolute_limit - total, ratio_limit - total) + read_size = min(_DECOMPRESSION_CHUNK_BYTES, max(1, remaining + 1)) + chunk = decoded.read(read_size) + if not chunk: + break + total += len(chunk) + if total > absolute_limit: + raise SDLParseError(f"OCI bundle uncompressed tar stream exceeds the {absolute_limit}-byte limit") + if total > ratio_limit: + raise SDLParseError("OCI bundle gzip expansion exceeds the configured compressed-to-uncompressed ratio") + stream.write(chunk) + stream.seek(0) + yield stream + + +def _recover_cache_root( + *, + slot: Path, + versions: Path, + expected_content_digest: str, + expected_manifest: dict[str, Any], + root_relative: PurePosixPath, +) -> Path | None: + """Find a valid current/orphan version and repair its pointer when needed.""" + + current = _read_version_pointer(slot=slot) + candidates: list[Path] = [] + if current is not None: + candidates.append(versions / current) + digest_prefix = _version_digest_prefix(expected_content_digest) + for version in _iter_version_directories( + versions, + error_message="Unable to inspect OCI module cache versions", + ): + if version.name.startswith(digest_prefix) and version not in candidates: + candidates.append(version) + for version in candidates[:64]: + root = _validated_cache_root( + version=version, + expected_manifest=expected_manifest, + root_relative=root_relative, + ) + if root is None: + continue + _prune_version_directories( + versions=versions, + retain_names={version.name, *(() if current is None else (current,))}, + error_message="Unable to prune stale OCI module cache versions", + ) + if current != version.name: + _write_version_pointer( + slot=slot, + version_name=version.name, + error_message="Unable to commit the OCI module cache pointer atomically", + ) + return root + return None diff --git a/implementations/python/packages/raes/module_registry/_cache_integrity.py b/implementations/python/packages/raes/module_registry/_cache_integrity.py new file mode 100644 index 000000000..22e7573f1 --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_cache_integrity.py @@ -0,0 +1,341 @@ +"""Bounded integrity manifests and tree admission for OCI cache versions.""" + +from __future__ import annotations + +import hashlib +import json +import os +import stat +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Any + +from .._errors import SDLParseError +from ._filesystem import _O_BINARY, _O_NOFOLLOW, _same_file_identity + +if TYPE_CHECKING: + from . import _OCIResourceLimits + +_CACHE_TREE_MANIFEST_NAME = ".raes-cache-tree.json" +_CACHE_TREE_SCHEMA = "raes-cache-tree/v1" +_CACHE_INTEGRITY_ERROR = "OCI module cache tree failed integrity validation" +_CACHE_MANIFEST_FIELDS = frozenset({"content_digest", "entries", "root_file", "schema", "tree_digest"}) +_DECOMPRESSION_CHUNK_BYTES = 1024 * 1024 + + +def _representable_directory_mode() -> int: + """Return the safe directory mode representable by the current platform.""" + + return {"nt": 0o777}.get(os.name, 0o700) + + +def _limits() -> _OCIResourceLimits: + # Keep the historical package-facade test/operator seam authoritative. + from . import _OCI_LIMITS + + return _OCI_LIMITS + + +def _hash_cache_file(path: Path, expected: os.stat_result) -> tuple[int, str]: + """Hash one regular file without following a last-component symlink.""" + + if expected.st_size > _limits().max_member_bytes: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + flags = os.O_RDONLY | _O_BINARY | _O_NOFOLLOW + try: + descriptor = os.open(path, flags) + except OSError as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + digest = hashlib.sha256() + total = 0 + try: + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + actual = os.fstat(handle.fileno()) + if ( + not stat.S_ISREG(actual.st_mode) + or not _same_file_identity(expected, actual) + or actual.st_size != expected.st_size + or stat.S_IMODE(actual.st_mode) != stat.S_IMODE(expected.st_mode) + ): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + while chunk := handle.read(_DECOMPRESSION_CHUNK_BYTES): + total += len(chunk) + if total > _limits().max_member_bytes: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + digest.update(chunk) + except OSError as exc: + if descriptor >= 0: + os.close(descriptor) + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + if total != expected.st_size: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + return total, f"sha256:{digest.hexdigest()}" + + +def _cache_lstat(path: Path) -> os.stat_result: + try: + return path.lstat() + except OSError as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + + +def _append_bounded_child(children: list[Path], child_path: str, *, max_children: int) -> None: + if len(children) >= max_children: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + children.append(Path(child_path)) + + +def _validated_directory_children( + path: Path, + *, + before: os.stat_result, + mode: int, + max_children: int, + excluded_names: frozenset[str], +) -> list[Path]: + try: + children: list[Path] = [] + with os.scandir(path) as iterator: + for child in iterator: + if child.name in excluded_names: + continue + _append_bounded_child(children, child.path, max_children=max_children) + children.sort(key=lambda child: child.name) + after = path.lstat() + except OSError as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + if not stat.S_ISDIR(after.st_mode) or not _same_file_identity(before, after) or stat.S_IMODE(after.st_mode) != mode: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + return children + + +def _cache_tree_node( + path: Path, + relative: PurePosixPath, + *, + max_children: int, +) -> tuple[dict[str, Any], list[Path], int]: + before = _cache_lstat(path) + mode = stat.S_IMODE(before.st_mode) + relative_name = relative.as_posix() + if stat.S_ISLNK(before.st_mode): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + if stat.S_ISREG(before.st_mode): + size, digest = _hash_cache_file(path, before) + entry = {"digest": digest, "mode": mode, "path": relative_name, "size": size, "type": "file"} + return entry, [], size + if not stat.S_ISDIR(before.st_mode): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + children = _validated_directory_children( + path, + before=before, + mode=mode, + max_children=max_children, + excluded_names=frozenset({_CACHE_TREE_MANIFEST_NAME}) if relative == PurePosixPath(".") else frozenset(), + ) + return {"mode": mode, "path": relative_name, "type": "directory"}, children, 0 + + +def _child_relative_path(parent: PurePosixPath, child: Path) -> PurePosixPath: + if parent == PurePosixPath("."): + return PurePosixPath(child.name) + return parent / child.name + + +def _cache_tree_entries(root: Path) -> list[dict[str, Any]]: + """Return a canonical, bounded inventory of a cache version's extracted tree.""" + + limits = _limits() + if limits.max_bundle_members < 0 or limits.max_tree_depth < 0: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + entries: list[dict[str, Any]] = [] + total_bytes = 0 + pending: list[tuple[Path, PurePosixPath, int]] = [(root, PurePosixPath("."), 0)] + entry_limit = limits.max_bundle_members + 1 + while pending: + path, relative, depth = pending.pop() + if depth > limits.max_tree_depth: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + entry, children, size = _cache_tree_node( + path, + relative, + max_children=entry_limit - len(entries) - len(pending) - 1, + ) + total_bytes += size + if total_bytes > limits.max_total_bytes: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + entries.append(entry) + for child in reversed(children): + pending.append((child, _child_relative_path(relative, child), depth + 1)) + return entries + + +def _canonical_json_bytes(value: object) -> bytes: + return (json.dumps(value, sort_keys=True, separators=(",", ":")) + "\n").encode("utf-8") + + +def _cache_tree_manifest(*, root: Path, content_digest: str, root_file: str) -> dict[str, Any]: + entries = _cache_tree_entries(root) + return { + "content_digest": content_digest, + "entries": entries, + "root_file": root_file, + "schema": _CACHE_TREE_SCHEMA, + "tree_digest": f"sha256:{hashlib.sha256(_canonical_json_bytes(entries)).hexdigest()}", + } + + +def _write_cache_tree_manifest(*, root: Path, content_digest: str, root_file: str) -> None: + manifest_path = root / _CACHE_TREE_MANIFEST_NAME + if manifest_path.exists() or manifest_path.is_symlink(): + raise SDLParseError("OCI module bundle contains a reserved cache metadata path") + manifest_bytes = _canonical_json_bytes( + _cache_tree_manifest(root=root, content_digest=content_digest, root_file=root_file) + ) + if len(manifest_bytes) > _limits().max_metadata_bytes: + raise SDLParseError("OCI module cache integrity manifest exceeds the metadata limit") + try: + manifest_path.write_bytes(manifest_bytes) + except OSError as exc: + raise SDLParseError("Unable to write the OCI module cache integrity manifest") from exc + + +def _read_cache_manifest_bytes(path: Path) -> bytes: + """Read a small regular manifest without following a last-component link.""" + + try: + expected = path.lstat() + except OSError as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + if not stat.S_ISREG(expected.st_mode): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + flags = os.O_RDONLY | _O_BINARY | _O_NOFOLLOW + descriptor = -1 + try: + descriptor = os.open(path, flags) + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + actual = os.fstat(handle.fileno()) + if not stat.S_ISREG(actual.st_mode) or not _same_file_identity(expected, actual): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + payload = handle.read(_limits().max_metadata_bytes + 1) + except OSError as exc: + if descriptor >= 0: + os.close(descriptor) + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + if len(payload) > _limits().max_metadata_bytes: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + return payload + + +def _trusted_entry_projection(entries: list[Any]) -> list[dict[str, Any]]: + """Project a local inventory onto properties determined by bundle bytes.""" + + projected: list[dict[str, Any]] = [] + for entry in entries: + if not isinstance(entry, dict) or not isinstance(entry.get("path"), str): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + if entry.get("type") == "directory" and isinstance(entry.get("mode"), int): + projected.append({"mode": entry["mode"], "path": entry["path"], "type": "directory"}) + elif entry.get("type") == "file" and all( + isinstance(entry.get(field), field_type) + for field, field_type in (("digest", str), ("mode", int), ("size", int)) + ): + projected.append( + { + "digest": entry["digest"], + "mode": entry["mode"], + "path": entry["path"], + "size": entry["size"], + "type": "file", + } + ) + else: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + return projected + + +def _require_cache_integrity(condition: bool) -> None: + if not condition: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + + +def _require_cache_manifest(value: object) -> dict[str, Any]: + _require_cache_integrity(isinstance(value, dict)) + manifest: dict[str, Any] = value + _require_cache_integrity(set(manifest) == _CACHE_MANIFEST_FIELDS) + _require_cache_integrity(manifest["schema"] == _CACHE_TREE_SCHEMA) + _require_cache_integrity(isinstance(manifest["entries"], list)) + return manifest + + +def _expected_manifest_projection(expected_manifest: object) -> tuple[dict[str, Any], list[dict[str, Any]]]: + manifest = _require_cache_manifest(expected_manifest) + entries = manifest["entries"] + expected_tree_digest = f"sha256:{hashlib.sha256(_canonical_json_bytes(entries)).hexdigest()}" + _require_cache_integrity(manifest["tree_digest"] == expected_tree_digest) + return manifest, _trusted_entry_projection(entries) + + +def _read_trusted_cache_manifest(version: Path) -> dict[str, Any]: + raw = _read_cache_manifest_bytes(version / _CACHE_TREE_MANIFEST_NAME) + try: + decoded = json.loads(raw.decode("utf-8")) + manifest = _require_cache_manifest(decoded) + canonical = _canonical_json_bytes(manifest) + except (RecursionError, ValueError) as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + _require_cache_integrity(raw == canonical) + return manifest + + +def _require_matching_manifest_identity( + manifest: dict[str, Any], + expected_manifest: dict[str, Any], +) -> None: + identity_fields = ("content_digest", "root_file", "schema") + _require_cache_integrity(all(manifest[field] == expected_manifest[field] for field in identity_fields)) + + +def _require_matching_cache_inventory( + *, + version: Path, + manifest: dict[str, Any], + expected_entries: list[dict[str, Any]], +) -> None: + entries = _cache_tree_entries(version) + _require_cache_integrity(manifest["entries"] == entries) + tree_digest = f"sha256:{hashlib.sha256(_canonical_json_bytes(entries)).hexdigest()}" + _require_cache_integrity(manifest["tree_digest"] == tree_digest) + _require_cache_integrity(_trusted_entry_projection(entries) == expected_entries) + + +def _validated_root_path(version: Path, root_relative: PurePosixPath) -> Path: + root_path = version.joinpath(*root_relative.parts) + _require_cache_integrity(not root_path.is_symlink()) + _require_cache_integrity(root_path.is_file()) + _require_cache_integrity(root_path.resolve(strict=True).is_relative_to(version.resolve(strict=True))) + return root_path + + +def _validated_cache_root( + *, + version: Path, + expected_manifest: dict[str, Any], + root_relative: PurePosixPath, +) -> Path | None: + """Return the root only when it matches inventory derived from verified bytes.""" + + try: + trusted_expected, expected_entries = _expected_manifest_projection(expected_manifest) + manifest = _read_trusted_cache_manifest(version) + _require_matching_manifest_identity(manifest, trusted_expected) + _require_matching_cache_inventory( + version=version, + manifest=manifest, + expected_entries=expected_entries, + ) + root_path = _validated_root_path(version, root_relative) + except (OSError, UnicodeError, json.JSONDecodeError, SDLParseError): + return None + return root_path diff --git a/implementations/python/packages/raes/module_registry/_extraction.py b/implementations/python/packages/raes/module_registry/_extraction.py new file mode 100644 index 000000000..8bd6e74db --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_extraction.py @@ -0,0 +1,209 @@ +"""Resource bounds and tar-member admission for OCI module extraction.""" + +from __future__ import annotations + +import os +import stat +import tarfile +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Any, Protocol, cast + +from .._errors import SDLParseError +from ._cache_integrity import _CACHE_TREE_MANIFEST_NAME +from ._filesystem import _same_file_identity + +_HTTP_TIMEOUT_SECONDS = 30 + + +class _DataFilterTarFile(Protocol): + """Python 3.11.4+ tar extraction surface with the security filter backport.""" + + def extractall(self, path: Path, members: list[tarfile.TarInfo], *, filter: str) -> None: ... + + +@dataclass(frozen=True) +class _OCIResourceLimits: + """Bounds for remote OCI fetches and bundle extraction (issue #12). + + The OCI import path pulls attacker-influenceable bytes from allowlisted + registries; without caps a compromised registry, mirror, or oversized module + can exhaust process memory (buffering an unbounded response) or disk/CPU + (extracting an unbounded bundle). Compressed-download limits are kept separate + from extracted-archive limits because a small gzip can expand into a large tar + payload. This is the single extensibility seam: operator-tunable overrides + should later extend ``RegistryTrustPolicy`` and merge with these defaults, + rather than threading limit arguments through parser/compiler/runtime/CLI. + """ + + timeout_seconds: int = _HTTP_TIMEOUT_SECONDS + max_metadata_bytes: int = 8 * 1024 * 1024 + max_bundle_bytes: int = 128 * 1024 * 1024 + max_bundle_members: int = 8192 + max_member_bytes: int = 64 * 1024 * 1024 + max_total_bytes: int = 256 * 1024 * 1024 + max_tar_stream_bytes: int = 320 * 1024 * 1024 + max_gzip_expansion_ratio: int = 1024 + max_tree_depth: int = 256 + + +def _invalid_member_name(member: tarfile.TarInfo) -> bool: + pure_name = PurePosixPath(member.name) + source_name = member.name.removesuffix("/") if member.isdir() else member.name + windows_absolute = bool(pure_name.parts) and len(pure_name.parts[0]) == 2 and pure_name.parts[0][1] == ":" + return any( + ( + not member.name, + "\x00" in member.name, + "\\" in member.name, + pure_name.is_absolute(), + ".." in pure_name.parts, + source_name != pure_name.as_posix(), + windows_absolute, + ) + ) + + +def _validated_member_path( + member: tarfile.TarInfo, + *, + dest: Path, + resolved_dest: Path, + limits: _OCIResourceLimits, +) -> Path: + member_path = (dest / member.name).resolve() + if not member_path.is_relative_to(resolved_dest): + raise SDLParseError(f"Path traversal detected in OCI bundle tar member: {member.name!r}") + if len(member_path.relative_to(resolved_dest).parts) > limits.max_tree_depth: + raise SDLParseError( + f"OCI bundle member {member.name!r} exceeds the {limits.max_tree_depth}-component path-depth limit" + ) + if member_path == (dest / _CACHE_TREE_MANIFEST_NAME).resolve(): + raise SDLParseError("OCI module bundle contains a reserved cache metadata path") + return member_path + + +def _require_supported_member_type(member: tarfile.TarInfo) -> None: + if member.issym() or member.islnk(): + raise SDLParseError(f"Links are not allowed in OCI bundle tar: {member.name!r}") + if not (member.isfile() or member.isdir()): + raise SDLParseError(f"Unsupported tar member type in OCI bundle: {member.name!r}") + + +def _record_unique_member(member: tarfile.TarInfo, member_path: Path, seen_paths: set[str]) -> None: + normalized = member_path.as_posix() + if normalized in seen_paths: + raise SDLParseError(f"Duplicate tar member path in OCI bundle: {member.name!r}") + seen_paths.add(normalized) + + +def _require_bounded_member_size(member: tarfile.TarInfo, limits: _OCIResourceLimits) -> None: + if member.isfile() and member.size > limits.max_member_bytes: + raise SDLParseError( + f"OCI bundle member {member.name!r} exceeds the {limits.max_member_bytes}-byte per-member limit" + ) + + +def _normalize_extracted_directory_modes(root: Path, entries: list[dict[str, Any]]) -> None: + """Apply and verify bundle-derived safe modes on every extracted directory.""" + + for entry in entries: + if entry["type"] != "directory": + continue + relative = PurePosixPath(entry["path"]) + path = root if relative == PurePosixPath(".") else root.joinpath(*relative.parts) + try: + before = path.lstat() + if not stat.S_ISDIR(before.st_mode): + raise SDLParseError("OCI cache directory mode normalization found a non-directory entry") + if os.name != "nt": + os.chmod(path, entry["mode"], follow_symlinks=False) + after = path.lstat() + except (NotImplementedError, OSError, TypeError) as exc: + raise SDLParseError("Unable to normalize OCI cache directory permissions") from exc + if ( + not stat.S_ISDIR(after.st_mode) + or not _same_file_identity(before, after) + or stat.S_IMODE(after.st_mode) != entry["mode"] + ): + raise SDLParseError("OCI cache directory permissions changed during normalization") + + +def _extract_tar_to_stage( + tar: tarfile.TarFile, + *, + members: list[tarfile.TarInfo], + staging: Path, + expected_entries: list[dict[str, Any]], +) -> None: + try: + cast(_DataFilterTarFile, tar).extractall(staging, members=members, filter="data") + except TypeError as exc: + raise SDLParseError("Safe OCI tar extraction requires Python 3.11.4 or newer") from exc + _normalize_extracted_directory_modes(staging, expected_entries) + + +def _validate_tar_member_shape( + member: tarfile.TarInfo, + *, + dest: Path, + resolved_dest: Path, + seen_paths: set[str], + limits: _OCIResourceLimits, +) -> None: + """Fail closed on an unsafe or oversized single tar member (issues #12/#13). + + Rejects path traversal, symlinks, hard links, special files, and duplicate + normalized paths, and enforces the per-member extracted-size cap. Records the + member's normalized path in ``seen_paths`` so a later duplicate is caught. + """ + + if _invalid_member_name(member): + raise SDLParseError(f"Path traversal detected in OCI bundle tar member: {member.name!r}") + member_path = _validated_member_path(member, dest=dest, resolved_dest=resolved_dest, limits=limits) + _require_supported_member_type(member) + _record_unique_member(member, member_path, seen_paths) + _require_bounded_member_size(member, limits) + + +def _safe_tar_members_with_limits( + tar: tarfile.TarFile, + dest: Path, + *, + limits: _OCIResourceLimits, +) -> list[tarfile.TarInfo]: + """Validate every member before extraction under explicit resource limits.""" + + safe: list[tarfile.TarInfo] = [] + resolved_dest = dest.resolve() + seen_paths: set[str] = set() + total_bytes = 0 + # Iterate lazily so an unbounded member list or extraction is rejected as + # soon as a cap is crossed, before the rest of the archive is decompressed. + for member_count, member in enumerate(tar, start=1): + if member_count > limits.max_bundle_members: + raise SDLParseError(f"OCI bundle exceeds the maximum of {limits.max_bundle_members} archive members") + _validate_tar_member_shape( + member, + dest=dest, + resolved_dest=resolved_dest, + seen_paths=seen_paths, + limits=limits, + ) + if member.isfile(): + total_bytes += member.size + if total_bytes > limits.max_total_bytes: + raise SDLParseError(f"OCI bundle exceeds the {limits.max_total_bytes}-byte total extraction limit") + member.mode &= 0o777 + safe.append(member) + return safe + + +__all__ = [ + "_HTTP_TIMEOUT_SECONDS", + "_OCIResourceLimits", + "_extract_tar_to_stage", + "_normalize_extracted_directory_modes", + "_safe_tar_members_with_limits", + "_validate_tar_member_shape", +] diff --git a/implementations/python/packages/raes/module_registry/_filesystem.py b/implementations/python/packages/raes/module_registry/_filesystem.py new file mode 100644 index 000000000..23b24ff9f --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_filesystem.py @@ -0,0 +1,354 @@ +"""Gapless filesystem publication helpers for OCI layouts and cache trees.""" + +from __future__ import annotations + +import contextlib +import errno +import os +import re +import shutil +import stat +import tempfile +from collections.abc import Iterator +from pathlib import Path +from uuid import uuid4 + +from .._errors import SDLParseError + +_CURRENT_POINTER_NAME = ".raes-current" +_POINTER_STAGE_PREFIX = f"{_CURRENT_POINTER_NAME}.staged-" +_VERSION_STAGE_PREFIX = ".staged-" +_VERSIONS_DIRECTORY_NAME = "versions" +_MAX_POINTER_BYTES = 256 +_MAX_RETAINED_VERSIONS = 8 +_VERSION_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,199}\Z") +_VERSION_DIGEST_PREFIX_HEX = 20 +_VERSION_NONCE_HEX = 20 +_DIRECTORY_FSYNC_SUPPORTED = os.name != "nt" +_O_BINARY = getattr(os, "O_BINARY", 0) +_O_CLOEXEC = getattr(os, "O_CLOEXEC", 0) +_O_DIRECTORY = getattr(os, "O_DIRECTORY", 0) +_O_NOFOLLOW = getattr(os, "O_NOFOLLOW", 0) +# Windows implements os.fsync with MSVCRT _commit, which rejects a read-only +# descriptor. Transaction stages are writable; POSIX retains its read-only open. +_REGULAR_FILE_FSYNC_ACCESS_MODE = os.O_RDWR if os.name == "nt" else os.O_RDONLY + + +def _version_digest_prefix(digest: str) -> str: + """Return the bounded content prefix used to find immutable versions.""" + + return digest.removeprefix("sha256:")[:_VERSION_DIGEST_PREFIX_HEX] + + +def _new_digest_version_name(digest: str) -> str: + """Return a version lookup discriminator with bounded path growth. + + The full digest remains authoritative in each domain's validated contents; + this prefix and nonce only name an immutable filesystem version. + """ + + nonce = uuid4().hex[:_VERSION_NONCE_HEX] + return f"{_version_digest_prefix(digest)}-{nonce}" + + +def _remove_path(path: Path) -> None: + """Remove one explicitly resolved transaction path without following links.""" + + if path.is_symlink() or (path.exists() and not path.is_dir()): + path.unlink() + elif path.exists(): + shutil.rmtree(path) + + +def _fsync_directory(path: Path, *, error_message: str) -> None: + """Persist directory-entry changes where the host exposes directory fsync.""" + + if not _DIRECTORY_FSYNC_SUPPORTED: + return + flags = os.O_RDONLY | _O_DIRECTORY | _O_CLOEXEC + descriptor = -1 + try: + descriptor = os.open(path, flags) + os.fsync(descriptor) + except OSError as exc: + unsupported = { + errno.EINVAL, + getattr(errno, "ENOTSUP", errno.EINVAL), + getattr(errno, "EOPNOTSUPP", errno.EINVAL), + } + if exc.errno in unsupported: + return + raise SDLParseError(error_message) from exc + finally: + if descriptor >= 0: + os.close(descriptor) + + +def _same_file_identity(left: os.stat_result, right: os.stat_result) -> bool: + if left.st_dev and right.st_dev and left.st_dev != right.st_dev: + return False + return not (left.st_ino and right.st_ino and left.st_ino != right.st_ino) + + +def _fsync_tree(root: Path, *, error_message: str) -> None: + """Persist a validated regular-file/directory tree before publishing it.""" + + pending = [root] + directories: list[Path] = [] + try: + while pending: + path = pending.pop() + expected = path.lstat() + if stat.S_ISLNK(expected.st_mode): + raise SDLParseError(error_message) + if stat.S_ISDIR(expected.st_mode): + directories.append(path) + with os.scandir(path) as iterator: + pending.extend(Path(entry.path) for entry in iterator) + continue + if not stat.S_ISREG(expected.st_mode): + raise SDLParseError(error_message) + flags = _REGULAR_FILE_FSYNC_ACCESS_MODE | _O_BINARY | _O_NOFOLLOW + descriptor = os.open(path, flags) + try: + actual = os.fstat(descriptor) + if not stat.S_ISREG(actual.st_mode) or not _same_file_identity(expected, actual): + raise SDLParseError(error_message) + os.fsync(descriptor) + finally: + os.close(descriptor) + for directory in sorted(directories, key=lambda item: len(item.parts), reverse=True): + _fsync_directory(directory, error_message=error_message) + except SDLParseError: + raise + except OSError as exc: + raise SDLParseError(error_message) from exc + + +def _require_directory(path: Path, *, error_message: str) -> None: + """Create one directory while rejecting a file or symlink at its name.""" + + if path.is_symlink() or (path.exists() and not path.is_dir()): + raise SDLParseError(error_message) + existed = path.exists() + try: + path.mkdir(parents=False, exist_ok=True) + except OSError as exc: + raise SDLParseError(error_message) from exc + # Recheck after mkdir so a concurrent replacement cannot turn the slot into a + # link between the preflight and use. Domain writers serialize access to a + # logical slot, while this check also fails closed on external interference. + if path.is_symlink() or not path.is_dir(): + raise SDLParseError(error_message) + if not existed: + _fsync_directory(path.parent, error_message=error_message) + + +def _prepare_versioned_slot(*, slot: Path, error_message: str) -> Path: + """Return a safe versions directory and remove crash-only temporary names. + + Complete versions are never removed here: one may be held by a reader or may + be the only recoverable version after a process died between installing it + and replacing the pointer. Only names that can never be published are + cleaned. + """ + + _require_directory(slot, error_message=error_message) + versions = slot / _VERSIONS_DIRECTORY_NAME + _require_directory(versions, error_message=error_message) + try: + for child in versions.iterdir(): + if child.name.startswith(_VERSION_STAGE_PREFIX): + _remove_path(child) + for child in slot.iterdir(): + if child.name.startswith(_POINTER_STAGE_PREFIX): + _remove_path(child) + except OSError as exc: + raise SDLParseError(error_message) from exc + return versions + + +def _valid_version_name(name: str) -> bool: + return bool(_VERSION_NAME.fullmatch(name)) and name not in {".", ".."} + + +def _iter_version_directories( + versions: Path, + *, + error_message: str = "Unable to inspect immutable version directories", +) -> Iterator[Path]: + """Yield immutable versions in a deterministic order, never following links.""" + + try: + children = sorted(versions.iterdir(), key=lambda path: path.name) + except OSError as exc: + raise SDLParseError(error_message) from exc + for child in children: + if _valid_version_name(child.name) and not child.is_symlink() and child.is_dir(): + yield child + + +def _prune_version_directories( + *, + versions: Path, + retain_names: set[str], + error_message: str, + max_versions: int = _MAX_RETAINED_VERSIONS, +) -> None: + """Bound retained immutable versions while preserving current/prior readers. + + Domain writers call this under the logical-slot lock after publishing. The + selected version and the pointer value observed before publication are + explicit retains; the newest remaining versions fill the bounded support + window. Content is never modified in place. + """ + + if max_versions < 2 or len(retain_names) > max_versions: + raise SDLParseError(error_message) + try: + ranked = sorted( + _iter_version_directories(versions, error_message=error_message), + key=lambda version: (version.stat().st_mtime_ns, version.name), + reverse=True, + ) + keep = {name for name in retain_names if _valid_version_name(name)} + for version in ranked: + if len(keep) >= max_versions: + break + keep.add(version.name) + for version in ranked: + if version.name not in keep: + _remove_path(version) + _fsync_directory(versions, error_message=error_message) + except OSError as exc: + raise SDLParseError(error_message) from exc + + +def _read_version_pointer(*, slot: Path) -> str | None: + """Read the current-version pointer, returning ``None`` when it is invalid.""" + + pointer = slot / _CURRENT_POINTER_NAME + payload = _read_version_pointer_payload(pointer) + name = _decode_version_pointer(payload) + if name is None: + return None + version = slot / _VERSIONS_DIRECTORY_NAME / name + return name if not version.is_symlink() and version.is_dir() else None + + +def _read_version_pointer_payload(pointer: Path) -> bytes | None: + """Read a bounded pointer payload through a stable regular-file descriptor.""" + + descriptor = -1 + payload: bytes | None = None + try: + expected = pointer.lstat() + if stat.S_ISREG(expected.st_mode): + flags = os.O_RDONLY | _O_BINARY | _O_NOFOLLOW + descriptor = os.open(pointer, flags) + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + actual = os.fstat(handle.fileno()) + if stat.S_ISREG(actual.st_mode) and _same_file_identity(expected, actual): + payload = handle.read(_MAX_POINTER_BYTES + 1) + except OSError: + pass + finally: + if descriptor >= 0: + os.close(descriptor) + return payload + + +def _decode_version_pointer(payload: bytes | None) -> str | None: + """Decode the exact canonical pointer representation, if valid.""" + + name: str | None = None + if payload is not None and len(payload) <= _MAX_POINTER_BYTES: + try: + candidate = payload.decode("ascii").removesuffix("\n") + except UnicodeDecodeError: + pass + else: + if payload == f"{candidate}\n".encode("ascii") and _valid_version_name(candidate): + name = candidate + return name + + +def _install_version_directory( + *, + staged: Path, + versions: Path, + version_name: str, + error_message: str, +) -> Path: + """Install a complete new immutable version with one directory rename.""" + + if not _valid_version_name(version_name): + _remove_path(staged) + raise SDLParseError(error_message) + target = versions / version_name + if target.exists() or target.is_symlink(): + _remove_path(staged) + raise SDLParseError(error_message) + try: + _fsync_tree(staged, error_message=error_message) + os.replace(staged, target) + _fsync_directory(versions, error_message=error_message) + except SDLParseError: + _remove_path(staged) + raise + except OSError as exc: + _remove_path(staged) + raise SDLParseError(error_message) from exc + return target + + +def _write_version_pointer( + *, + slot: Path, + version_name: str, + error_message: str, +) -> None: + """Publish a version by atomically replacing one small pointer file. + + The old pointer is never removed first. A crash before ``os.replace`` leaves + it untouched; a crash after ``os.replace`` leaves the complete new version + selected. The temporary pointer is safe startup residue and is removed by + :func:`_prepare_versioned_slot`. + """ + + if not _valid_version_name(version_name): + raise SDLParseError(error_message) + version = slot / _VERSIONS_DIRECTORY_NAME / version_name + if version.is_symlink() or not version.is_dir(): + raise SDLParseError(error_message) + pointer = slot / _CURRENT_POINTER_NAME + descriptor = -1 + temporary: Path | None = None + try: + descriptor, raw_path = tempfile.mkstemp(prefix=_POINTER_STAGE_PREFIX, dir=slot) + temporary = Path(raw_path) + payload = f"{version_name}\n".encode("ascii") + with os.fdopen(descriptor, "wb") as handle: + descriptor = -1 + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + os.replace(temporary, pointer) + _fsync_directory(slot, error_message=error_message) + except OSError as exc: + if descriptor >= 0: + os.close(descriptor) + if temporary is not None: + with contextlib.suppress(OSError): + _remove_path(temporary) + raise SDLParseError(error_message) from exc + + +def _new_version_stage(*, versions: Path, error_message: str) -> Path: + """Create a private sibling stage inside a version store.""" + + try: + return Path(tempfile.mkdtemp(prefix=_VERSION_STAGE_PREFIX, dir=versions)) + except OSError as exc: + raise SDLParseError(error_message) from exc diff --git a/implementations/python/packages/raes/module_registry/_verified_sources.py b/implementations/python/packages/raes/module_registry/_verified_sources.py new file mode 100644 index 000000000..65b1ab2d9 --- /dev/null +++ b/implementations/python/packages/raes/module_registry/_verified_sources.py @@ -0,0 +1,283 @@ +"""Descriptor-bound immutable SDL sources for verified OCI cache trees.""" + +from __future__ import annotations + +import hashlib +import os +import stat +from collections.abc import Mapping +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from types import MappingProxyType +from typing import TYPE_CHECKING, Any, BinaryIO + +from pydantic import ValidationError + +from .._errors import SDLParseError +from .._source_profile import SDLSourceParseOptions +from ..scenario import ImportDecl +from ._filesystem import _O_BINARY, _O_NOFOLLOW, _same_file_identity + +if TYPE_CHECKING: + from ..parser import SDLSourceDocument + +_CACHE_INTEGRITY_ERROR = "OCI module cache tree failed integrity validation" + + +@dataclass(frozen=True) +class _VerifiedSourceBundle: + """One immutable in-memory view of a verified OCI SDL import graph.""" + + cache_root: Path + documents: Mapping[str, SDLSourceDocument] + + def resolve_local(self, *, base_dir: Path, relative: str) -> tuple[Path, SDLSourceDocument]: + base_relative = _cache_relative_path(self.cache_root, base_dir) + target = _normalize_bundle_path(base_relative, relative) + document = self.documents.get(target) + if document is None: + raise SDLParseError(f"Imported SDL file not found: {relative}") + return self.cache_root.joinpath(*PurePosixPath(target).parts), document + + def identity_path(self, path: Path) -> Path: + """Return a lexical cache identity without consulting mutable paths.""" + + relative = _cache_relative_path(self.cache_root, path) + return self.cache_root.joinpath(*PurePosixPath(relative).parts) + + +def _cache_relative_path(cache_root: Path, path: Path) -> str: + root = cache_root.absolute() + candidate = path.absolute() + try: + relative = candidate.relative_to(root) + except ValueError as exc: + raise SDLParseError("Local import path escapes the verified OCI module cache") from exc + return relative.as_posix() or "." + + +def _local_resolved_source(import_path: Path, base_dir: Path, *, lexical: bool = False) -> str: + """Return one portable lock identity, without resolving admitted cache paths. + + Local lock identities are checkout-independent POSIX paths. Ordinary local + files retain canonical filesystem resolution, while a descriptor-bound OCI + source uses only its already-confined lexical identity. + """ + + anchor = base_dir.absolute() if lexical else base_dir.resolve() + return Path(os.path.relpath(import_path, anchor)).as_posix() + + +def _normalize_bundle_path(base_relative: str, relative: str) -> str: + normalized = relative.replace("\\", "/") + candidate = PurePosixPath(normalized) + invalid_reference = ( + not normalized + or "\x00" in normalized + or candidate.is_absolute() + or (candidate.parts and len(candidate.parts[0]) == 2 and candidate.parts[0][1] == ":") + ) + if invalid_reference: + raise SDLParseError(f"Local import path escapes base directory: {relative!r}") + parts = [] if base_relative == "." else list(PurePosixPath(base_relative).parts) + _apply_bundle_path_parts(parts, candidate.parts, relative=relative) + if not parts: + raise SDLParseError(f"Local import path escapes base directory: {relative!r}") + return PurePosixPath(*parts).as_posix() + + +def _apply_bundle_path_parts(parts: list[str], candidate_parts: tuple[str, ...], *, relative: str) -> None: + """Apply normalized path components without permitting an escape.""" + + for part in candidate_parts: + if part == "..": + if not parts: + raise SDLParseError(f"Local import path escapes base directory: {relative!r}") + parts.pop() + else: + parts.append(part) + + +def _cache_file_fingerprint(metadata: os.stat_result) -> tuple[bool, int, int]: + return stat.S_ISREG(metadata.st_mode), metadata.st_size, stat.S_IMODE(metadata.st_mode) + + +def _require_expected_cache_file( + metadata: os.stat_result, + *, + expected_size: int, + expected_mode: int, + identity: os.stat_result | None = None, +) -> None: + expected_fingerprint = (True, expected_size, expected_mode) + if _cache_file_fingerprint(metadata) != expected_fingerprint: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + if identity is not None and not _same_file_identity(identity, metadata): + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + + +def _capture_source_bytes(source: BinaryIO, *, capture_limit: int) -> tuple[bytes, int, str]: + """Hash an entire admitted source while retaining only the parser limit.""" + + digest = hashlib.sha256() + captured = bytearray() + byte_count = 0 + while chunk := source.read(1024 * 1024): + byte_count += len(chunk) + digest.update(chunk) + remaining = max(capture_limit - len(captured), 0) + captured.extend(chunk[:remaining]) + return bytes(captured), byte_count, digest.hexdigest() + + +def _read_verified_cache_source( + path: Path, + *, + expected_entry: dict[str, Any], + source_options: SDLSourceParseOptions, +) -> SDLSourceDocument: + """Read and authenticate exact source bytes through one no-follow descriptor.""" + + expected_size = expected_entry["size"] + expected_mode = expected_entry["mode"] + descriptor = -1 + try: + before = path.lstat() + _require_expected_cache_file(before, expected_size=expected_size, expected_mode=expected_mode) + flags = os.O_RDONLY | _O_BINARY | _O_NOFOLLOW + descriptor = os.open(path, flags) + with os.fdopen(descriptor, "rb") as source: + descriptor = -1 + opened = os.fstat(source.fileno()) + _require_expected_cache_file( + opened, + expected_size=expected_size, + expected_mode=expected_mode, + identity=before, + ) + captured, byte_count, digest = _capture_source_bytes( + source, + capture_limit=source_options.limits.max_input_bytes + 1, + ) + after = os.fstat(source.fileno()) + _require_expected_cache_file( + after, + expected_size=expected_size, + expected_mode=expected_mode, + identity=opened, + ) + except SDLParseError: + raise + except OSError as exc: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) from exc + finally: + if descriptor >= 0: + os.close(descriptor) + + if byte_count != expected_size or f"sha256:{digest}" != expected_entry["digest"]: + raise SDLParseError(_CACHE_INTEGRITY_ERROR) + + from ..parser import _source_document_from_bytes + + return _source_document_from_bytes(captured, path=path, limits=source_options.limits) + + +def _local_import_targets( + document: SDLSourceDocument, + *, + path: Path, + relative: str, + source_options: SDLSourceParseOptions, +) -> list[str]: + """Return normalized local targets declared by one verified source.""" + + from ..parser import _load_normalized_data + + payload = _load_normalized_data( + document.text, + path=path, + source_format=source_options.source_format, + migration_policy=source_options.migration_policy, + limits=source_options.limits, + ) + raw_imports = payload.get("imports", []) + if not isinstance(raw_imports, list): + raise SDLParseError("Imported SDL unit is structurally invalid", path=path) + parent = PurePosixPath(relative).parent.as_posix() + targets: list[str] = [] + for raw_import in raw_imports: + try: + import_decl = ImportDecl.model_validate(raw_import) + except ValidationError as exc: + raise SDLParseError("Imported SDL unit is structurally invalid", path=path) from exc + source = import_decl.normalized_source + if source.startswith("local:"): + targets.append(_normalize_bundle_path(parent, source.removeprefix("local:"))) + return targets + + +def _read_verified_source_bundle( + *, + cache_root: Path, + expected_manifest: dict[str, Any], + root_relative: PurePosixPath, + source_options: SDLSourceParseOptions, +) -> _VerifiedSourceBundle: + """Capture the complete reachable local SDL graph before releasing its lock.""" + + entries = { + entry["path"]: entry + for entry in expected_manifest["entries"] + if isinstance(entry, dict) and entry.get("type") == "file" + } + pending = [root_relative.as_posix()] + documents: dict[str, SDLSourceDocument] = {} + while pending: + relative = pending.pop() + if relative in documents: + continue + entry = entries.get(relative) + if entry is None: + raise SDLParseError(f"Imported SDL file not found: {relative}") + path = cache_root.joinpath(*PurePosixPath(relative).parts) + document = _read_verified_cache_source( + path, + expected_entry=entry, + source_options=source_options, + ) + documents[relative] = document + pending.extend( + _local_import_targets( + document, + path=path, + relative=relative, + source_options=source_options, + ) + ) + return _VerifiedSourceBundle( + cache_root=cache_root.absolute(), + documents=MappingProxyType(dict(sorted(documents.items()))), + ) + + +def _cache_source_result( + root: Path, + *, + expected_manifest: dict[str, Any], + root_relative: PurePosixPath, + source_options: SDLSourceParseOptions | None, +) -> Path | _VerifiedSourceBundle: + if source_options is None: + return root + cache_root = root + for _part in root_relative.parts: + cache_root = cache_root.parent + return _read_verified_source_bundle( + cache_root=cache_root, + expected_manifest=expected_manifest, + root_relative=root_relative, + source_options=source_options, + ) + + +__all__ = ["_VerifiedSourceBundle"] diff --git a/implementations/python/packages/raes/module_registry/models.py b/implementations/python/packages/raes/module_registry/models.py index 7138c9a1b..0bf4ca489 100644 --- a/implementations/python/packages/raes/module_registry/models.py +++ b/implementations/python/packages/raes/module_registry/models.py @@ -22,6 +22,7 @@ if TYPE_CHECKING: from ..parser import SDLSourceDocument + from ._verified_sources import _VerifiedSourceBundle class RegistryTrustPolicy(SDLModel): @@ -65,6 +66,7 @@ class ResolvedModule: content_digest: str = "" export_hash: str = "" signer_id: str = "" + verified_sources: _VerifiedSourceBundle | None = None def _scenario_module_descriptor(scenario: Scenario, *, source_id: str) -> ModuleDescriptor: diff --git a/implementations/python/packages/raes/module_registry/publishing.py b/implementations/python/packages/raes/module_registry/publishing.py index 9777af72f..a5f62e342 100644 --- a/implementations/python/packages/raes/module_registry/publishing.py +++ b/implementations/python/packages/raes/module_registry/publishing.py @@ -3,12 +3,17 @@ from __future__ import annotations import base64 +import gzip import io import json +import stat import tarfile +from collections.abc import Mapping +from dataclasses import dataclass from pathlib import Path from typing import Any +from cryptography.exceptions import UnsupportedAlgorithm from cryptography.hazmat.primitives import serialization from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey @@ -21,6 +26,18 @@ OCI_LAYOUT_SCHEMA_VERSION, ) from ._digests import _SHA256_PREFIX, _descriptor_digest +from ._filesystem import ( + _install_version_directory, + _iter_version_directories, + _new_digest_version_name, + _new_version_stage, + _prepare_versioned_slot, + _prune_version_directories, + _read_version_pointer, + _remove_path, + _version_digest_prefix, + _write_version_pointer, +) from .models import _scenario_module_descriptor # ``_sha256_digest`` / ``_signable_payload`` are resolved through the package @@ -32,17 +49,26 @@ def _collect_local_bundle_files( root_path: Path, *, + bundle_root: Path | None = None, seen: set[Path] | None = None, ) -> dict[Path, bytes]: from ..parser import _load_normalized_data, read_sdl_source seen = set() if seen is None else set(seen) - resolved = root_path.resolve() + try: + resolved = root_path.resolve(strict=True) + except OSError as exc: + raise SDLParseError("Unable to resolve the module publishing entrypoint") from exc + if not resolved.is_file(): + raise SDLParseError("The module publishing entrypoint must be a regular file") + bundle_root = resolved.parent if bundle_root is None else bundle_root + if not resolved.is_relative_to(bundle_root): + raise SDLParseError("Local import path escapes the canonical publishing root") if resolved in seen: raise SDLParseError(f"Import cycle detected at {resolved}") seen.add(resolved) - root_source = read_sdl_source(root_path) - payload = _load_normalized_data(root_source.text, path=root_path) + root_source = read_sdl_source(resolved) + payload = _load_normalized_data(root_source.text, path=resolved) files = {resolved: root_source.raw_bytes} for raw_import in payload.get("imports", []): import_decl = ImportDecl.model_validate(raw_import) @@ -53,20 +79,34 @@ def _collect_local_bundle_files( "publish a self-contained local module graph" ) child_path = (resolved.parent / source.removeprefix("local:")).resolve() - if not child_path.is_relative_to(resolved.parent): + if not child_path.is_relative_to(bundle_root): raise SDLParseError(f"Local import path escapes base directory: {source!r}") - files.update(_collect_local_bundle_files(child_path, seen=seen)) + files.update(_collect_local_bundle_files(child_path, bundle_root=bundle_root, seen=seen)) return files def _build_module_bundle(root_path: Path) -> bytes: - files = _collect_local_bundle_files(root_path) - relative_files = {path.relative_to(root_path.parent).as_posix(): content for path, content in files.items()} + canonical_root = root_path.resolve(strict=True) + bundle_root = canonical_root.parent + files = _collect_local_bundle_files(canonical_root, bundle_root=bundle_root) + relative_files = {path.relative_to(bundle_root).as_posix(): content for path, content in files.items()} bundle_buffer = io.BytesIO() - with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: + # Issue #1096 / GOV-913: both the tar headers and the gzip envelope are + # normalized. Identical canonical module bytes therefore produce identical + # content digests regardless of wall clock, checkout ownership, or file mode. + with ( + gzip.GzipFile(filename="", mode="wb", compresslevel=9, fileobj=bundle_buffer, mtime=0) as compressed, + tarfile.open(fileobj=compressed, mode="w", format=tarfile.PAX_FORMAT) as tar, + ): for relative_name, content in sorted(relative_files.items()): info = tarfile.TarInfo(name=relative_name) info.size = len(content) + info.mode = 0o644 + info.uid = 0 + info.gid = 0 + info.uname = "" + info.gname = "" + info.mtime = 0 tar.addfile(info, io.BytesIO(content)) return bundle_buffer.getvalue() @@ -81,12 +121,23 @@ def _build_signatures( ) -> list[dict[str, str]]: from . import _signable_payload - if not (signer_id and private_key_path is not None): + signer_configured = signer_id != "" + key_configured = private_key_path is not None + if signer_configured != key_configured: + raise SDLParseError("Publishing signing requires both signer_id and private_key_path") + if not signer_configured: return [] - private_key = serialization.load_pem_private_key( - private_key_path.read_bytes(), - password=None, - ) + if signer_id != signer_id.strip(): + raise SDLParseError("Publishing signer_id must not contain leading or trailing whitespace") + assert private_key_path is not None + try: + private_key_bytes = private_key_path.read_bytes() + except OSError as exc: + raise SDLParseError("Unable to read the publishing private key") from exc + try: + private_key = serialization.load_pem_private_key(private_key_bytes, password=None) + except (TypeError, ValueError, UnsupportedAlgorithm) as exc: + raise SDLParseError("Publishing private key is not a valid PEM private key") from exc if not isinstance(private_key, Ed25519PrivateKey): raise SDLParseError("Publishing key must be an Ed25519 private key") signature = private_key.sign(_signable_payload(descriptor, content_digest=content_digest, root_file=root_file)) @@ -98,22 +149,26 @@ def _build_signatures( ] -def _write_oci_layout( +@dataclass(frozen=True) +class _LayoutPublication: + """Immutable inputs shared by layout validation and publication.""" + + slot: Path + expected_files: Mapping[str, bytes] + manifest_digest: str + + +class _InvalidLayoutError(ValueError): + """Internal marker for a filesystem object that is not an OCI layout.""" + + +def _build_layout_index( *, - output_dir: Path, descriptor: ModuleDescriptor, blobs: dict[str, bytes], manifest_digest: str, -) -> Path: - # ``blobs`` maps each blob's ``sha256:`` digest to its bytes (config, bundle, - # manifest); ``manifest_digest`` selects which one the index references. - layout_dir = output_dir / f"{descriptor.id.replace('/', '_')}-{descriptor.version}.oci" - blobs_dir = layout_dir / "blobs" / "sha256" - blobs_dir.mkdir(parents=True, exist_ok=True) - (layout_dir / "oci-layout").write_text('{"imageLayoutVersion":"1.0.0"}\n', encoding="utf-8") - for digest, payload in blobs.items(): - (blobs_dir / digest.removeprefix(_SHA256_PREFIX)).write_bytes(payload) - (layout_dir / "index.json").write_text( +) -> bytes: + return ( json.dumps( { "schemaVersion": 2, @@ -132,10 +187,217 @@ def _write_oci_layout( indent=2, sort_keys=True, ) - + "\n", - encoding="utf-8", + + "\n" + ).encode("utf-8") + + +def _layout_expected_files( + *, + descriptor: ModuleDescriptor, + blobs: dict[str, bytes], + manifest_digest: str, +) -> dict[str, bytes]: + # ``blobs`` maps each blob's ``sha256:`` digest to its bytes (config, bundle, + # manifest); ``manifest_digest`` selects which one the index references. + return { + "oci-layout": b'{"imageLayoutVersion":"1.0.0"}\n', + "index.json": _build_layout_index( + descriptor=descriptor, + blobs=blobs, + manifest_digest=manifest_digest, + ), + **{f"blobs/sha256/{digest.removeprefix(_SHA256_PREFIX)}": payload for digest, payload in blobs.items()}, + } + + +def _layout_inventory(version: Path) -> tuple[set[str], set[str]]: + actual_files: set[str] = set() + actual_directories: set[str] = {"."} + for path in version.rglob("*"): + relative = path.relative_to(version).as_posix() + metadata = path.lstat() + if stat.S_ISLNK(metadata.st_mode): + raise _InvalidLayoutError + if stat.S_ISDIR(metadata.st_mode): + actual_directories.add(relative) + elif stat.S_ISREG(metadata.st_mode): + actual_files.add(relative) + else: + raise _InvalidLayoutError + return actual_files, actual_directories + + +def _layout_file_bytes_match(version: Path, expected_files: Mapping[str, bytes]) -> bool: + matches = True + for relative, expected in expected_files.items(): + path = version.joinpath(*relative.split("/")) + if path.is_symlink() or path.stat().st_size != len(expected) or path.read_bytes() != expected: + matches = False + break + return matches + + +def _valid_oci_layout(version: Path, expected_files: Mapping[str, bytes]) -> bool: + try: + actual_files, actual_directories = _layout_inventory(version) + expected_directories = {".", "blobs", "blobs/sha256"} + valid_inventory = actual_directories == expected_directories and actual_files == set(expected_files) + return valid_inventory and _layout_file_bytes_match(version, expected_files) + except (OSError, _InvalidLayoutError): + return False + + +def _layout_candidates( + publication: _LayoutPublication, + *, + versions: Path, + current: str | None, +) -> list[Path]: + candidates = [] if current is None else [versions / current] + digest_prefix = _version_digest_prefix(publication.manifest_digest) + for version in _iter_version_directories( + versions, + error_message="Unable to inspect OCI layout versions", + ): + if version.name.startswith(digest_prefix) and version not in candidates: + candidates.append(version) + return candidates[:64] + + +def _recover_oci_layout( + publication: _LayoutPublication, + *, + versions: Path, + current: str | None, +) -> Path | None: + for version in _layout_candidates(publication, versions=versions, current=current): + if not _valid_oci_layout(version, publication.expected_files): + continue + _prune_version_directories( + versions=versions, + retain_names={version.name, *(() if current is None else (current,))}, + error_message="Unable to prune stale OCI layout versions", + ) + if current != version.name: + _write_version_pointer( + slot=publication.slot, + version_name=version.name, + error_message="Unable to publish the OCI layout pointer atomically", + ) + return version + return None + + +def _reject_legacy_layout(layout_slot: Path) -> None: + if layout_slot.is_symlink() or not layout_slot.is_dir(): + return + legacy_paths = (layout_slot / "oci-layout", layout_slot / "index.json", layout_slot / "blobs") + if any(child.exists() or child.is_symlink() for child in legacy_paths): + raise SDLParseError( + "Existing OCI output uses the legacy root-layout format; move or remove " + f"'{layout_slot}' before publishing into its versioned layout slot" + ) + + +def _populate_layout_stage(staging: Path, expected_files: Mapping[str, bytes]) -> None: + (staging / "blobs" / "sha256").mkdir(parents=True) + for relative, payload in sorted(expected_files.items()): + staging.joinpath(*relative.split("/")).write_bytes(payload) + if not _valid_oci_layout(staging, expected_files): + raise SDLParseError("Staged OCI layout failed validation") + + +def _commit_layout_stage( + publication: _LayoutPublication, + *, + staging: Path, + versions: Path, + prior_version: str | None, +) -> Path: + version_name = _new_digest_version_name(publication.manifest_digest) + installed = _install_version_directory( + staged=staging, + versions=versions, + version_name=version_name, + error_message="Unable to publish the OCI layout atomically", + ) + if not _valid_oci_layout(installed, publication.expected_files): + raise SDLParseError("Published OCI layout failed validation") + _prune_version_directories( + versions=versions, + retain_names={version_name, *(() if prior_version is None else (prior_version,))}, + error_message="Unable to prune stale OCI layout versions", + ) + _write_version_pointer( + slot=publication.slot, + version_name=version_name, + error_message="Unable to publish the OCI layout pointer atomically", + ) + return installed + + +def _publish_new_layout( + publication: _LayoutPublication, + *, + versions: Path, + prior_version: str | None, +) -> Path: + staging = _new_version_stage( + versions=versions, + error_message="Unable to stage the OCI layout", + ) + try: + _populate_layout_stage(staging, publication.expected_files) + return _commit_layout_stage( + publication, + staging=staging, + versions=versions, + prior_version=prior_version, + ) + finally: + _remove_path(staging) + + +def _write_oci_layout( + *, + output_dir: Path, + descriptor: ModuleDescriptor, + blobs: dict[str, bytes], + manifest_digest: str, +) -> Path: + layout_slot = output_dir / f"{descriptor.id.replace('/', '_')}-{descriptor.version}.oci" + publication = _LayoutPublication( + slot=layout_slot, + expected_files=_layout_expected_files( + descriptor=descriptor, + blobs=blobs, + manifest_digest=manifest_digest, + ), + manifest_digest=manifest_digest, ) - return layout_dir + + try: + output_dir.mkdir(parents=True, exist_ok=True) + lock_path = output_dir / ".raes-layout-locks" / f"{layout_slot.name}.lock" + from . import _cache_entry_lock + + with _cache_entry_lock(lock_path): + _reject_legacy_layout(layout_slot) + versions = _prepare_versioned_slot( + slot=layout_slot, + error_message="Unable to publish the OCI layout atomically", + ) + prior_version = _read_version_pointer(slot=layout_slot) + recovered = _recover_oci_layout(publication, versions=versions, current=prior_version) + if recovered is not None: + return recovered + return _publish_new_layout( + publication, + versions=versions, + prior_version=prior_version, + ) + except OSError as exc: + raise SDLParseError("Unable to build the OCI layout") from exc def publish_module_to_oci_layout( @@ -148,23 +410,27 @@ def publish_module_to_oci_layout( from ..parser import parse_sdl_file from . import _sha256_digest - scenario = parse_sdl_file(root_path, skip_semantic_validation=True) + try: + canonical_root = root_path.resolve(strict=True) + except OSError as exc: + raise SDLParseError("Unable to resolve the module publishing entrypoint") from exc + scenario = parse_sdl_file(canonical_root, skip_semantic_validation=True) descriptor = _scenario_module_descriptor( scenario, - source_id=str(root_path.name), + source_id=canonical_root.name, ) - bundle_bytes = _build_module_bundle(root_path) + bundle_bytes = _build_module_bundle(canonical_root) content_digest = f"{_SHA256_PREFIX}{_sha256_digest(bundle_bytes)}" signatures = _build_signatures( descriptor, content_digest=content_digest, - root_file=root_path.name, + root_file=canonical_root.name, signer_id=signer_id, private_key_path=private_key_path, ) config_payload = { "schema_version": OCI_LAYOUT_SCHEMA_VERSION, - "root_file": root_path.name, + "root_file": canonical_root.name, "module": descriptor.model_dump(mode="python", by_alias=True), "signatures": signatures, } diff --git a/implementations/python/packages/raes/module_registry/resolution.py b/implementations/python/packages/raes/module_registry/resolution.py index 57289cf15..6472d6f8f 100644 --- a/implementations/python/packages/raes/module_registry/resolution.py +++ b/implementations/python/packages/raes/module_registry/resolution.py @@ -2,10 +2,9 @@ from __future__ import annotations -import json -import os +from dataclasses import dataclass from pathlib import Path -from typing import Any +from typing import Any, TypedDict, Unpack from urllib.parse import quote from packaging.version import InvalidVersion, Version @@ -16,6 +15,7 @@ from ..scenario import ImportDecl, ModuleDescriptor, Scenario from ._constants import LOCKFILE_NAME, OCI_BUNDLE_MEDIA_TYPE, OCI_LAYOUT_MEDIA_TYPE from ._digests import _SHA256_PREFIX, _descriptor_digest, _normalize_exact_or_range, _satisfies_version +from ._verified_sources import _local_resolved_source, _VerifiedSourceBundle from .models import ( Lockfile, LockRecord, @@ -33,6 +33,26 @@ # production calls use, exactly as the pre-split single-file module did. +@dataclass(frozen=True) +class _ResolutionContext: + """Invariant options shared by locked, local, and OCI resolution.""" + + base_dir: Path + lockfile: Lockfile | None + trust_policy: TrustPolicy + source_options: SDLSourceParseOptions + source_diagnostics: list[SDLParseDiagnostic] | None + verified_sources: _VerifiedSourceBundle | None + registry_base_dir: Path | None + + +class _ResolutionPrivateOptions(TypedDict, total=False): + """Compatibility names for descriptor-bound internal resolution state.""" + + verified_sources: _VerifiedSourceBundle | None + _registry_base_dir: Path | None + + def _parse_oci_source(source: str) -> tuple[str, str]: ref = source.removeprefix("oci:") if "://" in ref: @@ -116,22 +136,6 @@ def _validate_digest_pin(actual_digest: str, expected_digest: str, *, source: st raise SDLParseError(f"Digest mismatch for import '{source}': {expected_digest!r} != {actual_digest!r}") -def _local_resolved_source(import_path: Path, base_dir: Path) -> str: - """Persisted lock identity for a ``local:`` import (issue #551). - - The lockfile is committed and verified across machines and CI, so a local - import's ``resolved_source`` must be a checkout-independent identity rather - than an absolute, machine-specific path. Express it relative to the SDL base - directory using POSIX separators so the same lockfile verifies on any - checkout. ``ResolvedModule.root_file`` remains the absolute runtime ``Path`` - used for reads, digesting, parsing, and cycle detection; this is the single - normalization seam for persisted local lock identity (OCI imports keep their - registry/digest identity). - """ - relative = os.path.relpath(import_path, base_dir.resolve()) - return Path(relative).as_posix() - - def resolve_import( import_decl: ImportDecl, *, @@ -140,38 +144,35 @@ def resolve_import( trust_policy: TrustPolicy | None = None, source_options: SDLSourceParseOptions = DEFAULT_SOURCE_PARSE_OPTIONS, source_diagnostics: list[SDLParseDiagnostic] | None = None, + **private_options: Unpack[_ResolutionPrivateOptions], ) -> ResolvedModule: - trust_policy = trust_policy or TrustPolicy() + context = _ResolutionContext( + base_dir=base_dir, + lockfile=lockfile, + trust_policy=trust_policy or TrustPolicy(), + source_options=source_options, + source_diagnostics=source_diagnostics, + verified_sources=private_options.get("verified_sources"), + registry_base_dir=private_options.get("_registry_base_dir"), + ) + return _resolve_import_with_context(import_decl, context) + + +def _resolve_import_with_context(import_decl: ImportDecl, context: _ResolutionContext) -> ResolvedModule: source = import_decl.normalized_source if source.startswith("locked:"): - return _resolve_locked_import( - import_decl, - source, - base_dir=base_dir, - lockfile=lockfile, - trust_policy=trust_policy, - source_options=source_options, - source_diagnostics=source_diagnostics, - ) + return _resolve_locked_import(import_decl, source, context=context) if source.startswith("local:"): - return _resolve_local_import( - import_decl, - source, - base_dir=base_dir, - lockfile=lockfile, - trust_policy=trust_policy, - source_options=source_options, - source_diagnostics=source_diagnostics, - ) + return _resolve_local_import(import_decl, source, context=context) if not source.startswith("oci:"): raise SDLParseError(f"Unsupported import source '{source}'") return _resolve_oci_import( import_decl, source, - base_dir=base_dir, - lockfile=lockfile, - trust_policy=trust_policy, - source_options=source_options, + base_dir=context.base_dir if context.registry_base_dir is None else context.registry_base_dir, + lockfile=context.lockfile, + trust_policy=context.trust_policy, + source_options=context.source_options, ) @@ -179,13 +180,10 @@ def _resolve_locked_import( import_decl: ImportDecl, source: str, *, - base_dir: Path, - lockfile: Lockfile | None, - trust_policy: TrustPolicy, - source_options: SDLSourceParseOptions, - source_diagnostics: list[SDLParseDiagnostic] | None, + context: _ResolutionContext, ) -> ResolvedModule: locked_ref = source.removeprefix("locked:") + lockfile = context.lockfile if lockfile is None: raise SDLParseError(f"Locked import '{source}' requires {LOCKFILE_NAME}") record = next( @@ -205,43 +203,38 @@ def _resolve_locked_import( parameters=dict(import_decl.parameters), digest=import_decl.digest or record.content_digest, ) - return resolve_import( - delegated, - base_dir=base_dir, - lockfile=lockfile, - trust_policy=trust_policy, - source_options=source_options, - source_diagnostics=source_diagnostics, - ) + return _resolve_import_with_context(delegated, context) def _resolve_local_import( import_decl: ImportDecl, source: str, *, - base_dir: Path, - lockfile: Lockfile | None, - trust_policy: TrustPolicy, - source_options: SDLSourceParseOptions, - source_diagnostics: list[SDLParseDiagnostic] | None, + context: _ResolutionContext, ) -> ResolvedModule: from ..parser import _load_normalized_data, read_sdl_source from . import _sha256_digest relative = source.removeprefix("local:") - import_path = (base_dir / relative).resolve() - if not import_path.is_relative_to(base_dir.resolve()): - raise SDLParseError(f"Local import path escapes base directory: {relative!r}") - if not import_path.exists(): - raise SDLParseError(f"Imported SDL file not found: {relative}") - imported_source = read_sdl_source(import_path, limits=source_options.limits) + if context.verified_sources is None: + import_path = (context.base_dir / relative).resolve() + if not import_path.is_relative_to(context.base_dir.resolve()): + raise SDLParseError(f"Local import path escapes base directory: {relative!r}") + if not import_path.exists(): + raise SDLParseError(f"Imported SDL file not found: {relative}") + imported_source = read_sdl_source(import_path, limits=context.source_options.limits) + else: + import_path, imported_source = context.verified_sources.resolve_local( + base_dir=context.base_dir, + relative=relative, + ) imported_raw = _load_normalized_data( imported_source.text, path=import_path, - source_format=source_options.source_format, - migration_policy=source_options.migration_policy, - limits=source_options.limits, - source_diagnostics=source_diagnostics, + source_format=context.source_options.source_format, + migration_policy=context.source_options.migration_policy, + limits=context.source_options.limits, + source_diagnostics=context.source_diagnostics, ) imported_scenario = Scenario.model_validate(imported_raw) descriptor = _scenario_module_descriptor( @@ -253,12 +246,12 @@ def _resolve_local_import( raise SDLParseError( f"Import '{relative}' requested version {import_decl.version!r} but module declares {descriptor.version!r}" ) - if not trust_policy.allow_unsigned_local_sources: + if context.verified_sources is None and not context.trust_policy.allow_unsigned_local_sources: raise SDLParseError( "Local SDL imports are disabled by trust policy because unsigned local sources are not allowed" ) _validate_digest_pin(content_digest, import_decl.digest, source=source) - locked = _lock_record_for(lockfile, import_decl) + locked = _lock_record_for(context.lockfile, import_decl) if locked is not None and locked.content_digest: _validate_digest_pin(content_digest, locked.content_digest, source=source) _verify_allowed_parameters(import_decl, descriptor) @@ -267,9 +260,14 @@ def _resolve_local_import( module_descriptor=descriptor, root_file=import_path, source_document=imported_source, - resolved_source=_local_resolved_source(import_path, base_dir), + resolved_source=_local_resolved_source( + import_path, + context.base_dir, + lexical=context.verified_sources is not None, + ), content_digest=content_digest, export_hash=_descriptor_digest(descriptor.exports), + verified_sources=context.verified_sources, ) @@ -281,19 +279,22 @@ def _resolve_oci_manifest( locked: LockRecord | None, source: str, ) -> tuple[str, dict[str, Any]]: - from . import _bytes_request, _json_request, _sha256_digest + from . import _bytes_request, _decode_json_object, _json_request, _sha256_digest manifest_ref = locked.manifest_digest if locked is not None else None if manifest_ref is None: tags_payload = _json_request(f"{base_url}/v2/{quote(repository, safe='/')}/tags/list") - tags = list(tags_payload.get("tags") or []) + raw_tags = tags_payload.get("tags") or [] + if not isinstance(raw_tags, list) or any(not isinstance(tag, str) for tag in raw_tags): + raise SDLParseError(f"OCI tag metadata for '{source}' has an invalid tags list") + tags = raw_tags manifest_ref = _select_tag(tags, import_decl.version) manifest_bytes = _bytes_request( f"{base_url}/v2/{quote(repository, safe='/')}/manifests/{quote(str(manifest_ref), safe=':@/')}", headers={"Accept": OCI_LAYOUT_MEDIA_TYPE}, ) manifest_digest = f"{_SHA256_PREFIX}{_sha256_digest(manifest_bytes)}" - manifest = json.loads(manifest_bytes.decode("utf-8")) + manifest = _decode_json_object(manifest_bytes, context=f"OCI manifest for '{source}'") if locked is not None and locked.manifest_digest != manifest_digest: raise SDLParseError( f"Lockfile digest mismatch for import '{source}': {locked.manifest_digest!r} != {manifest_digest!r}" @@ -304,11 +305,18 @@ def _resolve_oci_manifest( def _resolve_oci_config( *, base_url: str, repository: str, manifest: dict[str, Any], source: str ) -> tuple[dict[str, Any], str]: - from . import _bytes_request, _sha256_digest + from . import _bytes_request, _decode_json_object, _sha256_digest config = manifest.get("config", {}) + layers = manifest.get("layers", []) + if ( + not isinstance(config, dict) + or not isinstance(layers, list) + or any(not isinstance(item, dict) for item in layers) + ): + raise SDLParseError(f"OCI module '{source}' has malformed config or layer descriptors") layer = next( - (candidate for candidate in manifest.get("layers", []) if candidate.get("mediaType") == OCI_BUNDLE_MEDIA_TYPE), + (candidate for candidate in layers if candidate.get("mediaType") == OCI_BUNDLE_MEDIA_TYPE), None, ) if not config or not layer: @@ -326,7 +334,7 @@ def _resolve_oci_config( ) if f"{_SHA256_PREFIX}{_sha256_digest(config_bytes)}" != config_digest: raise SDLParseError(f"OCI module '{source}' config digest verification failed") - config_payload = json.loads(config_bytes.decode("utf-8")) + config_payload = _decode_json_object(config_bytes, context=f"OCI config for '{source}'") return config_payload, layer_digest @@ -353,7 +361,7 @@ def _build_oci_descriptor( try: descriptor = ModuleDescriptor.model_validate(config_payload.get("module", {})) except ValidationError as exc: - raise SDLParseError(f"OCI module '{source}' has invalid module descriptor: {exc}") from exc + raise SDLParseError(f"OCI module '{source}' has invalid module descriptor") from exc if locked is not None and locked.module_id != descriptor.id: raise SDLParseError( f"Lockfile module id mismatch for import '{source}': {locked.module_id!r} != {descriptor.id!r}" @@ -384,7 +392,6 @@ def _resolve_oci_import( trust_policy: TrustPolicy, source_options: SDLSourceParseOptions, ) -> ResolvedModule: - from ..parser import read_sdl_source from . import _extract_bundle_to_cache, _verify_signatures registry, repository = _parse_oci_source(source) @@ -405,20 +412,30 @@ def _resolve_oci_import( ) signer_id = "" if registry_policy.require_signatures: + raw_signatures = config_payload.get("signatures", []) + if not isinstance(raw_signatures, list) or any(not isinstance(item, dict) for item in raw_signatures): + raise SDLParseError(f"OCI module '{source}' has malformed signature metadata") signer_id = _verify_signatures( - signatures=list(config_payload.get("signatures", [])), + signatures=raw_signatures, trust_policy=registry_policy, module_descriptor=descriptor, content_digest=content_digest, root_file=root_file, ) - resolved_root = _extract_bundle_to_cache( + verified_sources = _extract_bundle_to_cache( bundle_bytes=bundle_bytes, manifest_digest=manifest_digest.replace(_SHA256_PREFIX, ""), + content_digest=content_digest, root_file=root_file, base_dir=base_dir, + source_options=source_options, + ) + if not isinstance(verified_sources, _VerifiedSourceBundle): + raise SDLParseError("OCI module cache did not return verified source documents") + resolved_root, resolved_source_document = verified_sources.resolve_local( + base_dir=verified_sources.cache_root, + relative=root_file, ) - resolved_source_document = read_sdl_source(resolved_root, limits=source_options.limits) _verify_allowed_parameters(import_decl, descriptor) export_hash = _descriptor_digest(descriptor.exports) if locked is not None and locked.export_hash != export_hash: @@ -433,6 +450,7 @@ def _resolve_oci_import( content_digest=content_digest, export_hash=export_hash, signer_id=signer_id, + verified_sources=verified_sources, ) diff --git a/implementations/python/packages/raes/parser.py b/implementations/python/packages/raes/parser.py index 1b52f9661..7410a986a 100644 --- a/implementations/python/packages/raes/parser.py +++ b/implementations/python/packages/raes/parser.py @@ -72,6 +72,17 @@ def read_sdl_source( raise FileNotFoundError(f"SDL file not found: {path}") with path.open("rb") as source: raw_bytes = source.read(limits.max_input_bytes + 1) + return _source_document_from_bytes(raw_bytes, path=path, limits=limits) + + +def _source_document_from_bytes( + raw_bytes: bytes, + *, + path: Path, + limits: SDLParserLimits = DEFAULT_PARSER_LIMITS, +) -> SDLSourceDocument: + """Decode already-captured source bytes with the canonical diagnostics.""" + if len(raw_bytes) > limits.max_input_bytes: _raise_source_limit( f"SDL source exceeds the byte limit of {limits.max_input_bytes} bytes.", diff --git a/implementations/python/packages/raes_cli/sdl.py b/implementations/python/packages/raes_cli/sdl.py index e4be9a443..a4d0b1ee9 100644 --- a/implementations/python/packages/raes_cli/sdl.py +++ b/implementations/python/packages/raes_cli/sdl.py @@ -95,7 +95,7 @@ def publish( output_dir: Path = typer.Option( Path("dist"), "--output-dir", - help="Directory where the OCI layout will be written.", + help="Directory for the versioned OCI layout slot; result layout_dir is the usable immutable layout.", ), signer_id: str = typer.Option("", "--signer-id", help="Signer identity label."), private_key: Path | None = typer.Option( @@ -106,7 +106,7 @@ def publish( help="Optional Ed25519 PEM private key used to sign the module bundle.", ), ) -> None: - """Package an SDL module as an OCI image layout.""" + """Package an SDL module and report its usable immutable OCI layout.""" result = publish_module_to_oci_layout( path, output_dir=output_dir, diff --git a/implementations/python/tests/test_issue_1107_oci_source_snapshot.py b/implementations/python/tests/test_issue_1107_oci_source_snapshot.py new file mode 100644 index 000000000..700f5b521 --- /dev/null +++ b/implementations/python/tests/test_issue_1107_oci_source_snapshot.py @@ -0,0 +1,413 @@ +"""Issue #1107: descriptor-bound OCI source snapshots close cache TOCTOU gaps.""" + +from __future__ import annotations + +import gzip +import hashlib +import io +import os +import stat +import tarfile +from pathlib import Path, PurePosixPath + +import pytest +import raes.module_registry as module_registry +import raes.module_registry._verified_sources as verified_sources +import raes.module_registry.resolution as module_registry_resolution +from raes._errors import SDLParseError +from raes._source_profile import DEFAULT_SOURCE_PARSE_OPTIONS, SDLParserLimits, SDLSourceParseOptions +from raes.composition import expand_sdl_modules + + +def _entry(path: Path, relative: str, *, digest: str | None = None) -> dict[str, object]: + metadata = path.lstat() + payload = path.read_bytes() if stat.S_ISREG(metadata.st_mode) else b"" + return { + "path": relative, + "type": "file", + "size": metadata.st_size, + "mode": stat.S_IMODE(metadata.st_mode), + "digest": digest or f"sha256:{hashlib.sha256(payload).hexdigest()}", + } + + +def _manifest(root: Path, relative_paths: list[str]) -> dict[str, object]: + entries: list[object] = [None, {"path": ".", "type": "directory"}] + entries.extend(_entry(root.joinpath(*PurePosixPath(relative).parts), relative) for relative in relative_paths) + return {"entries": entries} + + +def _bundle(members: list[tuple[str, bytes]]) -> bytes: + compressed = io.BytesIO() + with ( + gzip.GzipFile(filename="", mode="wb", fileobj=compressed, mtime=0) as stream, + tarfile.open(fileobj=stream, mode="w") as archive, + ): + for name, payload in members: + member = tarfile.TarInfo(name=name) + member.mode = 0o644 + member.size = len(payload) + archive.addfile(member, io.BytesIO(payload)) + return compressed.getvalue() + + +def test_verified_snapshot_cache_miss_and_hit_capture_nested_graph( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + root_payload = b"""name: root +module: {id: acme/root, version: 1.0.0} +imports: + - source: local:child.yaml + namespace: child +""" + child_payload = b"name: child\nmodule: {id: acme/child, version: 1.0.0}\n" + bundle = _bundle([("nested/module.yaml", root_payload), ("nested/child.yaml", child_payload)]) + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + arguments = { + "bundle_bytes": bundle, + "manifest_digest": "snapshot-hit", + "content_digest": digest, + "root_file": "nested/module.yaml", + "base_dir": tmp_path, + "source_options": DEFAULT_SOURCE_PARSE_OPTIONS, + } + + missed = module_registry._extract_bundle_to_cache(**arguments) + + assert isinstance(missed, module_registry._VerifiedSourceBundle) + assert list(missed.documents) == ["nested/child.yaml", "nested/module.yaml"] + assert ( + missed.resolve_local(base_dir=missed.cache_root / "nested", relative="child.yaml")[1].raw_bytes == child_payload + ) + + def forbid_staging(*_args, **_kwargs): + pytest.fail("an admitted cache hit must not stage or extract another version") + + monkeypatch.setattr(module_registry, "_new_version_stage", forbid_staging) + monkeypatch.setattr(tarfile.TarFile, "extractall", forbid_staging) + hit = module_registry._extract_bundle_to_cache(**arguments) + + assert isinstance(hit, module_registry._VerifiedSourceBundle) + assert hit.cache_root == missed.cache_root + assert hit.documents == missed.documents + + +def test_verified_bundle_paths_are_lexical_confined_and_immutable(tmp_path: Path) -> None: + root = tmp_path / "cache" + (root / "nested").mkdir(parents=True) + (root / "nested" / "module.yaml").write_text( + """name: root +imports: + - source: local:../common.yaml + namespace: common +""", + encoding="utf-8", + ) + (root / "common.yaml").write_text( + """name: common +imports: + - source: local:nested/module.yaml + namespace: cycle + - source: oci:registry.example/acme/remote + namespace: remote +""", + encoding="utf-8", + ) + snapshot = verified_sources._read_verified_source_bundle( + cache_root=root, + expected_manifest=_manifest(root, ["nested/module.yaml", "common.yaml"]), + root_relative=PurePosixPath("nested/module.yaml"), + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + assert list(snapshot.documents) == ["common.yaml", "nested/module.yaml"] + assert snapshot.identity_path(root / "common.yaml") == root.absolute() / "common.yaml" + assert verified_sources._cache_relative_path(root, root) == "." + assert snapshot.resolve_local(base_dir=root / "nested", relative=r"..\common.yaml")[0] == root / "common.yaml" + with pytest.raises(TypeError): + snapshot.documents["new.yaml"] = snapshot.documents["common.yaml"] # type: ignore[index] + with pytest.raises(SDLParseError, match="not found"): + snapshot.resolve_local(base_dir=root, relative="absent.yaml") + with pytest.raises(SDLParseError, match="escapes the verified"): + snapshot.identity_path(tmp_path / "outside.yaml") + + +@pytest.mark.parametrize( + ("base_relative", "relative"), + [ + (".", ""), + (".", "bad\x00name"), + (".", "/absolute.yaml"), + (".", "C:/windows.yaml"), + (".", "../escape.yaml"), + ("nested", ".."), + ], +) +def test_verified_bundle_path_normalization_rejects_escape( + base_relative: str, + relative: str, +) -> None: + with pytest.raises(SDLParseError, match="escapes base directory"): + verified_sources._normalize_bundle_path(base_relative, relative) + + +@pytest.mark.parametrize("mismatch", ["type", "size", "mode"]) +def test_verified_source_rejects_initial_metadata_mismatch(tmp_path: Path, mismatch: str) -> None: + path = tmp_path / "source.yaml" + path.write_bytes(b"name: source\n") + expected = _entry(path, path.name) + if mismatch == "type": + path.unlink() + path.mkdir() + expected["size"] = path.stat().st_size + expected["mode"] = stat.S_IMODE(path.stat().st_mode) + elif mismatch == "size": + expected["size"] = int(expected["size"]) + 1 + else: + expected["mode"] = int(expected["mode"]) ^ 0o100 + + with pytest.raises(SDLParseError, match="integrity validation"): + verified_sources._read_verified_cache_source( + path, + expected_entry=expected, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + +def _changed_stat(metadata: os.stat_result, mismatch: str) -> os.stat_result: + values = list(metadata) + if mismatch == "type": + values[0] = stat.S_IFDIR | stat.S_IMODE(metadata.st_mode) + elif mismatch == "identity": + values[2] = metadata.st_dev + 1 + elif mismatch == "size": + values[6] = metadata.st_size + 1 + else: + values[0] ^= 0o100 + return os.stat_result(values) + + +@pytest.mark.parametrize("mismatch", ["type", "identity", "size", "mode"]) +def test_verified_source_rejects_opened_descriptor_mismatch( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + mismatch: str, +) -> None: + path = tmp_path / "source.yaml" + path.write_bytes(b"name: source\n") + expected = _entry(path, path.name) + real_fstat = os.fstat + + def changed_fstat(descriptor: int) -> os.stat_result: + return _changed_stat(real_fstat(descriptor), mismatch) + + monkeypatch.setattr(verified_sources.os, "fstat", changed_fstat) + with pytest.raises(SDLParseError, match="integrity validation"): + verified_sources._read_verified_cache_source( + path, + expected_entry=expected, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + +@pytest.mark.parametrize("mismatch", ["identity", "size", "mode"]) +def test_verified_source_rejects_descriptor_change_while_reading( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + mismatch: str, +) -> None: + path = tmp_path / "source.yaml" + path.write_bytes(b"name: source\n") + expected = _entry(path, path.name) + real_fstat = os.fstat + calls = 0 + + def changed_second_fstat(descriptor: int) -> os.stat_result: + nonlocal calls + calls += 1 + metadata = real_fstat(descriptor) + return metadata if calls == 1 else _changed_stat(metadata, mismatch) + + monkeypatch.setattr(verified_sources.os, "fstat", changed_second_fstat) + with pytest.raises(SDLParseError, match="integrity validation"): + verified_sources._read_verified_cache_source( + path, + expected_entry=expected, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + +def test_verified_source_rejects_io_digest_utf8_and_source_limit_failures( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + path = tmp_path / "source.yaml" + path.write_bytes(b"name: source\n") + expected = _entry(path, path.name) + with pytest.raises(SDLParseError, match="integrity validation"): + verified_sources._read_verified_cache_source( + tmp_path / "missing.yaml", + expected_entry=expected, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + bad_digest = dict(expected, digest="sha256:" + "0" * 64) + with pytest.raises(SDLParseError, match="integrity validation"): + verified_sources._read_verified_cache_source( + path, + expected_entry=bad_digest, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + path.write_bytes(b"\xff") + invalid_utf8 = _entry(path, path.name) + with pytest.raises(SDLParseError, match="UTF-8"): + verified_sources._read_verified_cache_source( + path, + expected_entry=invalid_utf8, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + oversized = b"name: source\n" * 100_000 + path.write_bytes(oversized) + oversized_entry = _entry(path, path.name) + limited = SDLSourceParseOptions(limits=SDLParserLimits(max_input_bytes=1)) + with pytest.raises(SDLParseError, match="byte limit"): + verified_sources._read_verified_cache_source( + path, + expected_entry=oversized_entry, + source_options=limited, + ) + + path.write_bytes(b"name: source\n") + expected = _entry(path, path.name) + + def fail_fdopen(*_args, **_kwargs): + raise OSError("sensitive descriptor failure") + + monkeypatch.setattr(verified_sources.os, "fdopen", fail_fdopen) + with pytest.raises(SDLParseError) as exc_info: + verified_sources._read_verified_cache_source( + path, + expected_entry=expected, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + assert str(exc_info.value) == "OCI module cache tree failed integrity validation" + + +@pytest.mark.parametrize( + ("payload", "message"), + [ + (b"name: root\nimports: {}\n", "structurally invalid"), + (b"name: root\nimports: [7]\n", "structurally invalid"), + ( + b"name: root\nimports:\n - source: local:missing.yaml\n namespace: missing\n", + "not found", + ), + ], +) +def test_verified_graph_rejects_invalid_or_missing_imports( + tmp_path: Path, + payload: bytes, + message: str, +) -> None: + root = tmp_path / "module.yaml" + root.write_bytes(payload) + expected_manifest = _manifest(tmp_path, ["module.yaml"]) + root_relative = PurePosixPath("module.yaml") + with pytest.raises(SDLParseError, match=message): + verified_sources._read_verified_source_bundle( + cache_root=tmp_path, + expected_manifest=expected_manifest, + root_relative=root_relative, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + +def test_legacy_cache_result_remains_a_path(tmp_path: Path) -> None: + root = tmp_path / "module.yaml" + assert ( + verified_sources._cache_source_result( + root, + expected_manifest={"entries": []}, + root_relative=PurePosixPath("module.yaml"), + source_options=None, + ) + == root + ) + + +def test_verified_expansion_context_has_a_safe_default_trust_policy(tmp_path: Path) -> None: + root = tmp_path / "module.yaml" + root.write_text("name: root\n", encoding="utf-8") + snapshot = verified_sources._read_verified_source_bundle( + cache_root=tmp_path, + expected_manifest=_manifest(tmp_path, ["module.yaml"]), + root_relative=PurePosixPath("module.yaml"), + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + expanded, _provenance = expand_sdl_modules( + {"name": "root"}, + path=root, + _verified_sources=snapshot, + ) + + assert expanded["name"] == "root" + + +def test_ordinary_local_resolution_preserves_missing_and_unsigned_policy_errors(tmp_path: Path) -> None: + missing = module_registry.ImportDecl(source="local:missing.yaml", namespace="missing") + with pytest.raises(SDLParseError, match="Imported SDL file not found"): + module_registry.resolve_import(missing, base_dir=tmp_path) + + (tmp_path / "module.yaml").write_text( + "name: local\nmodule: {id: acme/local, version: 1.0.0}\n", + encoding="utf-8", + ) + local = module_registry.ImportDecl(source="local:module.yaml", namespace="local") + reject_unsigned = module_registry.TrustPolicy(allow_unsigned_local_sources=False) + with pytest.raises(SDLParseError, match="unsigned local sources are not allowed"): + module_registry.resolve_import( + local, + base_dir=tmp_path, + trust_policy=reject_unsigned, + ) + + +def test_oci_resolution_requires_verified_source_documents( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source = "oci:registry.example/acme/module" + digest = "sha256:" + "0" * 64 + import_decl = module_registry.ImportDecl(source=source, namespace="module") + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_manifest", + lambda **_kwargs: (digest, {}), + ) + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_config", + lambda **_kwargs: ( + {"module": {"id": "acme/module", "version": "1.0.0"}}, + digest, + ), + ) + monkeypatch.setattr(module_registry_resolution, "_fetch_oci_bundle", lambda **_kwargs: b"bundle") + monkeypatch.setattr(module_registry, "_extract_bundle_to_cache", lambda **_kwargs: tmp_path / "module.yaml") + trust_policy = module_registry.TrustPolicy( + registries={"registry.example": module_registry.RegistryTrustPolicy(require_signatures=False)} + ) + + with pytest.raises(SDLParseError, match="did not return verified source documents"): + module_registry_resolution._resolve_oci_import( + import_decl, + source, + base_dir=tmp_path, + lockfile=None, + trust_policy=trust_policy, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) diff --git a/implementations/python/tests/test_sdl_module_registry.py b/implementations/python/tests/test_sdl_module_registry.py index dec3b7c93..67a312bf4 100644 --- a/implementations/python/tests/test_sdl_module_registry.py +++ b/implementations/python/tests/test_sdl_module_registry.py @@ -4,21 +4,36 @@ import base64 import dataclasses +import gzip import io import json +import os import shutil +import stat +import sys import tarfile import textwrap import threading +from concurrent.futures import ThreadPoolExecutor from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from pathlib import Path +from pathlib import Path, PurePosixPath +from urllib.error import URLError import pytest +import raes.composition._expand as composition_expand import raes.module_registry as module_registry +import raes.module_registry._archive as module_registry_archive +import raes.module_registry._cache as module_registry_cache +import raes.module_registry._extraction as module_registry_extraction +import raes.module_registry._filesystem as module_registry_filesystem +import raes.module_registry.publishing as module_registry_publishing +import raes.module_registry.resolution as module_registry_resolution import raes.parser as sdl_parser from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import rsa from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey from raes._errors import SDLParseError, SDLValidationError +from raes._source_profile import DEFAULT_SOURCE_PARSE_OPTIONS from raes.module_registry import ( LOCKFILE_NAME, load_lockfile, @@ -467,6 +482,1057 @@ def test_publishing_local_bundle_rejects_import_escape(tmp_path: Path): publish_module_to_oci_layout(root, output_dir=tmp_path / "dist") +def _published_blob(layout_dir: Path, digest: str) -> bytes: + return (layout_dir / "blobs" / "sha256" / digest.removeprefix("sha256:")).read_bytes() + + +def _version_slot(version: Path) -> Path: + assert version.parent.name == "versions" + return version.parent.parent + + +def _current_version(slot: Path) -> Path: + version_name = (slot / ".raes-current").read_text(encoding="ascii").removesuffix("\n") + return slot / "versions" / version_name + + +def _published_config(layout_dir: Path) -> dict[str, object]: + index = json.loads((layout_dir / "index.json").read_text(encoding="utf-8")) + manifest = json.loads(_published_blob(layout_dir, index["manifests"][0]["digest"])) + return json.loads(_published_blob(layout_dir, manifest["config"]["digest"])) + + +def test_published_bundle_is_byte_reproducible_and_normalized(tmp_path: Path): + module_path = _local_module(tmp_path / "source" / "shared.yaml") + first = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist-a") + os.chmod(module_path, 0o600) + os.utime(module_path, (2_000_000_000, 2_000_000_000)) + second = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist-b") + + assert first["content_digest"] == second["content_digest"] + assert first["manifest_digest"] == second["manifest_digest"] + first_bundle = _published_blob(Path(first["layout_dir"]), str(first["content_digest"])) + second_bundle = _published_blob(Path(second["layout_dir"]), str(second["content_digest"])) + assert first_bundle == second_bundle + assert int.from_bytes(first_bundle[4:8], "little") == 0 # gzip MTIME + with tarfile.open(fileobj=io.BytesIO(first_bundle), mode="r:gz") as archive: + members = archive.getmembers() + assert [member.name for member in members] == ["shared.yaml"] + assert {(member.uid, member.gid, member.uname, member.gname, member.mode, member.mtime) for member in members} == { + (0, 0, "", "", 0o644, 0) + } + + +def test_published_bundle_collects_nested_local_graph_against_one_root(tmp_path: Path): + _write( + tmp_path / "common.yaml", + """ + name: common + module: {id: acme/common, version: 1.0.0} + """, + ) + _write( + tmp_path / "nested" / "child.yaml", + """ + name: child + module: {id: acme/child, version: 1.0.0} + imports: + - source: local:../common.yaml + namespace: common + """, + ) + root = _write( + tmp_path / "root.yaml", + """ + name: root + module: {id: acme/root, version: 1.0.0} + imports: + - source: local:nested/child.yaml + namespace: child + """, + ) + + bundle = module_registry_publishing._build_module_bundle(root) + + with tarfile.open(fileobj=io.BytesIO(bundle), mode="r:gz") as archive: + assert [member.name for member in archive] == ["common.yaml", "nested/child.yaml", "root.yaml"] + + +def test_published_bundle_rejects_remote_import_graph(tmp_path: Path): + root = _write( + tmp_path / "root.yaml", + """ + name: root + module: {id: acme/root, version: 1.0.0} + imports: + - source: oci:registry.example/acme/child + namespace: child + """, + ) + + with pytest.raises(SDLParseError, match="self-contained local module graph"): + module_registry_publishing._build_module_bundle(root) + + +def test_bundle_collector_rejects_invalid_roots_cycles_and_escape(tmp_path: Path): + root = _write(tmp_path / "root.yaml", "name: root\nmodule: {id: acme/root, version: 1.0.0}") + with pytest.raises(SDLParseError, match="Unable to resolve"): + module_registry_publishing._collect_local_bundle_files(tmp_path / "missing.yaml") + with pytest.raises(SDLParseError, match="regular file"): + module_registry_publishing._collect_local_bundle_files(tmp_path) + with pytest.raises(SDLParseError, match="canonical publishing root"): + module_registry_publishing._collect_local_bundle_files(root, bundle_root=tmp_path / "other") + seen = {root.resolve()} + with pytest.raises(SDLParseError, match="Import cycle"): + module_registry_publishing._collect_local_bundle_files(root, seen=seen) + + _write(tmp_path / "outside.yaml", "name: outside\nmodule: {id: acme/outside, version: 1.0.0}") + nested = _write( + tmp_path / "nested" / "root.yaml", + """ + name: nested + module: {id: acme/nested, version: 1.0.0} + imports: + - source: local:../outside.yaml + namespace: outside + """, + ) + bundle_root = nested.parent.resolve() + with pytest.raises(SDLParseError, match="escapes base directory"): + module_registry_publishing._collect_local_bundle_files(nested, bundle_root=bundle_root) + + +def test_relative_and_symlink_publish_roots_have_one_canonical_identity( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +): + module_path = _local_module(tmp_path / "source" / "shared.yaml") + alias = tmp_path / "alias.yaml" + try: + alias.symlink_to(module_path) + except OSError: + pytest.skip("symlink creation is unavailable") + monkeypatch.chdir(tmp_path) + + relative = publish_module_to_oci_layout(Path("source/shared.yaml"), output_dir=tmp_path / "dist-relative") + linked = publish_module_to_oci_layout(Path("alias.yaml"), output_dir=tmp_path / "dist-linked") + + assert relative["content_digest"] == linked["content_digest"] + assert relative["manifest_digest"] == linked["manifest_digest"] + assert _published_config(Path(relative["layout_dir"]))["root_file"] == "shared.yaml" + assert _published_config(Path(linked["layout_dir"]))["root_file"] == "shared.yaml" + + +@pytest.mark.parametrize( + ("signer_id", "private_key"), + [("configured-signer", None), ("", Path("configured-key.pem"))], +) +def test_publish_rejects_partially_configured_signing(tmp_path: Path, signer_id: str, private_key: Path | None): + module_path = _local_module(tmp_path / "shared.yaml") + + with pytest.raises(SDLParseError, match="requires both"): + publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist", + signer_id=signer_id, + private_key_path=private_key, + ) + + +def test_publish_rejects_invalid_key_with_stable_error(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + invalid_key = tmp_path / "invalid.pem" + invalid_key.write_text("not a private key\nSECRET-NATIVE-DETAIL", encoding="utf-8") + + with pytest.raises(SDLParseError) as exc_info: + publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist", + signer_id="configured-signer", + private_key_path=invalid_key, + ) + + assert str(exc_info.value) == "Publishing private key is not a valid PEM private key" + assert "SECRET-NATIVE-DETAIL" not in str(exc_info.value) + + +def test_publish_rejects_whitespace_signer_id(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + key = Ed25519PrivateKey.generate() + key_path = tmp_path / "signing-key.pem" + key_path.write_bytes( + key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.PKCS8, + encryption_algorithm=serialization.NoEncryption(), + ) + ) + + with pytest.raises(SDLParseError, match="leading or trailing whitespace"): + publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist", + signer_id=" signer ", + private_key_path=key_path, + ) + + +def test_publish_with_complete_signing_configuration_emits_signature(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + key = Ed25519PrivateKey.generate() + key_path = tmp_path / "signing-key.pem" + key_path.write_bytes( + key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.PKCS8, + encryption_algorithm=serialization.NoEncryption(), + ) + ) + + published = publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist", + signer_id="signer", + private_key_path=key_path, + ) + + config = _published_config(Path(published["layout_dir"])) + assert config["signatures"][0]["signer_id"] == "signer" + assert config["signatures"][0]["signature"] + + +def test_publish_rejects_unreadable_or_non_ed25519_key(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + with pytest.raises(SDLParseError, match="Unable to read"): + publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist-a", + signer_id="signer", + private_key_path=tmp_path / "missing.pem", + ) + + rsa_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + rsa_path = tmp_path / "rsa.pem" + rsa_path.write_bytes( + rsa_key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.PKCS8, + encryption_algorithm=serialization.NoEncryption(), + ) + ) + with pytest.raises(SDLParseError, match="must be an Ed25519"): + publish_module_to_oci_layout( + module_path, + output_dir=tmp_path / "dist-b", + signer_id="signer", + private_key_path=rsa_path, + ) + + +def test_publish_missing_entrypoint_and_unwritable_layout_fail_stably(tmp_path: Path): + with pytest.raises(SDLParseError, match="Unable to resolve"): + publish_module_to_oci_layout(tmp_path / "missing.yaml", output_dir=tmp_path / "dist") + + module_path = _local_module(tmp_path / "shared.yaml") + output_file = tmp_path / "not-a-directory" + output_file.write_text("occupied\n", encoding="utf-8") + with pytest.raises(SDLParseError, match="Unable to build the OCI layout"): + publish_module_to_oci_layout(module_path, output_dir=output_file) + + +def test_publish_rejects_linked_layout_lock_parent_without_writing_target(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + output_dir = tmp_path / "dist" + output_dir.mkdir() + outside = tmp_path / "outside-locks" + outside.mkdir() + try: + (output_dir / ".raes-layout-locks").symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): + publish_module_to_oci_layout(module_path, output_dir=output_dir) + + assert not list(outside.iterdir()) + + +def test_republishing_layout_removes_stale_inventory(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + layout = Path(first["layout_dir"]) + (layout / "stale.txt").write_text("stale\n", encoding="utf-8") + (layout / "blobs" / "sha256" / ("f" * 64)).write_bytes(b"stale") + + second = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + layout = Path(second["layout_dir"]) + assert layout != Path(first["layout_dir"]) + assert (Path(first["layout_dir"]) / "stale.txt").read_text(encoding="utf-8") == "stale\n" + assert _current_version(_version_slot(layout)) == layout + actual_files = {path.relative_to(layout).as_posix() for path in layout.rglob("*") if path.is_file()} + expected_blobs = { + f"blobs/sha256/{str(second[key]).removeprefix('sha256:')}" for key in ("content_digest", "manifest_digest") + } + manifest = json.loads(_published_blob(layout, str(second["manifest_digest"]))) + expected_blobs.add(f"blobs/sha256/{manifest['config']['digest'].removeprefix('sha256:')}") + assert actual_files == {"index.json", "oci-layout", *expected_blobs} + + +def test_failed_layout_commit_restores_prior_layout(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + module_path = _local_module(tmp_path / "shared.yaml") + published = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + layout = Path(published["layout_dir"]) + slot = _version_slot(layout) + before = {path.relative_to(layout).as_posix(): path.read_bytes() for path in layout.rglob("*") if path.is_file()} + module_path.write_text(module_path.read_text(encoding="utf-8") + "# changed bytes\n", encoding="utf-8") + real_replace = module_registry_filesystem.os.replace + + def fail_staged_install(source, destination): + if Path(source).name.startswith(".raes-current.staged-") and Path(destination) == slot / ".raes-current": + raise OSError("simulated atomic install failure") + return real_replace(source, destination) + + monkeypatch.setattr(module_registry_filesystem.os, "replace", fail_staged_install) + + with pytest.raises(SDLParseError, match="atomically"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + after = {path.relative_to(layout).as_posix(): path.read_bytes() for path in layout.rglob("*") if path.is_file()} + assert after == before + assert _current_version(slot) == layout + assert len(list((slot / "versions").iterdir())) == 2 + assert not list(slot.glob(".raes-current.staged-*")) + + monkeypatch.undo() + repaired = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + assert repaired != layout + assert _current_version(slot) == repaired + + +def test_layout_prune_failure_cannot_advance_pointer(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + slot = _version_slot(first) + module_path.write_text(module_path.read_text(encoding="utf-8") + "# next publication\n", encoding="utf-8") + + def fail_prune(**kwargs): + del kwargs + raise SDLParseError("simulated prune failure") + + monkeypatch.setattr(module_registry_publishing, "_prune_version_directories", fail_prune) + + with pytest.raises(SDLParseError, match="simulated prune failure"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + assert _current_version(slot) == first + assert (first / "index.json").is_file() + + +def test_layout_rejects_non_directory_target(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + layout = tmp_path / "dist" / "acme_shared-1.2.3.oci" + layout.parent.mkdir() + layout.write_text("occupied\n", encoding="utf-8") + + with pytest.raises(SDLParseError, match="atomically"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + assert layout.read_text(encoding="utf-8") == "occupied\n" + + +def test_layout_build_failure_removes_staging_directory(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + module_path = _local_module(tmp_path / "shared.yaml") + real_write_bytes = Path.write_bytes + + def fail_blob_write(path: Path, payload: bytes): + if ".staged-" in path.as_posix() and "/blobs/sha256/" in path.as_posix(): + raise OSError("simulated blob write failure") + return real_write_bytes(path, payload) + + monkeypatch.setattr(Path, "write_bytes", fail_blob_write) + + with pytest.raises(SDLParseError, match="Unable to build the OCI layout"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + versions = tmp_path / "dist" / "acme_shared-1.2.3.oci" / "versions" + assert not list(versions.glob(".staged-*")) + + +def test_layout_repairs_missing_pointer_and_abandoned_stages(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + slot = _version_slot(first) + (slot / ".raes-current").unlink() + (slot / "versions" / ".staged-abandoned").mkdir() + (slot / ".raes-current.staged-abandoned").write_text("partial\n", encoding="utf-8") + + repaired = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + + assert repaired == first + assert _current_version(slot) == first + assert not list((slot / "versions").glob(".staged-*")) + assert not list(slot.glob(".raes-current.staged-*")) + + +def test_layout_prior_reader_survives_new_publication(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first_result = publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + first = Path(first_result["layout_dir"]) + first_index = (first / "index.json").read_bytes() + module_path.write_text(module_path.read_text(encoding="utf-8") + "# next publication\n", encoding="utf-8") + + second = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + + assert second != first + assert (first / "index.json").read_bytes() == first_index + assert first.is_dir() + assert _current_version(_version_slot(second)) == second + + +def test_layout_validation_rejects_symlink_without_touching_target(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + outside = tmp_path / "outside-layout" + outside.write_text("outside\n", encoding="utf-8") + layout_metadata = first / "oci-layout" + layout_metadata.unlink() + try: + layout_metadata.symlink_to(outside) + except OSError: + pytest.skip("symlink creation is unavailable") + + repaired = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + + assert repaired != first + assert (repaired / "oci-layout").read_bytes() == b'{"imageLayoutVersion":"1.0.0"}\n' + assert outside.read_text(encoding="utf-8") == "outside\n" + + +def test_layout_validation_rejects_modified_bytes_and_special_files(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + index = first / "index.json" + payload = index.read_bytes() + index.write_bytes(bytes([payload[0] ^ 1]) + payload[1:]) + + second = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + assert second != first + + if not hasattr(os, "mkfifo"): + return + os.mkfifo(second / "unexpected-pipe") + third = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + assert third != second + + +def test_layout_rejects_invalid_staged_and_installed_versions(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + module_path = _local_module(tmp_path / "shared.yaml") + real_rglob = Path.rglob + + def hide_staged_inventory(path: Path, pattern: str): + if path.name.startswith(".staged-"): + return iter(()) + return real_rglob(path, pattern) + + monkeypatch.setattr(Path, "rglob", hide_staged_inventory) + with pytest.raises(SDLParseError, match="Staged OCI layout failed validation"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist-a") + + monkeypatch.undo() + real_install = module_registry_publishing._install_version_directory + + def corrupt_installed(**kwargs): + installed = real_install(**kwargs) + (installed / "unexpected").write_text("corrupt\n", encoding="utf-8") + return installed + + monkeypatch.setattr(module_registry_publishing, "_install_version_directory", corrupt_installed) + with pytest.raises(SDLParseError, match="Published OCI layout failed validation"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist-b") + + +def test_layout_healthy_republish_reuses_current_immutable_version(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + + second = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + + assert second == first + assert _current_version(_version_slot(first)) == first + + +def test_layout_validation_io_failure_rebuilds_without_exposing_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + module_path = _local_module(tmp_path / "shared.yaml") + first = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + failed_path = first / "index.json" + real_lstat = Path.lstat + failed = False + + def fail_once(path: Path): + nonlocal failed + if path == failed_path and not failed: + failed = True + raise OSError("SECRET-LSTAT-DETAIL") + return real_lstat(path) + + monkeypatch.setattr(Path, "lstat", fail_once) + + repaired = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"]) + assert repaired != first + + +def test_layout_result_and_cli_identify_usable_immutable_version(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + publish = CliRunner().invoke( + app, + ["sdl", "publish", str(module_path), "--output-dir", str(tmp_path / "dist")], + ) + + assert publish.exit_code == 0, publish.output + result = json.loads(publish.stdout) + layout = Path(result["layout_dir"]) + slot = _version_slot(layout) + assert layout.is_dir() + assert (layout / "oci-layout").is_file() + assert (layout / "index.json").is_file() + assert slot.name == "acme_shared-1.2.3.oci" + assert not (slot / "oci-layout").exists() + assert _current_version(slot) == layout + + +def test_layout_fails_closed_on_pre_versioned_root_inventory(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + seed = Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "seed")["layout_dir"]) + legacy_slot = tmp_path / "dist" / "acme_shared-1.2.3.oci" + legacy_slot.parent.mkdir() + shutil.copytree(seed, legacy_slot) + before = { + path.relative_to(legacy_slot).as_posix(): path.read_bytes() for path in legacy_slot.rglob("*") if path.is_file() + } + module_path.write_text(module_path.read_text(encoding="utf-8") + "# changed publication\n", encoding="utf-8") + + with pytest.raises(SDLParseError, match="legacy root-layout format"): + publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist") + + after = { + path.relative_to(legacy_slot).as_posix(): path.read_bytes() for path in legacy_slot.rglob("*") if path.is_file() + } + assert after == before + assert not (legacy_slot / "versions").exists() + assert not (legacy_slot / ".raes-current").exists() + + +def test_layout_version_retention_is_bounded_and_preserves_immediate_prior_reader(tmp_path: Path): + module_path = _local_module(tmp_path / "shared.yaml") + published: list[Path] = [] + for revision in range(module_registry_filesystem._MAX_RETAINED_VERSIONS + 3): + module_path.write_text( + module_path.read_text(encoding="utf-8") + f"# revision {revision}\n", + encoding="utf-8", + ) + published.append(Path(publish_module_to_oci_layout(module_path, output_dir=tmp_path / "dist")["layout_dir"])) + + current = published[-1] + prior = published[-2] + versions = _version_slot(current) / "versions" + assert len(list(versions.iterdir())) == module_registry_filesystem._MAX_RETAINED_VERSIONS + assert current.is_dir() + assert prior.is_dir() + assert _current_version(_version_slot(current)) == current + + +def test_transaction_cleanup_does_not_follow_symlink(tmp_path: Path): + outside = tmp_path / "outside" + outside.mkdir() + (outside / "keep.txt").write_text("keep\n", encoding="utf-8") + link = tmp_path / "transaction-link" + try: + link.symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + module_registry_filesystem._remove_path(link) + + assert not link.exists() + assert (outside / "keep.txt").read_text(encoding="utf-8") == "keep\n" + + +def test_version_install_failure_preserves_current_tree(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + prior = versions / "prior" + prior.mkdir() + (prior / "prior.txt").write_text("prior\n", encoding="utf-8") + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="prior", + error_message="transaction failed", + ) + staged = module_registry_filesystem._new_version_stage(versions=versions, error_message="transaction failed") + real_replace = module_registry_filesystem.os.replace + + def fail_install(source, destination): + if Path(source) == staged: + raise OSError("simulated rename failure") + return real_replace(source, destination) + + monkeypatch.setattr(module_registry_filesystem.os, "replace", fail_install) + + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._install_version_directory( + staged=staged, + versions=versions, + version_name="next", + error_message="transaction failed", + ) + + assert _current_version(slot) == prior + assert (prior / "prior.txt").read_text(encoding="utf-8") == "prior\n" + assert not staged.exists() + + +def test_pointer_failure_never_removes_prior_pointer(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + for name in ("prior", "next"): + (versions / name).mkdir() + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="prior", + error_message="transaction failed", + ) + real_replace = module_registry_filesystem.os.replace + + def fail_pointer(source, destination): + if Path(destination) == slot / ".raes-current": + raise OSError("simulated pointer failure") + return real_replace(source, destination) + + monkeypatch.setattr(module_registry_filesystem.os, "replace", fail_pointer) + + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="next", + error_message="transaction failed", + ) + + assert _current_version(slot) == versions / "prior" + assert not list(slot.glob(".raes-current.staged-*")) + + +def test_version_install_and_pointer_sync_before_and_after_rename( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + staged = module_registry_filesystem._new_version_stage(versions=versions, error_message="transaction failed") + (staged / "payload").write_text("complete\n", encoding="utf-8") + events: list[str] = [] + real_replace = module_registry_filesystem.os.replace + + monkeypatch.setattr( + module_registry_filesystem, + "_fsync_tree", + lambda path, *, error_message: events.append("tree-fsync"), + ) + monkeypatch.setattr( + module_registry_filesystem, + "_fsync_directory", + lambda path, *, error_message: events.append("directory-fsync"), + ) + + def record_replace(source, destination): + events.append("replace") + return real_replace(source, destination) + + monkeypatch.setattr(module_registry_filesystem.os, "replace", record_replace) + module_registry_filesystem._install_version_directory( + staged=staged, + versions=versions, + version_name="version", + error_message="transaction failed", + ) + assert events == ["tree-fsync", "replace", "directory-fsync"] + + events.clear() + monkeypatch.setattr(module_registry_filesystem.os, "fsync", lambda descriptor: events.append("file-fsync")) + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="version", + error_message="transaction failed", + ) + assert events == ["file-fsync", "replace", "directory-fsync"] + + +def test_version_fsync_failures_and_unsafe_tree_nodes_fail_closed( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setattr(module_registry_filesystem, "_DIRECTORY_FSYNC_SUPPORTED", False) + monkeypatch.setattr( + module_registry_filesystem.os, + "open", + lambda *args, **kwargs: pytest.fail("unsupported directory fsync must not open a directory"), + ) + module_registry_filesystem._fsync_directory(tmp_path, error_message="sync failed") + + monkeypatch.undo() + if module_registry_filesystem._DIRECTORY_FSYNC_SUPPORTED: + real_open = module_registry_filesystem.os.open + + def fail_directory_open(path, *args, **kwargs): + if Path(path) == tmp_path: + raise OSError("SECRET-DIRECTORY-OPEN") + return real_open(path, *args, **kwargs) + + monkeypatch.setattr(module_registry_filesystem.os, "open", fail_directory_open) + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_directory(tmp_path, error_message="sync failed") + + monkeypatch.undo() + monkeypatch.setattr( + module_registry_filesystem.os, + "fsync", + lambda descriptor: (_ for _ in ()).throw(OSError("SECRET-FSYNC")), + ) + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_directory(tmp_path, error_message="sync failed") + + monkeypatch.undo() + monkeypatch.setattr( + module_registry_filesystem.os, + "fsync", + lambda descriptor: (_ for _ in ()).throw(OSError(module_registry_filesystem.errno.EINVAL, "unsupported")), + ) + module_registry_filesystem._fsync_directory(tmp_path, error_message="sync failed") + + monkeypatch.undo() + outside = tmp_path / "outside" + outside.write_text("outside\n", encoding="utf-8") + linked = tmp_path / "linked" + try: + linked.symlink_to(outside) + except OSError: + pytest.skip("symlink creation is unavailable") + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_tree(linked, error_message="sync failed") + + if hasattr(os, "mkfifo"): + fifo = tmp_path / "fifo" + os.mkfifo(fifo) + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_tree(fifo, error_message="sync failed") + + regular = tmp_path / "regular" + regular.write_text("regular\n", encoding="utf-8") + directory_stat = tmp_path.stat() + monkeypatch.setattr(module_registry_filesystem.os, "fstat", lambda descriptor: directory_stat) + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_tree(regular, error_message="sync failed") + + monkeypatch.undo() + with pytest.raises(SDLParseError, match="sync failed"): + module_registry_filesystem._fsync_tree(tmp_path / "missing", error_message="sync failed") + + original = tmp_path.stat() + changed_values = list(original) + changed_values[2] = original.st_dev + 1 + assert not module_registry_filesystem._same_file_identity(original, os.stat_result(changed_values)) + + +@pytest.mark.parametrize("access_mode", [os.O_RDONLY, os.O_RDWR]) +def test_tree_fsync_uses_platform_compatible_file_access( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + access_mode: int, +): + regular = tmp_path / "regular" + regular.write_text("regular\n", encoding="utf-8") + real_open = module_registry_filesystem.os.open + opened_flags: dict[int, int] = {} + synced: list[int] = [] + + def tracked_open(path, flags): + descriptor = real_open(path, flags) + opened_flags[descriptor] = flags + return descriptor + + def windows_compatible_fsync(descriptor): + flags = opened_flags[descriptor] + if access_mode == os.O_RDWR and flags & os.O_RDWR != os.O_RDWR: + raise OSError(module_registry_filesystem.errno.EBADF, "Windows requires a writable descriptor") + synced.append(descriptor) + + monkeypatch.setattr(module_registry_filesystem, "_REGULAR_FILE_FSYNC_ACCESS_MODE", access_mode) + monkeypatch.setattr(module_registry_filesystem.os, "open", tracked_open) + monkeypatch.setattr(module_registry_filesystem.os, "fsync", windows_compatible_fsync) + + module_registry_filesystem._fsync_tree(regular, error_message="sync failed") + + assert len(opened_flags) == 1 + assert next(iter(opened_flags.values())) & (os.O_WRONLY | os.O_RDWR) == access_mode + assert len(synced) == 1 + + +def test_generated_version_names_bound_internal_path_growth() -> None: + digest = "sha256:" + "a" * 64 + + version_name = module_registry_filesystem._new_digest_version_name(digest) + + assert module_registry_filesystem._version_digest_prefix(digest) == "a" * 20 + assert version_name.startswith("a" * 20 + "-") + assert len(version_name) == 41 + assert module_registry_filesystem._valid_version_name(version_name) + + +def test_version_install_cleans_stage_when_tree_sync_rejects(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + staged = module_registry_filesystem._new_version_stage(versions=versions, error_message="transaction failed") + + def fail_sync(*args, **kwargs): + del args, kwargs + raise SDLParseError("tree sync failed") + + monkeypatch.setattr(module_registry_filesystem, "_fsync_tree", fail_sync) + with pytest.raises(SDLParseError, match="tree sync failed"): + module_registry_filesystem._install_version_directory( + staged=staged, + versions=versions, + version_name="version", + error_message="transaction failed", + ) + assert not staged.exists() + + +@pytest.mark.parametrize( + "payload", + [b"", b"version", b"\xff\n", b"../escape\n", b"a" * 257], +) +def test_invalid_version_pointer_is_never_followed(tmp_path: Path, payload: bytes): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + (versions / "version").mkdir() + (slot / ".raes-current").write_bytes(payload) + + assert module_registry_filesystem._read_version_pointer(slot=slot) is None + + +def test_version_pointer_rejects_symlink_and_missing_version(tmp_path: Path): + slot = tmp_path / "slot" + module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + outside = tmp_path / "outside" + outside.write_text("missing\n", encoding="utf-8") + pointer = slot / ".raes-current" + try: + pointer.symlink_to(outside) + except OSError: + pytest.skip("symlink creation is unavailable") + assert module_registry_filesystem._read_version_pointer(slot=slot) is None + assert outside.read_text(encoding="utf-8") == "missing\n" + + pointer.unlink() + pointer.write_text("missing\n", encoding="ascii") + assert module_registry_filesystem._read_version_pointer(slot=slot) is None + + +def test_version_store_rejects_links_files_collisions_and_invalid_names(tmp_path: Path): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + valid = versions / "valid" + valid.mkdir() + (versions / "file").write_text("not a version\n", encoding="utf-8") + try: + (versions / "link").symlink_to(valid, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + assert list(module_registry_filesystem._iter_version_directories(versions)) == [valid] + for version_name in ("../escape", "valid"): + staged = module_registry_filesystem._new_version_stage(versions=versions, error_message="transaction failed") + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._install_version_directory( + staged=staged, + versions=versions, + version_name=version_name, + error_message="transaction failed", + ) + assert not staged.exists() + + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="../escape", + error_message="transaction failed", + ) + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="missing", + error_message="transaction failed", + ) + + +def test_version_slot_rejects_symlink_without_following_it(tmp_path: Path): + outside = tmp_path / "outside" + outside.mkdir() + (outside / "keep").write_text("keep\n", encoding="utf-8") + slot = tmp_path / "slot" + try: + slot.symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + + assert (outside / "keep").read_text(encoding="utf-8") == "keep\n" + + +def test_version_store_translates_directory_and_stage_creation_failures(tmp_path: Path): + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._prepare_versioned_slot( + slot=tmp_path / "missing-parent" / "slot", + error_message="transaction failed", + ) + with pytest.raises(SDLParseError, match="Unable to inspect immutable version directories"): + list(module_registry_filesystem._iter_version_directories(tmp_path / "missing-versions")) + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._new_version_stage( + versions=tmp_path / "missing-versions", + error_message="transaction failed", + ) + + +def test_version_pointer_translates_temporary_file_failure(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + (versions / "version").mkdir() + + def fail_mkstemp(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-TEMP-DETAIL") + + monkeypatch.setattr(module_registry_filesystem.tempfile, "mkstemp", fail_mkstemp) + + with pytest.raises(SDLParseError) as exc_info: + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="version", + error_message="transaction failed", + ) + assert str(exc_info.value) == "transaction failed" + + +def test_version_store_detects_post_create_replacement_and_cleanup_io( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + target = tmp_path / "target" + real_is_dir = Path.is_dir + + def hide_created_directory(path: Path) -> bool: + if path == target: + return False + return real_is_dir(path) + + monkeypatch.setattr(Path, "is_dir", hide_created_directory) + with pytest.raises(SDLParseError, match="transaction failed"): + module_registry_filesystem._require_directory(target, error_message="transaction failed") + + monkeypatch.undo() + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + real_iterdir = Path.iterdir + + def fail_cleanup_scan(path: Path): + if path == versions: + raise OSError("SECRET-SCAN-DETAIL") + return real_iterdir(path) + + monkeypatch.setattr(Path, "iterdir", fail_cleanup_scan) + with pytest.raises(SDLParseError) as exc_info: + module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + assert str(exc_info.value) == "transaction failed" + + +def test_version_pointer_rejects_identity_swap(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + (versions / "version").mkdir() + (slot / ".raes-current").write_text("version\n", encoding="ascii") + directory_stat = slot.stat() + monkeypatch.setattr(module_registry_filesystem.os, "fstat", lambda descriptor: directory_stat) + + assert module_registry_filesystem._read_version_pointer(slot=slot) is None + + +def test_version_pointer_closes_descriptor_when_fdopen_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + (versions / "version").mkdir() + + def fail_fdopen(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-FDOPEN-DETAIL") + + monkeypatch.setattr(module_registry_filesystem.os, "fdopen", fail_fdopen) + with pytest.raises(SDLParseError) as exc_info: + module_registry_filesystem._write_version_pointer( + slot=slot, + version_name="version", + error_message="transaction failed", + ) + assert str(exc_info.value) == "transaction failed" + assert not list(slot.glob(".raes-current.staged-*")) + + +def test_version_pointer_read_closes_descriptor_when_fdopen_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + slot = tmp_path / "slot" + versions = module_registry_filesystem._prepare_versioned_slot(slot=slot, error_message="transaction failed") + (versions / "version").mkdir() + (slot / ".raes-current").write_text("version\n", encoding="ascii") + + def fail_fdopen(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-FDOPEN-DETAIL") + + monkeypatch.setattr(module_registry_filesystem.os, "fdopen", fail_fdopen) + assert module_registry_filesystem._read_version_pointer(slot=slot) is None + + +def test_version_pruning_rejects_invalid_bound_and_translates_io(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + versions = tmp_path / "versions" + versions.mkdir() + version = versions / "version" + version.mkdir() + with pytest.raises(SDLParseError, match="prune failed"): + module_registry_filesystem._prune_version_directories( + versions=versions, + retain_names={"version"}, + max_versions=1, + error_message="prune failed", + ) + + real_stat = Path.stat + + def fail_version_stat(path: Path, *args, **kwargs): + if path == version: + raise OSError("SECRET-STAT-DETAIL") + return real_stat(path, *args, **kwargs) + + monkeypatch.setattr(Path, "stat", fail_version_stat) + with pytest.raises(SDLParseError) as exc_info: + module_registry_filesystem._prune_version_directories( + versions=versions, + retain_names={"version"}, + error_message="prune failed", + ) + assert str(exc_info.value) == "prune failed" + + def test_oci_registry_requests_use_bounded_timeouts(monkeypatch: pytest.MonkeyPatch): responses = [b'{"ok": true}', b"bundle-bytes"] timeouts: list[float | None] = [] @@ -492,40 +1558,238 @@ def fake_urlopen(request, *, timeout=None): return _Response(responses.pop(0)) monkeypatch.setattr(module_registry, "urlopen", fake_urlopen) + monkeypatch.setattr( + module_registry, "_OCI_LIMITS", dataclasses.replace(module_registry._OCI_LIMITS, timeout_seconds=7) + ) assert module_registry._json_request("https://registry.example/v2/acme/tags/list") == {"ok": True} assert module_registry._bytes_request("https://registry.example/v2/acme/blobs/sha256:abc") == b"bundle-bytes" - assert timeouts == [module_registry._HTTP_TIMEOUT_SECONDS, module_registry._HTTP_TIMEOUT_SECONDS] + assert timeouts == [7, 7] -def test_oci_bundle_rejects_root_file_escape(tmp_path: Path): - with pytest.raises(SDLParseError, match="Invalid OCI root_file path"): - module_registry._extract_bundle_to_cache( - bundle_bytes=b"", - manifest_digest="abc123", - root_file="../module.yaml", - base_dir=tmp_path, - ) +@pytest.mark.parametrize("request_name", ["_json_request", "_bytes_request"]) +def test_oci_registry_transport_errors_are_stable(monkeypatch: pytest.MonkeyPatch, request_name: str): + def fail_urlopen(*args, **kwargs): + raise URLError("SECRET-TRANSPORT-DETAIL") + monkeypatch.setattr(module_registry, "urlopen", fail_urlopen) + request = getattr(module_registry, request_name) -def test_oci_bundle_rejects_unsafe_tar_members(tmp_path: Path): - bundle_buffer = io.BytesIO() - with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: - payload = b"name: unsafe\n" - member = tarfile.TarInfo(name="../escape.yaml") - member.size = len(payload) - tar.addfile(member, io.BytesIO(payload)) - bundle_buffer.seek(0) + with pytest.raises(SDLParseError) as exc_info: + request("https://registry.example/v2/acme/resource") - with ( - tarfile.open(fileobj=bundle_buffer, mode="r:gz") as tar, - pytest.raises(SDLParseError, match="Path traversal detected"), - ): - module_registry._safe_tar_members(tar, tmp_path / "cache") + assert "SECRET-TRANSPORT-DETAIL" not in str(exc_info.value) -def test_oci_bundle_rejects_special_member_types(tmp_path: Path): - bundle_buffer = io.BytesIO() +def test_oci_manifest_rejects_malformed_tag_shape(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr(module_registry, "_json_request", lambda *args, **kwargs: {"tags": "1.0.0"}) + import_decl = module_registry.ImportDecl(source="oci:registry.example/acme/shared", namespace="shared") + + with pytest.raises(SDLParseError, match="invalid tags list"): + module_registry_resolution._resolve_oci_manifest( + base_url="https://registry.example", + repository="acme/shared", + import_decl=import_decl, + locked=None, + source="oci:registry.example/acme/shared", + ) + + +def test_oci_config_rejects_malformed_descriptor_shape(): + with pytest.raises(SDLParseError, match="malformed config or layer descriptors"): + module_registry_resolution._resolve_oci_config( + base_url="https://registry.example", + repository="acme/shared", + manifest={"config": [], "layers": ["not-an-object"]}, + source="oci:registry.example/acme/shared", + ) + + +def test_oci_module_descriptor_validation_is_normalized_to_sdl_parse_error() -> None: + import_decl = module_registry.ImportDecl( + source="oci:registry.example/acme/shared", + namespace="shared", + ) + + with pytest.raises(SDLParseError, match="invalid module descriptor"): + module_registry_resolution._build_oci_descriptor( + config_payload={"module": {"id": [], "version": "1.0.0"}}, + layer_digest="sha256:" + "0" * 64, + import_decl=import_decl, + locked=None, + source=import_decl.source, + ) + + +def test_signed_oci_import_rejects_non_object_signature_metadata( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source = "oci:registry.example/acme/shared" + import_decl = module_registry.ImportDecl(source=source, namespace="shared") + trust_policy = module_registry.TrustPolicy( + registries={ + "registry.example": module_registry.RegistryTrustPolicy( + require_signatures=True, + ) + } + ) + digest = "sha256:" + "0" * 64 + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_manifest", + lambda **_kwargs: (digest, {}), + ) + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_config", + lambda **_kwargs: ( + { + "module": {"id": "acme/shared", "version": "1.0.0"}, + "signatures": ["not-an-object"], + }, + digest, + ), + ) + monkeypatch.setattr( + module_registry_resolution, + "_fetch_oci_bundle", + lambda **_kwargs: b"unused-bundle", + ) + + with pytest.raises(SDLParseError, match="malformed signature metadata"): + module_registry_resolution._resolve_oci_import( + import_decl, + source, + base_dir=tmp_path, + lockfile=None, + trust_policy=trust_policy, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + +@pytest.mark.parametrize( + ("payload", "message"), + [ + (b'{"unterminated":', "is not valid UTF-8 JSON"), + (b'\xff{"valid": false}', "is not valid UTF-8 JSON"), + (b'["not", "an", "object"]', "must be a JSON object"), + pytest.param(b'{"integer":' + b"9" * 10_000 + b"}", "is not valid UTF-8 JSON", id="integer"), + ], +) +def test_oci_metadata_malformed_json_has_stable_error(monkeypatch: pytest.MonkeyPatch, payload: bytes, message: str): + monkeypatch.setattr(module_registry, "urlopen", _fake_urlopen_returning(_FakeResponse(payload))) + + with pytest.raises(SDLParseError) as exc_info: + module_registry._json_request("https://registry.example/v2/acme/tags/list") + + assert str(exc_info.value) == f"OCI metadata from https://registry.example/v2/acme/tags/list {message}" + assert "line 1 column" not in str(exc_info.value) + + +@pytest.mark.parametrize("decoder_error", [RecursionError("too deep"), ValueError("integer limit")]) +def test_oci_metadata_decoder_limit_errors_are_stable( + monkeypatch: pytest.MonkeyPatch, + decoder_error: Exception, +) -> None: + def fail_decode(_payload: str) -> object: + raise decoder_error + + monkeypatch.setattr(module_registry.json, "loads", fail_decode) + with pytest.raises(SDLParseError) as exc_info: + module_registry._decode_json_object(b"{}", context="OCI metadata") + + assert str(exc_info.value) == "OCI metadata is not valid UTF-8 JSON" + + +def test_oci_manifest_malformed_json_has_stable_error(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr(module_registry, "_bytes_request", lambda *args, **kwargs: b'{"broken":') + monkeypatch.setattr(module_registry, "_json_request", lambda *args, **kwargs: {"tags": ["1.0.0"]}) + import_decl = module_registry.ImportDecl(source="oci:registry.example/acme/shared", namespace="shared") + + with pytest.raises(SDLParseError) as exc_info: + module_registry_resolution._resolve_oci_manifest( + base_url="https://registry.example", + repository="acme/shared", + import_decl=import_decl, + locked=None, + source="oci:registry.example/acme/shared", + ) + + assert str(exc_info.value) == "OCI manifest for 'oci:registry.example/acme/shared' is not valid UTF-8 JSON" + + +def test_oci_config_malformed_json_has_stable_error(monkeypatch: pytest.MonkeyPatch): + malformed = b'{"broken":' + digest = f"sha256:{module_registry._sha256_digest(malformed)}" + monkeypatch.setattr(module_registry, "_bytes_request", lambda *args, **kwargs: malformed) + manifest = { + "config": {"digest": digest}, + "layers": [{"mediaType": module_registry.OCI_BUNDLE_MEDIA_TYPE, "digest": "sha256:" + "a" * 64}], + } + + with pytest.raises(SDLParseError) as exc_info: + module_registry_resolution._resolve_oci_config( + base_url="https://registry.example", + repository="acme/shared", + manifest=manifest, + source="oci:registry.example/acme/shared", + ) + + assert str(exc_info.value) == "OCI config for 'oci:registry.example/acme/shared' is not valid UTF-8 JSON" + + +@pytest.mark.parametrize( + "root_file", + ["../module.yaml", "a//module.yaml", "a/module.yaml/", "./module.yaml", ".", "C:/module.yaml"], +) +def test_oci_bundle_rejects_root_file_escape(tmp_path: Path, root_file: str): + with pytest.raises(SDLParseError, match="Invalid OCI root_file path"): + module_registry._extract_bundle_to_cache( + bundle_bytes=b"", + manifest_digest="abc123", + root_file=root_file, + base_dir=tmp_path, + ) + + +def test_oci_bundle_rejects_unsafe_tar_members(tmp_path: Path): + bundle_buffer = io.BytesIO() + with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: + payload = b"name: unsafe\n" + member = tarfile.TarInfo(name="../escape.yaml") + member.size = len(payload) + tar.addfile(member, io.BytesIO(payload)) + bundle_buffer.seek(0) + + with ( + tarfile.open(fileobj=bundle_buffer, mode="r:gz") as tar, + pytest.raises(SDLParseError, match="Path traversal detected"), + ): + module_registry._safe_tar_members(tar, tmp_path / "cache") + + +def test_oci_bundle_rejects_nul_in_pax_member_path(tmp_path: Path): + raw_bundle = io.BytesIO() + with tarfile.open(fileobj=raw_bundle, mode="w", format=tarfile.PAX_FORMAT) as tar: + payload = b"name: unsafe\n" + member = tarfile.TarInfo(name="safe.yaml") + member.pax_headers = {"path": "unsafe\x00name.yaml"} + member.size = len(payload) + tar.addfile(member, io.BytesIO(payload)) + + bundle = gzip.compress(raw_bundle.getvalue(), mtime=0) + with pytest.raises(SDLParseError, match="Path traversal detected"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="nul-pax-path", + root_file="safe.yaml", + base_dir=tmp_path, + ) + + +def test_oci_bundle_rejects_special_member_types(tmp_path: Path): + bundle_buffer = io.BytesIO() with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: fifo = tarfile.TarInfo(name="pipe") fifo.type = tarfile.FIFOTYPE @@ -539,6 +1803,15 @@ def test_oci_bundle_rejects_special_member_types(tmp_path: Path): module_registry._safe_tar_members(tar, tmp_path / "cache") +def test_oci_bundle_rejects_reserved_cache_manifest_member(tmp_path: Path): + bundle = _gzip_tar([(".raes-cache-tree.json", b"attacker controlled\n")]) + with ( + tarfile.open(fileobj=bundle, mode="r:gz") as archive, + pytest.raises(SDLParseError, match="reserved cache metadata"), + ): + module_registry._safe_tar_members(archive, tmp_path / "cache") + + def test_oci_bundle_rejects_symlink_members(tmp_path: Path): # A symlink whose own name passes the traversal check (e.g. name='module.yaml') # but whose linkname escapes the cache is a distinct attack vector and must be @@ -594,6 +1867,274 @@ def test_oci_bundle_strips_dangerous_mode_bits(tmp_path: Path): assert safe[0].mode == 0o755 +@pytest.mark.parametrize( + "member_name", + ["", "./module.yaml", "nested/../module.yaml", "nested//module.yaml", "C:/module.yaml", "nested\\module.yaml"], +) +def test_oci_bundle_rejects_noncanonical_cross_platform_member_paths(tmp_path: Path, member_name: str): + member = tarfile.TarInfo(name=member_name) + member.size = 0 + destination = tmp_path / "cache" + resolved_destination = destination.resolve() + + with pytest.raises(SDLParseError, match="Path traversal detected"): + module_registry._validate_tar_member_shape( + member, + dest=destination, + resolved_dest=resolved_destination, + seen_paths=set(), + limits=module_registry._OCI_LIMITS, + ) + + +def test_oci_bundle_member_must_remain_below_validated_destination(tmp_path: Path): + member = tarfile.TarInfo(name="module.yaml") + member.size = 0 + destination = tmp_path / "cache" + resolved_destination = (tmp_path / "different-cache").resolve() + + with pytest.raises(SDLParseError, match="Path traversal detected"): + module_registry._validate_tar_member_shape( + member, + dest=destination, + resolved_dest=resolved_destination, + seen_paths=set(), + limits=module_registry._OCI_LIMITS, + ) + + +def test_oci_bundle_inventory_is_streamed_with_filtered_modes_and_implicit_directories(tmp_path: Path): + bundle = io.BytesIO() + with tarfile.open(fileobj=bundle, mode="w:gz") as archive: + member = tarfile.TarInfo(name="nested/module.yaml") + member.mode = 0o111 + payload = b"name: executable\n" + member.size = len(payload) + archive.addfile(member, io.BytesIO(payload)) + bundle.seek(0) + + with tarfile.open(fileobj=bundle, mode="r:gz") as archive: + members = module_registry._safe_tar_members(archive, tmp_path / "inventory") + manifest = module_registry._expected_cache_tree_manifest( + tar=archive, + members=members, + content_digest="sha256:" + "0" * 64, + root_file="nested/module.yaml", + ) + + directory_mode = 0o777 if os.name == "nt" else 0o700 + file_mode = 0o666 if os.name == "nt" else 0o711 + assert manifest["entries"] == [ + {"mode": directory_mode, "path": ".", "type": "directory"}, + {"mode": directory_mode, "path": "nested", "type": "directory"}, + { + "digest": f"sha256:{module_registry._sha256_digest(payload)}", + "mode": file_mode, + "path": "nested/module.yaml", + "size": len(payload), + "type": "file", + }, + ] + assert not (tmp_path / "inventory").exists() + + +def test_oci_bundle_inventory_uses_windows_representable_modes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(module_registry_archive.os, "name", "nt") + bundle = _gzip_tar([("nested/module.yaml", b"name: windows\n")]) + with tarfile.open(fileobj=bundle, mode="r:gz") as archive: + members = module_registry._safe_tar_members(archive, tmp_path / "inventory") + manifest = module_registry._expected_cache_tree_manifest( + tar=archive, + members=members, + content_digest="sha256:" + "0" * 64, + root_file="nested/module.yaml", + ) + + entries = {entry["path"]: entry for entry in manifest["entries"]} + assert entries["."]["mode"] == 0o777 + assert entries["nested"]["mode"] == 0o777 + assert entries["nested/module.yaml"]["mode"] == 0o666 + + +def test_oci_directory_mode_normalization_does_not_use_windows_chmod( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + mode = stat.S_IMODE(tmp_path.stat().st_mode) + monkeypatch.setattr(module_registry_extraction.os, "name", "nt") + monkeypatch.setattr( + module_registry_extraction.os, + "chmod", + lambda *args, **kwargs: pytest.fail("Windows directory mode admission must not call chmod"), + ) + + module_registry_extraction._normalize_extracted_directory_modes( + tmp_path, + [{"mode": mode, "path": ".", "type": "directory"}], + ) + + +def test_oci_directory_mode_normalization_rejects_invalid_and_unstable_entries( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + not_directory = tmp_path / "file" + not_directory.write_text("payload", encoding="utf-8") + with pytest.raises(SDLParseError, match="non-directory entry"): + module_registry_extraction._normalize_extracted_directory_modes( + tmp_path, + [{"mode": 0o700, "path": "file", "type": "directory"}], + ) + + expected_mode = stat.S_IMODE(tmp_path.stat().st_mode) ^ 0o100 + monkeypatch.setattr(module_registry_extraction.os, "chmod", lambda *args, **kwargs: None) + with pytest.raises(SDLParseError, match="changed during normalization"): + module_registry_extraction._normalize_extracted_directory_modes( + tmp_path, + [{"mode": expected_mode, "path": ".", "type": "directory"}], + ) + + +def test_oci_directory_mode_normalization_wraps_platform_errors( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fail_chmod(*args, **kwargs): + raise OSError("injected mode failure") + + monkeypatch.setattr(module_registry_extraction.os, "name", "posix") + monkeypatch.setattr(module_registry_extraction.os, "chmod", fail_chmod) + with pytest.raises(SDLParseError, match="Unable to normalize"): + module_registry_extraction._normalize_extracted_directory_modes( + tmp_path, + [{"mode": 0o700, "path": ".", "type": "directory"}], + ) + + +@pytest.mark.parametrize( + "members", + [ + [("parent", b"file\n"), ("parent/module.yaml", b"name: nested\n")], + [("parent/module.yaml", b"name: nested\n"), ("parent", b"file\n")], + ], +) +def test_oci_bundle_inventory_rejects_file_directory_conflicts( + tmp_path: Path, + members: list[tuple[str, bytes]], +): + bundle_bytes = _gzip_tar(members).getvalue() + member_identity = repr(members).encode() + manifest_digest = "conflict-" + module_registry._sha256_digest(member_identity) + with pytest.raises(SDLParseError, match="conflicting file and directory paths"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_bytes, + manifest_digest=manifest_digest, + root_file="parent/module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_bundle_inventory_rejects_file_at_tree_root(tmp_path: Path): + bundle = io.BytesIO() + with tarfile.open(fileobj=bundle, mode="w:gz") as archive: + member = tarfile.TarInfo(name=".") + member.size = 0 + archive.addfile(member, io.BytesIO()) + + bundle_bytes = bundle.getvalue() + with pytest.raises(SDLParseError, match="cannot replace the cache tree root"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_bytes, + manifest_digest="root-file", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_bundle_inventory_defensive_stream_failures(monkeypatch: pytest.MonkeyPatch): + class StubTar: + def __init__(self, payload: bytes | None) -> None: + self.payload = payload + + def extractfile(self, member): + del member + return None if self.payload is None else io.BytesIO(self.payload) + + root_directory = tarfile.TarInfo(name=".") + root_directory.type = tarfile.DIRTYPE + manifest = module_registry._expected_cache_tree_manifest( + tar=StubTar(None), + members=[root_directory], + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + directory_mode = 0o777 if os.name == "nt" else 0o700 + assert manifest["entries"] == [{"mode": directory_mode, "path": ".", "type": "directory"}] + + file_member = tarfile.TarInfo(name="entry") + file_member.size = 1 + missing_payload_tar = StubTar(None) + with pytest.raises(SDLParseError, match="Unable to read"): + module_registry._expected_cache_tree_manifest( + tar=missing_payload_tar, + members=[file_member], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + oversized_payload_tar = StubTar(b"xx") + with pytest.raises(SDLParseError, match="exceeds its declared size"): + module_registry._expected_cache_tree_manifest( + tar=oversized_payload_tar, + members=[file_member], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + file_member.size = 2 + short_payload_tar = StubTar(b"x") + with pytest.raises(SDLParseError, match="shorter than its declared size"): + module_registry._expected_cache_tree_manifest( + tar=short_payload_tar, + members=[file_member], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + + file_member.size = 0 + directory_member = tarfile.TarInfo(name="entry") + directory_member.type = tarfile.DIRTYPE + conflicting_entry_tar = StubTar(b"") + with pytest.raises(SDLParseError, match="conflicting file and directory paths"): + module_registry._expected_cache_tree_manifest( + tar=conflicting_entry_tar, + members=[file_member, directory_member], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + + limits = module_registry._OCI_LIMITS + monkeypatch.setattr(module_registry, "_OCI_LIMITS", dataclasses.replace(limits, max_bundle_members=0)) + entry_limit_tar = StubTar(b"") + with pytest.raises(SDLParseError, match="tree entry limit"): + module_registry._expected_cache_tree_manifest( + tar=entry_limit_tar, + members=[file_member], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + monkeypatch.setattr(module_registry, "_OCI_LIMITS", dataclasses.replace(limits, max_metadata_bytes=1)) + metadata_limit_tar = StubTar(None) + with pytest.raises(SDLParseError, match="metadata limit"): + module_registry._expected_cache_tree_manifest( + tar=metadata_limit_tar, + members=[], + content_digest="sha256:" + "0" * 64, + root_file="entry", + ) + + def test_oci_bundle_extracts_safe_members(tmp_path: Path): payload = b"name: ok\n" bundle_buffer = io.BytesIO() @@ -607,113 +2148,1574 @@ def test_oci_bundle_extracts_safe_members(tmp_path: Path): bundle_bytes=bundle_buffer.getvalue(), manifest_digest="deadbeef", root_file="module.yaml", - base_dir=tmp_path, + base_dir=tmp_path, + ) + + assert root_path.is_file() + assert root_path.read_bytes() == payload + cache = module_registry._oci_cache_dir(tmp_path) / "deadbeef" + assert root_path.resolve().is_relative_to(cache.resolve()) + + +def test_oci_bundle_without_data_filter_fails_closed(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + # Simulate Python 3.11.0–3.11.3, where TarFile.extractall lacks the PEP 706 + # `filter` keyword (backported in 3.11.4). There is deliberately no unsafe + # compatibility fallback at this filesystem-write boundary. + + def no_filter_extractall(self, path=None, members=None, **kwargs): + del self, path, members + if "filter" in kwargs: + raise TypeError("extractall() got an unexpected keyword argument 'filter'") + pytest.fail("unfiltered extraction must never be attempted") + + monkeypatch.setattr(tarfile.TarFile, "extractall", no_filter_extractall) + + payload = b"name: ok\n" + bundle_buffer = io.BytesIO() + with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: + member = tarfile.TarInfo(name="module.yaml") + member.size = len(payload) + tar.addfile(member, io.BytesIO(payload)) + bundle_buffer.seek(0) + bundle_bytes = bundle_buffer.getvalue() + + with pytest.raises(SDLParseError, match="Python 3.11.4 or newer"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_bytes, + manifest_digest="cafef00d", + root_file="module.yaml", + base_dir=tmp_path, + ) + versions = module_registry._oci_cache_dir(tmp_path) / "cafef00d" / "versions" + assert not list(versions.iterdir()) + + +def test_oci_bundle_fallback_rejects_traversal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + # The fallback path used on Python 3.11.0–3.11.3 must still reject traversal. + real_extractall = tarfile.TarFile.extractall + + def no_filter_extractall(self, path=None, members=None, **kwargs): + if "filter" in kwargs: + raise TypeError("extractall() got an unexpected keyword argument 'filter'") + return real_extractall(self, path, members=members) + + monkeypatch.setattr(tarfile.TarFile, "extractall", no_filter_extractall) + + payload = b"owned\n" + bundle_buffer = io.BytesIO() + with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: + member = tarfile.TarInfo(name="../escape.yaml") + member.size = len(payload) + tar.addfile(member, io.BytesIO(payload)) + bundle_buffer.seek(0) + + with pytest.raises(SDLParseError, match="Path traversal detected"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_buffer.getvalue(), + manifest_digest="badbad", + root_file="module.yaml", + base_dir=tmp_path, + ) + assert not (tmp_path / "escape.yaml").exists() + + +def test_oci_bundle_rejects_root_file_directory(tmp_path: Path): + bundle_buffer = io.BytesIO() + with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: + directory = tarfile.TarInfo(name="module.yaml") + directory.type = tarfile.DIRTYPE + directory.mode = 0o755 + tar.addfile(directory) + bundle_buffer.seek(0) + + with pytest.raises(SDLParseError, match="root file"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_buffer.getvalue(), + manifest_digest="d1rd1r", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_bundle_rechecks_declared_root_after_extraction( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setattr(tarfile.TarFile, "extractall", lambda *args, **kwargs: None) + bundle_bytes = _cache_test_bundle(b"name: absent\n") + + with pytest.raises(SDLParseError, match="missing declared root file"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle_bytes, + manifest_digest="missing-after-extraction", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_bundle_cache_hit_enforces_root_file_containment(tmp_path: Path): + # Simulate a cache populated by an earlier unsafe extractor: a symlink at the + # root_file location resolving outside the digest cache. The cache-hit fast path + # must still fail closed rather than returning the escaping path. + bundle = _cache_test_bundle(b"name: safe\n") + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="stale", + root_file="module.yaml", + base_dir=tmp_path, + ) + outside = tmp_path / "outside.yaml" + outside.write_text("name: evil\n", encoding="utf-8") + root.unlink() + root.symlink_to(outside) + + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="stale", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired.parent != root.parent + assert repaired.read_bytes() == b"name: safe\n" + assert root.is_symlink() + + +def _cache_test_bundle(payload: bytes) -> bytes: + return _gzip_tar([("module.yaml", payload)]).getvalue() + + +def _cache_graph_bundle() -> bytes: + return _gzip_tar( + [ + ( + "module.yaml", + b"name: root\nimports:\n - source: local:nested/child.yaml\n namespace: child\n", + ), + ("nested/child.yaml", b"name: child\n"), + ] + ).getvalue() + + +def test_verified_oci_cache_source_graph_is_captured_on_miss_and_hit( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + bundle = _cache_graph_bundle() + first = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="verified-source-graph", + root_file="module.yaml", + base_dir=tmp_path, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + assert isinstance(first, module_registry._VerifiedSourceBundle) + assert list(first.documents) == ["module.yaml", "nested/child.yaml"] + assert first.resolve_local(base_dir=first.cache_root, relative="module.yaml")[1].raw_bytes.startswith(b"name: root") + + monkeypatch.setattr( + tarfile.TarFile, + "extractall", + lambda *_args, **_kwargs: pytest.fail("a verified cache hit must not extract"), + ) + second = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="verified-source-graph", + root_file="module.yaml", + base_dir=tmp_path, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + assert isinstance(second, module_registry._VerifiedSourceBundle) + assert second.cache_root == first.cache_root + assert second.documents == first.documents + + +@pytest.mark.parametrize("relative", ["module.yaml", "nested/child.yaml"]) +def test_verified_oci_cache_source_graph_rejects_replacement_after_tree_validation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + relative: str, +) -> None: + bundle = _cache_graph_bundle() + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="verified-source-race", + root_file="module.yaml", + base_dir=tmp_path, + ) + real_validate = module_registry_cache._validated_cache_root + replaced = False + + def replace_after_validation(**kwargs): + nonlocal replaced + root = real_validate(**kwargs) + if root is not None and not replaced: + target = root.parent.joinpath(*PurePosixPath(relative).parts) + payload = target.read_bytes() + replacement = target.with_name(f".{target.name}.replacement") + replacement.write_bytes(payload.replace(b"root", b"evil").replace(b"child", b"other")) + replacement.chmod(stat.S_IMODE(target.stat().st_mode)) + os.replace(replacement, target) + replaced = True + return root + + monkeypatch.setattr(module_registry_cache, "_validated_cache_root", replace_after_validation) + + with pytest.raises(SDLParseError, match="cache tree failed integrity validation"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="verified-source-race", + root_file="module.yaml", + base_dir=tmp_path, + source_options=DEFAULT_SOURCE_PARSE_OPTIONS, + ) + + assert replaced is True + + +def test_oci_cache_entry_is_bound_to_bundle_digest(tmp_path: Path): + first_bundle = _cache_test_bundle(b"name: first\n") + second_bundle = _cache_test_bundle(b"name: second\n") + first_digest = f"sha256:{module_registry._sha256_digest(first_bundle)}" + second_digest = f"sha256:{module_registry._sha256_digest(second_bundle)}" + + first = module_registry._extract_bundle_to_cache( + bundle_bytes=first_bundle, + manifest_digest="shared-manifest", + content_digest=first_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + assert first.read_bytes() == b"name: first\n" + second = module_registry._extract_bundle_to_cache( + bundle_bytes=second_bundle, + manifest_digest="shared-manifest", + content_digest=second_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert second.read_bytes() == b"name: second\n" + manifest = json.loads((second.parent / ".raes-cache-tree.json").read_text(encoding="utf-8")) + assert manifest["content_digest"] == second_digest + assert _current_version(_version_slot(second.parent)) == second.parent + + +def test_oci_cache_hit_hashes_bundle_without_extraction_or_version_staging( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + bundle = _cache_graph_bundle() + first = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="streaming-hit", + root_file="module.yaml", + base_dir=tmp_path, + ) + + def forbidden(*args, **kwargs): + del args, kwargs + pytest.fail("a valid cache hit must not extract or stage a version") + + monkeypatch.setattr(tarfile.TarFile, "extractall", forbidden) + monkeypatch.setattr(module_registry, "_new_version_stage", forbidden) + monkeypatch.setattr(module_registry, "_install_version_directory", forbidden) + + second = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="streaming-hit", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert second == first + assert len(list(first.parent.parent.iterdir())) == 1 + + +def test_oci_cache_rejects_bundle_bytes_that_do_not_match_expected_digest(tmp_path: Path): + bundle = _cache_test_bundle(b"name: mismatch\n") + + with pytest.raises(SDLParseError, match="expected content digest"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="mismatch", + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert not (module_registry._oci_cache_dir(tmp_path) / "mismatch").exists() + + +def test_oci_cache_concurrent_writers_extract_once(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + bundle = _cache_test_bundle(b"name: concurrent\n") + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + barrier = threading.Barrier(2) + commits = 0 + real_commit = module_registry._install_version_directory + + def counted_commit(**kwargs): + nonlocal commits + commits += 1 + return real_commit(**kwargs) + + monkeypatch.setattr(module_registry, "_install_version_directory", counted_commit) + + def resolve() -> bytes: + barrier.wait(timeout=5) + return module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="concurrent", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ).read_bytes() + + with ThreadPoolExecutor(max_workers=2) as executor: + results = list(executor.map(lambda _: resolve(), range(2))) + + assert results == [b"name: concurrent\n", b"name: concurrent\n"] + assert commits == 1 + versions = module_registry._oci_cache_dir(tmp_path) / "concurrent" / "versions" + assert not list(versions.glob(".staged-*")) + + +def test_oci_cache_failed_pointer_preserves_prior_entry(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + first_bundle = _cache_test_bundle(b"name: first\n") + second_bundle = _cache_test_bundle(b"name: second\n") + first_digest = f"sha256:{module_registry._sha256_digest(first_bundle)}" + second_digest = f"sha256:{module_registry._sha256_digest(second_bundle)}" + first = module_registry._extract_bundle_to_cache( + bundle_bytes=first_bundle, + manifest_digest="transaction", + content_digest=first_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + slot = _version_slot(first.parent) + + def fail_commit(**kwargs): + raise SDLParseError("simulated cache transaction failure") + + monkeypatch.setattr(module_registry, "_write_version_pointer", fail_commit) + with pytest.raises(SDLParseError, match="simulated cache transaction failure"): + module_registry._extract_bundle_to_cache( + bundle_bytes=second_bundle, + manifest_digest="transaction", + content_digest=second_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert first.read_bytes() == b"name: first\n" + assert ( + json.loads((first.parent / ".raes-cache-tree.json").read_text(encoding="utf-8"))["content_digest"] + == first_digest + ) + assert _current_version(slot) == first.parent + assert not list((slot / "versions").glob(".staged-*")) + + +def test_oci_cache_prune_failure_cannot_advance_pointer(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + first_bundle = _cache_test_bundle(b"name: first\n") + second_bundle = _cache_test_bundle(b"name: second\n") + first = module_registry._extract_bundle_to_cache( + bundle_bytes=first_bundle, + manifest_digest="prune-order", + root_file="module.yaml", + base_dir=tmp_path, + ) + slot = _version_slot(first.parent) + + def fail_prune(**kwargs): + del kwargs + raise SDLParseError("simulated cache prune failure") + + monkeypatch.setattr(module_registry, "_prune_version_directories", fail_prune) + with pytest.raises(SDLParseError, match="simulated cache prune failure"): + module_registry._extract_bundle_to_cache( + bundle_bytes=second_bundle, + manifest_digest="prune-order", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert _current_version(slot) == first.parent + assert first.read_bytes() == b"name: first\n" + + +def test_oci_cache_invalid_archive_leaves_no_partial_entry(tmp_path: Path): + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w:gz") as archive: + good = tarfile.TarInfo(name="module.yaml") + good_payload = b"name: partial\n" + good.size = len(good_payload) + archive.addfile(good, io.BytesIO(good_payload)) + bad = tarfile.TarInfo(name="../escape.yaml") + bad_payload = b"escaped\n" + bad.size = len(bad_payload) + archive.addfile(bad, io.BytesIO(bad_payload)) + bundle = buffer.getvalue() + content_digest = f"sha256:{module_registry._sha256_digest(bundle)}" + + with pytest.raises(SDLParseError, match="Path traversal"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="partial", + content_digest=content_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + cache_root = module_registry._oci_cache_dir(tmp_path) + slot = cache_root / "partial" + assert not (slot / ".raes-current").exists() + assert not list((slot / "versions").iterdir()) + assert not (tmp_path / "escape.yaml").exists() + + +def test_oci_cache_rejects_invalid_manifest_cache_key(tmp_path: Path): + bundle = _cache_test_bundle(b"name: invalid-key\n") + with pytest.raises(SDLParseError, match="cache key"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="../escape", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_cache_repairs_unreadable_tree_manifest(tmp_path: Path): + bundle = _cache_test_bundle(b"name: marker\n") + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="marker", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + (root.parent / ".raes-cache-tree.json").write_bytes(b"\xff") + + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="marker", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired.read_bytes() == b"name: marker\n" + assert repaired.parent != root.parent + manifest = json.loads((repaired.parent / ".raes-cache-tree.json").read_text(encoding="utf-8")) + assert manifest["content_digest"] == digest + + +def test_oci_cache_rejects_invalid_gzip_without_partial_entry(tmp_path: Path): + payload = b"not a gzip tar" + content_digest = f"sha256:{module_registry._sha256_digest(payload)}" + with pytest.raises(SDLParseError, match="not a valid gzip-compressed tar archive"): + module_registry._extract_bundle_to_cache( + bundle_bytes=payload, + manifest_digest="invalid-archive", + content_digest=content_digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + slot = module_registry._oci_cache_dir(tmp_path) / "invalid-archive" + assert not (slot / ".raes-current").exists() + assert not list((slot / "versions").iterdir()) + + +def test_oci_cache_detects_noncommitting_transaction(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + bundle = _cache_test_bundle(b"name: no-commit\n") + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + monkeypatch.setattr( + module_registry, + "_install_version_directory", + lambda *, versions, **kwargs: versions / "missing", + ) + + with pytest.raises(SDLParseError, match="failed validation"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="no-commit", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + versions = module_registry._oci_cache_dir(tmp_path) / "no-commit" / "versions" + assert not list(versions.glob(".staged-*")) + + +@pytest.mark.parametrize("tamper", ["content", "mode", "type", "missing", "extra", "symlink"]) +def test_oci_cache_hit_revalidates_complete_extracted_tree(tmp_path: Path, tamper: str): + bundle = _cache_graph_bundle() + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest=f"tree-{tamper}", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + version = root.parent + child = version / "nested" / "child.yaml" + completion = version / ".raes-cache-tree.json" + retained_completion = completion.read_bytes() + + if tamper == "content": + child.write_bytes(b"name: modified\n") + elif tamper == "mode": + if os.name == "nt": + os.chmod(child, stat.S_IREAD) + else: + os.chmod(child, stat.S_IMODE(child.stat().st_mode) ^ 0o100) + elif tamper == "type": + child.unlink() + child.mkdir() + elif tamper == "missing": + child.unlink() + elif tamper == "extra": + (version / "extra.yaml").write_text("name: extra\n", encoding="utf-8") + else: + outside = tmp_path / "outside.yaml" + outside.write_text("name: outside\n", encoding="utf-8") + child.unlink() + try: + child.symlink_to(outside) + except OSError: + pytest.skip("symlink creation is unavailable") + + assert completion.read_bytes() == retained_completion + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest=f"tree-{tamper}", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired.parent != version + assert (repaired.parent / "nested" / "child.yaml").read_bytes() == b"name: child\n" + manifest = json.loads((repaired.parent / ".raes-cache-tree.json").read_text(encoding="utf-8")) + entries = {entry["path"]: entry for entry in manifest["entries"]} + assert entries["nested"]["type"] == "directory" + assert entries["nested/child.yaml"]["digest"].startswith("sha256:") + assert entries["nested/child.yaml"]["mode"] == (repaired.parent / "nested" / "child.yaml").stat().st_mode & 0o777 + + +@pytest.mark.parametrize("forgery", ["content", "directory-mode"]) +def test_oci_cache_rejects_tree_and_manifest_forged_together(tmp_path: Path, forgery: str): + bundle = _cache_graph_bundle() if forgery == "directory-mode" else _cache_test_bundle(b"name: authentic\n") + digest = f"sha256:{module_registry._sha256_digest(bundle)}" + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="coordinated-forgery", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + forged_version = root.parent + authentic_root = root.read_bytes() + if forgery == "content": + root.write_bytes(b"name: forged\n") + else: + if os.name == "nt": + pytest.skip("Windows does not expose mutable directory permission bits") + directory = forged_version / "nested" + directory.chmod(stat.S_IMODE(directory.stat().st_mode) ^ 0o020) + forged_manifest = module_registry._cache_tree_manifest( + root=forged_version, + content_digest=digest, + root_file="module.yaml", + ) + (forged_version / ".raes-cache-tree.json").write_bytes(module_registry._canonical_json_bytes(forged_manifest)) + + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="coordinated-forgery", + content_digest=digest, + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired.parent != forged_version + assert repaired.read_bytes() == authentic_root + + +@pytest.mark.parametrize( + "corruption", + ["malformed", "deep", "integer", "shape", "noncanonical", "tree-digest", "entry-mode"], +) +def test_oci_cache_rebuilds_for_invalid_completion_manifest(tmp_path: Path, corruption: str): + bundle = _cache_graph_bundle() + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest=f"completion-{corruption}", + root_file="module.yaml", + base_dir=tmp_path, + ) + completion = root.parent / ".raes-cache-tree.json" + manifest = json.loads(completion.read_text(encoding="utf-8")) + if corruption == "malformed": + completion.write_bytes(b"{") + elif corruption == "deep": + completion.write_bytes(b'{"entries":' + b"[" * 10_000 + b"]" * 10_000 + b"}") + elif corruption == "integer": + completion.write_bytes(b'{"integer":' + b"9" * 10_000 + b"}") + elif corruption == "shape": + completion.write_bytes(module_registry._canonical_json_bytes({"schema": "wrong"})) + elif corruption == "noncanonical": + completion.write_text(json.dumps(manifest, indent=2), encoding="utf-8") + elif corruption == "tree-digest": + manifest["tree_digest"] = "sha256:" + "0" * 64 + completion.write_bytes(module_registry._canonical_json_bytes(manifest)) + else: + manifest["entries"][0]["mode"] ^= 0o100 + manifest["tree_digest"] = ( + f"sha256:{module_registry._sha256_digest(module_registry._canonical_json_bytes(manifest['entries']))}" + ) + completion.write_bytes(module_registry._canonical_json_bytes(manifest)) + + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest=f"completion-{corruption}", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired.parent != root.parent + assert repaired.read_bytes().startswith(b"name: root") + + +def test_oci_cache_repairs_missing_pointer_and_crash_residue(tmp_path: Path): + bundle = _cache_test_bundle(b"name: recoverable\n") + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="recover-pointer", + root_file="module.yaml", + base_dir=tmp_path, + ) + slot = _version_slot(root.parent) + (slot / ".raes-current").unlink() + abandoned_version_stage = slot / "versions" / ".staged-abandoned" + abandoned_version_stage.mkdir() + (abandoned_version_stage / "partial").write_text("partial\n", encoding="utf-8") + (slot / ".raes-current.staged-abandoned").write_text("partial\n", encoding="utf-8") + + repaired = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="recover-pointer", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert repaired == root + assert _current_version(slot) == root.parent + assert not abandoned_version_stage.exists() + assert not list(slot.glob(".raes-current.staged-*")) + + +def test_oci_cache_old_reader_remains_valid_while_pointer_advances(tmp_path: Path): + first_bundle = _cache_test_bundle(b"name: first\n") + second_bundle = _cache_test_bundle(b"name: second\n") + first = module_registry._extract_bundle_to_cache( + bundle_bytes=first_bundle, + manifest_digest="reader-writer", + root_file="module.yaml", + base_dir=tmp_path, + ) + barrier = threading.Barrier(2) + + def read_prior() -> bytes: + barrier.wait(timeout=5) + return first.read_bytes() + + def publish_next() -> Path: + barrier.wait(timeout=5) + return module_registry._extract_bundle_to_cache( + bundle_bytes=second_bundle, + manifest_digest="reader-writer", + root_file="module.yaml", + base_dir=tmp_path, + ) + + with ThreadPoolExecutor(max_workers=2) as executor: + prior_future = executor.submit(read_prior) + next_future = executor.submit(publish_next) + assert prior_future.result(timeout=5) == b"name: first\n" + second = next_future.result(timeout=5) + + assert first.read_bytes() == b"name: first\n" + assert second.read_bytes() == b"name: second\n" + assert _current_version(_version_slot(second.parent)) == second.parent + + +def test_oci_cache_version_retention_is_bounded_and_preserves_immediate_prior_reader(tmp_path: Path): + resolved: list[Path] = [] + for revision in range(module_registry_filesystem._MAX_RETAINED_VERSIONS + 3): + bundle = _cache_test_bundle(f"name: revision-{revision}\n".encode()) + resolved.append( + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="bounded-retention", + root_file="module.yaml", + base_dir=tmp_path, + ) + ) + + current = resolved[-1] + prior = resolved[-2] + versions = _version_slot(current.parent) / "versions" + assert len(list(versions.iterdir())) == module_registry_filesystem._MAX_RETAINED_VERSIONS + assert current.read_text(encoding="utf-8") == ( + f"name: revision-{module_registry_filesystem._MAX_RETAINED_VERSIONS + 2}\n" + ) + assert prior.read_text(encoding="utf-8") == ( + f"name: revision-{module_registry_filesystem._MAX_RETAINED_VERSIONS + 1}\n" + ) + assert _current_version(_version_slot(current.parent)) == current.parent + + +def test_oci_cache_tree_inventory_enforces_limits(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + limits = module_registry._OCI_LIMITS + root = tmp_path / "tree" + root.mkdir() + (root / "first").write_bytes(b"1234") + (root / "second").write_bytes(b"5678") + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(limits, max_member_bytes=4, max_total_bytes=7), + ) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(limits, max_bundle_members=-1), + ) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(limits, max_member_bytes=3, max_total_bytes=100), + ) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace( + limits, + max_member_bytes=100, + max_total_bytes=100, + max_bundle_members=1, + ), + ) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + +@pytest.mark.parametrize("last_entry", ["file", "directory"]) +def test_oci_cache_tree_inventory_counts_nested_entries( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + last_entry: str, +): + root = tmp_path / "tree" + (root / "a").mkdir(parents=True) + (root / "a" / "child").write_bytes(b"x") + if last_entry == "file": + (root / "z").write_bytes(b"z") + else: + (root / "z").mkdir() + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_bundle_members=2), + ) + + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + +def test_oci_cache_tree_inventory_rejects_special_file(tmp_path: Path): + if not hasattr(os, "mkfifo"): + pytest.skip("FIFO creation is unavailable") + root = tmp_path / "tree" + root.mkdir() + os.mkfifo(root / "pipe") + + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + +def test_oci_cache_manifest_size_and_reserved_path_fail_closed(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + root = tmp_path / "tree" + root.mkdir() + (root / "module.yaml").write_text("name: module\n", encoding="utf-8") + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_metadata_bytes=1), + ) + with pytest.raises(SDLParseError, match="metadata limit"): + module_registry._write_cache_tree_manifest( + root=root, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + + (root / ".raes-cache-tree.json").write_text("occupied\n", encoding="utf-8") + with pytest.raises(SDLParseError, match="reserved cache metadata"): + module_registry._write_cache_tree_manifest( + root=root, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + + +def test_oci_cache_manifest_io_failures_are_bounded_and_stable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + manifest = tmp_path / "manifest" + manifest.mkdir() + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._read_cache_manifest_bytes(manifest) + + manifest.rmdir() + manifest.write_bytes(b"xx") + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_metadata_bytes=1), + ) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._read_cache_manifest_bytes(manifest) + + root = tmp_path / "tree" + root.mkdir() + (root / "module.yaml").write_text("name: module\n", encoding="utf-8") + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_metadata_bytes=1024), + ) + real_write_bytes = Path.write_bytes + + def fail_manifest_write(path: Path, payload: bytes): + if path.name == ".raes-cache-tree.json": + raise OSError("SECRET-WRITE-DETAIL") + return real_write_bytes(path, payload) + + monkeypatch.setattr(Path, "write_bytes", fail_manifest_write) + with pytest.raises(SDLParseError) as exc_info: + module_registry._write_cache_tree_manifest( + root=root, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + assert str(exc_info.value) == "Unable to write the OCI module cache integrity manifest" + + +def test_oci_cache_manifest_rejects_open_and_identity_races(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + manifest = tmp_path / "manifest" + manifest.write_text("{}\n", encoding="utf-8") + real_open = module_registry.os.open + + def fail_open(path, flags): + if Path(path) == manifest: + raise OSError("SECRET-OPEN-DETAIL") + return real_open(path, flags) + + monkeypatch.setattr(module_registry.os, "open", fail_open) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._read_cache_manifest_bytes(manifest) + + monkeypatch.undo() + directory_stat = tmp_path.stat() + monkeypatch.setattr(module_registry.os, "fstat", lambda descriptor: directory_stat) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._read_cache_manifest_bytes(manifest) + + monkeypatch.undo() + + def fail_fdopen(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-FDOPEN-DETAIL") + + monkeypatch.setattr(module_registry.os, "fdopen", fail_fdopen) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._read_cache_manifest_bytes(manifest) + + +def test_oci_cache_tree_translates_missing_and_scan_failures(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(tmp_path / "missing") + + root = tmp_path / "tree" + root.mkdir() + real_scandir = module_registry.os.scandir + + def fail_scandir(path): + if Path(path) == root: + raise OSError("SECRET-SCAN-DETAIL") + return real_scandir(path) + + monkeypatch.setattr(module_registry.os, "scandir", fail_scandir) + with pytest.raises(SDLParseError) as exc_info: + module_registry._cache_tree_entries(root) + assert str(exc_info.value) == "OCI module cache tree failed integrity validation" + + +@pytest.mark.parametrize("race", ["missing", "mode"]) +def test_oci_cache_tree_detects_directory_race(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, race: str): + root = tmp_path / "tree" + root.mkdir() + original = root.lstat() + calls = 0 + real_lstat = Path.lstat + + def race_lstat(path: Path): + nonlocal calls + if path != root: + return real_lstat(path) + calls += 1 + if calls == 1: + return original + if race == "missing": + raise OSError("SECRET-RACE-DETAIL") + changed = list(original) + changed[0] ^= 0o100 + return os.stat_result(changed) + + monkeypatch.setattr(Path, "lstat", race_lstat) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(root) + + +def test_oci_cache_file_open_and_identity_failures_are_stable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + existing = tmp_path / "existing" + existing.write_text("data\n", encoding="utf-8") + expected = existing.lstat() + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._hash_cache_file(tmp_path / "missing", expected) + + different_values = list(expected) + different_values[2] = expected.st_dev + 1 + different = os.stat_result(different_values) + assert not module_registry._same_file_identity(expected, different) + + +def test_oci_cache_file_hash_detects_fstat_read_and_size_races(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + path = tmp_path / "file" + path.write_bytes(b"1234") + expected = path.lstat() + directory_stat = tmp_path.stat() + monkeypatch.setattr(module_registry.os, "fstat", lambda descriptor: directory_stat) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._hash_cache_file(path, expected) + + monkeypatch.undo() + real_fstat = module_registry.os.fstat + + def fail_fstat(descriptor: int): + del descriptor + raise OSError("SECRET-FSTAT-DETAIL") + + monkeypatch.setattr(module_registry.os, "fstat", fail_fstat) + with pytest.raises(SDLParseError) as exc_info: + module_registry._hash_cache_file(path, expected) + assert str(exc_info.value) == "OCI module cache tree failed integrity validation" + + monkeypatch.undo() + truncated = False + + def truncate_after_stat(descriptor: int): + nonlocal truncated + result = real_fstat(descriptor) + if not truncated: + truncated = True + path.write_bytes(b"") + return result + + monkeypatch.setattr(module_registry.os, "fstat", truncate_after_stat) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._hash_cache_file(path, expected) + + monkeypatch.undo() + + def fail_fdopen(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-FDOPEN-DETAIL") + + monkeypatch.setattr(module_registry.os, "fdopen", fail_fdopen) + expected = path.lstat() + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._hash_cache_file(path, expected) + + +def test_oci_cache_file_hash_detects_growth_after_fstat(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + path = tmp_path / "file" + path.write_bytes(b"1234") + expected = path.lstat() + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_member_bytes=4), + ) + real_fstat = module_registry.os.fstat + grown = False + + def grow_after_stat(descriptor: int): + nonlocal grown + result = real_fstat(descriptor) + if not grown: + grown = True + with path.open("ab") as handle: + handle.write(b"5") + return result + + monkeypatch.setattr(module_registry.os, "fstat", grow_after_stat) + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._hash_cache_file(path, expected) + + +def test_oci_cache_rejects_valid_manifest_without_declared_root(tmp_path: Path): + version = tmp_path / "version" + version.mkdir() + (version / "different.yaml").write_text("name: different\n", encoding="utf-8") + expected_manifest = module_registry._cache_tree_manifest( + root=version, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + module_registry._write_cache_tree_manifest( + root=version, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", ) - assert root_path.is_file() - assert root_path.read_bytes() == payload - cache = module_registry._oci_cache_dir(tmp_path) / "deadbeef" - assert root_path.resolve().is_relative_to(cache.resolve()) - - -@pytest.mark.filterwarnings("ignore::DeprecationWarning") -def test_oci_bundle_fallback_extraction_validates_members(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): - # Simulate Python 3.11.0–3.11.3, where TarFile.extractall lacks the PEP 706 - # `filter` keyword (backported in 3.11.4). The fallback path must still extract - # only validated members rather than performing an unfiltered extraction. - real_extractall = tarfile.TarFile.extractall + assert ( + module_registry._validated_cache_root( + version=version, + expected_manifest=expected_manifest, + root_relative=module_registry.PurePosixPath("module.yaml"), + ) + is None + ) - def no_filter_extractall(self, path=None, members=None, **kwargs): - if "filter" in kwargs: - raise TypeError("extractall() got an unexpected keyword argument 'filter'") - return real_extractall(self, path, members=members) - monkeypatch.setattr(tarfile.TarFile, "extractall", no_filter_extractall) +def test_oci_cache_rejects_invalid_trusted_manifest_shape_and_tree_digest(tmp_path: Path): + version = tmp_path / "version" + version.mkdir() + (version / "module.yaml").write_text("name: module\n", encoding="utf-8") + valid = module_registry._cache_tree_manifest( + root=version, + content_digest="sha256:" + "0" * 64, + root_file="module.yaml", + ) + manifest_path = version / ".raes-cache-tree.json" - payload = b"name: ok\n" - bundle_buffer = io.BytesIO() - with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: - member = tarfile.TarInfo(name="module.yaml") - member.size = len(payload) - tar.addfile(member, io.BytesIO(payload)) - bundle_buffer.seek(0) + assert ( + module_registry._validated_cache_root( + version=version, + expected_manifest={}, + root_relative=module_registry.PurePosixPath("module.yaml"), + ) + is None + ) - root_path = module_registry._extract_bundle_to_cache( - bundle_bytes=bundle_buffer.getvalue(), - manifest_digest="cafef00d", - root_file="module.yaml", - base_dir=tmp_path, + invalid_schema = {**valid, "schema": "wrong"} + manifest_path.write_bytes(module_registry._canonical_json_bytes(invalid_schema)) + assert ( + module_registry._validated_cache_root( + version=version, + expected_manifest=invalid_schema, + root_relative=module_registry.PurePosixPath("module.yaml"), + ) + is None ) - assert root_path.is_file() - assert root_path.read_bytes() == payload + manifest_path.write_bytes(module_registry._canonical_json_bytes(invalid_schema)) + assert ( + module_registry._validated_cache_root( + version=version, + expected_manifest=valid, + root_relative=module_registry.PurePosixPath("module.yaml"), + ) + is None + ) + invalid_digest = {**valid, "tree_digest": "sha256:" + "f" * 64} + manifest_path.write_bytes(module_registry._canonical_json_bytes(invalid_digest)) + assert ( + module_registry._validated_cache_root( + version=version, + expected_manifest=invalid_digest, + root_relative=module_registry.PurePosixPath("module.yaml"), + ) + is None + ) -def test_oci_bundle_fallback_rejects_traversal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): - # The fallback path used on Python 3.11.0–3.11.3 must still reject traversal. - real_extractall = tarfile.TarFile.extractall - def no_filter_extractall(self, path=None, members=None, **kwargs): - if "filter" in kwargs: - raise TypeError("extractall() got an unexpected keyword argument 'filter'") - return real_extractall(self, path, members=members) +@pytest.mark.parametrize( + "entries", + [ + [None], + [{"path": "entry", "type": "special"}], + [{"path": "entry", "type": "file", "digest": "sha256:0", "mode": 0}], + ], +) +def test_oci_cache_trusted_inventory_projection_rejects_invalid_entries(entries: list[object]): + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._trusted_entry_projection(entries) - monkeypatch.setattr(tarfile.TarFile, "extractall", no_filter_extractall) - payload = b"owned\n" - bundle_buffer = io.BytesIO() - with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: - member = tarfile.TarInfo(name="../escape.yaml") - member.size = len(payload) - tar.addfile(member, io.BytesIO(payload)) - bundle_buffer.seek(0) +def test_oci_cache_staging_validation_failure_cleans_stage(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + bundle = _cache_test_bundle(b"name: invalid-stage\n") + monkeypatch.setattr(module_registry, "_validated_cache_root", lambda **kwargs: None) - with pytest.raises(SDLParseError, match="Path traversal detected"): + with pytest.raises(SDLParseError, match="Staged OCI module cache entry failed validation"): module_registry._extract_bundle_to_cache( - bundle_bytes=bundle_buffer.getvalue(), - manifest_digest="badbad", + bundle_bytes=bundle, + manifest_digest="invalid-stage", root_file="module.yaml", base_dir=tmp_path, ) - assert not (tmp_path / "escape.yaml").exists() + versions = module_registry._oci_cache_dir(tmp_path) / "invalid-stage" / "versions" + assert not list(versions.iterdir()) -def test_oci_bundle_rejects_root_file_directory(tmp_path: Path): - bundle_buffer = io.BytesIO() - with tarfile.open(fileobj=bundle_buffer, mode="w:gz") as tar: - directory = tarfile.TarInfo(name="module.yaml") - directory.type = tarfile.DIRTYPE - directory.mode = 0o755 - tar.addfile(directory) - bundle_buffer.seek(0) - with pytest.raises(SDLParseError, match="root file"): +def test_oci_cache_creation_failure_is_stable(tmp_path: Path): + occupied = tmp_path / "occupied" + occupied.write_text("not a directory\n", encoding="utf-8") + bundle = _cache_test_bundle(b"name: blocked\n") + + with pytest.raises(SDLParseError, match="Unable to create the OCI module cache"): module_registry._extract_bundle_to_cache( - bundle_bytes=bundle_buffer.getvalue(), - manifest_digest="d1rd1r", + bundle_bytes=bundle, + manifest_digest="blocked", + root_file="module.yaml", + base_dir=occupied, + ) + + +def test_oci_cache_lock_open_failure_is_stable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + lock_path = tmp_path / "locks" / "entry.lock" + real_open = os.open + + def fail_lock_open(path, *args, **kwargs): + if Path(path) in {lock_path, Path(lock_path.name)}: + raise OSError("SECRET-LOCK-DETAIL") + return real_open(path, *args, **kwargs) + + monkeypatch.setattr(os, "open", fail_lock_open) + + with pytest.raises(SDLParseError) as exc_info, module_registry._cache_entry_lock(lock_path): + pass + + assert str(exc_info.value) == "Unable to open the OCI module cache lock" + + +def test_oci_cache_lock_rejects_identity_and_fdopen_races(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + lock_path = lock_parent / "entry.lock" + lock_path.write_bytes(b"") + directory_stat = lock_parent.stat() + monkeypatch.setattr(module_registry.os, "fstat", lambda descriptor: directory_stat) + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): + module_registry._open_cache_lock(lock_path) + + monkeypatch.undo() + + def fail_fdopen(*args, **kwargs): + del args, kwargs + raise OSError("SECRET-FDOPEN-DETAIL") + + monkeypatch.setattr(module_registry.os, "fdopen", fail_fdopen) + with pytest.raises(SDLParseError) as exc_info: + module_registry._open_cache_lock(lock_path) + assert str(exc_info.value) == "Unable to open the OCI module cache lock" + + +def test_oci_cache_lock_rechecks_parent_type_and_descriptor_identity( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + occupied_parent = tmp_path / "occupied" + occupied_parent.write_bytes(b"") + monkeypatch.setattr(module_registry_cache, "_require_directory", lambda *args, **kwargs: None) + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): + module_registry_cache._open_cache_lock(occupied_parent / "entry.lock") + + monkeypatch.undo() + if not module_registry_cache._LOCK_DIR_FD_SUPPORTED: + pytest.skip("parent descriptor identity is unavailable") + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + monkeypatch.setattr(module_registry_cache.os, "fstat", lambda descriptor: tmp_path.stat()) + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): + module_registry_cache._open_cache_lock(lock_parent / "entry.lock") + + +@pytest.mark.parametrize("linked_component", ["metadata", "cache"]) +def test_oci_cache_rejects_linked_cache_root_without_writing_target(tmp_path: Path, linked_component: str): + outside = tmp_path / "outside-cache" + outside.mkdir() + metadata = tmp_path / ".raes" + try: + if linked_component == "metadata": + metadata.symlink_to(outside, target_is_directory=True) + else: + metadata.mkdir() + (metadata / "module-cache").symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + bundle = _cache_test_bundle(b"name: blocked\n") + with pytest.raises(SDLParseError, match="Unable to create the OCI module cache"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="linked-root", root_file="module.yaml", base_dir=tmp_path, ) + assert not list(outside.iterdir()) -def test_oci_bundle_cache_hit_enforces_root_file_containment(tmp_path: Path): - # Simulate a cache populated by an earlier unsafe extractor: a symlink at the - # root_file location resolving outside the digest cache. The cache-hit fast path - # must still fail closed rather than returning the escaping path. - cache_root = module_registry._oci_cache_dir(tmp_path) / "stale" - cache_root.mkdir(parents=True) - outside = tmp_path / "outside.yaml" - outside.write_text("name: evil\n", encoding="utf-8") - (cache_root / "module.yaml").symlink_to(outside) - with pytest.raises(SDLParseError, match="root file"): +def test_oci_cache_rejects_linked_real_lock_parent_without_writing_target(tmp_path: Path): + cache_root = module_registry._oci_cache_dir(tmp_path) + cache_root.mkdir(parents=True) + outside = tmp_path / "outside-locks" + outside.mkdir() + try: + (cache_root / ".locks").symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + bundle = _cache_test_bundle(b"name: blocked\n") + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): module_registry._extract_bundle_to_cache( - bundle_bytes=b"", - manifest_digest="stale", + bundle_bytes=bundle, + manifest_digest="linked-lock-parent", root_file="module.yaml", base_dir=tmp_path, ) + assert not list(outside.iterdir()) + + +def test_oci_cache_lock_rejects_linked_parent_and_file(tmp_path: Path): + outside = tmp_path / "outside-locks" + outside.mkdir() + linked_parent = tmp_path / "linked-locks" + try: + linked_parent.symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("symlink creation is unavailable") + + with ( + pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"), + module_registry._cache_entry_lock(linked_parent / "entry.lock"), + ): + pytest.fail("a linked lock parent must not be entered") + assert not (outside / "entry.lock").exists() + + real_parent = tmp_path / "real-locks" + real_parent.mkdir() + outside_file = tmp_path / "outside-file" + outside_file.write_bytes(b"") + (real_parent / "entry.lock").symlink_to(outside_file) + with ( + pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"), + module_registry._cache_entry_lock(real_parent / "entry.lock"), + ): + pytest.fail("a linked lock file must not be entered") + assert outside_file.read_bytes() == b"" + + +def test_oci_cache_lock_parent_swap_cannot_escape_anchored_directory( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + if not module_registry_cache._LOCK_DIR_FD_SUPPORTED: + pytest.skip("anchored openat is unavailable") + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + moved_parent = tmp_path / "moved-locks" + outside = tmp_path / "outside-locks" + outside.mkdir() + lock_path = lock_parent / "entry.lock" + real_open = module_registry_cache.os.open + swapped = False + + def swap_parent_before_lock_open(path, flags, *args, **kwargs): + nonlocal swapped + if Path(path) == Path(lock_path.name) and kwargs.get("dir_fd") is not None and not swapped: + swapped = True + lock_parent.rename(moved_parent) + lock_parent.symlink_to(outside, target_is_directory=True) + return real_open(path, flags, *args, **kwargs) + + monkeypatch.setattr(module_registry_cache.os, "open", swap_parent_before_lock_open) + with ( + pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"), + module_registry._cache_entry_lock(lock_path), + ): + pytest.fail("a replaced lock parent must not be entered") + + assert swapped + assert not (outside / "entry.lock").exists() + assert (moved_parent / "entry.lock").is_file() + + +def test_oci_cache_lock_parent_identity_fallback_is_rechecked( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + lock_path = tmp_path / "locks" / "entry.lock" + monkeypatch.setattr(module_registry_cache, "_LOCK_DIR_FD_SUPPORTED", False) + + handle = module_registry_cache._open_cache_lock(lock_path) + handle.close() + + assert lock_path.is_file() + + +@pytest.mark.parametrize("anchored_parent", [True, False]) +def test_oci_cache_lock_reopens_peer_created_regular_file_after_exclusive_race( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + anchored_parent: bool, +): + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + lock_path = lock_parent / "entry.lock" + real_open = module_registry_cache.os.open + collided = False + if not anchored_parent: + monkeypatch.setattr(module_registry_cache, "_LOCK_DIR_FD_SUPPORTED", False) + + def create_as_peer_then_report_collision(path, flags, *args, **kwargs): + nonlocal collided + if flags & os.O_EXCL and not collided: + collided = True + descriptor = real_open(path, flags, *args, **kwargs) + os.close(descriptor) + raise FileExistsError("simulated concurrent lock creation") + return real_open(path, flags, *args, **kwargs) + + monkeypatch.setattr(module_registry_cache.os, "open", create_as_peer_then_report_collision) + + handle = module_registry_cache._open_cache_lock(lock_path) + handle.close() + + assert collided is True + assert lock_path.is_file() + + +def test_oci_cache_lock_rejects_peer_created_link_after_exclusive_race( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + lock_path = lock_parent / "entry.lock" + outside = tmp_path / "outside.lock" + outside.write_bytes(b"") + real_open = module_registry_cache.os.open + collided = False + monkeypatch.setattr(module_registry_cache, "_LOCK_DIR_FD_SUPPORTED", False) + + def create_link_as_peer_then_report_collision(path, flags, *args, **kwargs): + nonlocal collided + if flags & os.O_EXCL and not collided: + collided = True + Path(path).symlink_to(outside) + raise FileExistsError("simulated concurrent linked lock creation") + return real_open(path, flags, *args, **kwargs) + + monkeypatch.setattr(module_registry_cache.os, "open", create_link_as_peer_then_report_collision) + + with pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"): + module_registry_cache._open_cache_lock(lock_path) + + assert collided is True + assert lock_path.is_symlink() + assert outside.read_bytes() == b"" + + +def test_oci_cache_lock_rejects_fifo_without_blocking(tmp_path: Path): + if not hasattr(os, "mkfifo"): + pytest.skip("FIFO creation is unavailable") + lock_parent = tmp_path / "locks" + lock_parent.mkdir() + lock_path = lock_parent / "entry.lock" + os.mkfifo(lock_path) + + with ( + pytest.raises(SDLParseError, match="Unable to open the OCI module cache lock"), + module_registry._cache_entry_lock(lock_path), + ): + pytest.fail("a FIFO lock must not be entered") + + +def test_oci_cache_lock_closes_handle_when_acquisition_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + lock_path = tmp_path / "locks" / "entry.lock" + + def fail_acquire(handle): + assert not handle.closed + raise SDLParseError("simulated acquisition failure") + + monkeypatch.setattr(module_registry, "_acquire_file_lock", fail_acquire) + + with ( + pytest.raises(SDLParseError, match="simulated acquisition failure"), + module_registry._cache_entry_lock(lock_path), + ): + pytest.fail("an unacquired lock must not yield") + + assert lock_path.is_file() + + +def test_oci_cache_windows_lock_backend_is_exercised_without_platform_exclusion( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + calls: list[int] = [] + + class FakeMSVCRT: + LK_NBLCK = 1 + LK_UNLCK = 2 + + @staticmethod + def locking(descriptor: int, operation: int, length: int) -> None: + assert descriptor >= 0 + assert length == 1 + calls.append(operation) + + lock_path = tmp_path / "windows.lock" + lock_path.write_bytes(b"") + monkeypatch.setitem(sys.modules, "msvcrt", FakeMSVCRT) + monkeypatch.setattr(module_registry_cache, "_WINDOWS_LOCKING", True) + + with lock_path.open("r+b") as handle: + module_registry_cache._acquire_file_lock(handle) + module_registry_cache._release_file_lock(handle) + module_registry_cache._acquire_file_lock(handle) + module_registry_cache._release_file_lock(handle) + + assert lock_path.read_bytes() == b"\0" + assert calls == [FakeMSVCRT.LK_NBLCK, FakeMSVCRT.LK_UNLCK, FakeMSVCRT.LK_NBLCK, FakeMSVCRT.LK_UNLCK] + + +def test_oci_cache_lock_timeout_is_bounded(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + if os.name == "nt": + pytest.skip("POSIX lock contention seam") + import fcntl + + times = iter((0.0, 0.5, 2.0)) + sleeps: list[float] = [] + monkeypatch.setattr(module_registry.time, "monotonic", lambda: next(times)) + monkeypatch.setattr(module_registry.time, "sleep", sleeps.append) + monkeypatch.setattr( + module_registry, "_OCI_LIMITS", dataclasses.replace(module_registry._OCI_LIMITS, timeout_seconds=1) + ) + monkeypatch.setattr(fcntl, "flock", lambda *args, **kwargs: (_ for _ in ()).throw(BlockingIOError())) + + with (tmp_path / "lock").open("a+b") as handle, pytest.raises(SDLParseError, match="Timed out"): + module_registry._acquire_file_lock(handle) + + assert sleeps == [0.01] + + +def test_oci_import_composes_verified_root_and_nested_sources_after_cache_replacement( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + root_bytes = b"""name: remote-root +version: 1.0.0 +module: {id: acme/shared, version: 1.0.0, exports: {nodes: [child.vm], infrastructure: [child.vm]}} +imports: + - {source: local:nested/child.yaml, namespace: child} +""" + child_bytes = b"""name: child +version: 1.0.0 +module: {id: acme/child, version: 1.0.0, exports: {nodes: [vm], infrastructure: [vm]}} +nodes: {vm: {type: compute, os: linux, resources: {ram: 1 gib, cpu: 1}}} +infrastructure: {vm: 1} +""" + bundle = _gzip_tar([("module.yaml", root_bytes), ("nested/child.yaml", child_bytes)]).getvalue() + content_digest = f"sha256:{module_registry._sha256_digest(bundle)}" + manifest_digest = "sha256:" + "a" * 64 + descriptor = { + "id": "acme/shared", + "version": "1.0.0", + "exports": {"nodes": ["child.vm"], "infrastructure": ["child.vm"]}, + } + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_manifest", + lambda **_kwargs: (manifest_digest, {}), + ) + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_config", + lambda **_kwargs: ({"module": descriptor, "root_file": "module.yaml"}, content_digest), + ) + monkeypatch.setattr(module_registry_resolution, "_fetch_oci_bundle", lambda **_kwargs: bundle) + real_extract = module_registry._extract_bundle_to_cache + tampered_roots: list[Path] = [] + + def replace_after_capture(**kwargs): + sources = real_extract(**kwargs) + assert isinstance(sources, module_registry._VerifiedSourceBundle) + tampered_roots.append(sources.cache_root) + (sources.cache_root / "module.yaml").write_text("name: replaced-root\n", encoding="utf-8") + (sources.cache_root / "nested" / "child.yaml").write_text("name: replaced-child\n", encoding="utf-8") + for directory in (sources.cache_root, sources.cache_root / "nested"): + (directory / module_registry.TRUST_POLICY_NAME).write_text("not: [valid\n", encoding="utf-8") + (directory / module_registry.LOCKFILE_NAME).write_text("{not-json", encoding="utf-8") + return sources + + monkeypatch.setattr(module_registry, "_extract_bundle_to_cache", replace_after_capture) + _write( + tmp_path / module_registry.TRUST_POLICY_NAME, + """ + schema_version: raes-trust/v1 + registries: + registry.example: {require_signatures: false} + """, + ) + root = _root_import( + tmp_path / "root.yaml", + "source: oci:registry.example/acme/shared\n namespace: remote", + ) + + scenario = parse_sdl_file(root) + + assert set(scenario.nodes) == {"remote.child.vm"} + assert tampered_roots + tampered_child = tampered_roots[0] / "nested" / "child.yaml" + assert tampered_child.read_text(encoding="utf-8") == "name: replaced-child\n" + provenance = {entry.namespace: entry for entry in scenario.expansion_provenance.imports} + assert provenance[("remote",)].content_digest == content_digest + assert provenance[("remote",)].manifest_digest == manifest_digest + assert provenance[("remote", "child")].content_digest == (f"sha256:{module_registry._sha256_digest(child_bytes)}") + @pytest.mark.integration def test_signed_oci_import_resolution_and_publish_cli(tmp_path: Path): @@ -908,12 +3910,328 @@ def _gzip_tar(members: list[tuple[str, bytes]]) -> io.BytesIO: return buffer +@pytest.mark.parametrize("replaced_relative", ["module.yaml", "nested/child.yaml"]) +def test_oci_composition_uses_one_snapshot_after_cache_replacement( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + replaced_relative: str, +) -> None: + bundle = _gzip_tar( + [ + ( + "module.yaml", + b"""name: remote-root +module: {id: acme/remote-root, version: 1.0.0} +imports: + - source: local:nested/child.yaml + namespace: child +""", + ), + ( + "nested/child.yaml", + b"""name: child +module: {id: acme/child, version: 1.0.0} +""", + ), + ] + ).getvalue() + content_digest = f"sha256:{module_registry._sha256_digest(bundle)}" + descriptor = module_registry.ModuleDescriptor(id="acme/remote-root", version="1.0.0") + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_manifest", + lambda **_kwargs: ("sha256:" + "a" * 64, {}), + ) + monkeypatch.setattr( + module_registry_resolution, + "_resolve_oci_config", + lambda **_kwargs: ( + {"module": descriptor.model_dump(mode="python"), "root_file": "module.yaml"}, + content_digest, + ), + ) + monkeypatch.setattr(module_registry_resolution, "_fetch_oci_bundle", lambda **_kwargs: bundle) + + admitted = False + original_extract = module_registry._extract_bundle_to_cache + original_resolve = Path.resolve + original_load_lockfile = composition_expand.load_lockfile + original_load_trust_policy = composition_expand.load_trust_policy + + def replace_after_snapshot(**kwargs): + nonlocal admitted + snapshot = original_extract(**kwargs) + assert isinstance(snapshot, module_registry._VerifiedSourceBundle) + target = snapshot.cache_root.joinpath(*replaced_relative.split("/")) + replacement = target.with_name(f".{target.name}.replacement") + replacement.write_bytes(b"\xff") + os.replace(replacement, target) + admitted = True + return snapshot + + def reject_cache_resolution(path: Path, *args, **kwargs): + if admitted and ".raes" in path.parts: + pytest.fail("admitted OCI cache paths must retain lexical identities") + return original_resolve(path, *args, **kwargs) + + def reject_cache_lockfile_discovery(base_dir: Path): + if admitted and ".raes" in base_dir.parts: + pytest.fail("admitted OCI bundles must not discover cache-local lockfiles") + return original_load_lockfile(base_dir) + + def reject_cache_policy_discovery(base_dir: Path): + if admitted and ".raes" in base_dir.parts: + pytest.fail("admitted OCI bundles must not discover cache-local trust policy") + return original_load_trust_policy(base_dir) + + monkeypatch.setattr(module_registry, "_extract_bundle_to_cache", replace_after_snapshot) + monkeypatch.setattr(Path, "resolve", reject_cache_resolution) + monkeypatch.setattr(composition_expand, "load_lockfile", reject_cache_lockfile_discovery) + monkeypatch.setattr(composition_expand, "load_trust_policy", reject_cache_policy_discovery) + _write( + tmp_path / "raes-trust.yaml", + """ + schema_version: raes-trust/v1 + allow_unsigned_local_sources: false + registries: + registry.example: + require_signatures: false + """, + ) + root = _root_import( + tmp_path / "root.yaml", + "source: oci:registry.example/acme/remote-root\n namespace: remote", + ) + + scenario = parse_sdl_file(root) + + assert admitted + assert len(scenario.expansion_provenance.imports) == 2 + + +def test_nested_oci_import_replaces_the_parent_source_snapshot( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + bundles = { + "acme/parent": _gzip_tar( + [ + ( + "module.yaml", + b"""name: parent +module: {id: acme/parent, version: 1.0.0} +imports: + - source: oci:registry.example/acme/child + namespace: child +""", + ) + ] + ).getvalue(), + "acme/child": _gzip_tar( + [("module.yaml", b"name: child\nmodule: {id: acme/child, version: 1.0.0}\n")] + ).getvalue(), + } + manifest_digests = { + "acme/parent": "sha256:" + "a" * 64, + "acme/child": "sha256:" + "b" * 64, + } + captured_snapshots: list[module_registry._VerifiedSourceBundle] = [] + + def resolve_manifest(*, repository: str, **_kwargs): + return manifest_digests[repository], {"repository": repository} + + def resolve_config(*, manifest: dict[str, str], **_kwargs): + repository = manifest["repository"] + return ( + { + "module": {"id": repository, "version": "1.0.0"}, + "root_file": "module.yaml", + }, + f"sha256:{module_registry._sha256_digest(bundles[repository])}", + ) + + def fetch_bundle(*, repository: str, **_kwargs): + return bundles[repository] + + monkeypatch.setattr(module_registry_resolution, "_resolve_oci_manifest", resolve_manifest) + monkeypatch.setattr(module_registry_resolution, "_resolve_oci_config", resolve_config) + monkeypatch.setattr(module_registry_resolution, "_fetch_oci_bundle", fetch_bundle) + original_extract = module_registry._extract_bundle_to_cache + + def capture_snapshot(**kwargs): + snapshot = original_extract(**kwargs) + assert isinstance(snapshot, module_registry._VerifiedSourceBundle) + captured_snapshots.append(snapshot) + return snapshot + + monkeypatch.setattr(module_registry, "_extract_bundle_to_cache", capture_snapshot) + _write( + tmp_path / "raes-trust.yaml", + """ + schema_version: raes-trust/v1 + registries: + registry.example: + require_signatures: false + """, + ) + root = _root_import( + tmp_path / "root.yaml", + "source: oci:registry.example/acme/parent\n namespace: parent", + ) + + scenario = parse_sdl_file(root) + + assert [record.module_id for record in scenario.expansion_provenance.imports] == ["acme/parent", "acme/child"] + assert len(captured_snapshots) == 2 + assert not captured_snapshots[1].cache_root.is_relative_to(captured_snapshots[0].cache_root) + + +def _gzip_metadata_archive(kind: str) -> tuple[bytes, bytes]: + raw = io.BytesIO() + archive_format = tarfile.PAX_FORMAT if kind == "pax" else tarfile.GNU_FORMAT + with tarfile.open(fileobj=raw, mode="w", format=archive_format) as archive: + info = tarfile.TarInfo(name="module.yaml" if kind == "pax" else f"{'a' * 65536}/module.yaml") + if kind == "pax": + info.pax_headers = {"comment": "x" * 65536} + payload = b"name: metadata\n" + info.size = len(payload) + archive.addfile(info, io.BytesIO(payload)) + decoded = raw.getvalue() + return gzip.compress(decoded, compresslevel=9, mtime=0), decoded + + def test_oci_resource_limits_separate_compressed_from_extracted(): limits = module_registry._OCI_LIMITS # Compressed-download and extracted-archive caps are deliberately distinct so a # small gzip cannot smuggle a large extraction past the download limit. assert limits.max_bundle_bytes > limits.max_metadata_bytes assert limits.max_total_bytes >= limits.max_member_bytes + assert limits.max_tar_stream_bytes > limits.max_total_bytes + assert limits.max_gzip_expansion_ratio > 1 + + +def test_oci_tree_depth_is_bounded_before_extraction_and_during_inventory( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_tree_depth=3), + ) + deep_name = "a/b/c/module.yaml" + bundle = _gzip_tar([(deep_name, b"name: deep\n")]).getvalue() + + with pytest.raises(SDLParseError, match="path-depth limit"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="deep-archive", + root_file=deep_name, + base_dir=tmp_path, + ) + versions = module_registry._oci_cache_dir(tmp_path) / "deep-archive" / "versions" + assert not list(versions.glob(".staged-*")) + + tree = tmp_path / "inventory" + tree.mkdir() + current = tree + for component in ("a", "b", "c", "d"): + current = current / component + current.mkdir() + with pytest.raises(SDLParseError, match="integrity validation"): + module_registry._cache_tree_entries(tree) + + +@pytest.mark.parametrize("metadata_kind", ["pax", "gnu"]) +def test_oci_metadata_bomb_is_bounded_before_tar_parsing( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + metadata_kind: str, +): + bundle, decoded = _gzip_metadata_archive(metadata_kind) + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace( + module_registry._OCI_LIMITS, + max_tar_stream_bytes=len(decoded) - 1, + max_gzip_expansion_ratio=1_000_000, + ), + ) + + def parsed_too_early(*args, **kwargs): + del args, kwargs + pytest.fail("tarfile must not parse metadata before decoded-stream admission") + + monkeypatch.setattr(module_registry.tarfile, "open", parsed_too_early) + + with pytest.raises(SDLParseError, match="uncompressed tar stream"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest=f"metadata-{metadata_kind}", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_gzip_ratio_is_bounded_before_tar_parsing(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + bundle, decoded = _gzip_metadata_archive("pax") + exact_ratio = (len(decoded) + len(bundle) - 1) // len(bundle) + assert exact_ratio > 1 + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace( + module_registry._OCI_LIMITS, + max_tar_stream_bytes=len(decoded) + 1, + max_gzip_expansion_ratio=exact_ratio - 1, + ), + ) + + def parsed_too_early(*args, **kwargs): + del args, kwargs + pytest.fail("tarfile must not parse metadata before gzip-ratio admission") + + monkeypatch.setattr(module_registry.tarfile, "open", parsed_too_early) + + with pytest.raises(SDLParseError, match="expansion"): + module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="metadata-ratio", + root_file="module.yaml", + base_dir=tmp_path, + ) + + +def test_oci_decoded_tar_accepts_exact_absolute_and_ratio_boundaries(monkeypatch: pytest.MonkeyPatch): + bundle, decoded = _gzip_metadata_archive("pax") + exact_ratio = (len(decoded) + len(bundle) - 1) // len(bundle) + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace( + module_registry._OCI_LIMITS, + max_tar_stream_bytes=len(decoded), + max_gzip_expansion_ratio=exact_ratio, + ), + ) + + with module_registry._bounded_gzip_tar_stream(bundle) as admitted: + assert admitted.read() == decoded + + +def test_oci_decoded_tar_rejects_invalid_limit_configuration(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_tar_stream_bytes=-1), + ) + + with ( + pytest.raises(SDLParseError, match="non-negative"), + module_registry._bounded_gzip_tar_stream(b""), + ): + pass def test_oci_metadata_request_rejects_oversized_response(monkeypatch: pytest.MonkeyPatch): @@ -959,6 +4277,16 @@ def test_oci_response_rejects_invalid_content_length(monkeypatch: pytest.MonkeyP module_registry._bytes_request("https://registry.example/v2/acme/blobs/sha256:abc", max_bytes=16) +def test_oci_response_rejects_negative_content_length(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr( + module_registry, + "urlopen", + _fake_urlopen_returning(_FakeResponse(b"x", content_length="-1")), + ) + with pytest.raises(SDLParseError, match="negative Content-Length"): + module_registry._bytes_request("https://registry.example/v2/acme/blobs/sha256:abc", max_bytes=16) + + def test_oci_bundle_rejects_excess_member_count(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): monkeypatch.setattr( module_registry, @@ -973,6 +4301,24 @@ def test_oci_bundle_rejects_excess_member_count(tmp_path: Path, monkeypatch: pyt module_registry._safe_tar_members(tar, tmp_path / "cache") +def test_oci_cache_manifest_does_not_consume_member_budget(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr( + module_registry, + "_OCI_LIMITS", + dataclasses.replace(module_registry._OCI_LIMITS, max_bundle_members=1), + ) + bundle = _gzip_tar([("module.yaml", b"name: bounded\n")]).getvalue() + + root = module_registry._extract_bundle_to_cache( + bundle_bytes=bundle, + manifest_digest="one-member-boundary", + root_file="module.yaml", + base_dir=tmp_path, + ) + + assert root.read_bytes() == b"name: bounded\n" + + def test_oci_bundle_rejects_oversized_member(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): monkeypatch.setattr( module_registry,