diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..cbde56c --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +# Preserve the exact generated bytes whose SHA-256 identities are recorded in +# the EXP1 v1.1 freeze and output manifests, independent of core.autocrlf. +/ephemeral-sandbox-v1/experiments/analysis/exp1-v11-freeze-record-*.json -text +/ephemeral-sandbox-v1/experiments/analysis/final-v11-*-tables-*/** -text diff --git a/.gitignore b/.gitignore index 6e9d6a8..3cc2e0f 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,28 @@ *~ .DS_Store +# Python bytecode and tool caches +__pycache__/ +*.py[cod] +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +.coverage +htmlcov/ + # Paper-local benchmark runtime state and Python environment /ephemeral-sandbox-v1/.benchmark-state/ /ephemeral-sandbox-v1/.venv/ + +# Local EXP1 runtime state and large/reproducible generated artifacts +/ephemeral-sandbox-v1/tmp/ +/ephemeral-sandbox-v1/experiments/runs/ +/ephemeral-sandbox-v1/experiments/diagnostics/ +/ephemeral-sandbox-v1/experiments/exp1-pre-edit-baseline.json + +# Exploratory EXP1 projections and table regenerations. The final-v11 table +# trees and freeze record are intentionally versioned and are not matched here. +/ephemeral-sandbox-v1/experiments/analysis/paper-*-expanded*.json +/ephemeral-sandbox-v1/experiments/analysis/paper-*-expanded*.stderr.txt +/ephemeral-sandbox-v1/experiments/analysis/pilot-*.json +/ephemeral-sandbox-v1/experiments/analysis/pilot-*-tables-*/ diff --git a/ephemeral-sandbox-v1/ARTIFACTS.md b/ephemeral-sandbox-v1/ARTIFACTS.md new file mode 100644 index 0000000..0adc937 --- /dev/null +++ b/ephemeral-sandbox-v1/ARTIFACTS.md @@ -0,0 +1,42 @@ +# Artifact guide + +## Scope + +This artifact package supports a narrow source-and-treatment claim: Ephemeral +Sandbox implements private LayerStack workspace sessions and controlled +filesystem-delta publication; the paper reports a single frozen local treatment +of startup, public-CLI, and selected resource observations. It is not a +comparative benchmark or a general coding-agent productivity evaluation. + +## Immutable evidence + +| Artifact | Location | Integrity anchor | Use | +| --- | --- | --- | --- | +| Eligible experiment archive | `experiments/runs/019fb86c-096e-7589-a0a4-a6d6ef5d7f8b/` | content tree `606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986` | Raw reports, manifests, and run facts. | +| Frozen table output | `experiments/analysis/final-v11-019fb86c-tables-a/` | output tree `27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04` | Source for manuscript displays. | +| Numeric registry | `numeric_evidence.json` | Project-side selector projection of the frozen provenance CSV | Verifies every manuscript number in result tables. | +| Tagged product source | `plan/source_revalidation.md` | `paper-v1.1-freeze` peeled commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` | Source-defined mechanism and interface. | +| Citation lock | `citation_requests.json`, `citation_lock.json` | Terminal arXiv/Crossref metadata checks | Scholarly bibliography provenance. | + +The archive and frozen table output are read-only evidence. The manuscript +generator reads them but never writes inside either directory. + +## Paper-facing outputs + +- `main.tex` and `sections/` — buildable manuscript source. +- `sections/generated_results_tables.tex` — generated LaTeX tables; do not edit + it directly. +- `scripts/generate_latex_results.py` — deterministic table and numeric-registry + projection. +- `scripts/generate_bibliography.py` — bibliography projection from the verified + citation lock. +- `main.pdf` — locally built review PDF. +- `literature/` — inventory, comparison matrix, and claim-positioning record. +- `figures/concept-figure-review.md` — final figure QA and explicit waivers. + +## Deliberate exclusions + +No experiment rerun, new benchmark, external baseline, fault campaign, +security evaluation, or multi-agent productivity study was performed during +paper completion. Qualification, smoke, pilot, earlier protocol, and broader +research-plan material remain excluded from result claims. diff --git a/ephemeral-sandbox-v1/BUILD.md b/ephemeral-sandbox-v1/BUILD.md index 3193bd3..955f50c 100644 --- a/ephemeral-sandbox-v1/BUILD.md +++ b/ephemeral-sandbox-v1/BUILD.md @@ -38,7 +38,10 @@ latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex - PDF: `main.pdf` - Primary LaTeX log: `main.log` -- Skill build-attestation log: `plan/pw0-build-attempt.log` +- PW0 skill build-attestation log: `plan/pw0-build-attempt.log` +- PW1 skill build-attestation log: `plan/pw1-build.log` +- PW2 skill build-attestation log: `plan/pw2-build.log` +- PW3 skill build-attestation log: `plan/pw3-build.log` - Auxiliary files: `main.aux`, `main.bbl`, `main.blg`, `main.fdb_latexmk`, `main.fls`, and `main.out` when produced by the installed toolchain ## PW0 recorded build @@ -52,3 +55,65 @@ The skill build recorder executed the declared command successfully on 2026-07-3 - PDF size and format: 192,209 bytes, four US-letter pages, PDF 1.7. The log contains no build errors, undefined citations or references, missing files, or overfull boxes. It contains five underfull-box warnings in the provisional source-derived cost table and an empty-bibliography warning. The latter produces an intentionally empty References page while `references.bib` remains comment-only. + +## PW1 recorded build + +After drafting Sections 2 and 3, the skill build recorder executed the same declared command successfully on 2026-07-30 at `2026-07-30T00:48:19.855325+00:00`. + +- Toolchain: Latexmk 4.88 (2026-03-09); pdfTeX 3.141592653-2.6-1.40.29 (TeX Live 2026); BibTeX 0.99e (TeX Live 2026). +- Exit code: 0. +- PDF: `main.pdf`, 209,575 bytes and six US-letter pages. +- PDF SHA-256: `ba4963d3d5f6352e1829946290265671599432e9984e301d5626de7316435327`. +- Attested input SHA-256: `8a9d0f97487ccf937efd82eb6a5726b2c7c8b5e3e31be1434945826a00731708`. +- Build log: `plan/pw1-build.log`. +- Build-log SHA-256: `719696c0aa0efb9fce4796efaab3cce5b27dd17563c4b60d7f4951b539aa7f30`. +- Attestation: executed. + +The parsed LaTeX log contains zero errors, emergency stops, undefined citations, undefined references, missing files, or overfull boxes. The five underfull boxes remain confined to the pre-existing provisional cost table in Section 7, and the comment-only bibliography still produces the expected empty-bibliography warning. + +## PW2 recorded build + +After drafting Sections 4 and 5, the skill build recorder executed the same declared command successfully on 2026-07-30 at `2026-07-30T01:10:40.550071+00:00`. + +- Toolchain: Latexmk 4.88 (2026-03-09); pdfTeX 3.141592653-2.6-1.40.29 (TeX Live 2026); BibTeX 0.99e (TeX Live 2026). +- Exit code: 0. +- PDF: `main.pdf`, 221,013 bytes and eight US-letter pages. +- PDF SHA-256: `801ac91c302ae3ea7d5827d34dd4da09278e8f537409e3426fd4ff30c8ed36e7`. +- Attested input SHA-256: `36877891dbce1e066589d8a295e436654ccb097970620003cdaf45871f74311b`. +- Build log: `plan/pw2-build.log`. +- Build-log SHA-256: `9fc4261ab327472e004f4f62466bf2266218d1b17a66cc5b852ee5dbe7b23265`. +- Attestation: executed. + +The final LaTeX pass contains zero errors, emergency stops, undefined citations, undefined references, missing files, or overfull boxes. The five underfull boxes remain confined to the pre-existing provisional cost table in Section 7, and the comment-only bibliography still produces the expected empty-bibliography warning. + +## PW3 recorded build + +After completing Sections 6--7 and integrating the four author-supplied +concept figures, the skill build recorder executed the declared command +successfully on 2026-07-30 at +`2026-07-30T03:22:41.491473+00:00`. + +- Toolchain: Latexmk 4.88 (2026-03-09); pdfTeX 1.40.29 (TeX Live + 2026); BibTeX 0.99e (TeX Live 2026). +- Exit code: 0. +- PDF: `main.pdf`, 4,485,845 bytes and 14 US-letter pages. +- PDF SHA-256: + `bf893a5ac17e396233232b18d552e77ddddfc2bcba50786d457fd58db71604eb`. +- Attested input SHA-256: + `e31aa83e7db9c45cdf0c3c1f731fe66670fc6641dc20d755704a7673a552e18a`. +- Build log: `plan/pw3-build.log`. +- Build-log SHA-256: + `39a836a6226fc9b8007dd23fda5deb9e2cd8bf3c453faf17539c9b356871e59f`. +- Attestation: executed. + +The final LaTeX pass contains zero errors, emergency stops, undefined +citations, undefined references, missing files, or overfull boxes. Seven +underfull boxes occur in the two narrow Section 7 tables, and the comment-only +bibliography produces the expected empty-bibliography warning. + +All 14 pages were rendered at 144 dpi for visual review, with an additional +96 dpi grayscale pass. The four figures appear on pages 3, 6, 8, and 10; both +Section 7 tables remain after their introducing prose. No clipping, overlap, +broken glyph, or page-number defect was found. The figures remain unchanged +drafting-stage review assets, and their recorded topology, resolution, +grayscale-contrast, and style-family disparities remain PW7 work. diff --git a/ephemeral-sandbox-v1/PRD.md b/ephemeral-sandbox-v1/PRD.md index 8315d1e..202a51f 100644 --- a/ephemeral-sandbox-v1/PRD.md +++ b/ephemeral-sandbox-v1/PRD.md @@ -1,166 +1,87 @@ -# Product Requirements Document — *Ephemeral Sandbox v1* Paper - -**Status:** Draft — evidence-gated -**Paper folder:** `ephemeral-sandbox-v1/` -**Initial experimental baseline:** upstream `ephemeral-sandbox` `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5` (2026-07-29) -**Final paper freeze:** Pending — create an annotated tag after paper-specific fixes and benchmark completion; every run must record its exact commit. - -**Primary submission target:** arXiv `cs.OS`; consider `cs.SE` and `cs.AI` cross-lists only after the evaluation supports their relevance. - -## 1. Product objective - -Produce a reproducible, technically precise arXiv preprint that records the v1 design of Ephemeral Sandbox: a runtime for concurrent coding agents that gives each workspace a private copy-on-write execution view of a shared project, then integrates accepted work through conflict-aware atomic publication. - -The paper is a systems-design and evaluation paper. It must let a reader distinguish the implemented v1 mechanisms from hypotheses and from measured results. It is not a journal submission, security certification, or marketing document. - -## 2. Research problem and thesis - -Parallel coding agents need to make and execute changes without sharing mutable working state, while their useful results still need controlled integration into a common project. Ordinary shared worktrees risk interference; fully independent copies make synchronization, storage, and integration more expensive or less controlled. - -**Thesis:** Ephemeral Sandbox realizes concurrent coding-agent workspaces as leased projections of immutable filesystem history. Each workspace mutates a private overlay upper layer and executes in scoped namespaces; capture validates changed paths against a snapshot and atomically publishes a new immutable layer, merging eligible text changes and rejecting incompatible conflicts. - -## 3. Scope - -The paper must document these v1 mechanisms, tied to the eventual final paper snapshot: - -- LayerStack: immutable base, published, and squash layers represented by an active manifest. -- Lease acquisition over a manifest snapshot, with lease lifetime pinning the referenced layers. -- OverlayFS workspaces whose lower layers are shared read-only and whose upper/work directories are private. -- Persistent namespace holders and one-shot runners that enter the holder namespaces for commands and file operations. -- The public CLI contract: separate management, runtime, and read-only observability clients; sandbox-scoped runtime operations; stable request identity; gateway discovery/authentication; JSON output and exit-status behavior; and catalog-derived help. -- Capture and publication: changed-path validation, eligible text three-way merge, rejection of ineligible/binary conflicts, creation of one immutable published layer, and atomic manifest replacement. -- Lease-aware squash/remount behavior, including fail-closed remount gating. -- The implemented layered isolation model and its stated boundaries. - -The paper must explain the design invariants before implementation details: - -1. A workspace observes the manifest snapshot leased at creation. -2. Workspace writes remain private until successful publication. -3. Publication either creates one new durable layer from a validated result or leaves the active manifest unchanged. -4. Compaction cannot reclaim layers still pinned by leases. - -## 4. Explicit non-goals and prohibited claims - -Do **not** claim any of the following without new, directly supporting evidence: - -- a formal security proof, sandbox-escape resistance, or universal network-egress blocking; -- process-state checkpoint/rollback, CRIU-like restore, or DeltaBox-equivalent execution rollback; -- superior speed, scalability, storage savings, cost, or agent productivity; -- Windows reflink support or Windows performance results; -- a general LLM-agent benchmark result based only on scripted runs; -- test success merely because test code exists. - -The current code rejects the `rfc1918_egress=deny` path, so the paper must describe network behavior narrowly and accurately. The isolation discussion is a threat-model and implementation-boundary discussion, not a security evaluation. - -## 5. Readers and success criteria - -Primary readers are operating-systems, software-engineering, and agent-infrastructure researchers, plus engineers building multi-agent coding platforms. - -A successful reader should be able to answer: - -- What state is shared, what state is private, and for how long? -- How does a workspace obtain and retain a consistent view? -- What exactly happens when concurrent changes overlap? -- What is durable after publication, and what occurs on rejection or cleanup? -- Which claims have been measured on which platform and configuration? - -## 6. Required paper structure - -1. **Introduction** — problem, contribution, and precise thesis. -2. **Goals, non-goals, and threat model** — concurrency, durability, and containment boundaries. -3. **System model and invariants** — LayerStack, manifests, leases, workspace state. -4. **Workspace execution** — OverlayFS projection, private writes, holder/runner namespaces. -5. **Capture and publication** — optimistic validation, merge/reject policy, atomic manifest update. -6. **Lifecycle and recovery** — cleanup, squash, and lease-aware remount. -7. **Implementation and CLI boundary** — v1 component map plus the management, runtime, and observability client contracts. -8. **Evaluation** — correctness first; performance only after reproducible measurements. -9. **Limitations and related work** — clear distinctions from DeltaBox, Shepherd, and AgentBay. -10. **Conclusion** — bounded design contribution and open questions. - -## 7. Evidence and evaluation requirements - -No numeric result may enter the manuscript unless its raw data, command/configuration, environment, source SHA, binary/image digest, and analysis script are retained in this folder or a linked archival artifact. - -### 7.1 Correctness gates - -Run and record tests for: - -- private-write isolation between simultaneous workspaces; -- three isolated namespaces and intended connectivity behavior; -- non-overlapping concurrent publications; -- overlapping changes that exercise merge, conflict rejection, and retry; -- cleanup after destroy/failure and manifest integrity after interrupted paths; -- lease-protected squash and remount behavior; -- CLI contract behavior: role separation, required sandbox scope, request correlation, JSON response/error envelopes, and exit-status classes. - -### 7.2 Performance protocol - -The primary platform should be Linux with cgroup v2. Freeze kernel, container/runtime versions, filesystem, hardware, configuration, and seeds. Compare against an independently provisioned container/worktree-per-agent baseline. - -Sweep at least: - -- agents: 1, 5, 20; -- payload sizes: 4 KiB, 256 KiB, 3 MiB; -- layer depths: 1, 10, 50, 100. - -Report raw samples plus p50/p95/p99 for workspace creation, command execution, publication, squash, and remount. Also report physical disk use, RSS/CPU, throughput, failures, and confidence intervals. Explain baseline setup and any features it cannot match. - -### 7.3 Agent-workload gate - -The historical ten-lane scripted run (482/482 planned public operations in 56.4 seconds) is exploratory only: it predates v1 and lacks source/binary digest. Re-run it with full provenance before describing it as a reproducibility demonstration. Treat an LLM-agent benchmark as a later study requiring fixed model, prompt, tasks, budget, coordinator policy, success rubric, cost accounting, and cleanup checks. - -### 7.4 Known blocker - -The Windows Docker Desktop/WSL 2 reflink experiment failed with `errno=95`. This is a platform limitation to disclose, not a result to generalize. The existing public benchmark preset has no completed data and must not be cited as a measurement. - -## 8. Related-work positioning - -- **DeltaBox** targets high-frequency coupled filesystem and process-state checkpoint/rollback for search and RL. Ephemeral Sandbox v1 instead focuses on long-lived private filesystem workspaces, leases, and durable conflict-aware publication; it does not implement process-state rollback. -- **Shepherd** models reversible agent/environment traces as Git-like commits, forks, and reversions. Ephemeral Sandbox focuses on the workspace-runtime and integration boundary; the two are potentially complementary layers. -- **AgentBay** provides multi-platform cloud sandboxes with human graphical takeover and adaptive streaming. Its architecture is broader in interaction surface; Ephemeral Sandbox concentrates on versioned filesystem workspaces and publish semantics. Do not use AgentBay's hypothetical evaluation data as empirical comparison. - -## 9. Required paper artifacts - -Create and maintain the following under this folder: - -- `paper_story.md` — one-sentence thesis, reader, contribution, and section arc. -- `project_inventory.md` — source paths, v1 snapshot, test locations, experiments, environment facts. -- `claim_evidence_map.md` — every substantive claim mapped to code, measurement, or qualified limitation. -- `cli_contract_matrix.md` — public command families mapped to runtime operations, state effects, and contract tests. -- `main.tex`, `references.bib`, and a reproducible build command. -- `figures/` — architecture diagram, workspace/publication state sequence, and evaluation figures generated from data. -- `experiments/` — immutable run manifests, raw samples, analysis scripts, and result tables. -- `ARTIFACTS.md` — how to reproduce evaluation and locate binaries/images/data. -- `paper.pdf` — generated release candidate only after all claim gates pass. - -## 10. Acceptance criteria - -The paper is ready for arXiv submission only when all conditions hold: - -- The title, abstract, and conclusion make bounded, evidence-supported claims. -- Every design statement links to the final annotated paper tag or a documented architectural decision. -- Every number is reproducible from retained raw data and scripts. -- Evaluation distinguishes implementation tests, scripted demonstrations, and LLM-agent experiments. -- Related-work citations are verified and describe differences without unsupported performance comparisons. -- CLI behavior in the manuscript is cross-checked against source and the versioned CLI reference. -- Limitations include current network-policy boundaries and the Windows reflink failure. -- The LaTeX source compiles in the arXiv-compatible environment and ships without undeclared generated dependencies. -- A final skeptical pass finds no language implying security, performance, or generality beyond the evidence. - -## 11. Milestones - -1. Freeze scope, commit SHA, and architecture inventory. -2. Write `paper_story.md`, inventory, and the claim–evidence map. -3. Implement/run correctness experiments and archive results. -4. Run the baseline comparison and analyze results. -5. Draft figures, tables, and main text around validated evidence. -6. Complete citation, reproducibility, and arXiv build checks. -7. Publish the source/artifact repository; submit the LaTeX source to arXiv; link the arXiv record and artifacts from Hugging Face. - -## 12. Sources reviewed - -- [DeltaBox](https://arxiv.org/pdf/2605.22781) -- [AgentBay](https://arxiv.org/html/2512.04367v1) -- [Shepherd](https://arxiv.org/pdf/2605.10913) -- [Ephemeral Sandbox architecture](https://ephemeral-sandbox.com/architecture) -- [Ephemeral Sandbox CLI reference](https://ephemeral-sandbox.com/docs/cli) +# Product Requirements Document - Ephemeral Sandbox v1 paper + +**Status:** PW4--PW7 scope reconciled on 2026-08-02 +**Target:** arXiv `cs.OS` preprint; cross-lists are an author decision +**Measured product revision:** [`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8), annotated tag `paper-v1.1-freeze` (object `834c84534359f37653fb25ac45304091e82c37a6) +**Eligible quantitative evidence:** only EXP1 v1.1 final `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` + +## Objective + +Produce a reproducible systems preprint about a source-grounded workspace-session +and publication protocol, with a focused, descriptive RQ3 measurement study. +The paper must distinguish implementation evidence, contract tests, and the +completed performance corpus. It is not a security certification, competitive +benchmark, or multi-agent productivity study. + +## Final thesis + +Ephemeral Sandbox provides private executable workspace sessions over leased +LayerStack history and a controlled capture-to-publication path; on one +disclosed Windows/Docker Desktop treatment, EXP1 v1.1 describes end-to-end +public-CLI startup, operation, and selected resource observations without +claiming general performance or correctness. + +## Contribution boundary + +1. A source-grounded LayerStack workspace-session protocol: a sessionless + `exec_command` uses an implicit private session, while explicit sessions + retain private state across selected command/file operations. +2. A source-grounded capture and current-head publication protocol with bounded + text reconciliation, whole-changeset reject behavior, and an active-manifest + data-visibility boundary. +3. A regenerated 8/10/8 role-separated CLI contract at the measured tag, + including endpoint selection and request-ID behavior. +4. A fully archived focused RQ3 campaign describing startup, public-CLI, and + preregistered resource values for the disclosed treatment. + +## Evaluation question + +**EQ1 (completed):** Under the frozen native-Windows/Docker Desktop treatment, +with `product_cli`, a pinned Ubuntu 24.04 image, the `paper-100m` fixture, and +concurrency 1 or 5, what end-to-end CLI startup, operation, and preregistered +resource observations are recorded by EXP1 v1.1? + +The following remain limitations and future evaluation, not answered research +questions: isolation correctness (former RQ1), publication/fault correctness +(former RQ2), competitive/deeper scaling and useful work (former RQ4), and +attribution/restart recovery completeness (former RQ5). + +## Evidence rules + +- Numbers may originate only from the final v1.1 archive through the two + byte-identical output trees and their 153 selector-bound entries. +- Do not use v1.0, qualifier, smoke, pilot, projection, setup, verification, + teardown, partial, simulated, or unavailable values as results. +- Describe source implementation and test existence as source/contract evidence, + never as a measured correctness result. +- Scope every EXP1 statement to its host, Docker Desktop engine, pinned image, + fixture, product/benchmark revisions, public CLI boundary, payloads, and + concurrency 1/5. + +## Claims excluded from this paper + +- superiority, speedup, broad scalability, efficiency, production readiness, + reliability, security, isolation correctness, or multi-agent productivity; +- semantic merge correctness, serializable snapshot isolation, process rollback, + universal egress control, or cross-platform durability; +- measured publication/fault, baseline, useful-work, or recovery behavior; +- Windows reflink support, `O(1)` reflink behavior, or LayerStack 2.0 results. + +## Required limitations + +The final paper preserves the post-commit best-effort attribution boundary, +protected-drop-policy uncertainty, lease/substitution restart gap, +source-derived diff-trace memory risk, Linux/OverlayFS implementation boundary, +Windows/Docker Desktop measurement boundary, unavailable resource metrics, and +the narrow Windows reflink feasibility failure. LayerStack 2.0 is future work. + +## Acceptance criteria + +The preprint can be called submission-ready only when its source package builds +from declared inputs; every displayed number resolves to the frozen registry; +cited keys have terminal citation-lock records and sentence-level support; +figures receive final visual decisions; artifact documentation avoids +local-path-only reproduction; and author names, affiliations, licensing, +category, availability links, and any required AI-assistance disclosure are +provided by the author. Until then, the manuscript remains a buildable draft +with explicit blockers. diff --git a/ephemeral-sandbox-v1/README.md b/ephemeral-sandbox-v1/README.md index e7e1a55..f9dba41 100644 --- a/ephemeral-sandbox-v1/README.md +++ b/ephemeral-sandbox-v1/README.md @@ -1,50 +1,34 @@ -# Ephemeral Sandbox v1 - -Design paper for the v1 implementation of [ephemeral-sandbox](https://github.com/Ephemeral-AI-Lab/Ephemeral-AI-Lab). - -## Project management - -- [PRD](PRD.md) — evidence-gated requirements and submission criteria -- [Progress tracker](progress.md) — current milestones, parallel work lanes, evidence gates, and blockers -- [Paper and work skeleton](paper_skeleton.md) — section plan, research questions, figures/tables, work packages, and dependencies -- [Paper-writing lane](lanes/paper-writing.md) — source-grounded drafting order, evidence contract, and completion gates -- [Experiment lane](lanes/experiments.md) — protocol lock, pilots, freeze, final runs, analysis, and manuscript handoff -- [Focused performance protocol](experiment_inventory.md) — phase gates, - acceptance tracker, workload, metrics, and stopping rules -- [Environment setup](experiments/environment_setup.md) — prebuilt staging, - first-step verification, smoke, and good-pass commands -- [Expected tables](experiments/expected_tables.md) — final schemas and - clearly labeled simulated previews -- [Experiment log](experiments/experiment_log.md) — append-only attempts, - failures, amendments, and dispositions - -## Evidence and story documents - -- [Paper story](paper_story.md) — title, thesis, abstracts, introduction opening, contributions, and claim boundaries -- [Project inventory](project_inventory.md) — source, test, documentation, experiment, and environment evidence -- [Claim–evidence map](claim_evidence_map.md) — intended claims mapped to source, tests, measurements, or limitations -- [Complexity and evolution](complexity_and_evolution.md) — source-derived time/space cost model, scaling risks, limitations, and LayerStack 2.0 targets -- [CLI contract matrix](cli_contract_matrix.md) — source-derived management, runtime, and observability interfaces -- [Related-work audit](references/related_work.md) — verified metadata, citation safety, differentiation, and novelty risks - -## Folder structure - -```text -ephemeral-sandbox-v1/ -├── README.md, PRD.md, progress.md, NEXT_AGENT_PROMPT.md -├── paper_skeleton.md -├── paper_story.md, project_inventory.md -├── claim_evidence_map.md, complexity_and_evolution.md, cli_contract_matrix.md -├── lanes/ # paper-writing and experiment charters -├── main.tex, references.bib # created when manuscript drafting begins -├── sections/ # LaTeX section files -├── references/ # related-work and citation-verification records -├── figures/ # concept sources and generated/result assets -├── experiments/ # protocols, immutable runs, and analysis -├── ARTIFACTS.md -└── paper.pdf # generated only after evidence gates pass -``` - -The required evidence documents remain at the paper-folder root so the PRD, progress tracker, and handoff links stay stable. Large or generated assets belong in the dedicated subdirectories. - -Status: baseline discovery and paper-story work are complete; protocol lock, frozen evidence collection, and LaTeX drafting are next. +# Ephemeral Sandbox paper artifact + +This directory contains the source, evidence ledger, and reproducible PDF for +the Ephemeral Sandbox design paper. The claim is intentionally narrow: the +source implements private LayerStack workspace sessions and a controlled +filesystem-delta publication protocol; one frozen local treatment reports +startup, public-CLI, and selected resource observations. + +## Start here + +- `main.tex` and `main.pdf` — manuscript source and current review build. +- `ARTIFACTS.md` — frozen evidence inventory and integrity anchors. +- `REPRODUCIBILITY.md` — deterministic projection, verification, and build + commands. +- `REVIEWER_GUIDE.md` — claim-to-artifact reading path. +- `SUBMISSION.md` and `submission_readiness.md` — external authoring and + publication gate status. + +## Evidence and provenance + +- `numeric_evidence.json` and `sections/results_numeric_bindings.md` bind each + displayed result number to the immutable Table-A provenance CSV. +- `citation_requests.json`, `citation_lock.json`, and + `citation_verification.md` provide terminal primary-source citation records. +- `plan/source_revalidation.md` and `cli_contract_matrix.md` anchor source + claims to the tagged product snapshot. +- `literature/` records the closest-work audit and the bounded positioning. + +## Status + +The scientific and build review is complete. The package is not yet +submission-ready because the paper owner must supply confirmed author names, +affiliations, intended venue/category, and disclosure text. No experiment rerun +or pull-request merge is part of this paper artifact. diff --git a/ephemeral-sandbox-v1/REPRODUCIBILITY.md b/ephemeral-sandbox-v1/REPRODUCIBILITY.md new file mode 100644 index 0000000..4e0780f --- /dev/null +++ b/ephemeral-sandbox-v1/REPRODUCIBILITY.md @@ -0,0 +1,46 @@ +# Reproducibility instructions + +## Prerequisites + +- Python with the repository's analysis dependencies. +- TinyTeX or another LaTeX installation with `latexmk`, `pdflatex`, and + `bibtex` available on `PATH`. +- Read access to the immutable archive and frozen Table-A output directories + listed in `ARTIFACTS.md`. + +No gateway, Docker daemon, product build, network request, or experiment rerun +is required for the paper projection and build. + +## Rebuild the manuscript projection + +Run from this paper directory: + +```text +python scripts/generate_latex_results.py +python scripts/generate_bibliography.py +python C:\Users\yifan\.codex\skills\ai-research-writing\scripts\check_citations.py main.tex references.bib +python C:\Users\yifan\.codex\skills\ai-research-writing\scripts\check_citation_lock.py . --max-age-days 180 +python C:\Users\yifan\.codex\skills\ai-research-writing\scripts\check_numeric_evidence.py . +latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex +``` + +The numeric checker validates the manuscript-side registry against selectors in +the frozen `numeric-provenance.csv`; it is not a trust-by-copy check. If the +citations must be refreshed, set the local CA-bundle path when necessary and +run the verifier before regenerating the bibliography: + +```text +set SSL_CERT_FILE= +python C:\Users\yifan\.codex\skills\ai-research-writing\scripts\verify_citations.py . +python scripts/generate_bibliography.py +``` + +Refreshing the citation lock changes only bibliographic metadata and requires a +subsequent paper build. It does not authorize any experiment action. + +## Verify frozen evidence + +Use the archive verifier documented in the experiment package, then compare +the resulting content-tree and output-tree hashes against `ARTIFACTS.md`. +The expected Table-A generator sources, table JSON, provenance CSV, and output +manifest are already included in the frozen output directory. diff --git a/ephemeral-sandbox-v1/REVIEWER_GUIDE.md b/ephemeral-sandbox-v1/REVIEWER_GUIDE.md new file mode 100644 index 0000000..d322ac9 --- /dev/null +++ b/ephemeral-sandbox-v1/REVIEWER_GUIDE.md @@ -0,0 +1,24 @@ +# Reviewer guide + +## Fast path + +Read the abstract, Sections 3--5 for the mechanism boundary, Section 8 for +the frozen treatment, and Section 9 for the claim exclusions. Then run the +commands in `REPRODUCIBILITY.md`. + +## What each evidence class supports + +| Question | Evidence to inspect | Supported conclusion | +| --- | --- | --- | +| What is the mechanism? | `plan/source_revalidation.md`, Sections 3--7 | Source-defined private workspace and publication behavior. | +| Which numbers are reportable? | `sections/results_numeric_bindings.md`, `numeric_evidence.json`, frozen Table-A output | Only the rendered local-treatment values. | +| Can values be traced to raw data? | Frozen `numeric-provenance.csv` and archive manifest | Selector-level provenance for the numerical tables. | +| Are citations current? | `citation_requests.json`, `citation_lock.json`, `citation_verification.md` | Terminal metadata and sentence-level support records. | +| Are visuals evidence? | `figures/concept-figure-review.md` | No; they explain the mechanism only. | + +## Negative evidence boundary + +The package does not support comparative performance, a useful-work concurrency +ceiling, semantic merge correctness, crash recovery, security, process-state +rollback, cross-platform behavior, or team productivity. Those omissions are +intentional and are not hidden as unresolved result claims. diff --git a/ephemeral-sandbox-v1/SUBMISSION.md b/ephemeral-sandbox-v1/SUBMISSION.md new file mode 100644 index 0000000..bce8f1b --- /dev/null +++ b/ephemeral-sandbox-v1/SUBMISSION.md @@ -0,0 +1,34 @@ +# Submission handoff + +## Package state + +The manuscript source, results projection, citation lock, bibliography, and +PDF build are complete for editorial review. `main.tex` deliberately uses +`Anonymous authors`: real author names, affiliations, and any required +conflict/disclosure metadata have not been supplied by the paper owner. +Therefore this package is **not submission-ready** and must not be uploaded as +a final non-anonymous arXiv record or submitted to a venue as-is. + +## Before authoring a submission + +- Replace `Anonymous authors` in `main.tex` with the confirmed author list and + affiliations. +- Confirm the intended venue, license, category, acknowledgements, and any + conflict/disclosure text. +- Rebuild, record hashes, rerender the PDF, and rerun every command in + `REPRODUCIBILITY.md`. +- Recheck citations if the lock is older than the requested submission window. +- Confirm that the paper's intentionally narrow evidence boundary remains + acceptable for the selected venue. + +## Do not change without reopening evidence review + +Do not add performance/comparison claims, replace numerical displays, alter +the frozen source/tag references, or use new experiment outputs without a new +claim-evidence review. Do not mutate the archive or frozen Table-A output. + +## Git and hosting + +The paper branch and its draft pull request are review context only. Merging a +paper or product pull request remains a separate user-authorized action; this +campaign does not merge either pull request. diff --git a/ephemeral-sandbox-v1/benchmark/PAPER_ARTIFACT.md b/ephemeral-sandbox-v1/benchmark/PAPER_ARTIFACT.md index 1172c6d..5cc9c52 100644 --- a/ephemeral-sandbox-v1/benchmark/PAPER_ARTIFACT.md +++ b/ephemeral-sandbox-v1/benchmark/PAPER_ARTIFACT.md @@ -1,7 +1,26 @@ # Paper-local benchmark artifact -This directory is a frozen, runnable snapshot of the benchmark implementation -used by the Ephemeral Sandbox paper. +This directory is the paper-local, runnable benchmark implementation used by +the Ephemeral Sandbox paper. EXP1 protocol v1.0 was frozen at paper commit +`eb10c26d1bfd632772baf1bc331c985d0231f52d`; the measured product is anchored +by annotated `paper-v1-freeze` tag object +`0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d`. The sole final attempt failed and +is ineligible, so this freeze supports provenance but no final performance +claim. + +Completed v1.1 work is governed by +[`../experiments/exp1-v1.1-protocol-amendment.md`](../experiments/exp1-v1.1-protocol-amendment.md). +It retains one fresh native CLI subprocess per measured request but replaces +loopback TCP with one isolated Windows named pipe per gateway execution block. +The v1.1 product is clean direct `main` commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, annotated tag object +`834c84534359f37653fb25ac45304091e82c37a6`. Qualification, smoke, pilot, +projection, measurement freeze, sole final, immutable archival, deterministic +analysis, and numeric handoff passed. Final run +`019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` is archive-derived into +`../experiments/analysis/final-v11-019fb86c-tables-a`; its exact identities and +post-freeze Table-1 reader compatibility erratum are in the +[`final Gate report`](../experiments/analysis/exp1-gate0-7-final-report.md). ## Provenance @@ -12,22 +31,154 @@ used by the Ephemeral Sandbox paper. - Import verification: all 171 upstream files matched by SHA-256 immediately after copying -Paper-local additions that are not present in the upstream benchmark subtree: +Paper-local additions that are not present in the imported upstream subtree: -- `PAPER_ARTIFACT.md` -- `defaults/workspace-profiles/paper-100m.yml` -- `presets/paper-env-smoke.yml` -- `presets/paper-good-pass.yml` +- `PAPER_ARTIFACT.md` records the complete paper-local delta and the qualified + execution shape. +- `defaults/workspace-profiles/paper-100m.yml` defines the fixed 100 MiB, + 4,000-file, depth-100 paper workspace. +- `presets/paper-env-smoke.yml`, `presets/paper-pilot.yml`, and + `presets/paper-good-pass.yml` define the 19-cell CLI-only smoke, exploratory + pilot, and final matrices. +- `backend/benchmark_lab/product_cli.py` is the native manager/runtime/ + observability CLI subprocess adapter, including end-to-end timing, exact + executable selection, response validation, cancellation/timeout reaping, + pre-staged content-file transfer, canonical `--gateway-endpoint` routing, + redacted argv, and retained stdout/stderr metadata. +- `backend/benchmark_lab/ipc_qualification.py` implements the preregistered + qualification-only 25,000-invocation native manager-CLI gate, including + fixed identity bindings, Windows event/TCP/process sampling, fail-closed + lifecycle evidence, and deterministic summary/manifest output. +- `backend/tests/conftest.py` supplies a fail-closed Windows symlink helper + which skips only WinError 1314. +- `backend/tests/unit/test_product_cli.py` covers argument vectors, token + redaction, executable routing, JSON and product errors, stderr, timeout, + cancellation, timing, concurrency, canonical endpoint routing, and raw CLI + evidence. +- `backend/tests/unit/test_ipc_qualification.py` covers the fixed qualifier + workload, response and identity gates, sampling cadence, exact resource + bounds, TCP/event failures, manifest retention, cleanup, and CLI exit status. +- `tests/fixtures/golden/artifacts/operation-evidence-v1-squash.json` is the + explicit frozen schema-v1 squash evidence fixture used for compatibility + checks. -Paper-local modifications to the upstream snapshot: +Paper-local modifications to imported upstream files, including the active +v1.1 prequalification worktree delta, are complete as follows: -- `backend/benchmark_lab/fixtures.py` raises the admitted maximum fixture depth - from 64 to the product ceiling of 499 so that the depth-100 paper profile is - valid. -- `backend/tests/unit/test_fixtures.py` verifies admission at depth 100, exact - generated path depth, and rejection beyond the new limit. +- `backend/benchmark_lab/artifacts.py` adds content-addressed, colon-free, + bounded evidence v2 storage, durable batch journal appends, and retained + schema-v1 corpus reads. +- `backend/benchmark_lab/catalog.py` validates Windows `.exe` identities and + falls back from an unavailable catalog exporter to reviewed released-CLI + help probes. +- `backend/benchmark_lab/cli.py` exposes the fail-closed + `qualify-exp1-ipc` command and returns a nonzero status for a failed gate. +- `backend/benchmark_lab/derivation.py` raises the derivation revision and + records the CLI subprocess timing boundary, executable/evidence provenance, + and unavailable-metric reasons required by numeric evidence v2. +- `backend/benchmark_lab/fixtures.py` admits depth 100, uses extended-length + native Windows paths and binary-mode low-level writes, validates the cached + fixture tree and reparse/plain-entry identity before reuse, and materializes + independent native Windows copies through bounded multithreaded Robocopy + with a validated Python fallback while preserving fixed fixture bytes and + identity. +- `backend/benchmark_lab/gateway.py` stages native Windows configuration, + launches isolated released gateways, performs readiness through the manager + CLI for `product_cli`, scopes readiness request IDs by gateway instance, + assigns one unique named-pipe endpoint per v1.1 execution block, preserves + endpoint ownership evidence, passes arbitrary tokens with parser-safe equals + syntax, fails fast on deterministic product rejection, and cleans long owned + paths. +- `backend/benchmark_lab/metadata.py` records native Windows host, product + source, image, daemon, gateway, three CLI executable identities, and the + protocol-visible gateway transport/scope/rotation identity. +- `backend/benchmark_lab/models.py` admits the canonical `product_cli` cohort. +- `backend/benchmark_lab/observability.py` validates daemon self-metric + responses used by resource sampling. +- `backend/benchmark_lab/paths.py` resolves the paper-local source and staged + native Windows binary layout safely. +- `backend/benchmark_lab/planning.py` validates paper-only `product_cli`, + expands the distinct manager-CLI sandbox-create/base-mount cell, applies + `paper-100m` to every cell, derives exact batch/request counts, and persists + the seeded cell permutation for each sequential family execution block. +- `backend/benchmark_lab/product.py` exposes the product-observability and + lifecycle protocol used interchangeably by direct test doubles and the CLI + adapter. +- `backend/benchmark_lab/resource_sampling.py` launches daemon, cgroup, + snapshot, filesystem, process, and host-volume collection on fixed 100 ms + deadlines, permits at most one expensive collection in flight to bound + measurement-induced load, records every missed deadline as explicit + unavailability with scheduled and observed collection start/completion + offsets, and defers ordered durable persistence to the trial boundary. +- `backend/benchmark_lab/runner.py` routes all paper operations, verification, + observability, and cleanup through `product_cli`; measures batch makespan + from the admitted-task barrier release through the maximum validated CLI + response end while excluding evidence persistence; commits setup, + waiting-at-barrier, and all-tasks-ready events as one ordered durable + transaction before release, then batches timestamped in-flight and terminal + states after the operation; pre-stages write payloads; emits deterministic + checks and bounded evidence; distinguishes sandbox create from session + create; records the v1.1 gateway policy; and durably persists each unique + execution-block endpoint before client construction or trial timing. +- `backend/benchmark_lab/safety.py` removes only marked owned trees using safe + native extended-length paths. +- `backend/benchmark_lab/service.py` requires reviewed product catalog/CLI + availability before validation or execution. +- `backend/benchmark_lab/sessions.py` uses the shared product-access protocol + without constructing a direct gateway transport. +- `backend/benchmark_lab/transport.py` exposes bounded product rejection + classification plus strict loopback-TCP/named-pipe endpoint parsing and a + one-attempt Windows named-pipe request path with no fallback. +- `backend/tests/compatibility/test_artifacts.py` and + `backend/tests/unit/test_runner_squash.py` verify schema-v1 compatibility + using the explicit frozen evidence fixture and v2 indexing/download. +- `backend/tests/compatibility/test_reports.py` verifies canonical CSV bytes + independently of Git checkout newline conversion. +- `backend/tests/contract/test_api.py`, `backend/tests/contract/test_planning.py`, + and `backend/tests/contract/test_catalog.py` verify public serialization, + paper-preset rejection/fallback rules, exact 19-cell expansions, request + counts, `paper-100m`, executable probing, and fail-closed catalog behavior. +- `backend/tests/contract/test_zero_rust_guard.py` excludes the prepared + virtual environment while retaining the zero-product-source coupling guard. +- `backend/tests/integration/test_gateway_lifecycle.py` covers native Windows + launch/configuration, CLI readiness, gateway-scoped IDs, leading-hyphen + rejection behavior, per-block named-pipe uniqueness/ownership, cleanup, + stale recovery, and symlink refusal. +- `backend/tests/integration/test_gateway_transport.py` covers real and mocked + named-pipe round trips, concurrent requests, unsafe endpoint rejection, and + absence of TCP fallback. +- `backend/tests/integration/test_runner.py` keeps its fakes aligned with CLI + readiness, daemon sampling, explicit gateway cleanup semantics, synchronized + barrier timing, validated-response makespan, write pre-staging, v1.1 policy + provenance, endpoint uniqueness, and crash-prefix manifest durability. +- `backend/tests/unit/test_exp1_archive.py` verifies explicit protocol version, + unique safe named-pipe provenance, v1.1 freeze tags, and the complete frozen + protocol/analysis/tracker scope. +- `backend/tests/unit/test_exp1_runtime_projection.py` verifies v1.1 protocol + and named-pipe provenance before any final-runtime projection is accepted. +- `backend/tests/unit/test_metadata.py` verifies the recorded named-pipe + transport, local scope, and per-execution-block rotation identity. +- `backend/tests/unit/test_resource_sampling.py` verifies fixed-deadline + collection, deferred ordered persistence, and explicit saturation evidence. +- `backend/tests/unit/test_derivation.py` verifies timing/provenance and + numeric-evidence derivation. +- `backend/tests/unit/test_fixtures.py` verifies depth-100 generation and the + revised admission boundary. +- `backend/tests/unit/test_runner_command.py` verifies the released line-window + command representation and command evidence. +- `backend/tests/unit/test_safety.py` verifies deep Windows cleanup and + fail-closed symlink behavior. +- `defaults/definition-catalog.json` defines the separate sandbox lifecycle + create/base-mount operation and EXP1 timing/evidence semantics. +- `presets/paper-env-smoke.yml` and `presets/paper-good-pass.yml` select only + `product_cli`, retain the fixed 19 cells, and apply `paper-100m`; the final + preset fixes two warmups and 100 measured trials without retry or outlier + removal. +- `web/src/api/types.ts`, `web/src/components/DefaultPlanLauncher.tsx`, and + `web/tests/unit/plan-workflow.test.tsx` expose and verify the `product_cli` + cohort in the plan UI. -All other files copied from the upstream snapshot remain unchanged. +No imported benchmark file outside this list is modified by EXP1. The upstream test repository remains the engineering source of truth. This copy is intentionally frozen so that paper results can be tied to an exact @@ -50,41 +201,71 @@ Generated fixtures, run workspaces, raw observations, and reports are written under `ephemeral-sandbox-v1/.benchmark-state/`. That directory is ignored by Git and must not be treated as benchmark source. -## Install and validate on the Linux benchmark host +## Prepare and validate on the qualified Windows host -From `research-papers/ephemeral-sandbox-v1`: +The selected host is native Windows x64. Docker Desktop supplies the Linux +AMD64 engine and the pinned Ubuntu sandbox image; Ubuntu is not the host. The +released native Windows gateway and manager/runtime/observability CLIs are the +only permitted product access path. -```sh -python3.13 -m venv .venv -. .venv/bin/activate -python -m pip install --upgrade pip -python -m pip install -e "./benchmark[test]" +The benchmark requires Python 3.13 or newer for its own orchestration. Prepare +its virtual environment and dependencies off-clock. No package installation, +build, image pull, or source mutation may occur during a pilot or final +campaign. -sandbox-benchmark validate \ - --test-repository-root "$PWD" \ - --product-root /absolute/path/to/ephemeral-sandbox \ - --product-bin-dir /absolute/path/to/ephemeral-sandbox/target/release \ - --plan paper-env-smoke -``` +The three paper presets select `product_cli`; `direct_client` and `cli_e2e` +remain prohibited. The frozen v1.1 product is local clean `main` at +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`. It retains the earlier +released-CLI, shared-base-cache, and resource-sampling repairs and adds the +no-fallback local-IPC treatment plus TCP endpoint compatibility. Its staged +native Windows package and archive are: + +- `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5c48dae1` +- `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5c48dae1.zip` +- archive SHA-256 + `11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506` + +From `research-papers\ephemeral-sandbox-v1`, the validated Windows command +shape is: -Validation checks the plan and the product catalog but does not establish -performance. Run the smoke campaign before a measured campaign: +```powershell +$paper = 'C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1' +$product = 'C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox' +$productBin = "$product\target\windows-exp1-5c48dae1\bin" -```sh -sandbox-benchmark run \ - --test-repository-root "$PWD" \ - --product-root /absolute/path/to/ephemeral-sandbox \ - --product-bin-dir /absolute/path/to/ephemeral-sandbox/target/release \ +Set-Location -LiteralPath $paper +py -3.13 -m venv .venv +& .\.venv\Scripts\python.exe -m pip install -e '.\benchmark[test]' + +& .\.venv\Scripts\sandbox-benchmark.exe validate ` + --test-repository-root $paper ` + --product-root $product ` + --product-bin-dir $productBin ` --plan paper-env-smoke ``` -The paper smoke and good-pass presets pin the selected Linux AMD64 Ubuntu image -by digest. The final host must pull and independently inspect that exact digest -before collecting numbers intended for the paper. +Validation checks the plan and product catalog but does not establish +performance. A campaign command must not be issued until the preset expands to +`product_cli`, the exact released executables and image digest pass preflight, +and the phase gate in +[`../experiment_inventory.md`](../experiment_inventory.md) authorizes the run. + +The paper presets pin the selected Linux AMD64 Ubuntu image by digest. Docker +Desktop must already contain and independently inspect that exact digest before +any pilot or final measurement begins. + +At the frozen v1.1 revision, validation produces: + +| preset | plan hash | cells | batches | CLI requests | +| --- | --- | ---: | ---: | ---: | +| `paper-env-smoke` | `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` | 19 | 19 | 55 | +| `paper-pilot` | `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` | 19 | 133 | 385 | +| `paper-good-pass` | `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` | 19 | 1,938 | 5,610 | ## Paper base workspace -Use workspace profile `paper-100m` for the paper's depth-stress measurements. -It generates exactly 100 MiB (104,857,600 bytes) across 4,000 deterministic -files with a maximum directory depth of 100. The cached seed is copied into a -clean per-cell workspace before sandbox creation. +Use workspace profile `paper-100m` for every final measured cell. It generates +exactly 100 MiB (104,857,600 bytes) across 4,000 deterministic files with a +maximum directory depth of 100. The cached seed is copied into a clean +benchmark-owned per-cell workspace before sandbox creation; operation-specific +file targets are prepared there outside the primary timing interval. diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-311.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-311.pyc deleted file mode 100644 index 2cff2f0..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-311.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-313.pyc deleted file mode 100644 index e979615..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__init__.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__main__.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__main__.cpython-313.pyc deleted file mode 100644 index 358f2f6..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/__main__.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/api.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/api.cpython-313.pyc deleted file mode 100644 index b3ea38e..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/api.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/artifacts.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/artifacts.cpython-313.pyc deleted file mode 100644 index 1bcc8c3..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/artifacts.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/catalog.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/catalog.cpython-313.pyc deleted file mode 100644 index fd02051..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/catalog.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/cli.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/cli.cpython-313.pyc deleted file mode 100644 index 4615bdd..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/cli.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/comparison.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/comparison.cpython-313.pyc deleted file mode 100644 index 892bb83..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/comparison.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/derivation.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/derivation.cpython-313.pyc deleted file mode 100644 index 34c6230..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/derivation.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-311.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-311.pyc deleted file mode 100644 index 77bb293..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-311.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-313.pyc deleted file mode 100644 index 593ee6a..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/fixtures.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/gateway.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/gateway.cpython-313.pyc deleted file mode 100644 index eeab6fb..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/gateway.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/metadata.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/metadata.cpython-313.pyc deleted file mode 100644 index 5a85f49..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/metadata.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-311.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-311.pyc deleted file mode 100644 index e3bd3d1..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-311.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-313.pyc deleted file mode 100644 index 58c8a42..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/models.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/observability.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/observability.cpython-313.pyc deleted file mode 100644 index 0a8ae19..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/observability.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/paths.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/paths.cpython-313.pyc deleted file mode 100644 index cfd0e01..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/paths.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-311.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-311.pyc deleted file mode 100644 index 02c243f..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-311.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-313.pyc deleted file mode 100644 index befa434..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/planning.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/product.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/product.cpython-313.pyc deleted file mode 100644 index 0936fc3..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/product.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/recovery.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/recovery.cpython-313.pyc deleted file mode 100644 index 36c1b49..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/recovery.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/redaction.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/redaction.cpython-313.pyc deleted file mode 100644 index dd35f7a..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/redaction.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/reports.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/reports.cpython-313.pyc deleted file mode 100644 index 88fa8a9..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/reports.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resource_sampling.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resource_sampling.cpython-313.pyc deleted file mode 100644 index 1475bed..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resource_sampling.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resources.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resources.cpython-313.pyc deleted file mode 100644 index 97d72e8..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/resources.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/runner.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/runner.cpython-313.pyc deleted file mode 100644 index 1caa0e3..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/runner.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/safety.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/safety.cpython-313.pyc deleted file mode 100644 index 87756ef..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/safety.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/service.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/service.cpython-313.pyc deleted file mode 100644 index 4d3e084..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/service.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/sessions.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/sessions.cpython-313.pyc deleted file mode 100644 index 3bcb63b..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/sessions.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/statistics.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/statistics.cpython-313.pyc deleted file mode 100644 index 375db5e..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/statistics.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/transport.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/transport.cpython-313.pyc deleted file mode 100644 index 794a784..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/__pycache__/transport.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/artifacts.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/artifacts.py index c236799..88ab3e9 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/artifacts.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/artifacts.py @@ -18,6 +18,7 @@ MAX_DOWNLOAD_BYTES = 16 * 1024 * 1024 MAX_BOUNDED_EVIDENCE_BYTES = 1024 * 1024 RECOVERY_QUARANTINE_DIRECTORY = ".recovery-quarantine" +_BINARY_FLAG = getattr(os, "O_BINARY", 0) class ArtifactError(ValueError): @@ -105,8 +106,8 @@ class ArtifactSpec: "bounded-evidence", "application/json", "eos_benchmark_operation_evidence", - 1, - frozenset({1}), + 2, + frozenset({1, 2}), ), } @@ -461,14 +462,32 @@ def replace_plain(self, run_id: str, artifact_id: ArtifactId, content: bytes) -> self._replace(self.run_path(run_id) / spec.file_name, content) def append_record(self, run_id: str, artifact_id: ArtifactId, data: dict[str, Any]) -> None: + self.append_records(run_id, artifact_id, [data]) + + def append_records( + self, + run_id: str, + artifact_id: ArtifactId, + records: list[dict[str, Any]], + ) -> None: spec = ARTIFACT_SPECS[artifact_id] if not spec.journal: raise ArtifactError(f"artifact {artifact_id} is not a journal") + if not records: + return path = self.run_path(run_id) / spec.file_name - payload = json.dumps( - _envelope_value(spec, data), separators=(",", ":"), ensure_ascii=False - ).encode() + b"\n" - descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600) + payload = b"".join( + json.dumps( + _envelope_value(spec, data), + separators=(",", ":"), + ensure_ascii=False, + ).encode() + + b"\n" + for data in records + ) + descriptor = os.open( + path, os.O_WRONLY | os.O_CREAT | os.O_APPEND | _BINARY_FLAG, 0o600 + ) try: _write_all(descriptor, payload) os.fsync(descriptor) @@ -517,7 +536,7 @@ def quarantine_partial_tail( raise ArtifactError(f"quarantine content conflict at {destination}") else: self._write_new(destination, tail) - descriptor = os.open(path, os.O_WRONLY) + descriptor = os.open(path, os.O_WRONLY | _BINARY_FLAG) try: os.ftruncate(descriptor, complete_length) os.fsync(descriptor) @@ -544,12 +563,15 @@ def write_trial_evidence( _validate_component(cell_id) _validate_component(trial_id) run = self.run_path(run_id) - evidence = run - for component in ("cells", cell_id, "trials", trial_id, "bounded-evidence"): - evidence /= component - self._ensure_plain_directory(evidence) + evidence = run / "bounded-evidence" + self._ensure_plain_directory(evidence) spec = ARTIFACT_SPECS[ArtifactId.BOUNDED_EVIDENCE] - payload = _envelope_bytes(spec, data) + if "cell_id" in data or "trial_id" in data: + raise ArtifactError("bounded evidence payload duplicates identity fields") + payload = _envelope_bytes( + spec, + {"cell_id": cell_id, "trial_id": trial_id, **data}, + ) if len(payload) > MAX_BOUNDED_EVIDENCE_BYTES: raise ArtifactError("bounded evidence exceeds the byte cap") for secret in forbidden_secrets: @@ -639,7 +661,11 @@ def _replace(self, path: Path, payload: bytes) -> None: def _stage(self, payload: bytes) -> Path: temporary = self._roots.tmp / f"artifact-{os.getpid()}-{secrets.token_hex(12)}.tmp" - descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + descriptor = os.open( + temporary, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) try: _write_all(descriptor, payload) os.fsync(descriptor) @@ -690,11 +716,22 @@ def _reference_for_evidence(self, run: Path, path: Path) -> ArtifactReference: @staticmethod def _evidence_paths(run: Path) -> list[Path]: + found: list[Path] = [] + flat = run / "bounded-evidence" + if flat.exists(): + _require_plain_directory(flat) + for path in sorted(flat.iterdir()): + if ( + path.is_symlink() + or not path.is_file() + or not _EVIDENCE_NAME.fullmatch(path.name) + ): + raise ArtifactError(f"unexpected bounded evidence entry: {path}") + found.append(path) cells = run / "cells" if not cells.exists(): - return [] + return found _require_plain_directory(cells) - found: list[Path] = [] for cell in sorted(cells.iterdir()): _validate_component(cell.name) _require_plain_directory(cell) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/catalog.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/catalog.py index 9203482..8bb32ec 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/catalog.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/catalog.py @@ -14,8 +14,39 @@ CATALOG_EXECUTABLE = "sandbox-catalog-export" +MANAGER_CLI_EXECUTABLE = "sandbox-manager-cli" +RUNTIME_CLI_EXECUTABLE = "sandbox-runtime-cli" +OBSERVABILITY_CLI_EXECUTABLE = "sandbox-observability-cli" MAX_CATALOG_BYTES = 1024 * 1024 MAX_CATALOG_ERROR_BYTES = 16 * 1024 +_REVIEWED_CLI_OPERATIONS = { + MANAGER_CLI_EXECUTABLE: { + "create_sandbox", + "destroy_sandbox", + "inspect_sandbox", + "list_sandboxes", + "squash_layerstacks", + }, + RUNTIME_CLI_EXECUTABLE: { + "create_workspace_session", + "destroy_workspace_session", + "exec_command", + "file_blame", + "file_edit", + "file_read", + "file_write", + "publish_workspace_session", + }, + OBSERVABILITY_CLI_EXECUTABLE: { + "cgroup", + "daemon", + "layerstack", + "resources", + "snapshot", + "topology", + "trace", + }, +} class CatalogError(ValueError): @@ -31,7 +62,7 @@ class CatalogFamily(StrictModel): class CatalogArgument(StrictModel): name: str = Field(min_length=1) - kind: Literal["integer", "json_array", "path", "string"] + kind: Literal["float", "integer", "json_array", "path", "string"] required: bool help: str default: str | None @@ -136,8 +167,48 @@ def export_catalog(roots: BenchmarkRoots, *, timeout_seconds: float = 10.0) -> C ) +def probe_released_cli_operations( + roots: BenchmarkRoots, *, timeout_seconds: float = 10.0 +) -> frozenset[str]: + roots.validate_state() + operations: set[str] = set() + for executable_name, expected in _REVIEWED_CLI_OPERATIONS.items(): + executable = _prebuilt_executable(roots, executable_name) + completed = subprocess.run( + [os.fspath(executable), "help"], + cwd=roots.product_bin_dir.parent, + env={ + name: os.environ[name] + for name in ("PATH", "SystemRoot", "WINDIR") + if name in os.environ + }, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout_seconds, + check=False, + ) + if ( + completed.returncode != 0 + or completed.stderr + or len(completed.stdout) > MAX_CATALOG_BYTES + ): + raise CatalogError( + f"released CLI catalog probe failed: {executable_name}" + ) + text = completed.stdout.decode("utf-8", "strict") + missing = sorted(operation for operation in expected if operation not in text) + if missing: + raise CatalogError( + f"released CLI catalog omitted reviewed operations: {missing}" + ) + operations.update(expected) + return frozenset(operations) + + def _prebuilt_executable(roots: BenchmarkRoots, name: str) -> Path: - path = roots.product_bin_dir / name + file_name = f"{name}.exe" if os.name == "nt" and not name.endswith(".exe") else name + path = roots.product_bin_dir / file_name try: metadata = path.lstat() except OSError as error: diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/cli.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/cli.py index badf595..ce99ab4 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/cli.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/cli.py @@ -4,18 +4,27 @@ import asyncio import ipaddress import json +from collections.abc import Sequence from pathlib import Path -from typing import Any, Sequence +from typing import Any import uvicorn from .api import create_app +from .ipc_qualification import qualify_exp1_ipc from .paths import BenchmarkRoots from .planning import load_preset from .service import CampaignService, ServiceError - -COMMANDS = ("serve", "validate", "run", "compare", "recover", "cleanup") +COMMANDS = ( + "serve", + "validate", + "run", + "compare", + "recover", + "cleanup", + "qualify-exp1-ipc", +) def parser() -> argparse.ArgumentParser: @@ -27,7 +36,9 @@ def parser() -> argparse.ArgumentParser: command.add_argument("--product-root", type=Path, required=True) command.add_argument("--product-bin-dir", type=Path, required=True) if name in {"validate", "run"}: - command.add_argument("--plan", required=True, help="plan YAML path or installed preset id") + command.add_argument( + "--plan", required=True, help="plan YAML path or installed preset id" + ) elif name == "compare": command.add_argument("--reference", required=True) command.add_argument("--candidate", required=True) @@ -64,7 +75,11 @@ def _dispatch(arguments: argparse.Namespace, service: CampaignService) -> int: if not 1 <= arguments.port <= 65535: raise SystemExit("serve port must be between 1 and 65535") dist = _resolve_web_dist(arguments.web_dist, service.roots) - authority = f"[{arguments.host}]:{arguments.port}" if address.version == 6 else f"{arguments.host}:{arguments.port}" + authority = ( + f"[{arguments.host}]:{arguments.port}" + if address.version == 6 + else f"{arguments.host}:{arguments.port}" + ) app = create_app(service, authority=authority, web_dist=dist) uvicorn.run(app, host=arguments.host, port=arguments.port, access_log=False) return 0 @@ -86,7 +101,11 @@ def _dispatch(arguments: argparse.Namespace, service: CampaignService) -> int: _print(asyncio.run(service.run_foreground(plan, expanded["plan_hash"]))) return 0 if arguments.command == "compare": - _print(service.compare(arguments.reference, arguments.candidate, arguments.descriptive_override)) + _print( + service.compare( + arguments.reference, arguments.candidate, arguments.descriptive_override + ) + ) return 0 if arguments.command == "recover": result = asyncio.run(service.recover()) @@ -95,6 +114,10 @@ def _dispatch(arguments: argparse.Namespace, service: CampaignService) -> int: if arguments.command == "cleanup": _print(asyncio.run(service.cleanup(arguments.run_id))) return 0 + if arguments.command == "qualify-exp1-ipc": + result = asyncio.run(qualify_exp1_ipc(service.roots)) + _print(result) + return 0 if result["status"] == "passed" else 2 raise AssertionError(arguments.command) @@ -103,7 +126,9 @@ def _resolve_plan(service: CampaignService, value: str) -> dict[str, Any]: if candidate.exists(): if candidate.is_symlink() or not candidate.is_file(): raise SystemExit("plan path must be a plain file") - return CampaignService._load_plan(candidate.resolve(strict=True)).model_dump(mode="json") + return CampaignService._load_plan(candidate.resolve(strict=True)).model_dump( + mode="json" + ) for path in sorted((service.roots.benchmark_source_root / "presets").glob("*.yml")): preset = load_preset(path) if preset.id == value: diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/derivation.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/derivation.py index fe6b0dc..3252985 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/derivation.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/derivation.py @@ -19,14 +19,16 @@ _FAMILY_LABELS = { + "sandbox_lifecycle": "Sandbox Lifecycle", "command": "Command", "files": "File Operations", "workspace_lifecycle": "Workspace Lifecycle", "layer_stack": "LayerStack", } +_REPORT_DERIVATION_REVISION = 4 _DERIVED = { "batch_makespan_ns": ("Batch makespan", "Barrier release until the last issued product request reaches a terminal response.", "nanoseconds", "mean", "lower_is_preferred", "runner_monotonic_batch_barrier"), - "request_latency_ns": ("Request latency", "One issued product request from send until its final response is decoded.", "nanoseconds", "mean", "lower_is_preferred", "raw_asyncio_socket_monotonic"), + "request_latency_ns": ("Request latency", "One issued product request across the declared client-cohort boundary.", "nanoseconds", "mean", "lower_is_preferred", "client_cohort_monotonic_boundary"), "throughput_ops_s": ("Throughput", "Successful issued product requests divided by batch makespan seconds.", "operations_per_second", "mean", "higher_is_preferred", "successful_requests_per_batch_makespan"), "setup_ns": ("Setup", "Harness setup time outside the primary operation window.", "nanoseconds", "mean", "descriptive_only", "runner_monotonic_lifecycle"), "verify_ns": ("Verification", "Correctness verification time outside the primary operation window.", "nanoseconds", "mean", "descriptive_only", "runner_monotonic_lifecycle"), @@ -42,10 +44,20 @@ "sandbox_block_read_bytes": ("Sandbox block reads", "Sandbox cumulative block-read byte counter delta over the trial window."), "sandbox_block_write_bytes": ("Sandbox block writes", "Sandbox cumulative block-write byte counter delta over the trial window."), "workspace_logical_bytes": ("Workspace logical bytes", "Maximum logical file bytes in the named workspace scope."), - "workspace_allocated_bytes": ("Workspace allocated bytes", "Maximum allocated filesystem bytes in the named workspace scope."), + "workspace_allocated_bytes": ( + "Host fixture allocated bytes", + "Maximum allocated filesystem bytes in the immutable host fixture workspace; " + "this gauge is not a live session/upperdir delta and is explicitly unavailable " + "when host metadata lacks allocated-block counts.", + ), "workspace_file_count": ("Workspace files", "Maximum file count in the workspace scope."), "layerstack_bytes": ("LayerStack bytes", "Maximum allocated LayerStack storage reported by the product."), - "upperdir_bytes": ("Upperdir bytes", "Maximum allocated bytes in live workspace upperdirs."), + "upperdir_bytes": ( + "Live workspace upperdir allocated delta", + "Completed post-operation minus completed pre-operation allocated filesystem " + "bytes, summed across all live workspace-session upperdirs reported completely " + "by the product; unavailable if either boundary is incomplete.", + ), "host_free_bytes": ("Host free space", "Minimum free bytes on the benchmark volume."), } def build_report( @@ -63,7 +75,14 @@ def build_report( operation_defs = {item["id"]: item for item in definitions["operations"]} metric_defs = {item["id"]: item for item in definitions["metrics"]} cells = [ - _cell_report(cell, records_by_cell[cell["cell_id"]], plan, operation_defs[cell["operation_id"]], metric_defs) + _cell_report( + cell, + records_by_cell[cell["cell_id"]], + plan, + operation_defs[cell["operation_id"]], + metric_defs, + environment["client_cohort"], + ) for cell in plan["cells"] ] measured = sum(cell["counts"]["measured_attempted"] for cell in cells) @@ -104,7 +123,9 @@ def build_report( warnings.append({"code": "missing_correctness_observations", "message": "One or more registered checks lacks a measured-trial verdict."}) treatment = environment["treatment"] return BenchmarkReportV4.model_validate({ - "schema_version": 4, "report_derivation_revision": 3, "run_id": run_id, + "schema_version": 4, + "report_derivation_revision": _REPORT_DERIVATION_REVISION, + "run_id": run_id, "state": state, "provisional": state not in {"completed", "failed", "cancelled"}, "correctness_verdict": correctness, "design_counts": design, "research_question": f'How does {plan["canonical_plan"]["name"]} behave under its declared factors?', @@ -126,6 +147,7 @@ def utc_now() -> str: def _cell_report( cell: dict[str, Any], records: list[dict[str, Any]], plan: dict[str, Any], operation_def: dict[str, Any], metric_defs: dict[str, dict[str, Any]], + client_cohort: str, ) -> dict[str, Any]: trials = [record["data"] for record in records if record["record"] == "trial"] measured = {item["trial_id"]: item for item in trials if not item["warmup"]} @@ -135,7 +157,9 @@ def _cell_report( phases = [record["data"] for record in records if record["record"] == "phase"] checks = [record["data"] for record in records if record["record"] == "check" and record["data"]["trial_id"] in measured] operation_evidence = [record["data"] for record in records if record["record"] == "operation" and record["data"]["trial_id"] in measured] - metrics = _derived_metrics(cell["cell_id"], measured, eligible, requests, plan) + metrics = _derived_metrics( + cell["cell_id"], measured, eligible, requests, plan, client_cohort + ) metrics.extend(_resource_metrics(cell["cell_id"], measured, eligible, resources, metric_defs, plan)) metrics.sort(key=lambda item: item["identity"]["id"]) status_counts = Counter(item["status"] for item in measured.values()) @@ -179,7 +203,7 @@ def _cell_report( def _derived_metrics( cell_id: str, measured: dict[str, dict[str, Any]], eligible: set[str], - requests: list[dict[str, Any]], plan: dict[str, Any], + requests: list[dict[str, Any]], plan: dict[str, Any], client_cohort: str, ) -> list[dict[str, Any]]: metrics = [] request_attempts = sum(item.get("request_count", 0) for item in measured.values()) @@ -187,7 +211,7 @@ def _derived_metrics( (item["trial_id"], item["request_id"], item["latency_ns"], None, item["latency_ns"]) for item in requests if item["trial_id"] in eligible and item["status"] == "success" ] - metrics.append(_metric(_derived_identity("request_latency_ns"), request_attempts, request_attempts - len(request_points), request_points, cell_id, plan)) + metrics.append(_metric(_derived_identity("request_latency_ns", client_cohort), request_attempts, request_attempts - len(request_points), request_points, cell_id, plan)) for metric_id, field in (("batch_makespan_ns", "latency_ns"), ("setup_ns", "setup_ns"), ("verify_ns", "verify_ns"), ("teardown_ns", "teardown_ns")): points = [(trial_id, None, item.get(field), None, item.get(field)) for trial_id, item in measured.items() if trial_id in eligible and item.get(field) is not None] metrics.append(_metric(_derived_identity(metric_id), len(measured), len(measured) - len(eligible), points, cell_id, plan)) @@ -217,23 +241,34 @@ def _resource_metrics( if trial_id not in eligible: continue readings = sorted(grouped[(trial_id, metric_id)], key=lambda item: item["monotonic_offset_ns"]) - value, reason = _aggregate_resource(readings, definition["aggregation"]) + value, reason = _aggregate_resource( + readings, + definition["aggregation"], + definition["kind"], + ) points.append((trial_id, None, value, reason, int(value) if value is not None and value.is_integer() else None)) identity = _resource_identity(definition, grouped) output.append(_metric(identity, len(measured), len(measured) - len(eligible), points, cell_id, plan)) return output -def _aggregate_resource(readings: list[dict[str, Any]], aggregation: str) -> tuple[float | None, str | None]: +def _aggregate_resource( + readings: list[dict[str, Any]], + aggregation: str, + kind: str, +) -> tuple[float | None, str | None]: if not readings: return None, "resource observation was not emitted" available = [item for item in readings if item["value"]["availability"] == "available"] if aggregation == "delta": if len(available) < 2 or available[0] is not readings[0] or available[-1] is not readings[-1]: - return None, _unavailable_reason(readings, "counter window lacks available boundary samples") + return None, _unavailable_reason( + readings, + "delta window lacks available boundary samples", + ) first = float(available[0]["value"]["value"]) last = float(available[-1]["value"]["value"]) - if last < first: + if kind == "monotonic_counter" and last < first: return None, "monotonic counter reset during the trial window" return last - first, None if not available: @@ -271,16 +306,32 @@ def _metric( } -def _derived_identity(metric_id: str) -> dict[str, Any]: +def _derived_identity( + metric_id: str, client_cohort: str | None = None +) -> dict[str, Any]: label, help_text, unit, aggregation, direction, source = _DERIVED[metric_id] - return {"id": metric_id, "label": label, "help": help_text, "semantic_revision": 1, "unit": unit, "scope": "operation", "kind": "gauge", "availability": "explicit_unavailable", "aggregation": aggregation, "direction": direction, "source": source, "ratio_scale": True, "report_derivation_revision": 3} + if metric_id == "request_latency_ns": + if client_cohort == "product_cli": + source = "product_cli_subprocess_monotonic_spawn_to_validated_json" + help_text = ( + "One native product CLI invocation from immediately before " + "subprocess creation through exit, pipe collection, and " + "one-line JSON schema validation; evidence persistence is excluded." + ) + else: + source = "raw_asyncio_socket_monotonic" + help_text = ( + "One issued product request from send until its final response " + "is decoded." + ) + return {"id": metric_id, "label": label, "help": help_text, "semantic_revision": 1, "unit": unit, "scope": "operation", "kind": "gauge", "availability": "explicit_unavailable", "aggregation": aggregation, "direction": direction, "source": source, "ratio_scale": True, "report_derivation_revision": _REPORT_DERIVATION_REVISION} def _resource_identity(definition: dict[str, Any], grouped: dict[tuple[str, str], list[dict[str, Any]]]) -> dict[str, Any]: metric_id = definition["id"] label, help_text = _RESOURCE_TEXT[metric_id] source = next((readings[0]["source"] for (trial, identifier), readings in grouped.items() if identifier == metric_id and readings), resource_metric_source(metric_id)) - return {**definition, "label": label, "help": help_text, "source": source, "ratio_scale": True, "report_derivation_revision": 3} + return {**definition, "label": label, "help": help_text, "source": source, "ratio_scale": True, "report_derivation_revision": _REPORT_DERIVATION_REVISION} def _check_summary(definition: dict[str, Any], checks: list[dict[str, Any]]) -> dict[str, Any]: @@ -447,7 +498,75 @@ def _methods(plan: dict[str, Any], definitions: dict[str, Any], environment: dic for identifier, spec in ARTIFACT_SPECS.items() if identifier in PRODUCER_ARTIFACT_IDS } - return {"schema_version": 4, "report_derivation_revision": 3, "artifact_reader_revision": 3, "plan_schema_version": plan["schema_version"], "plan_seed": plan["canonical_plan"]["seed"], "cell_order": "randomized_blocks", "resource_sample_interval_ms": plan["canonical_plan"]["protocol"]["resource_interval_ms"], "design_counts": design, "fixture_generator_revision": 2, "fixture_hashes": fixture_hashes, "producer": {"package": "ephemeralos-benchmark", "version": "0.1.0"}, "artifact_schemas": artifact_schemas, "operation_authorities": authorities, "metric_revisions": [{"metric_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in sorted(definitions["metrics"], key=lambda item: item["id"])], "derived_metric_revisions": [{"metric_id": item, "semantic_revision": 1} for item in _DERIVED], "check_revisions": [{"check_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in checks], "phase_revisions": [{"phase_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in phases], "environment": environment, "raw_time_unit": "nanoseconds", "monotonic_clock": "time.monotonic_ns", "quantile_interpolation": "Hyndman-Fan Type 7", "confidence_interval": "10,000-resample deterministic percentile bootstrap", "bootstrap_resamples": 10_000, "outlier_policy": "Tukey 1.5 IQR flag only; never excluded", "warmup_policy": "warmups retained but excluded from aggregates", "failure_policy": "only successful, verified, cleanup-valid measured trials are reportable", "resource_policy": "explicit unavailable observations; never zero-filled", "comparison_policy": "versioned scientific compatibility before aggregation"} + treatment = environment["treatment"] + product_cli = environment["client_cohort"] == "product_cli" + return { + "schema_version": 4, + "report_derivation_revision": _REPORT_DERIVATION_REVISION, + "artifact_reader_revision": 3, + "plan_schema_version": plan["schema_version"], + "plan_seed": plan["canonical_plan"]["seed"], + "cell_order": "randomized_blocks", + "resource_sample_interval_ms": plan["canonical_plan"]["protocol"]["resource_interval_ms"], + "design_counts": design, + "fixture_generator_revision": 2, + "fixture_hashes": fixture_hashes, + "producer": {"package": "ephemeralos-benchmark", "version": "0.1.0"}, + "artifact_schemas": artifact_schemas, + "operation_authorities": authorities, + "metric_revisions": [{"metric_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in sorted(definitions["metrics"], key=lambda item: item["id"])], + "derived_metric_revisions": [{"metric_id": item, "semantic_revision": 1} for item in _DERIVED], + "check_revisions": [{"check_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in checks], + "phase_revisions": [{"phase_id": item["id"], "semantic_revision": item["semantic_revision"]} for item in phases], + "environment": environment, + "primary_timing_boundary": { + "client_cohort": environment["client_cohort"], + "request_latency_ns": ( + "time.monotonic_ns immediately before native CLI subprocess " + "creation through exit, stdout/stderr collection, and validated " + "one-line JSON response; evidence persistence excluded" + if product_cli + else "time.monotonic_ns around one raw asynchronous gateway request" + ), + "batch_makespan_ns": ( + "time.monotonic_ns from runner barrier release through terminal " + "completion of every issued client-cohort request" + ), + }, + "cli_evidence": ( + { + "directory": "cli-subprocesses", + "per_invocation": [ + "sanitized argv metadata", + "raw stdout", + "raw stderr", + ], + "credential_policy": "gateway authentication tokens are redacted", + } + if product_cli + else None + ), + "executable_identities": { + key: treatment.get(key) + for key in ( + "daemon_binary_hash", + "gateway_binary_hash", + "manager_cli_binary_hash", + "runtime_cli_binary_hash", + "observability_cli_binary_hash", + ) + }, + "raw_time_unit": "nanoseconds", + "monotonic_clock": "time.monotonic_ns", + "quantile_interpolation": "Hyndman-Fan Type 7", + "confidence_interval": "10,000-resample deterministic percentile bootstrap", + "bootstrap_resamples": 10_000, + "outlier_policy": "Tukey 1.5 IQR flag only; never excluded", + "warmup_policy": "warmups retained but excluded from aggregates", + "failure_policy": "only successful, verified, cleanup-valid measured trials are reportable", + "resource_policy": "explicit unavailable observations; never zero-filled", + "comparison_policy": "versioned scientific compatibility before aggregation", + } def _factor_value(value: Any) -> dict[str, Any]: diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/fixtures.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/fixtures.py index dbd1d6e..0d64783 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/fixtures.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/fixtures.py @@ -4,7 +4,11 @@ import json import os import shutil +import subprocess +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass from pathlib import Path +from threading import Lock from typing import Any @@ -13,6 +17,70 @@ class FixtureError(RuntimeError): _MAXIMUM_DEPTH = 499 +_BINARY_FLAG = getattr(os, "O_BINARY", 0) +_FIXTURE_VALIDATION_WORKERS = 8 +_MAXIMUM_MATERIALIZATION_WORKERS = 8 +_ROBOCOPY_TIMEOUT_SECONDS = 300.0 +_ROBOCOPY_OPTIONS = ( + "/E", + "/COPY:D", + "/DCOPY:D", + "/R:0", + "/W:0", + "/MT:4", + "/XJ", + "/NFL", + "/NDL", + "/NJH", + "/NP", +) +_VALIDATED_FIXTURE_CACHES: dict[str, "_ValidatedFixtureCache"] = {} +_VALIDATED_FIXTURE_CACHES_LOCK = Lock() + + +@dataclass(frozen=True) +class _TreeInventory: + directories: frozenset[str] + files: tuple[tuple[str, int], ...] + fingerprints: tuple[tuple[str, int, int, int], ...] + + +@dataclass(frozen=True) +class _ValidatedFixtureCache: + fixture_hash: str + tree_hash: str + manifest_sha256: str + manifest_state: tuple[int, int, int] + inventory: _TreeInventory + + +def native_filesystem_path(path: Path) -> Path: + if os.name != "nt": + return path + absolute = os.path.abspath(os.fspath(path)) + if absolute.startswith("\\\\?\\"): + return Path(absolute) + if absolute.startswith("\\\\"): + return Path(f"\\\\?\\UNC\\{absolute[2:]}") + return Path(f"\\\\?\\{absolute}") + + +def same_native_path(left: str | Path, right: str | Path) -> bool: + left_resolved = os.fspath(Path(left).resolve(strict=True)) + right_resolved = os.fspath(Path(right).resolve(strict=True)) + if os.name != "nt": + return left_resolved == right_resolved + return os.path.normcase( + _without_extended_prefix(left_resolved) + ) == os.path.normcase(_without_extended_prefix(right_resolved)) + + +def _without_extended_prefix(value: str) -> str: + if value.startswith("\\\\?\\UNC\\"): + return f"\\\\{value[8:]}" + if value.startswith("\\\\?\\"): + return value[4:] + return value def workspace_fixture_identity( @@ -44,16 +112,93 @@ def materialize_workspace( be confused with the historical Rust generator's cache format. """ + manifest, cache = prepare_workspace_fixture(fixtures_root, profile, seed) + inventory = _validated_fixture_cache(cache, profile, manifest) + _copy_tree(cache, workspace, inventory) + _verify_materialized_manifest(cache, workspace) + return manifest + + +def materialize_workspaces( + fixtures_root: Path, + workspaces: list[Path] | tuple[Path, ...], + profile: dict[str, Any], + seed: int, + *, + max_workers: int = 4, +) -> dict[str, Any]: + """Copy one validated fixture into distinct fresh workspaces concurrently. + + The source cache receives one authoritative validation before the bounded + copy batch and an inventory revalidation after every worker has stopped. + Every destination is checked for an exact directory/file-size manifest and + for file identity independence from the source and its sibling copies. + """ + + destinations = tuple(workspaces) + if ( + isinstance(max_workers, bool) + or not isinstance(max_workers, int) + or not 1 <= max_workers <= _MAXIMUM_MATERIALIZATION_WORKERS + ): + raise FixtureError("fixture materialization worker limit is invalid") + if not destinations: + raise FixtureError("fixture materialization requires a workspace") + + manifest, cache = prepare_workspace_fixture(fixtures_root, profile, seed) + _validate_batch_destinations(cache, destinations) + inventory = _validated_fixture_cache(cache, profile, manifest) + first_error: FixtureError | OSError | None = None + with ThreadPoolExecutor( + max_workers=min(max_workers, len(destinations)) + ) as executor: + futures = [ + executor.submit( + _materialize_validated_workspace, + cache, + destination, + inventory, + ) + for destination in destinations + ] + for future in futures: + try: + future.result() + except (FixtureError, OSError) as error: + if first_error is None: + first_error = error + + if first_error is None: + try: + _verify_materialized_file_independence(cache, destinations, inventory) + except FixtureError as error: + first_error = error + if _validated_tree_inventory(cache, "fixture cache") != inventory: + raise FixtureError("fixture cache drifted during batch materialization") + if first_error is not None: + raise first_error + return manifest + + +def prepare_workspace_fixture( + fixtures_root: Path, + profile: dict[str, Any], + seed: int, +) -> tuple[dict[str, Any], Path]: fixture = profile.get("fixture") if not isinstance(fixture, dict): raise FixtureError("workspace profile fixture is invalid") file_count, logical_bytes, maximum_depth = _validated_fixture_dimensions(fixture) identity, fixture_hash = workspace_fixture_identity(profile, seed) - cache = fixtures_root / str(profile["id"]) / fixture_hash + cache_name = fixture_hash.replace(":", "-") + cache = fixtures_root / str(profile["id"]) / cache_name manifest_path = cache / "fixture-manifest.json" if cache.exists(): manifest = _read_manifest(manifest_path) - if manifest.get("fixture_hash") != fixture_hash or manifest.get("identity") != identity: + if ( + manifest.get("fixture_hash") != fixture_hash + or manifest.get("identity") != identity + ): raise FixtureError("fixture cache failed identity validation") else: staging = cache.parent / f".{cache.name}.tmp-{os.getpid()}" @@ -75,10 +220,9 @@ def materialize_workspace( staging.rename(cache) _sync_directory(cache.parent) except BaseException: - shutil.rmtree(staging, ignore_errors=True) + shutil.rmtree(native_filesystem_path(staging), ignore_errors=True) raise - _copy_tree(cache, workspace) - return manifest + return manifest, cache def _validated_fixture_dimensions(fixture: dict[str, Any]) -> tuple[int, int, int]: @@ -97,13 +241,15 @@ def _validated_fixture_dimensions(fixture: dict[str, Any]) -> tuple[int, int, in return file_count, logical_bytes, maximum_depth -def _build_tree(root: Path, file_count: int, logical_bytes: int, depth: int, seed: int) -> str: +def _build_tree( + root: Path, file_count: int, logical_bytes: int, depth: int, seed: int +) -> str: quotient, remainder = divmod(logical_bytes, file_count) tree = hashlib.sha256() for index in range(file_count): size = quotient + (index < remainder) relative = _relative_path(index, depth) - path = root / relative + path = native_filesystem_path(root / relative) path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) content_digest = hashlib.sha256() with path.open("xb") as stream: @@ -111,8 +257,12 @@ def _build_tree(root: Path, file_count: int, logical_bytes: int, depth: int, see block_index = 0 while remaining: block_size = min(64 * 1024, remaining) - token = hashlib.sha256(f"{seed}:{index}:{block_index}".encode()).digest() - block = (token * ((64 * 1024 + len(token) - 1) // len(token)))[: 64 * 1024] + token = hashlib.sha256( + f"{seed}:{index}:{block_index}".encode() + ).digest() + block = (token * ((64 * 1024 + len(token) - 1) // len(token)))[ + : 64 * 1024 + ] payload = block[:block_size] stream.write(payload) content_digest.update(payload) @@ -137,21 +287,409 @@ def _relative_path(index: int, maximum_depth: int) -> Path: return path / f"file-{index:08}.bin" -def _copy_tree(source: Path, destination: Path) -> None: +def _copy_tree( + source: Path, + destination: Path, + source_inventory: _TreeInventory | None = None, +) -> _TreeInventory: if any(destination.iterdir()): raise FixtureError("workspace must be empty before fixture materialization") - for path in source.rglob("*"): - relative = path.relative_to(source) - target = destination / relative - if path.is_symlink(): - raise FixtureError("fixture cache contains a symbolic link") - if path.is_dir(): - target.mkdir(mode=0o700, parents=True, exist_ok=True) - elif path.is_file(): - target.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - shutil.copyfile(path, target) - else: - raise FixtureError("fixture cache contains a non-plain entry") + if source_inventory is None: + source_inventory = _validated_tree_inventory(source, "fixture cache") + if _uses_native_windows_copy(): + _copy_tree_with_robocopy(source, destination) + else: + _copy_tree_with_python(source, destination, source_inventory) + if _validated_tree_inventory(source, "fixture cache") != source_inventory: + raise FixtureError("fixture cache drifted during materialization") + _verify_copied_files(destination, source_inventory) + return source_inventory + + +def _validate_batch_destinations(source: Path, destinations: tuple[Path, ...]) -> None: + try: + source_resolved = source.resolve(strict=True) + except OSError as error: + raise FixtureError("fixture cache cannot be resolved safely") from error + resolved: list[Path] = [] + for destination in destinations: + try: + destination_resolved = destination.resolve(strict=True) + except OSError as error: + raise FixtureError("workspace cannot be resolved safely") from error + if _is_link_or_junction(destination) or not destination.is_dir(): + raise FixtureError("workspace must be a plain directory") + try: + if any(destination.iterdir()): + raise FixtureError( + "workspace must be empty before fixture materialization" + ) + except OSError as error: + raise FixtureError("workspace cannot be traversed safely") from error + if ( + destination_resolved == source_resolved + or destination_resolved in source_resolved.parents + or source_resolved in destination_resolved.parents + ): + raise FixtureError("workspace must be independent from fixture cache") + if destination_resolved in resolved: + raise FixtureError("fixture materialization workspaces must be distinct") + resolved.append(destination_resolved) + + +def _materialize_validated_workspace( + source: Path, + destination: Path, + source_inventory: _TreeInventory, +) -> None: + if any(destination.iterdir()): + raise FixtureError("workspace must be empty before fixture materialization") + if _uses_native_windows_copy(): + _copy_tree_with_robocopy(source, destination) + else: + _copy_tree_with_python(source, destination, source_inventory) + destination_inventory = _validated_tree_inventory( + destination, "materialized fixture" + ) + if ( + destination_inventory.directories != source_inventory.directories + or destination_inventory.files != source_inventory.files + ): + raise FixtureError("materialized fixture drifted from the fixture cache") + _verify_materialized_manifest(source, destination) + + +def _verify_materialized_file_independence( + source: Path, + destinations: tuple[Path, ...], + source_inventory: _TreeInventory, +) -> None: + def file_identities( + item: tuple[str, int], + ) -> tuple[tuple[int, int], ...]: + relative, _ = item + paths = [ + native_filesystem_path(root / Path(relative)) + for root in (source, *destinations) + ] + try: + return tuple( + (state.st_dev, state.st_ino) + for state in (path.stat() for path in paths) + ) + except OSError as error: + raise FixtureError( + "materialized fixture identity cannot be verified" + ) from error + + identities: set[tuple[int, int]] = set() + with ThreadPoolExecutor(max_workers=_FIXTURE_VALIDATION_WORKERS) as executor: + for file_identity_group in executor.map( + file_identities, source_inventory.files + ): + for identity in file_identity_group: + if identity in identities: + raise FixtureError("materialized fixture files are not independent") + identities.add(identity) + + +def _validated_tree_inventory(root: Path, label: str) -> _TreeInventory: + if _is_link_or_junction(root): + raise FixtureError(f"{label} contains a symbolic link or junction") + if not root.is_dir(): + raise FixtureError(f"{label} contains a non-plain entry") + native_root = native_filesystem_path(root) + directories = {"."} + files: list[tuple[str, int]] = [] + fingerprints: list[tuple[str, int, int, int]] = [] + + def scan_directory( + item: tuple[str, str], + ) -> tuple[ + list[tuple[str, str]], + list[tuple[str, int]], + list[tuple[str, int, int, int]], + ]: + relative_directory, directory = item + discovered_directories: list[tuple[str, str]] = [] + discovered_files: list[tuple[str, int]] = [] + discovered_fingerprints: list[tuple[str, int, int, int]] = [] + try: + with os.scandir(directory) as entries: + for entry in entries: + relative = ( + entry.name + if relative_directory == "." + else f"{relative_directory}/{entry.name}" + ) + if entry.is_symlink() or ( + hasattr(entry, "is_junction") and entry.is_junction() + ): + raise FixtureError( + f"{label} contains a symbolic link or junction" + ) + if entry.is_dir(follow_symlinks=False): + discovered_directories.append((relative, entry.path)) + elif entry.is_file(follow_symlinks=False): + state = entry.stat(follow_symlinks=False) + discovered_files.append((relative, state.st_size)) + discovered_fingerprints.append( + (relative, state.st_size, state.st_mtime_ns, state.st_ino) + ) + else: + raise FixtureError(f"{label} contains a non-plain entry") + except OSError as error: + raise FixtureError(f"{label} cannot be traversed safely") from error + return ( + discovered_directories, + discovered_files, + discovered_fingerprints, + ) + + pending = [(".", os.fspath(native_root))] + with ThreadPoolExecutor(max_workers=_FIXTURE_VALIDATION_WORKERS) as executor: + while pending: + next_pending: list[tuple[str, str]] = [] + for ( + discovered_directories, + discovered_files, + discovered_fingerprints, + ) in executor.map(scan_directory, pending): + next_pending.extend(discovered_directories) + directories.update(relative for relative, _ in discovered_directories) + files.extend(discovered_files) + fingerprints.extend(discovered_fingerprints) + pending = next_pending + return _TreeInventory( + directories=frozenset(directories), + files=tuple(sorted(files)), + fingerprints=tuple(sorted(fingerprints)), + ) + + +def _verify_copied_files(destination: Path, source_inventory: _TreeInventory) -> None: + if _is_link_or_junction(destination) or not destination.is_dir(): + raise FixtureError("materialized fixture contains a non-plain entry") + + def validate_file(item: tuple[str, int]) -> None: + relative, expected_size = item + path = native_filesystem_path(destination / Path(relative)) + try: + if _is_link_or_junction(path) or not path.is_file(): + raise FixtureError( + "materialized fixture drifted from the fixture cache" + ) + if path.stat().st_size != expected_size: + raise FixtureError( + "materialized fixture drifted from the fixture cache" + ) + except OSError as error: + raise FixtureError( + "materialized fixture drifted from the fixture cache" + ) from error + + with ThreadPoolExecutor(max_workers=_FIXTURE_VALIDATION_WORKERS) as executor: + tuple(executor.map(validate_file, source_inventory.files)) + + +def _is_link_or_junction(path: Path) -> bool: + return path.is_symlink() or (hasattr(path, "is_junction") and path.is_junction()) + + +def _uses_native_windows_copy() -> bool: + return os.name == "nt" + + +def _robocopy_executable() -> Path: + system_root = os.environ.get("SystemRoot") + if not system_root: + raise FixtureError("native Windows fixture copy executable is unavailable") + executable = Path(system_root) / "System32" / "robocopy.exe" + if _is_link_or_junction(executable) or not executable.is_file(): + raise FixtureError("native Windows fixture copy executable is unavailable") + return executable + + +def _copy_tree_with_robocopy(source: Path, destination: Path) -> None: + executable = _robocopy_executable() + arguments = [ + os.fspath(executable), + _without_extended_prefix(os.fspath(source.resolve(strict=True))), + _without_extended_prefix(os.fspath(destination.resolve(strict=True))), + *_ROBOCOPY_OPTIONS, + ] + try: + completed = subprocess.run( + arguments, + shell=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=_ROBOCOPY_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise FixtureError("native Windows fixture copy failed to execute") from error + if completed.stdout is None or completed.stderr is None: + raise FixtureError("native Windows fixture copy output is missing") + if not completed.stdout: + raise FixtureError("native Windows fixture copy summary is missing") + if not 0 <= completed.returncode <= 7: + raise FixtureError( + f"native Windows fixture copy failed with exit code {completed.returncode}" + ) + + +def _copy_tree_with_python( + source: Path, + destination: Path, + inventory: _TreeInventory, +) -> None: + for relative in sorted( + inventory.directories, key=lambda value: (value.count("/"), value) + ): + if relative != ".": + native_filesystem_path(destination / Path(relative)).mkdir( + mode=0o700, parents=True, exist_ok=True + ) + for relative, _ in inventory.files: + path = Path(relative) + shutil.copyfile( + native_filesystem_path(source / path), + native_filesystem_path(destination / path), + ) + + +def _validated_fixture_cache( + source: Path, + profile: dict[str, Any], + manifest: dict[str, Any], +) -> _TreeInventory: + cache_key = os.path.normcase(os.fspath(source.resolve(strict=True))) + inventory = _validated_tree_inventory(source, "fixture cache") + manifest_sha256, manifest_state, manifest_bytes = _manifest_identity(source) + fixture_hash = manifest.get("fixture_hash") + tree_hash = manifest.get("tree_hash") + if not isinstance(fixture_hash, str) or not isinstance(tree_hash, str): + raise FixtureError("fixture manifest identity is invalid") + with _VALIDATED_FIXTURE_CACHES_LOCK: + validated = _VALIDATED_FIXTURE_CACHES.get(cache_key) + if validated is not None: + if validated != _ValidatedFixtureCache( + fixture_hash=fixture_hash, + tree_hash=tree_hash, + manifest_sha256=manifest_sha256, + manifest_state=manifest_state, + inventory=inventory, + ): + raise FixtureError("fixture cache drifted after content validation") + return inventory + + expected_directories, expected_files, expected_payloads = _expected_fixture_layout( + profile, len(manifest_bytes) + ) + if inventory.directories != expected_directories: + raise FixtureError("fixture cache directory identity drifted") + if dict(inventory.files) != expected_files: + raise FixtureError("fixture cache file identity drifted") + if _fixture_tree_hash(source, expected_payloads) != tree_hash: + raise FixtureError("fixture cache content hash drifted") + final_inventory = _validated_tree_inventory(source, "fixture cache") + final_sha256, final_state, final_bytes = _manifest_identity(source) + if ( + final_inventory != inventory + or final_sha256 != manifest_sha256 + or final_state != manifest_state + or final_bytes != manifest_bytes + ): + raise FixtureError("fixture cache drifted during content validation") + validated = _ValidatedFixtureCache( + fixture_hash=fixture_hash, + tree_hash=tree_hash, + manifest_sha256=manifest_sha256, + manifest_state=manifest_state, + inventory=final_inventory, + ) + with _VALIDATED_FIXTURE_CACHES_LOCK: + current = _VALIDATED_FIXTURE_CACHES.get(cache_key) + if current is not None and current != validated: + raise FixtureError("fixture cache validation identity conflicted") + _VALIDATED_FIXTURE_CACHES[cache_key] = validated + return final_inventory + + +def _manifest_identity(source: Path) -> tuple[str, tuple[int, int, int], bytes]: + path = native_filesystem_path(source / "fixture-manifest.json") + try: + before = path.stat() + content = path.read_bytes() + after = path.stat() + except OSError as error: + raise FixtureError("fixture manifest identity cannot be verified") from error + before_state = (before.st_size, before.st_mtime_ns, before.st_ino) + after_state = (after.st_size, after.st_mtime_ns, after.st_ino) + if before_state != after_state: + raise FixtureError("fixture manifest drifted during identity validation") + return hashlib.sha256(content).hexdigest(), after_state, content + + +def _expected_fixture_layout( + profile: dict[str, Any], manifest_size: int +) -> tuple[frozenset[str], dict[str, int], list[tuple[Path, int]]]: + fixture = profile.get("fixture") + if not isinstance(fixture, dict): + raise FixtureError("workspace profile fixture is invalid") + file_count, logical_bytes, maximum_depth = _validated_fixture_dimensions(fixture) + quotient, remainder = divmod(logical_bytes, file_count) + expected_files: dict[str, int] = {} + expected_directories = {"."} + expected_payloads: list[tuple[Path, int]] = [] + for index in range(file_count): + size = quotient + (index < remainder) + relative = _relative_path(index, maximum_depth) + expected_payloads.append((relative, size)) + expected_files[relative.as_posix()] = size + parent = relative.parent + while parent != Path("."): + expected_directories.add(parent.as_posix()) + parent = parent.parent + expected_files["fixture-manifest.json"] = manifest_size + return frozenset(expected_directories), expected_files, expected_payloads + + +def _fixture_tree_hash(root: Path, payloads: list[tuple[Path, int]]) -> str: + def content_digest(item: tuple[Path, int]) -> bytes: + relative, _ = item + digest = hashlib.sha256() + try: + with native_filesystem_path(root / relative).open("rb") as stream: + while block := stream.read(1024 * 1024): + digest.update(block) + except OSError as error: + raise FixtureError("fixture content cannot be verified") from error + return digest.digest() + + tree = hashlib.sha256() + with ThreadPoolExecutor(max_workers=_FIXTURE_VALIDATION_WORKERS) as executor: + digests = executor.map(content_digest, payloads) + for (relative, size), digest in zip(payloads, digests, strict=True): + tree.update(relative.as_posix().encode()) + tree.update(b"\0") + tree.update(size.to_bytes(8, "little")) + tree.update(digest) + return f"sha256:{tree.hexdigest()}" + + +def _verify_materialized_manifest(source: Path, destination: Path) -> None: + source_manifest = native_filesystem_path(source / "fixture-manifest.json") + destination_manifest = native_filesystem_path(destination / "fixture-manifest.json") + try: + source_manifest_bytes = source_manifest.read_bytes() + destination_manifest_bytes = destination_manifest.read_bytes() + except OSError as error: + raise FixtureError("materialized fixture manifest is missing") from error + if destination_manifest_bytes != source_manifest_bytes: + raise FixtureError("materialized fixture manifest content drifted") def _read_manifest(path: Path) -> dict[str, Any]: @@ -166,7 +704,11 @@ def _read_manifest(path: Path) -> dict[str, Any]: def _write_new(manifest_path: Path, value: dict[str, Any]) -> None: payload = json.dumps(value, indent=2, ensure_ascii=False).encode() + b"\n" - descriptor = os.open(manifest_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + descriptor = os.open( + manifest_path, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) try: view = memoryview(payload) while view: @@ -180,6 +722,8 @@ def _write_new(manifest_path: Path, value: dict[str, Any]) -> None: def _sync_directory(path: Path) -> None: + if os.name == "nt": + return descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)) try: os.fsync(descriptor) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/gateway.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/gateway.py index 7bd821b..5dd2b26 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/gateway.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/gateway.py @@ -18,19 +18,29 @@ from .catalog import CatalogError, _prebuilt_executable from .models import OwnedPathMarker, StrictModel -from .paths import BenchmarkRoots, MARKER_NAME, _sync_directory +from .paths import MARKER_NAME, BenchmarkRoots, _sync_directory +from .product import ProductAccessError from .redaction import BoundedLogCapture, LogRecord, SecretRedactor from .safety import OwnershipError, OwnershipLedger -from .transport import GatewayClient, GatewayEndpoint, GatewayError - +from .transport import ( + GatewayClient, + GatewayEndpoint, + GatewayError, + GatewayProductError, +) GATEWAY_EXECUTABLE = "sandbox-gateway" AUTH_ENV = "SANDBOX_GATEWAY_AUTH_TOKEN" SHARED_CACHE_ENV = "EOS_SHARED_BASE_CACHE" -GIT_TOOLCHAIN_ENV = "SANDBOX_GIT_TOOLCHAIN_DIR" +_BINARY_FLAG = getattr(os, "O_BINARY", 0) ALLOWED_ENV = ( "PATH", "HOME", + "USERPROFILE", + "SystemRoot", + "WINDIR", + "TEMP", + "TMP", "TMPDIR", "XDG_RUNTIME_DIR", "DOCKER_HOST", @@ -42,6 +52,14 @@ "SSL_CERT_DIR", ) _SAFE_ID = re.compile(r"^[A-Za-z0-9_.:-]{1,256}$") +_DOCKER_CONTAINER_ID = re.compile(r"^[0-9a-f]{12,64}$") +_DOCKER_VOLUME_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,255}$") +_SHARED_BASE_VOLUME_NAME = re.compile(r"^eos-shared-base-([0-9a-f]{64})$") +_GATEWAY_INSTANCE_LABEL = "eos.gateway_instance_id" +_SHARED_BASE_ROOT_HASH_LABEL = "eos.shared_base.root_hash" +_SHARED_BASE_TARGET_LABEL = "eos.shared_base.target" +_SHARED_BASE_READONLY_LABEL = "eos.shared_base.readonly" +_SHARED_BASE_TARGET = "/eos/layer-stack/base" class GatewayLifecycleError(RuntimeError): @@ -53,15 +71,16 @@ class SandboxList(StrictModel): class OwnerProcess(StrictModel): - schema_version: int = Field(ge=1, le=1) + schema_version: int = Field(ge=1, le=2) pid: int = Field(gt=0) process_group: int = Field(gt=0) process_identity: str = Field(min_length=1, max_length=4096) gateway_executable: str = Field(min_length=1) gateway_binary_sha256: str = Field(pattern=r"^sha256:[0-9a-f]{64}$") gateway_instance_id: str = Field(min_length=1, max_length=256) - endpoint_host: str - endpoint_port: int = Field(ge=1, le=65535) + endpoint_uri: str | None = Field(default=None, min_length=1, max_length=512) + endpoint_host: str | None = None + endpoint_port: int | None = Field(default=None, ge=1, le=65535) @dataclass(frozen=True, slots=True) @@ -76,9 +95,9 @@ class GatewayIdentity: ProcessFactory = Callable[..., Awaitable[Any]] ClientFactory = Callable[[GatewayEndpoint, str], GatewayClient] -KillGroup = Callable[[int, signal.Signals], None] +KillGroup = Callable[[int, int], None] ProcessIdentity = Callable[[int], str | None] -OrphanCleanup = Callable[[str], Awaitable[None]] +OrphanCleanup = Callable[[str, bool], Awaitable[None]] class GatewayLauncher: @@ -88,15 +107,15 @@ def __init__( *, process_factory: ProcessFactory = asyncio.create_subprocess_exec, client_factory: ClientFactory = GatewayClient, - kill_group: KillGroup = os.killpg, - process_identity: ProcessIdentity = lambda pid: _process_identity(pid), + kill_group: KillGroup | None = None, + process_identity: ProcessIdentity | None = None, orphan_cleanup: OrphanCleanup | None = None, ) -> None: self._roots = roots self._process_factory = process_factory self._client_factory = client_factory - self._kill_group = kill_group - self._process_identity = process_identity + self._kill_group = kill_group or _default_kill_group + self._process_identity = process_identity or _process_identity self._orphan_cleanup = orphan_cleanup or _cleanup_docker_resources async def start( @@ -105,6 +124,7 @@ async def start( *, remount_sweep_width: int = 1, readiness_timeout_seconds: float = 60, + readiness_via_cli: bool = False, ) -> "IsolatedGateway": _validate_identity(run_id) if not 1 <= remount_sweep_width <= 1024: @@ -114,9 +134,10 @@ async def start( try: gateway_binary = _prebuilt_executable(self._roots, GATEWAY_EXECUTABLE) daemon_binary = _container_daemon_executable(self._roots) - git_toolchains = _fixed_git_toolchains(self._roots) except CatalogError as error: - raise GatewayLifecycleError("gateway preflight rejected a product binary") from error + raise GatewayLifecycleError( + "gateway preflight rejected a product binary" + ) from error ledger = OwnershipLedger(self._roots) runtime_path = self._roots.runtime / run_id @@ -126,7 +147,9 @@ async def start( ledger.register(runtime_path, runtime_marker) except (OSError, OwnershipError) as error: _cleanup_failed_registration(runtime_path, ledger, runtime_marker) - raise GatewayLifecycleError("isolated runtime ownership setup failed") from error + raise GatewayLifecycleError( + "isolated runtime ownership setup failed" + ) from error process: Any | None = None log_tasks: tuple[asyncio.Task[None], asyncio.Task[None]] | None = None @@ -136,9 +159,8 @@ async def start( capture: BoundedLogCapture | None = None gateway_instance_id: str | None = None try: - shared_cache = runtime_path / "shared-base-cache" - shared_cache.mkdir(mode=0o700) - endpoint = _reserve_loopback_endpoint() + shared_cache = _run_shared_cache(self._roots, run_id) + endpoint = _execution_block_endpoint(readiness_via_cli) gateway_instance_id = f"benchmark-gateway-{secrets.token_hex(16)}" config_path = runtime_path / "effective-config.yml" pid_path = runtime_path / "gateway.pid" @@ -158,28 +180,41 @@ async def start( config_bytes = yaml.safe_dump(config, sort_keys=True).encode() _write_new_private(config_path, config_bytes) _write_new_private(token_path, token.encode() + b"\n") - log_descriptor = os.open(log_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + log_descriptor = os.open( + log_path, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) def persist_log(record: LogRecord) -> None: assert log_descriptor is not None - payload = json.dumps( - {"stream": record.stream, "text": record.text}, - ensure_ascii=False, - separators=(",", ":"), - ).encode() + b"\n" + payload = ( + json.dumps( + {"stream": record.stream, "text": record.text}, + ensure_ascii=False, + separators=(",", ":"), + ).encode() + + b"\n" + ) os.write(log_descriptor, payload) capture = BoundedLogCapture( redactor, max_total_bytes=128 * 1024, sink=persist_log ) - environment = {name: os.environ[name] for name in ALLOWED_ENV if name in os.environ} + environment = { + name: os.environ[name] for name in ALLOWED_ENV if name in os.environ + } environment.update( { AUTH_ENV: token, SHARED_CACHE_ENV: os.fspath(shared_cache), - GIT_TOOLCHAIN_ENV: os.fspath(git_toolchains), } ) + process_options: dict[str, Any] = {} + if os.name == "nt": + process_options["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP + else: + process_options["start_new_session"] = True process = await self._process_factory( os.fspath(gateway_binary), "serve", @@ -187,15 +222,21 @@ def persist_log(record: LogRecord) -> None: "docker", "--config-yaml", os.fspath(config_path), - cwd=self._roots.product_root, + cwd=( + self._roots.product_bin_dir.parent + if os.name == "nt" + else self._roots.product_root + ), env=environment, stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, - start_new_session=True, + **process_options, ) if process.pid is None or process.stdout is None or process.stderr is None: - raise GatewayLifecycleError("gateway process streams or identity are unavailable") + raise GatewayLifecycleError( + "gateway process streams or identity are unavailable" + ) process_identity = self._process_identity(process.pid) if process_identity is None: raise GatewayLifecycleError("gateway process identity is unavailable") @@ -203,16 +244,16 @@ def persist_log(record: LogRecord) -> None: _write_new_private( runtime_path / "owner-process.json", OwnerProcess( - schema_version=1, + schema_version=2, pid=process.pid, process_group=process.pid, process_identity=process_identity, gateway_executable=os.fspath(gateway_binary), gateway_binary_sha256=gateway_binary_sha256, gateway_instance_id=gateway_instance_id, - endpoint_host=endpoint.host, - endpoint_port=endpoint.port, - ).model_dump_json() + endpoint_uri=endpoint.uri, + ) + .model_dump_json(exclude_none=True) .encode() + b"\n", ) @@ -222,11 +263,35 @@ def persist_log(record: LogRecord) -> None: asyncio.create_task(capture.drain(process.stderr, "stderr")), ) client = self._client_factory(endpoint, token) + readiness_probe = None + if readiness_via_cli: + from .product_cli import ProductCliAccess + + cli = ProductCliAccess( + endpoint, + token, + self._roots, + self._roots.results / run_id, + ) + readiness_attempt = 0 + + async def probe_cli_readiness() -> None: + nonlocal readiness_attempt + attempt = readiness_attempt + readiness_attempt += 1 + await cli.assert_no_sandboxes( + request_id=_cli_readiness_request_id( + run_id, gateway_instance_id, attempt + ) + ) + + readiness_probe = probe_cli_readiness await _wait_until_ready( process, pid_path, client, readiness_timeout_seconds=readiness_timeout_seconds, + readiness_probe=readiness_probe, ) identity = GatewayIdentity( run_id=run_id, @@ -249,6 +314,7 @@ def persist_log(record: LogRecord) -> None: kill_group=self._kill_group, log_descriptor=log_descriptor, orphan_cleanup=self._orphan_cleanup, + auth_token=token, ) except BaseException as error: issues: list[str] = [] @@ -266,7 +332,7 @@ def persist_log(record: LogRecord) -> None: issues.append("log persistence") if process is not None and gateway_instance_id is not None: try: - await self._orphan_cleanup(gateway_instance_id) + await self._orphan_cleanup(gateway_instance_id, False) except Exception: issues.append("Docker resource cleanup") token_path = runtime_path / "gateway.token" @@ -282,7 +348,9 @@ def persist_log(record: LogRecord) -> None: if isinstance(error, asyncio.CancelledError) and not issues: raise suffix = f"; incomplete cleanup: {', '.join(issues)}" if issues else "" - raise GatewayLifecycleError(f"isolated gateway startup failed{suffix}") from error + raise GatewayLifecycleError( + f"isolated gateway startup failed{suffix}" + ) from error class IsolatedGateway: @@ -301,6 +369,7 @@ def __init__( kill_group: KillGroup, log_descriptor: int, orphan_cleanup: OrphanCleanup, + auth_token: str, ) -> None: self.identity = identity self.client = client @@ -314,26 +383,41 @@ def __init__( self._kill_group = kill_group self._log_descriptor = log_descriptor self._orphan_cleanup = orphan_cleanup + self._auth_token = auth_token self._closed = False @property def logs(self) -> tuple[Any, ...]: return self._capture.records - async def close(self) -> None: + @property + def auth_token(self) -> str: + if self._closed: + raise GatewayLifecycleError("gateway credential is unavailable after close") + return self._auth_token + + async def close( + self, + *, + destroy_sandboxes_via_gateway: bool = True, + retain_shared_base_volumes: bool = False, + ) -> None: if self._closed: return issues: list[str] = [] - try: - await _destroy_all_sandboxes(self.client) - except Exception: - issues.append("sandbox cleanup") + if destroy_sandboxes_via_gateway: + try: + await _destroy_all_sandboxes(self.client) + except Exception: + issues.append("sandbox cleanup") try: await _terminate_process(self._process, self._kill_group) except Exception: issues.append("process termination") try: - await self._orphan_cleanup(self.identity.gateway_instance_id) + await self._orphan_cleanup( + self.identity.gateway_instance_id, retain_shared_base_volumes + ) except Exception: issues.append("Docker resource cleanup") await _join_log_tasks(self._log_tasks) @@ -358,6 +442,7 @@ async def close(self) -> None: f"isolated gateway cleanup was incomplete: {', '.join(issues)}" ) self._closed = True + self._auth_token = "" async def _wait_until_ready( @@ -366,6 +451,7 @@ async def _wait_until_ready( client: GatewayClient, *, readiness_timeout_seconds: float, + readiness_probe: Callable[[], Awaitable[None]] | None = None, ) -> None: deadline = asyncio.get_running_loop().time() + readiness_timeout_seconds while True: @@ -377,12 +463,19 @@ async def _wait_until_ready( pid_matches = False if pid_matches: try: - response = await client.request( - "list_sandboxes", {"kind": "system"}, {}, timeout_seconds=2 - ) - SandboxList.model_validate(response.value) + if readiness_probe is None: + response = await client.request( + "list_sandboxes", {"kind": "system"}, {}, timeout_seconds=2 + ) + SandboxList.model_validate(response.value) + else: + await readiness_probe() return - except (GatewayError, ValidationError): + except GatewayProductError as error: + raise GatewayLifecycleError( + "gateway readiness CLI request was rejected" + ) from error + except (GatewayError, ProductAccessError, ValidationError): pass if asyncio.get_running_loop().time() >= deadline: raise GatewayLifecycleError("gateway readiness timed out") @@ -396,7 +489,9 @@ async def _destroy_all_sandboxes(client: GatewayClient) -> None: try: listing = SandboxList.model_validate(response.value) except ValidationError as error: - raise GatewayLifecycleError("gateway sandbox listing schema is invalid") from error + raise GatewayLifecycleError( + "gateway sandbox listing schema is invalid" + ) from error failed = False for record in listing.sandboxes: sandbox_id = record.get("id") @@ -413,20 +508,22 @@ async def _destroy_all_sandboxes(client: GatewayClient) -> None: except GatewayError: failed = True if failed: - raise GatewayLifecycleError("one or more isolated sandboxes could not be removed") + raise GatewayLifecycleError( + "one or more isolated sandboxes could not be removed" + ) async def _terminate_process(process: Any, kill_group: KillGroup) -> None: if process.returncode is None: try: - kill_group(process.pid, signal.SIGTERM) + kill_group(process.pid, int(signal.SIGTERM)) except ProcessLookupError: pass try: await asyncio.wait_for(process.wait(), 10) except TimeoutError: try: - kill_group(process.pid, signal.SIGKILL) + kill_group(process.pid, _force_kill_signal()) except ProcessLookupError: pass await asyncio.wait_for(process.wait(), 10) @@ -449,36 +546,50 @@ def _effective_config( gateway_instance_id: str, remount_sweep_width: int, ) -> dict[str, Any]: - template = roots.benchmark_source_root / "defaults/gateway.yml" + template = ( + roots.product_bin_dir.parent / "config/windows-amd64.yml" + if os.name == "nt" + else roots.benchmark_source_root / "defaults/gateway.yml" + ) if template.is_symlink() or not template.is_file(): - raise GatewayLifecycleError("gateway configuration template is missing or unsafe") + raise GatewayLifecycleError( + "gateway configuration template is missing or unsafe" + ) config = yaml.safe_load(template.read_bytes()) if not isinstance(config, dict): raise GatewayLifecycleError("gateway configuration template is invalid") try: - config["gateway"].update( + gateway_config = config.setdefault("gateway", {}) + manager_config = config["manager"] + if not isinstance(gateway_config, dict) or not isinstance(manager_config, dict): + raise TypeError + gateway_config.update( { - "bind_addr": f"{endpoint.host}:{endpoint.port}", + "bind_addr": endpoint.address, "pid_path": os.fspath(pid_path), "max_concurrent_connections": 256, } ) - config["manager"].update( + manager_config.update( { "registry_path": os.fspath(registry_path), "workspace_roots": [os.fspath(roots.runs)], } ) - config["manager"]["docker"].update( + manager_config["docker"].update( { "daemon_binary_path": os.fspath(daemon_binary), - "daemon_config_yaml_path": os.fspath(config_path), + "daemon_config_yaml_path": os.fspath( + template if os.name == "nt" else config_path + ), "gateway_instance_id": gateway_instance_id, } ) config["runtime"]["layerstack"]["remount_sweep_width"] = remount_sweep_width except (KeyError, TypeError, AttributeError) as error: - raise GatewayLifecycleError("gateway configuration template shape is invalid") from error + raise GatewayLifecycleError( + "gateway configuration template shape is invalid" + ) from error return config @@ -491,16 +602,39 @@ def _require_owned_run(roots: BenchmarkRoots, run_id: str) -> None: raise GatewayLifecycleError("run workspace ownership is invalid") from error +def _run_shared_cache(roots: BenchmarkRoots, run_id: str) -> Path: + run_path = roots.runs / run_id + shared_cache = run_path / "shared-base-cache" + try: + run_root = run_path.resolve(strict=True) + shared_cache.mkdir(mode=0o700, exist_ok=True) + resolved = shared_cache.resolve(strict=True) + except OSError as error: + raise GatewayLifecycleError( + "run-scoped shared-base cache is unavailable" + ) from error + if ( + shared_cache.is_symlink() + or not shared_cache.is_dir() + or resolved == run_root + or not resolved.is_relative_to(run_root) + ): + raise GatewayLifecycleError("run-scoped shared-base cache is unsafe") + return resolved + + async def recover_stale_gateway( roots: BenchmarkRoots, run_id: str, *, client_factory: ClientFactory = GatewayClient, - kill_group: KillGroup = os.killpg, - process_identity: ProcessIdentity = lambda pid: _process_identity(pid), - orphan_cleanup: Callable[[str], Awaitable[None]] | None = None, + kill_group: KillGroup | None = None, + process_identity: ProcessIdentity | None = None, + orphan_cleanup: OrphanCleanup | None = None, ) -> None: _validate_identity(run_id) + kill_group = kill_group or _default_kill_group + process_identity = process_identity or _process_identity roots.validate_state() runtime_path = roots.runtime / run_id runtime_marker = OwnedPathMarker(role="runtime", identity={"run_id": run_id}) @@ -511,7 +645,8 @@ async def recover_stale_gateway( if metadata_path.is_symlink() or metadata_path.stat().st_size > 16 * 1024: raise GatewayLifecycleError("stale process metadata is unsafe") metadata = OwnerProcess.model_validate_json(metadata_path.read_bytes()) - except (OSError, OwnershipError, ValidationError) as error: + endpoint = _owner_process_endpoint(metadata) + except (OSError, OwnershipError, ValidationError, ValueError) as error: raise GatewayLifecycleError("stale process ownership proof failed") from error expected_binary = _prebuilt_executable(roots, GATEWAY_EXECUTABLE) if ( @@ -536,15 +671,13 @@ async def recover_stale_gateway( if token_path.is_symlink() or token_metadata.st_mode & 0o077 or not token: issues.append("credential recovery") else: - client = client_factory( - GatewayEndpoint(metadata.endpoint_host, metadata.endpoint_port), token - ) + client = client_factory(endpoint, token) try: await _destroy_all_sandboxes(client) except Exception: issues.append("sandbox cleanup") try: - kill_group(metadata.process_group, signal.SIGTERM) + kill_group(metadata.process_group, int(signal.SIGTERM)) except ProcessLookupError: pass except Exception: @@ -554,7 +687,7 @@ async def recover_stale_gateway( while process_identity(metadata.pid) is not None: if asyncio.get_running_loop().time() >= deadline: try: - kill_group(metadata.process_group, signal.SIGKILL) + kill_group(metadata.process_group, _force_kill_signal()) except ProcessLookupError: pass except Exception: @@ -567,7 +700,7 @@ async def recover_stale_gateway( cleanup = orphan_cleanup or _cleanup_docker_resources try: - await cleanup(metadata.gateway_instance_id) + await cleanup(metadata.gateway_instance_id, False) except Exception: issues.append("Docker resource cleanup") try: @@ -585,22 +718,6 @@ async def recover_stale_gateway( ) -def _fixed_git_toolchains(roots: BenchmarkRoots) -> Path: - directory = roots.product_root / "dist/git" - if directory.is_symlink() or not directory.is_dir() or directory.resolve(strict=True) != directory: - raise GatewayLifecycleError("fixed Git toolchain directory is unsafe") - for name in ("linux-arm64.tar", "linux-amd64.tar"): - archive = directory / name - if ( - archive.is_symlink() - or not archive.is_file() - or archive.stat().st_size == 0 - or archive.resolve(strict=True) != archive - ): - raise GatewayLifecycleError("fixed Git toolchain archive is unsafe") - return directory - - def _container_daemon_executable(roots: BenchmarkRoots) -> Path: architecture = platform.machine().lower() package = { @@ -610,15 +727,24 @@ def _container_daemon_executable(roots: BenchmarkRoots) -> Path: "amd64": "sandbox-daemon-linux-amd64", }.get(architecture) if package is None: - raise GatewayLifecycleError("host architecture has no fixed container daemon package") - candidate = roots.product_root / "dist" / package + raise GatewayLifecycleError( + "host architecture has no fixed container daemon package" + ) + package_candidate = roots.product_bin_dir.parent / "dist" / package + candidate = ( + package_candidate + if package_candidate.is_file() + else roots.product_root / "dist" / package + ) try: metadata = candidate.lstat() canonical = candidate.resolve(strict=True) with canonical.open("rb") as stream: magic = stream.read(4) except OSError as error: - raise GatewayLifecycleError("container daemon package is unavailable") from error + raise GatewayLifecycleError( + "container daemon package is unavailable" + ) from error if ( candidate.is_symlink() or not candidate.is_file() @@ -627,7 +753,9 @@ def _container_daemon_executable(roots: BenchmarkRoots) -> Path: or metadata.st_size <= 4 or magic != b"\x7fELF" ): - raise GatewayLifecycleError("container daemon package is unsafe or is not Linux ELF") + raise GatewayLifecycleError( + "container daemon package is unsafe or is not Linux ELF" + ) return canonical @@ -638,8 +766,37 @@ def _reserve_loopback_endpoint() -> GatewayEndpoint: return GatewayEndpoint(host, port) +def _execution_block_endpoint(readiness_via_cli: bool) -> GatewayEndpoint: + if os.name == "nt" and readiness_via_cli: + return GatewayEndpoint.windows_named_pipe( + "npipe://./pipe/" + f"ephemeral-sandbox-benchmark-{secrets.token_hex(16)}" + ) + return _reserve_loopback_endpoint() + + +def _owner_process_endpoint(metadata: OwnerProcess) -> GatewayEndpoint: + if metadata.schema_version == 2: + if ( + metadata.endpoint_uri is None + or metadata.endpoint_host is not None + or metadata.endpoint_port is not None + ): + raise ValueError("gateway owner endpoint metadata is invalid") + return GatewayEndpoint.parse(metadata.endpoint_uri) + if ( + metadata.endpoint_uri is not None + or metadata.endpoint_host is None + or metadata.endpoint_port is None + ): + raise ValueError("legacy gateway owner endpoint metadata is invalid") + return GatewayEndpoint(metadata.endpoint_host, metadata.endpoint_port) + + def _write_new_private(path: Path, content: bytes) -> None: - descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + descriptor = os.open( + path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, 0o600 + ) try: os.write(descriptor, content) os.fsync(descriptor) @@ -661,7 +818,38 @@ def _validate_identity(value: str) -> None: raise GatewayLifecycleError("run identity is invalid") +def _cli_readiness_request_id( + run_id: str, gateway_instance_id: str, attempt: int +) -> str: + return f"{run_id}.{gateway_instance_id}.ready.{attempt}" + + def _process_identity(pid: int) -> str | None: + if os.name == "nt": + command = ( + "$p=Get-Process -Id " + + str(pid) + + " -ErrorAction SilentlyContinue;" + + "if($null -eq $p){exit 1};" + + "$v=$p.Path+'|'+$p.StartTime.ToUniversalTime().Ticks;" + + "[Console]::Out.Write($v)" + ) + try: + completed = subprocess.run( + ["powershell.exe", "-NoProfile", "-Command", command], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + timeout=2, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise GatewayLifecycleError("process identity could not be read") from error + if completed.returncode == 1 or not completed.stdout: + return None + if completed.returncode != 0 or len(completed.stdout) > 16 * 1024: + raise GatewayLifecycleError("process identity could not be read") + return f"sha256:{hashlib.sha256(completed.stdout).hexdigest()}" proc = Path("/proc") / str(pid) if proc.exists(): try: @@ -689,24 +877,60 @@ def _process_identity(pid: int) -> str | None: return f"sha256:{hashlib.sha256(completed.stdout).hexdigest()}" -async def _cleanup_docker_resources(gateway_instance_id: str) -> None: +def _default_kill_group(pid: int, selected_signal: int) -> None: + if os.name == "nt": + os.kill(pid, signal.SIGTERM) + return + os.killpg(pid, selected_signal) + + +def _force_kill_signal() -> int: + return int(getattr(signal, "SIGKILL", signal.SIGTERM)) + + +async def cleanup_gateway_docker_resources( + gateway_instance_ids: tuple[str, ...], +) -> None: + failures = 0 + for gateway_instance_id in dict.fromkeys(gateway_instance_ids): + try: + await _cleanup_docker_resources( + gateway_instance_id, retain_shared_base_volumes=False + ) + except GatewayLifecycleError: + failures += 1 + if failures: + raise GatewayLifecycleError( + "retained shared-base Docker cleanup was incomplete" + ) + + +async def _cleanup_docker_resources( + gateway_instance_id: str, + retain_shared_base_volumes: bool = False, +) -> None: _validate_identity(gateway_instance_id) - label = f"label=eos.gateway_instance_id={gateway_instance_id}" + label = f"label={_GATEWAY_INSTANCE_LABEL}={gateway_instance_id}" container_output = await _docker_output("ps", "-aq", "--filter", label) - containers = _validated_lines(container_output, re.compile(r"^[0-9a-f]{12,64}$")) - volume_output = await _docker_output( - "volume", "ls", "--quiet", "--filter", label - ) - volumes = _validated_lines( - volume_output, re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,255}$") - ) + containers = _validated_lines(container_output, _DOCKER_CONTAINER_ID) + volume_output = await _docker_output("volume", "ls", "--quiet", "--filter", label) + volumes = _validated_lines(volume_output, _DOCKER_VOLUME_NAME) + removable_volumes: list[str] = [] failures = 0 + for volume in volumes: + try: + shared_base = await _validated_owned_volume(gateway_instance_id, volume) + except GatewayLifecycleError: + failures += 1 + continue + if not shared_base or not retain_shared_base_volumes: + removable_volumes.append(volume) for container in containers: try: await _docker_output("rm", "--force", container) except GatewayLifecycleError: failures += 1 - for volume in volumes: + for volume in removable_volumes: try: await _docker_output("volume", "rm", volume) except GatewayLifecycleError: @@ -715,6 +939,54 @@ async def _cleanup_docker_resources(gateway_instance_id: str) -> None: raise GatewayLifecycleError("Docker cleanup was incomplete") +async def _validated_owned_volume(gateway_instance_id: str, volume: str) -> bool: + content = await _docker_output("volume", "inspect", volume) + try: + records = json.loads(content) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise GatewayLifecycleError( + "Docker volume ownership output is invalid" + ) from error + if ( + not isinstance(records, list) + or len(records) != 1 + or not isinstance(records[0], dict) + or records[0].get("Name") != volume + or not isinstance(records[0].get("Labels"), dict) + ): + raise GatewayLifecycleError("Docker volume ownership output is invalid") + labels = records[0]["Labels"] + if ( + any( + not isinstance(name, str) or not isinstance(value, str) + for name, value in labels.items() + ) + or labels.get(_GATEWAY_INSTANCE_LABEL) != gateway_instance_id + ): + raise GatewayLifecycleError("Docker volume ownership proof is invalid") + match = _SHARED_BASE_VOLUME_NAME.fullmatch(volume) + shared_labels = { + _SHARED_BASE_ROOT_HASH_LABEL, + _SHARED_BASE_TARGET_LABEL, + _SHARED_BASE_READONLY_LABEL, + } + has_shared_labels = bool(shared_labels.intersection(labels)) + if match is None: + if has_shared_labels: + raise GatewayLifecycleError( + "Docker shared-base volume identity is ambiguous" + ) + return False + root_hash = match.group(1) + if ( + labels.get(_SHARED_BASE_ROOT_HASH_LABEL) != root_hash + or labels.get(_SHARED_BASE_TARGET_LABEL) != _SHARED_BASE_TARGET + or labels.get(_SHARED_BASE_READONLY_LABEL) != "true" + ): + raise GatewayLifecycleError("Docker shared-base volume identity is ambiguous") + return True + + async def _docker_output(*arguments: str) -> bytes: environment = {name: os.environ[name] for name in ALLOWED_ENV if name in os.environ} try: @@ -738,7 +1010,9 @@ def _validated_lines(content: bytes, pattern: re.Pattern[str]) -> tuple[str, ... try: lines = tuple(line for line in content.decode("ascii").splitlines() if line) except UnicodeDecodeError as error: - raise GatewayLifecycleError("Docker cleanup identity output is invalid") from error + raise GatewayLifecycleError( + "Docker cleanup identity output is invalid" + ) from error if len(lines) > 64 or any(pattern.fullmatch(line) is None for line in lines): raise GatewayLifecycleError("Docker cleanup identity output is invalid") return lines diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/ipc_qualification.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/ipc_qualification.py new file mode 100644 index 0000000..89d99ca --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/ipc_qualification.py @@ -0,0 +1,2359 @@ +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import platform +import re +import secrets +import sys +import time +import uuid +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Protocol + +from .catalog import _prebuilt_executable +from .paths import BenchmarkRoots, _sync_directory + +EXP1_IPC_INVOCATION_COUNT = 25_000 +EXP1_IPC_CONCURRENCY = 5 +EXP1_IPC_BATCH_COUNT = 5_000 +QUALIFICATION_ONLY = True +PERFORMANCE_EVIDENCE = False +TCPIP_EVENT_IDS = (4227, 4231) +MAX_GATEWAY_HANDLE_GROWTH = 32 +MAX_GATEWAY_PRIVATE_BYTES_GROWTH = 16 * 1024 * 1024 +MAX_GATEWAY_RSS_BYTES_GROWTH = 16 * 1024 * 1024 +RESOURCE_GROWTH_POLICY_SOURCE = "EXP1 v1.1 IPC qualification policy preregistration" +PROCESS_SAMPLE_EVERY_BATCHES = 100 +BENCHMARK_GIT_EXCLUSIONS = ( + ":(exclude,glob)benchmark/**/.pytest_cache/**", + ":(exclude,glob)benchmark/**/.venv/**", + ":(exclude,glob)benchmark/**/__pycache__/**", + ":(exclude,glob)benchmark/**/dist/**", + ":(exclude,glob)benchmark/**/node_modules/**", + ":(exclude,glob)benchmark/**/playwright-report/**", + ":(exclude,glob)benchmark/**/test-results/**", + ":(exclude,glob)benchmark/**/*.pyc", +) +PAPER_FROZEN_SCOPE = ( + "benchmark", + "progress.md", + "plan/task-packets/exp1-cli-performance-campaign.md", + "experiment_inventory.md", + "experiments/exp1-v1.1-protocol-amendment.md", + "experiments/environment_setup.md", + "experiments/expected_tables.md", + "experiments/experiment_log.md", + "paper_state.json", + "plan/progress.md", + "experiments/scripts/archive_exp1_run.py", + "experiments/scripts/project_exp1_final_runtime.py", + "experiments/analysis/scripts/generate_exp1_tables.py", +) +PREREGISTERED_PRODUCT_COMMIT = "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8" +PREREGISTERED_PRODUCT_BRANCH = "main" +PREREGISTERED_PACKAGE_SHA256 = ( + "sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506" +) +PREREGISTERED_ARTIFACT_SHA256 = { + "gateway": ( + "sha256:42e7642dd025487811abbcd78dcc5513760f2aaa1e6057cfdfa3e74c03748358" + ), + "manager_cli": ( + "sha256:e1faa2fe0e9f4909fa2d694166784ac65dde40ba82795b7e0c503eb5fea86513" + ), + "runtime_cli": ( + "sha256:e18827cf765945c958e169748575b89645c730b310ee5ffc1b42c382b44a0e26" + ), + "observability_cli": ( + "sha256:2b1c13bba36c9486f768824178d1e2ea8d2b1da019bd21cd1f9ea250d5da34c5" + ), + "linux_daemon": ( + "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a" + ), + "windows_config": ( + "sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a" + ), +} +EXP1_EXPECTED_HOST = { + "computer_name": "DESKTOP-OLP1ADS", + "os_build_number": 26200, + "architecture": "x64", + "logical_processors": 48, + "total_memory_bytes": 137_438_953_472, +} +_INVOCATION_TIMEOUT_SECONDS = 30.0 +_GATEWAY_READY_TIMEOUT_SECONDS = 10.0 +_GATEWAY_STOP_TIMEOUT_SECONDS = 10.0 +_COLLECTOR_TIMEOUT_SECONDS = 30.0 +_MAX_CAPTURE_BYTES = 1024 * 1024 +_MAX_TCPIP_EVENT_RECORDS = 1024 +_IDENTITY = re.compile(r"[a-z0-9][a-z0-9-]{0,127}\Z") +_NPIPE_NAME = re.compile(r"[A-Za-z0-9/_.-]+\Z") +_NPIPE_PREFIX = "npipe://./pipe/" +_BINARY_FLAG = getattr(os, "O_BINARY", 0) + + +@dataclass(frozen=True, slots=True) +class QualificationWorkload: + batches: int + concurrency: int + + @property + def invocation_count(self) -> int: + return self.batches * self.concurrency + + +PRODUCTION_WORKLOAD = QualificationWorkload( + batches=EXP1_IPC_BATCH_COUNT, + concurrency=EXP1_IPC_CONCURRENCY, +) + + +@dataclass(frozen=True, slots=True) +class InvocationCapture: + request_id: str + started_monotonic_ns: int + ended_monotonic_ns: int + started_utc_ns: int + ended_utc_ns: int + return_code: int | None + stdout: bytes + stderr: bytes + execution_error: str | None = None + credential_exposed: bool = False + + +@dataclass(frozen=True, slots=True) +class EventLogCursor: + phase: str + captured_monotonic_ns: int + captured_utc_ns: int + log_name: str + last_record_id: int + source: str = "windows_event_log" + + +@dataclass(frozen=True, slots=True) +class TcpipEvent: + event_id: int + record_id: int + created_at_utc: str + provider_name: str = "Tcpip" + + +@dataclass(frozen=True, slots=True) +class GatewayProcessSample: + phase: str + captured_monotonic_ns: int + captured_utc_ns: int + pid: int + completed_batches: int + handle_count: int + private_bytes: int + rss_bytes: int + source: str = "windows_get_process" + + +@dataclass(frozen=True, slots=True) +class OwnedTcpConnection: + state: str + local_address: str + local_port: int + remote_address: str + remote_port: int + + +@dataclass(frozen=True, slots=True) +class GatewayTcpSample: + phase: str + captured_monotonic_ns: int + captured_utc_ns: int + pid: int + completed_batches: int + connections: tuple[OwnedTcpConnection, ...] + source: str = "windows_get_net_tcp_connection" + + +class IpcQualificationRunner(Protocol): + endpoint: str + + @property + def gateway_pid(self) -> int | None: ... + + @property + def executable_paths(self) -> dict[str, str]: ... + + @property + def sanitized_commands(self) -> dict[str, Any]: ... + + async def start(self) -> None: ... + + async def invoke(self, request_id: str) -> InvocationCapture: ... + + async def stop(self) -> dict[str, Any]: ... + + +class IpcQualificationCollector(Protocol): + async def qualification_identity( + self, + roots: BenchmarkRoots, + runner: IpcQualificationRunner, + ) -> dict[str, Any]: ... + + async def event_cursor(self, phase: str) -> EventLogCursor: ... + + async def tcpip_events( + self, + after_record_id: int, + through_record_id: int, + ) -> tuple[TcpipEvent, ...]: ... + + async def gateway_process_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayProcessSample: ... + + async def gateway_tcp_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayTcpSample: ... + + +class QualificationError(RuntimeError): + pass + + +async def qualify_exp1_ipc(roots: BenchmarkRoots) -> dict[str, Any]: + qualification_id = uuid.uuid4().hex + evidence_root = _create_evidence_root(roots, qualification_id) + runner = NativeWindowsIpcRunner(roots, evidence_root, qualification_id) + return await _execute_qualification( + roots, + runner, + NativeWindowsQualificationCollector(), + evidence_root, + qualification_id, + PRODUCTION_WORKLOAD, + ) + + +async def _execute_qualification( + roots: BenchmarkRoots, + runner: IpcQualificationRunner, + collector: IpcQualificationCollector, + evidence_root: Path, + qualification_id: str, + workload: QualificationWorkload, +) -> dict[str, Any]: + roots.validate_state() + _validate_identity(qualification_id) + _validate_workload(workload) + _validate_npipe_endpoint(runner.endpoint) + evidence_root = evidence_root.resolve(strict=True) + if ( + evidence_root == roots.results + or not evidence_root.is_relative_to(roots.results) + or evidence_root.is_symlink() + or not evidence_root.is_dir() + ): + raise QualificationError("qualification evidence root is unsafe") + + invocations_path = evidence_root / "invocations.ndjson" + writer = _NdjsonWriter(invocations_path) + started_monotonic_ns = time.monotonic_ns() + started_utc_ns = time.time_ns() + attempted = 0 + succeeded = 0 + failed = 0 + batches_started = 0 + batches_completed = 0 + first_failure: dict[str, Any] | None = None + failure_kind: str | None = None + gateway_evidence: dict[str, Any] = {} + identity_evidence: dict[str, Any] | None = None + pre_cursor: EventLogCursor | None = None + post_cursor: EventLogCursor | None = None + process_samples: list[GatewayProcessSample] = [] + tcp_samples: list[GatewayTcpSample] = [] + tcpip_events: tuple[TcpipEvent, ...] = () + collector_errors: list[dict[str, str]] = [] + stop_validation_errors: list[dict[str, str]] = [] + runner_started = False + gateway_pid: int | None = None + + async def collect_checkpoint(phase: str, completed: int) -> None: + assert gateway_pid is not None + try: + sample = await collector.gateway_process_sample( + gateway_pid, + phase, + completed, + ) + _validate_process_sample(sample, gateway_pid, phase, completed) + process_samples.append(sample) + except Exception as error: + collector_errors.append( + _collector_error(f"{phase}_process_sample_{completed}", error) + ) + raise QualificationError( + f"gateway {phase} process sample unavailable" + ) from error + try: + tcp_sample = await collector.gateway_tcp_sample( + gateway_pid, + phase, + completed, + ) + _validate_tcp_sample(tcp_sample, gateway_pid, phase, completed) + tcp_samples.append(tcp_sample) + except Exception as error: + collector_errors.append( + _collector_error(f"{phase}_tcp_sample_{completed}", error) + ) + raise QualificationError( + f"gateway {phase} TCP evidence unavailable" + ) from error + if tcp_sample.connections: + raise QualificationError(f"gateway owns TCP endpoints at {phase}") + + try: + try: + identity_evidence = await collector.qualification_identity(roots, runner) + _validate_qualification_identity(identity_evidence, roots, runner) + except Exception as error: + collector_errors.append(_collector_error("qualification_identity", error)) + raise QualificationError( + "qualification identity evidence unavailable" + ) from error + try: + pre_cursor = await collector.event_cursor("pre_readiness") + _validate_event_cursor(pre_cursor, "pre_readiness") + except Exception as error: + collector_errors.append(_collector_error("pre_readiness_cursor", error)) + raise QualificationError( + "pre-readiness event cursor unavailable" + ) from error + await runner.start() + runner_started = True + gateway_pid = runner.gateway_pid + if gateway_pid is None or gateway_pid <= 0: + raise QualificationError("qualification gateway PID is unavailable") + await collect_checkpoint("readiness", 0) + for batch_index in range(workload.batches): + batches_started += 1 + request_ids = tuple( + _request_id(qualification_id, batch_index, slot) + for slot in range(workload.concurrency) + ) + captures = await asyncio.gather( + *(_invoke_once(runner, request_id) for request_id in request_ids) + ) + records = [ + _invocation_record( + capture, + batch_index=batch_index, + slot=slot, + invocation_index=batch_index * workload.concurrency + slot, + ) + for slot, capture in enumerate(captures) + ] + writer.append(records) + attempted += len(records) + succeeded += sum(record["result"] == "passed" for record in records) + failed += sum(record["result"] == "failed" for record in records) + if failed: + first_failure = next( + record for record in records if record["result"] == "failed" + ) + failure_kind = str(first_failure["validation"]) + break + batches_completed += 1 + if batches_completed % PROCESS_SAMPLE_EVERY_BATCHES == 0: + await collect_checkpoint("cadence", batches_completed) + except Exception as error: # noqa: BLE001 + failure_kind = f"qualification_error:{type(error).__name__}" + finally: + if runner_started and gateway_pid is not None: + try: + await collect_checkpoint("pre_stop", batches_completed) + except Exception as error: # noqa: BLE001 + if failure_kind is None: + failure_kind = f"qualification_error:{type(error).__name__}" + try: + gateway_evidence = await runner.stop() + if runner_started and gateway_pid is not None: + _validate_stop_evidence( + gateway_evidence, + runner, + gateway_pid, + ) + except Exception as error: # noqa: BLE001 + stop_validation_errors.append( + _collector_error("gateway_stop_evidence", error) + ) + if failure_kind is None: + failure_kind = f"gateway_stop_error:{type(error).__name__}" + if runner_started and gateway_pid is not None: + try: + cleanup_tcp = await collector.gateway_tcp_sample( + gateway_pid, + "after_cleanup", + batches_completed, + ) + _validate_tcp_sample( + cleanup_tcp, + gateway_pid, + "after_cleanup", + batches_completed, + ) + tcp_samples.append(cleanup_tcp) + except Exception as error: # noqa: BLE001 + collector_errors.append( + _collector_error("after_cleanup_tcp_sample", error) + ) + try: + post_cursor = await collector.event_cursor("post_cleanup") + _validate_event_cursor(post_cursor, "post_cleanup") + except Exception as error: # noqa: BLE001 + collector_errors.append(_collector_error("post_cleanup_cursor", error)) + if pre_cursor is not None and post_cursor is not None: + try: + tcpip_events = await collector.tcpip_events( + pre_cursor.last_record_id, + post_cursor.last_record_id, + ) + _validate_tcpip_events(tcpip_events, pre_cursor, post_cursor) + except Exception as error: # noqa: BLE001 + collector_errors.append(_collector_error("tcpip_event_query", error)) + writer.close() + + ended_monotonic_ns = time.monotonic_ns() + ended_utc_ns = time.time_ns() + host_evidence, host_gate_failures = _host_evidence( + pre_cursor=pre_cursor, + post_cursor=post_cursor, + identity_evidence=identity_evidence, + process_samples=process_samples, + tcp_samples=tcp_samples, + completed_batches=batches_completed, + tcpip_events=tcpip_events, + collector_errors=collector_errors, + stop_validation_errors=stop_validation_errors, + runner_started=runner_started, + ) + if failure_kind is None and host_gate_failures: + failure_kind = host_gate_failures[0] + workload_complete = ( + failed == 0 + and attempted == workload.invocation_count + and succeeded == workload.invocation_count + and batches_completed == workload.batches + ) + if failure_kind is None and not workload_complete: + failure_kind = "incomplete_qualification" + gate_failures = list(dict.fromkeys([failure_kind, *host_gate_failures])) + gate_failures = [failure for failure in gate_failures if failure is not None] + passed = failure_kind is None and not host_gate_failures and workload_complete + summary = { + "schema_version": 1, + "kind": "exp1_ipc_native_cli_qualification", + "qualification_id": qualification_id, + "disposition": "qualification_only", + "qualification_only": QUALIFICATION_ONLY, + "performance_evidence": PERFORMANCE_EVIDENCE, + "status": "passed" if passed else "failed", + "failure_kind": failure_kind, + "gate_failures": gate_failures, + "transport": { + "kind": "windows_named_pipe", + "endpoint": runner.endpoint, + "tcp_used": False, + "fallback_allowed": False, + "retry_allowed": False, + "pacing_allowed": False, + "gateway_count": 1, + }, + "workload": { + "operation": "list_sandboxes", + "executable_role": "manager", + "native_process_per_invocation": True, + "planned_invocations": workload.invocation_count, + "planned_batches": workload.batches, + "concurrency": workload.concurrency, + "attempted_invocations": attempted, + "successful_invocations": succeeded, + "failed_invocations": failed, + "batches_started": batches_started, + "batches_completed": batches_completed, + }, + "started_monotonic_ns": started_monotonic_ns, + "ended_monotonic_ns": ended_monotonic_ns, + "elapsed_ns": ended_monotonic_ns - started_monotonic_ns, + "started_utc_ns": started_utc_ns, + "ended_utc_ns": ended_utc_ns, + "first_failure": first_failure, + "gateway": gateway_evidence, + "host_evidence": host_evidence, + "evidence": { + "directory": os.fspath(evidence_root), + "invocations_ndjson": invocations_path.name, + "invocations_ndjson_bytes": invocations_path.stat().st_size, + "invocations_ndjson_sha256": _sha256_file(invocations_path), + "invocation_records": attempted, + "host_evidence_json": "host-evidence.json", + "qualification_manifest_json": "qualification-manifest.json", + "summary": "summary.json", + }, + } + host_evidence_path = evidence_root / "host-evidence.json" + _write_new_json(host_evidence_path, host_evidence) + summary["evidence"]["host_evidence_json_bytes"] = host_evidence_path.stat().st_size + summary["evidence"]["host_evidence_json_sha256"] = _sha256_file(host_evidence_path) + manifest = { + "schema_version": 1, + "kind": "exp1_ipc_qualification_manifest", + "qualification_id": qualification_id, + "qualification_only": QUALIFICATION_ONLY, + "performance_evidence": PERFORMANCE_EVIDENCE, + "status": "passed" if passed else "failed", + "gate_failures": gate_failures, + "transport": summary["transport"], + "workload": summary["workload"], + "started_monotonic_ns": started_monotonic_ns, + "ended_monotonic_ns": ended_monotonic_ns, + "started_utc_ns": started_utc_ns, + "ended_utc_ns": ended_utc_ns, + "gateway_pid": gateway_pid, + "gateway_stop": gateway_evidence, + "identity": identity_evidence, + "policy": host_evidence["policy"], + "event_log": host_evidence["event_log"], + "gateway_process": host_evidence["gateway_process"], + "gateway_tcp": host_evidence["gateway_tcp"], + "artifacts": { + "invocations_ndjson": { + "path": invocations_path.name, + "bytes": invocations_path.stat().st_size, + "sha256": _sha256_file(invocations_path), + }, + "host_evidence_json": { + "path": host_evidence_path.name, + "bytes": host_evidence_path.stat().st_size, + "sha256": _sha256_file(host_evidence_path), + }, + }, + } + manifest_path = evidence_root / "qualification-manifest.json" + _write_new_json(manifest_path, manifest) + summary["evidence"]["qualification_manifest_json_bytes"] = ( + manifest_path.stat().st_size + ) + summary["evidence"]["qualification_manifest_json_sha256"] = _sha256_file( + manifest_path + ) + summary_path = evidence_root / "summary.json" + _write_new_json(summary_path, summary) + return summary + + +class NativeWindowsIpcRunner: + def __init__( + self, + roots: BenchmarkRoots, + evidence_root: Path, + qualification_id: str, + ) -> None: + if os.name != "nt": + raise QualificationError( + "qualify-exp1-ipc requires native Windows execution" + ) + self.endpoint = f"{_NPIPE_PREFIX}ephemeral-sandbox-exp1-ipc-{qualification_id}" + self._roots = roots + self._evidence_root = evidence_root + self._gateway = _prebuilt_executable(roots, "sandbox-gateway") + self._manager = _prebuilt_executable(roots, "sandbox-manager-cli") + self._runtime = _prebuilt_executable(roots, "sandbox-runtime-cli") + self._observability = _prebuilt_executable( + roots, + "sandbox-observability-cli", + ) + self._auth_token = secrets.token_urlsafe(48) + self._pid_path = evidence_root / "gateway.pid" + self._process: asyncio.subprocess.Process | None = None + self._stdout_digest: asyncio.Task[dict[str, Any]] | None = None + self._stderr_digest: asyncio.Task[dict[str, Any]] | None = None + + @property + def gateway_pid(self) -> int | None: + return None if self._process is None else self._process.pid + + @property + def executable_paths(self) -> dict[str, str]: + return { + "gateway": os.fspath(self._gateway), + "manager_cli": os.fspath(self._manager), + "runtime_cli": os.fspath(self._runtime), + "observability_cli": os.fspath(self._observability), + } + + @property + def sanitized_commands(self) -> dict[str, Any]: + working_directory = os.fspath(self._roots.product_bin_dir.parent) + return { + "gateway_serve": { + "executable_path": os.fspath(self._gateway), + "argv": [ + "serve", + "--backend", + "none", + "--gateway-endpoint", + self.endpoint, + "--auth-token=", + "--pid-file", + os.fspath(self._pid_path), + ], + "working_directory": working_directory, + "stdin": "null", + "stdout": "pipe_digest_only", + "stderr": "pipe_digest_only", + }, + "manager_list_sandboxes": { + "executable_path": os.fspath(self._manager), + "argv_template": [ + "--gateway-endpoint", + self.endpoint, + "--gateway-auth-token=", + "--request-id", + "", + "list_sandboxes", + ], + "working_directory": working_directory, + "stdin": "null", + "stdout": "strict_single_json_line_hash_only", + "stderr": "must_be_empty_hash_only", + }, + } + + async def start(self) -> None: + if self._process is not None: + raise QualificationError("qualification gateway was already started") + self._process = await asyncio.create_subprocess_exec( + os.fspath(self._gateway), + "serve", + "--backend", + "none", + "--gateway-endpoint", + self.endpoint, + f"--auth-token={self._auth_token}", + "--pid-file", + os.fspath(self._pid_path), + cwd=self._roots.product_bin_dir.parent, + env=_native_environment(), + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + assert self._process.stdout is not None + assert self._process.stderr is not None + self._stdout_digest = asyncio.create_task(_digest_stream(self._process.stdout)) + self._stderr_digest = asyncio.create_task(_digest_stream(self._process.stderr)) + try: + await asyncio.wait_for( + self._wait_until_ready(), + timeout=_GATEWAY_READY_TIMEOUT_SECONDS, + ) + except Exception: + await self.stop() + raise + + async def invoke(self, request_id: str) -> InvocationCapture: + _validate_identity(request_id) + started_monotonic_ns = time.monotonic_ns() + started_utc_ns = time.time_ns() + process: asyncio.subprocess.Process | None = None + stdout = b"" + stderr = b"" + return_code: int | None = None + execution_error: str | None = None + try: + process = await asyncio.create_subprocess_exec( + os.fspath(self._manager), + "--gateway-endpoint", + self.endpoint, + f"--gateway-auth-token={self._auth_token}", + "--request-id", + request_id, + "list_sandboxes", + cwd=self._roots.product_bin_dir.parent, + env=_native_environment(), + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(), + timeout=_INVOCATION_TIMEOUT_SECONDS, + ) + except TimeoutError: + process.kill() + stdout, stderr = await process.communicate() + execution_error = "timeout" + return_code = process.returncode + except OSError: + execution_error = "process_creation_failed" + ended_monotonic_ns = time.monotonic_ns() + ended_utc_ns = time.time_ns() + credential = self._auth_token.encode() + return InvocationCapture( + request_id=request_id, + started_monotonic_ns=started_monotonic_ns, + ended_monotonic_ns=ended_monotonic_ns, + started_utc_ns=started_utc_ns, + ended_utc_ns=ended_utc_ns, + return_code=return_code, + stdout=stdout, + stderr=stderr, + execution_error=execution_error, + credential_exposed=credential in stdout or credential in stderr, + ) + + async def stop(self) -> dict[str, Any]: + process = self._process + termination = "not_started" + if process is not None and process.returncode is None: + termination = "terminate" + process.terminate() + try: + await asyncio.wait_for( + process.wait(), + timeout=_GATEWAY_STOP_TIMEOUT_SECONDS, + ) + except TimeoutError: + termination = "kill_after_terminate_timeout" + process.kill() + await process.wait() + elif process is not None: + termination = "already_exited" + stdout = ( + await self._stdout_digest + if self._stdout_digest is not None + else _empty_stream_digest() + ) + stderr = ( + await self._stderr_digest + if self._stderr_digest is not None + else _empty_stream_digest() + ) + try: + self._pid_path.unlink() + except FileNotFoundError: + pass + return { + "schema_version": 1, + "gateway_pid": None if process is None else process.pid, + "executable_path": os.fspath(self._gateway), + "process_exited": process is not None and process.returncode is not None, + "pid_file_removed": not self._pid_path.exists(), + "sanitized_command": self.sanitized_commands["gateway_serve"], + "termination": termination, + "return_code": None if process is None else process.returncode, + "stdout": stdout, + "stderr": stderr, + "auth_token_recorded": False, + } + + async def _wait_until_ready(self) -> None: + assert self._process is not None + while True: + if self._pid_path.is_file(): + value = self._pid_path.read_text(encoding="utf-8").strip() + if value == str(self._process.pid): + return + raise QualificationError("qualification gateway PID marker is invalid") + if self._process.returncode is not None: + raise QualificationError( + "qualification gateway exited before readiness" + ) + await asyncio.sleep(0.01) + + +class NativeWindowsQualificationCollector: + async def qualification_identity( + self, + roots: BenchmarkRoots, + runner: IpcQualificationRunner, + ) -> dict[str, Any]: + commit_output = await _run_strict_command( + ("git", "rev-parse", "HEAD"), + cwd=roots.product_root, + ) + try: + commit = commit_output.decode("ascii").strip() + except UnicodeDecodeError as error: + raise QualificationError("product commit identity is invalid") from error + if re.fullmatch(r"[0-9a-f]{40}", commit) is None: + raise QualificationError("product commit identity is invalid") + branch_output = await _run_strict_command( + ("git", "branch", "--show-current"), + cwd=roots.product_root, + ) + try: + branch = branch_output.decode("utf-8").strip() + except UnicodeDecodeError as error: + raise QualificationError("product branch identity is invalid") from error + if not branch: + raise QualificationError("product branch identity is invalid") + status = await _run_strict_command( + ("git", "status", "--porcelain=v1", "-z"), + cwd=roots.product_root, + ) + paper_commit_output = await _run_strict_command( + ("git", "rev-parse", "HEAD"), + cwd=roots.test_repository_root, + ) + try: + paper_commit = paper_commit_output.decode("ascii").strip() + except UnicodeDecodeError as error: + raise QualificationError("paper commit identity is invalid") from error + if re.fullmatch(r"[0-9a-f]{40}", paper_commit) is None: + raise QualificationError("paper commit identity is invalid") + paper_status = await _run_strict_command( + ( + "git", + "status", + "--porcelain=v1", + "-z", + "--", + *PAPER_FROZEN_SCOPE, + *BENCHMARK_GIT_EXCLUSIONS, + ), + cwd=roots.test_repository_root, + ) + + package_directory = roots.product_bin_dir.parent.resolve(strict=True) + package_zip = package_directory.with_suffix(".zip") + _require_safe_directory(package_directory) + _require_safe_file(package_zip) + expected_package_name = f"windows-exp1-{commit[:8]}" + executable_paths = runner.executable_paths + if set(executable_paths) != { + "gateway", + "manager_cli", + "runtime_cli", + "observability_cli", + }: + raise QualificationError("qualification executable identities are invalid") + executables = { + role: _file_identity(Path(path), roots.product_bin_dir) + for role, path in executable_paths.items() + } + packaged_support = { + "linux_daemon": _file_identity( + package_directory / "dist" / "sandbox-daemon-linux-amd64", + package_directory, + ), + "windows_config": _file_identity( + package_directory / "config" / "windows-amd64.yml", + package_directory, + ), + } + + qualifier_path = Path(__file__).resolve(strict=True) + cli_path = qualifier_path.with_name("cli.py").resolve(strict=True) + launcher_path = ( + roots.product_bin_dir / "start-sandbox-windows-docker-gateway.ps1" + ) + qualifier_test_path = ( + roots.benchmark_source_root + / "backend" + / "tests" + / "unit" + / "test_ipc_qualification.py" + ) + amendment_path = ( + roots.test_repository_root + / "experiments" + / "exp1-v1.1-protocol-amendment.md" + ) + sources = { + "ipc_qualification": _file_identity( + qualifier_path, + roots.test_repository_root, + ), + "benchmark_cli": _file_identity(cli_path, roots.test_repository_root), + "packaged_gateway_launcher": _file_identity( + launcher_path, + roots.product_bin_dir, + ), + "ipc_qualification_test": _file_identity( + qualifier_test_path, + roots.test_repository_root, + ), + "protocol_amendment": _file_identity( + amendment_path, + roots.test_repository_root, + ), + } + host = await self._host_identity() + commands = _expected_sanitized_commands(roots, runner) + return { + "schema_version": 1, + "captured_monotonic_ns": time.monotonic_ns(), + "captured_utc_ns": time.time_ns(), + "product": { + "commit": commit, + "branch": branch, + "status_clean": not status, + "status_bytes": len(status), + "status_sha256": _sha256_bytes(status), + "package_directory": os.fspath(package_directory), + "package_directory_name": package_directory.name, + "expected_package_directory_name": expected_package_name, + "package_name_matches_commit": ( + package_directory.name == expected_package_name + ), + "package_zip_path": os.fspath(package_zip), + "package_zip_bytes": package_zip.stat().st_size, + "package_zip_sha256": _sha256_file(package_zip), + }, + "paper": { + "commit": paper_commit, + "scoped_status_clean": not paper_status, + "scoped_status_bytes": len(paper_status), + "scoped_status_sha256": _sha256_bytes(paper_status), + "paper_root": os.fspath(roots.test_repository_root), + "frozen_scope": list(PAPER_FROZEN_SCOPE), + "generated_exclusions": list(BENCHMARK_GIT_EXCLUSIONS), + "policy_source": ( + "experiments/scripts/archive_exp1_run.py:" + "PAPER_FROZEN_SCOPE+BENCHMARK_GIT_EXCLUSIONS" + ), + }, + "executables": executables, + "packaged_support": packaged_support, + "qualifier_sources": sources, + "host": host, + "build": { + "package_identity": package_directory.name, + "product_commit": commit, + "python_executable": os.fspath( + Path(sys.executable).resolve(strict=True) + ), + "python_implementation": platform.python_implementation(), + "python_version": platform.python_version(), + "python_architecture": platform.machine().lower(), + "package_build_command": [ + "powershell.exe", + "-NoProfile", + "-ExecutionPolicy", + "Bypass", + "-File", + ".\\bin\\package-windows-amd64-release.ps1", + "-PackageName", + package_directory.name, + "-OutDir", + "target", + "-Profile", + "release", + ], + }, + "sanitized_commands": commands, + } + + async def _host_identity(self) -> dict[str, Any]: + value = await _run_powershell_json( + """ +$ErrorActionPreference = 'Stop' +$os = Get-CimInstance Win32_OperatingSystem +$computer = Get-CimInstance Win32_ComputerSystem +[pscustomobject]@{ + computer_name = [string]$computer.Name + os_caption = [string]$os.Caption + os_version = [string]$os.Version + os_build_number = [long]$os.BuildNumber + architecture = [System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture.ToString().ToLowerInvariant() + logical_processors = [long]$computer.NumberOfLogicalProcessors + total_memory_bytes = [long]$computer.TotalPhysicalMemory +} | ConvertTo-Json -Compress +""".strip() + ) + keys = { + "computer_name", + "os_caption", + "os_version", + "os_build_number", + "architecture", + "logical_processors", + "total_memory_bytes", + } + _require_exact_keys(value, keys) + if ( + any( + not isinstance(value[name], str) or not value[name].strip() + for name in ( + "computer_name", + "os_caption", + "os_version", + "architecture", + ) + ) + or not _is_positive_int(value["os_build_number"]) + or not _is_positive_int(value["logical_processors"]) + or not _is_positive_int(value["total_memory_bytes"]) + ): + raise QualificationError("Windows host/build identity is invalid") + return value + + async def event_cursor(self, phase: str) -> EventLogCursor: + _validate_phase(phase, {"pre_readiness", "post_cleanup"}) + value = await _run_powershell_json( + """ +$ErrorActionPreference = 'Stop' +$event = Get-WinEvent -LogName System -MaxEvents 1 -ErrorAction Stop +[pscustomobject]@{ + log_name = 'System' + last_record_id = [long]$event.RecordId +} | ConvertTo-Json -Compress +""".strip() + ) + _require_exact_keys(value, {"log_name", "last_record_id"}) + if value["log_name"] != "System" or not _is_nonnegative_int( + value["last_record_id"] + ): + raise QualificationError("Windows event cursor response is invalid") + return EventLogCursor( + phase=phase, + captured_monotonic_ns=time.monotonic_ns(), + captured_utc_ns=time.time_ns(), + log_name="System", + last_record_id=value["last_record_id"], + ) + + async def tcpip_events( + self, + after_record_id: int, + through_record_id: int, + ) -> tuple[TcpipEvent, ...]: + if ( + not _is_nonnegative_int(after_record_id) + or not _is_nonnegative_int(through_record_id) + or through_record_id < after_record_id + ): + raise QualificationError("Windows event query bounds are invalid") + xpath = ( + "*[System[Provider[@Name='Tcpip'] and " + "(EventID=4227 or EventID=4231) and " + f"EventRecordID > {after_record_id} and " + f"EventRecordID <= {through_record_id}]]" + ) + value = await _run_powershell_json( + f""" +$ErrorActionPreference = 'Stop' +$query = [System.Diagnostics.Eventing.Reader.EventLogQuery]::new( + 'System', + [System.Diagnostics.Eventing.Reader.PathType]::LogName, + "{xpath}" +) +$reader = [System.Diagnostics.Eventing.Reader.EventLogReader]::new($query) +$records = [System.Collections.Generic.List[object]]::new() +try {{ + while ($record = $reader.ReadEvent()) {{ + try {{ + if ($records.Count -ge {_MAX_TCPIP_EVENT_RECORDS}) {{ + throw 'TCP/IP qualification event limit exceeded' + }} + $records.Add([pscustomobject]@{{ + event_id = [long]$record.Id + record_id = [long]$record.RecordId + created_at_utc = $record.TimeCreated.ToUniversalTime().ToString('O') + provider_name = $record.ProviderName + }}) + }} finally {{ + $record.Dispose() + }} + }} +}} finally {{ + $reader.Dispose() +}} +ConvertTo-Json -InputObject $records.ToArray() -Compress +""".strip() + ) + if not isinstance(value, list): + raise QualificationError("Windows TCP/IP event response is invalid") + records: list[TcpipEvent] = [] + for item in value: + _require_exact_keys( + item, + {"event_id", "record_id", "created_at_utc", "provider_name"}, + ) + if ( + not _is_nonnegative_int(item["event_id"]) + or not _is_nonnegative_int(item["record_id"]) + or not isinstance(item["created_at_utc"], str) + or not item["created_at_utc"] + or item["provider_name"] != "Tcpip" + ): + raise QualificationError("Windows TCP/IP event record is invalid") + records.append( + TcpipEvent( + event_id=item["event_id"], + record_id=item["record_id"], + created_at_utc=item["created_at_utc"], + ) + ) + return tuple(sorted(records, key=lambda record: record.record_id)) + + async def gateway_process_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayProcessSample: + _validate_checkpoint(phase, completed_batches, include_cleanup=False) + if not _is_positive_int(pid): + raise QualificationError("gateway process PID is invalid") + value = await _run_powershell_json( + f""" +$ErrorActionPreference = 'Stop' +$process = Get-Process -Id {pid} -ErrorAction Stop +[pscustomobject]@{{ + pid = [long]$process.Id + handle_count = [long]$process.HandleCount + private_bytes = [long]$process.PrivateMemorySize64 + rss_bytes = [long]$process.WorkingSet64 +}} | ConvertTo-Json -Compress +""".strip() + ) + _require_exact_keys( + value, + {"pid", "handle_count", "private_bytes", "rss_bytes"}, + ) + if ( + value["pid"] != pid + or not _is_nonnegative_int(value["handle_count"]) + or not _is_nonnegative_int(value["private_bytes"]) + or not _is_nonnegative_int(value["rss_bytes"]) + ): + raise QualificationError("Windows gateway process sample is invalid") + return GatewayProcessSample( + phase=phase, + captured_monotonic_ns=time.monotonic_ns(), + captured_utc_ns=time.time_ns(), + pid=pid, + completed_batches=completed_batches, + handle_count=value["handle_count"], + private_bytes=value["private_bytes"], + rss_bytes=value["rss_bytes"], + ) + + async def gateway_tcp_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayTcpSample: + _validate_checkpoint(phase, completed_batches, include_cleanup=True) + if not _is_positive_int(pid): + raise QualificationError("gateway TCP owner PID is invalid") + value = await _run_powershell_json( + f""" +$ErrorActionPreference = 'Stop' +$connections = @( + Get-NetTCPConnection -ErrorAction Stop | + Where-Object {{ $_.OwningProcess -eq {pid} }} | + ForEach-Object {{ + [pscustomobject]@{{ + state = [string]$_.State + local_address = [string]$_.LocalAddress + local_port = [long]$_.LocalPort + remote_address = [string]$_.RemoteAddress + remote_port = [long]$_.RemotePort + }} + }} +) +if ($connections.Count -gt {_MAX_TCPIP_EVENT_RECORDS}) {{ + throw 'Gateway TCP ownership record limit exceeded' +}} +ConvertTo-Json -InputObject $connections -Compress +""".strip() + ) + if not isinstance(value, list): + raise QualificationError("Windows gateway TCP response is invalid") + connections: list[OwnedTcpConnection] = [] + keys = { + "state", + "local_address", + "local_port", + "remote_address", + "remote_port", + } + for item in value: + _require_exact_keys(item, keys) + if ( + not isinstance(item["state"], str) + or not item["state"] + or not isinstance(item["local_address"], str) + or not item["local_address"] + or not _is_nonnegative_int(item["local_port"]) + or item["local_port"] > 65_535 + or not isinstance(item["remote_address"], str) + or not item["remote_address"] + or not _is_nonnegative_int(item["remote_port"]) + or item["remote_port"] > 65_535 + ): + raise QualificationError( + "Windows gateway TCP ownership record is invalid" + ) + connections.append( + OwnedTcpConnection( + state=item["state"], + local_address=item["local_address"], + local_port=item["local_port"], + remote_address=item["remote_address"], + remote_port=item["remote_port"], + ) + ) + return GatewayTcpSample( + phase=phase, + captured_monotonic_ns=time.monotonic_ns(), + captured_utc_ns=time.time_ns(), + pid=pid, + completed_batches=completed_batches, + connections=tuple(connections), + ) + + +async def _invoke_once( + runner: IpcQualificationRunner, + request_id: str, +) -> InvocationCapture: + started_monotonic_ns = time.monotonic_ns() + started_utc_ns = time.time_ns() + try: + capture = await runner.invoke(request_id) + except Exception as error: # noqa: BLE001 + ended_monotonic_ns = time.monotonic_ns() + ended_utc_ns = time.time_ns() + return InvocationCapture( + request_id=request_id, + started_monotonic_ns=started_monotonic_ns, + ended_monotonic_ns=ended_monotonic_ns, + started_utc_ns=started_utc_ns, + ended_utc_ns=ended_utc_ns, + return_code=None, + stdout=b"", + stderr=b"", + execution_error=f"runner_error:{type(error).__name__}", + ) + if capture.request_id != request_id: + return InvocationCapture( + request_id=request_id, + started_monotonic_ns=capture.started_monotonic_ns, + ended_monotonic_ns=capture.ended_monotonic_ns, + started_utc_ns=capture.started_utc_ns, + ended_utc_ns=capture.ended_utc_ns, + return_code=capture.return_code, + stdout=capture.stdout, + stderr=capture.stderr, + execution_error="request_id_mismatch", + credential_exposed=capture.credential_exposed, + ) + return capture + + +def _host_evidence( + *, + pre_cursor: EventLogCursor | None, + post_cursor: EventLogCursor | None, + identity_evidence: dict[str, Any] | None, + process_samples: list[GatewayProcessSample], + tcp_samples: list[GatewayTcpSample], + completed_batches: int, + tcpip_events: tuple[TcpipEvent, ...], + collector_errors: list[dict[str, str]], + stop_validation_errors: list[dict[str, str]], + runner_started: bool, +) -> tuple[dict[str, Any], list[str]]: + failures: list[str] = [] + if collector_errors: + failures.append("host_evidence_collection_error") + if identity_evidence is None or any( + error["phase"] == "qualification_identity" for error in collector_errors + ): + failures.append("qualification_identity_invalid") + if runner_started and stop_validation_errors: + failures.append("gateway_stop_evidence_invalid") + + event_query_completed = ( + pre_cursor is not None + and post_cursor is not None + and not any(error["phase"] == "tcpip_event_query" for error in collector_errors) + ) + if not event_query_completed: + failures.append("tcpip_event_evidence_incomplete") + elif tcpip_events: + failures.append("tcpip_event_detected") + + expected_checkpoints = _expected_checkpoint_keys(completed_batches) + actual_process_checkpoints = [ + (sample.phase, sample.completed_batches) for sample in process_samples + ] + process_growth: dict[str, Any] | None = None + if actual_process_checkpoints != expected_checkpoints: + failures.append("gateway_process_evidence_incomplete") + else: + readiness_sample = process_samples[0] + final_sample = process_samples[-1] + limits = { + "handle_count": MAX_GATEWAY_HANDLE_GROWTH, + "private_bytes": MAX_GATEWAY_PRIVATE_BYTES_GROWTH, + "rss_bytes": MAX_GATEWAY_RSS_BYTES_GROWTH, + } + peak_decisions = {} + final_decisions = {} + for name, limit in limits.items(): + readiness_value = getattr(readiness_sample, name) + peak_value = max(getattr(sample, name) for sample in process_samples) + final_value = getattr(final_sample, name) + peak_decisions[name] = { + "readiness": readiness_value, + "peak": peak_value, + "growth": peak_value - readiness_value, + "maximum_allowed_growth": limit, + "within_bound": peak_value - readiness_value <= limit, + } + final_decisions[name] = { + "readiness": readiness_value, + "final": final_value, + "growth": final_value - readiness_value, + "maximum_allowed_growth": limit, + "within_bound": final_value - readiness_value <= limit, + } + process_growth = { + "peak_over_readiness": { + "decisions": peak_decisions, + "passed": all( + decision["within_bound"] for decision in peak_decisions.values() + ), + }, + "final_pre_stop_over_readiness": { + "decisions": final_decisions, + "passed": all( + decision["within_bound"] for decision in final_decisions.values() + ), + }, + } + for metric, label in ( + ("handle_count", "handle"), + ("private_bytes", "private_bytes"), + ("rss_bytes", "rss"), + ): + if not peak_decisions[metric]["within_bound"]: + failures.append(f"gateway_{label}_peak_growth_exceeded") + if not final_decisions[metric]["within_bound"]: + failures.append(f"gateway_{label}_final_growth_exceeded") + + expected_tcp_checkpoints = [ + *expected_checkpoints, + ("after_cleanup", completed_batches), + ] + actual_tcp_checkpoints = [ + (sample.phase, sample.completed_batches) for sample in tcp_samples + ] + tcp_complete = actual_tcp_checkpoints == expected_tcp_checkpoints + if not tcp_complete: + failures.append("gateway_tcp_evidence_incomplete") + owned_tcp_count = sum(len(sample.connections) for sample in tcp_samples) + if owned_tcp_count: + failures.append("gateway_owned_tcp_detected") + + failures = list(dict.fromkeys(failures)) + evidence = { + "schema_version": 1, + "kind": "exp1_ipc_host_qualification_evidence", + "qualification_only": QUALIFICATION_ONLY, + "performance_evidence": PERFORMANCE_EVIDENCE, + "status": "passed" if not failures else "failed", + "gate_failures": failures, + "policy": { + "source": RESOURCE_GROWTH_POLICY_SOURCE, + "source_requirement": ( + "zero new qualifier-attributable System/Tcpip 4227/4231 " + "events, zero gateway-owned TCP endpoints, bounded gateway " + "process handle/private/RSS growth, and complete cleanup evidence" + ), + "threshold_origin": ( + "fixed preregistered fail-closed qualification policy; not a " + "performance result" + ), + "handle_threshold_basis": ( + "one handle per maximum pending named-pipe instance; product " + "listener cap is 32" + ), + "memory_threshold_basis": ( + "fixed 16 MiB post-readiness allowance for both private and " + "resident bytes; any larger retained growth blocks freeze" + ), + "event_ids": list(TCPIP_EVENT_IDS), + "process_sample_every_completed_batches": PROCESS_SAMPLE_EVERY_BATCHES, + "process_growth_gates": [ + "peak_over_readiness", + "final_pre_stop_over_readiness", + ], + "maximum_post_readiness_growth": { + "handle_count": MAX_GATEWAY_HANDLE_GROWTH, + "private_bytes": MAX_GATEWAY_PRIVATE_BYTES_GROWTH, + "rss_bytes": MAX_GATEWAY_RSS_BYTES_GROWTH, + }, + "missing_or_invalid_evidence_fails": True, + }, + "event_log": { + "log_name": "System", + "provider_name": "Tcpip", + "attribution_policy": ( + "every new matching event in the qualification cursor interval " + "is conservatively treated as qualifier-attributable" + ), + "pre_readiness_cursor": _event_cursor_record(pre_cursor), + "post_cleanup_cursor": _event_cursor_record(post_cursor), + "query_interval": "(pre_readiness_record_id, post_cleanup_record_id]", + "query_completed": event_query_completed, + "new_event_count": len(tcpip_events), + "events": [_tcpip_event_record(event) for event in tcpip_events], + "passed": event_query_completed and not tcpip_events, + }, + "gateway_process": { + "expected_checkpoints": [ + {"phase": phase, "completed_batches": batches} + for phase, batches in expected_checkpoints + ], + "samples": [_process_sample_record(sample) for sample in process_samples], + "growth": process_growth, + "passed": ( + process_growth is not None + and process_growth["peak_over_readiness"]["passed"] + and process_growth["final_pre_stop_over_readiness"]["passed"] + ), + }, + "gateway_tcp": { + "expected_checkpoints": [ + {"phase": phase, "completed_batches": batches} + for phase, batches in expected_tcp_checkpoints + ], + "samples": [_tcp_sample_record(sample) for sample in tcp_samples], + "owned_connection_count": owned_tcp_count, + "passed": tcp_complete and owned_tcp_count == 0, + }, + "gateway_stop": { + "required": runner_started, + "validation_passed": runner_started and not stop_validation_errors, + "validation_errors": stop_validation_errors, + }, + "identity": identity_evidence, + "collector_errors": collector_errors, + } + return evidence, failures + + +def _expected_checkpoint_keys(completed_batches: int) -> list[tuple[str, int]]: + return [ + ("readiness", 0), + *[ + ("cadence", batch) + for batch in range( + PROCESS_SAMPLE_EVERY_BATCHES, + completed_batches + 1, + PROCESS_SAMPLE_EVERY_BATCHES, + ) + ], + ("pre_stop", completed_batches), + ] + + +def _event_cursor_record(cursor: EventLogCursor | None) -> dict[str, Any] | None: + if cursor is None: + return None + return { + "phase": cursor.phase, + "captured_monotonic_ns": cursor.captured_monotonic_ns, + "captured_utc_ns": cursor.captured_utc_ns, + "log_name": cursor.log_name, + "last_record_id": cursor.last_record_id, + "source": cursor.source, + } + + +def _tcpip_event_record(event: TcpipEvent) -> dict[str, Any]: + return { + "event_id": event.event_id, + "record_id": event.record_id, + "created_at_utc": event.created_at_utc, + "provider_name": event.provider_name, + } + + +def _process_sample_record( + sample: GatewayProcessSample | None, +) -> dict[str, Any] | None: + if sample is None: + return None + return { + "phase": sample.phase, + "captured_monotonic_ns": sample.captured_monotonic_ns, + "captured_utc_ns": sample.captured_utc_ns, + "pid": sample.pid, + "completed_batches": sample.completed_batches, + "handle_count": sample.handle_count, + "private_bytes": sample.private_bytes, + "rss_bytes": sample.rss_bytes, + "source": sample.source, + } + + +def _tcp_sample_record(sample: GatewayTcpSample) -> dict[str, Any]: + return { + "phase": sample.phase, + "captured_monotonic_ns": sample.captured_monotonic_ns, + "captured_utc_ns": sample.captured_utc_ns, + "pid": sample.pid, + "completed_batches": sample.completed_batches, + "source": sample.source, + "connection_count": len(sample.connections), + "connections": [ + { + "state": connection.state, + "local_address": connection.local_address, + "local_port": connection.local_port, + "remote_address": connection.remote_address, + "remote_port": connection.remote_port, + } + for connection in sample.connections + ], + } + + +def _validate_event_cursor(cursor: EventLogCursor, phase: str) -> None: + if ( + cursor.phase != phase + or cursor.log_name != "System" + or cursor.source != "windows_event_log" + or not _is_positive_int(cursor.captured_monotonic_ns) + or not _is_positive_int(cursor.captured_utc_ns) + or not _is_nonnegative_int(cursor.last_record_id) + ): + raise QualificationError("Windows event cursor evidence is invalid") + + +def _validate_process_sample( + sample: GatewayProcessSample, + pid: int, + phase: str, + completed_batches: int, +) -> None: + if ( + sample.phase != phase + or sample.pid != pid + or sample.completed_batches != completed_batches + or sample.source != "windows_get_process" + or not _is_positive_int(sample.captured_monotonic_ns) + or not _is_positive_int(sample.captured_utc_ns) + or not _is_nonnegative_int(sample.handle_count) + or not _is_nonnegative_int(sample.private_bytes) + or not _is_nonnegative_int(sample.rss_bytes) + ): + raise QualificationError("Windows gateway process evidence is invalid") + + +def _validate_tcp_sample( + sample: GatewayTcpSample, + pid: int, + phase: str, + completed_batches: int, +) -> None: + if ( + sample.phase != phase + or sample.pid != pid + or sample.completed_batches != completed_batches + or sample.source != "windows_get_net_tcp_connection" + or not _is_positive_int(sample.captured_monotonic_ns) + or not _is_positive_int(sample.captured_utc_ns) + or not isinstance(sample.connections, tuple) + ): + raise QualificationError("Windows gateway TCP evidence is invalid") + for connection in sample.connections: + if ( + not isinstance(connection, OwnedTcpConnection) + or not connection.state + or not connection.local_address + or not _is_nonnegative_int(connection.local_port) + or connection.local_port > 65_535 + or not connection.remote_address + or not _is_nonnegative_int(connection.remote_port) + or connection.remote_port > 65_535 + ): + raise QualificationError("Windows gateway TCP evidence is invalid") + + +def _validate_stop_evidence( + evidence: dict[str, Any], + runner: IpcQualificationRunner, + gateway_pid: int, +) -> None: + _require_exact_keys( + evidence, + { + "schema_version", + "gateway_pid", + "executable_path", + "process_exited", + "pid_file_removed", + "sanitized_command", + "termination", + "return_code", + "stdout", + "stderr", + "auth_token_recorded", + }, + ) + if ( + evidence["schema_version"] != 1 + or evidence["gateway_pid"] != gateway_pid + or evidence["executable_path"] != runner.executable_paths["gateway"] + or evidence["process_exited"] is not True + or evidence["pid_file_removed"] is not True + or evidence["sanitized_command"] != runner.sanitized_commands["gateway_serve"] + or evidence["termination"] + not in { + "terminate", + "already_exited", + } + or type(evidence["return_code"]) is not int + or evidence["auth_token_recorded"] is not False + ): + raise QualificationError("gateway stop evidence is invalid") + _validate_stream_digest(evidence["stdout"]) + _validate_stream_digest(evidence["stderr"]) + + +def _validate_stream_digest(value: Any) -> None: + _require_exact_keys(value, {"bytes", "sha256"}) + if ( + not _is_nonnegative_int(value["bytes"]) + or not isinstance(value["sha256"], str) + or re.fullmatch(r"sha256:[0-9a-f]{64}", value["sha256"]) is None + ): + raise QualificationError("gateway stream digest evidence is invalid") + + +def _validate_qualification_identity( + value: dict[str, Any], + roots: BenchmarkRoots, + runner: IpcQualificationRunner, +) -> None: + _require_exact_keys( + value, + { + "schema_version", + "captured_monotonic_ns", + "captured_utc_ns", + "product", + "paper", + "executables", + "packaged_support", + "qualifier_sources", + "host", + "build", + "sanitized_commands", + }, + ) + if ( + value["schema_version"] != 1 + or not _is_positive_int(value["captured_monotonic_ns"]) + or not _is_positive_int(value["captured_utc_ns"]) + ): + raise QualificationError("qualification identity evidence is invalid") + + product = value["product"] + _require_exact_keys( + product, + { + "commit", + "branch", + "status_clean", + "status_bytes", + "status_sha256", + "package_directory", + "package_directory_name", + "expected_package_directory_name", + "package_name_matches_commit", + "package_zip_path", + "package_zip_bytes", + "package_zip_sha256", + }, + ) + commit = product["commit"] + expected_package = f"windows-exp1-{commit[:8]}" if isinstance(commit, str) else "" + package_directory = os.fspath(roots.product_bin_dir.parent) + if ( + not isinstance(commit, str) + or re.fullmatch(r"[0-9a-f]{40}", commit) is None + or commit != PREREGISTERED_PRODUCT_COMMIT + or product["branch"] != PREREGISTERED_PRODUCT_BRANCH + or product["status_clean"] is not True + or product["status_bytes"] != 0 + or product["status_sha256"] != _sha256_bytes(b"") + or product["package_directory"] != package_directory + or product["package_directory_name"] != expected_package + or product["expected_package_directory_name"] != expected_package + or product["package_name_matches_commit"] is not True + or product["package_zip_path"] != f"{package_directory}.zip" + or not _is_positive_int(product["package_zip_bytes"]) + or not _is_sha256(product["package_zip_sha256"]) + or product["package_zip_sha256"] != PREREGISTERED_PACKAGE_SHA256 + ): + raise QualificationError("product/package identity evidence is invalid") + + paper = value["paper"] + _require_exact_keys( + paper, + { + "commit", + "scoped_status_clean", + "scoped_status_bytes", + "scoped_status_sha256", + "paper_root", + "frozen_scope", + "generated_exclusions", + "policy_source", + }, + ) + if ( + not isinstance(paper["commit"], str) + or re.fullmatch(r"[0-9a-f]{40}", paper["commit"]) is None + or paper["scoped_status_clean"] is not True + or paper["scoped_status_bytes"] != 0 + or paper["scoped_status_sha256"] != _sha256_bytes(b"") + or paper["paper_root"] != os.fspath(roots.test_repository_root) + or paper["frozen_scope"] != list(PAPER_FROZEN_SCOPE) + or paper["generated_exclusions"] != list(BENCHMARK_GIT_EXCLUSIONS) + or paper["policy_source"] + != ( + "experiments/scripts/archive_exp1_run.py:" + "PAPER_FROZEN_SCOPE+BENCHMARK_GIT_EXCLUSIONS" + ) + ): + raise QualificationError("paper scoped identity evidence is invalid") + + executables = value["executables"] + if not isinstance(executables, dict) or set(executables) != set( + runner.executable_paths + ): + raise QualificationError("executable identity evidence is invalid") + for role, expected_path in runner.executable_paths.items(): + _validate_file_identity(executables[role], expected_path) + if executables[role]["sha256"] != PREREGISTERED_ARTIFACT_SHA256[role]: + raise QualificationError("preregistered executable identity drifted") + + packaged_support = value["packaged_support"] + if not isinstance(packaged_support, dict) or set(packaged_support) != { + "linux_daemon", + "windows_config", + }: + raise QualificationError("packaged support identity evidence is invalid") + package_directory_path = Path(package_directory) + for role, relative in ( + ("linux_daemon", Path("dist/sandbox-daemon-linux-amd64")), + ("windows_config", Path("config/windows-amd64.yml")), + ): + _validate_file_identity( + packaged_support[role], + os.fspath(package_directory_path / relative), + ) + if packaged_support[role]["sha256"] != PREREGISTERED_ARTIFACT_SHA256[role]: + raise QualificationError("preregistered support identity drifted") + + qualifier_path = os.fspath(Path(__file__).resolve(strict=True)) + sources = value["qualifier_sources"] + if not isinstance(sources, dict) or set(sources) != { + "ipc_qualification", + "benchmark_cli", + "packaged_gateway_launcher", + "ipc_qualification_test", + "protocol_amendment", + }: + raise QualificationError("qualifier source identity evidence is invalid") + _validate_file_identity(sources["ipc_qualification"], qualifier_path) + _validate_file_identity( + sources["benchmark_cli"], + os.fspath(Path(qualifier_path).with_name("cli.py")), + ) + _validate_file_identity( + sources["packaged_gateway_launcher"], + os.fspath(roots.product_bin_dir / "start-sandbox-windows-docker-gateway.ps1"), + ) + _validate_file_identity( + sources["ipc_qualification_test"], + os.fspath( + roots.benchmark_source_root + / "backend" + / "tests" + / "unit" + / "test_ipc_qualification.py" + ), + ) + _validate_file_identity( + sources["protocol_amendment"], + os.fspath( + roots.test_repository_root + / "experiments" + / "exp1-v1.1-protocol-amendment.md" + ), + ) + + host = value["host"] + _require_exact_keys( + host, + { + "computer_name", + "os_caption", + "os_version", + "os_build_number", + "architecture", + "logical_processors", + "total_memory_bytes", + }, + ) + if ( + any( + not isinstance(host[name], str) or not host[name].strip() + for name in ( + "computer_name", + "os_caption", + "os_version", + "architecture", + ) + ) + or not _is_positive_int(host["os_build_number"]) + or not _is_positive_int(host["logical_processors"]) + or not _is_positive_int(host["total_memory_bytes"]) + or any(host[key] != expected for key, expected in EXP1_EXPECTED_HOST.items()) + ): + raise QualificationError("host identity evidence is invalid") + + build = value["build"] + _require_exact_keys( + build, + { + "package_identity", + "product_commit", + "python_executable", + "python_implementation", + "python_version", + "python_architecture", + "package_build_command", + }, + ) + if ( + build["package_identity"] != expected_package + or build["product_commit"] != commit + or not all( + isinstance(build[field], str) and build[field] + for field in ( + "python_executable", + "python_implementation", + "python_version", + "python_architecture", + ) + ) + or build["package_build_command"] + != [ + "powershell.exe", + "-NoProfile", + "-ExecutionPolicy", + "Bypass", + "-File", + ".\\bin\\package-windows-amd64-release.ps1", + "-PackageName", + expected_package, + "-OutDir", + "target", + "-Profile", + "release", + ] + ): + raise QualificationError("build identity evidence is invalid") + if value["sanitized_commands"] != _expected_sanitized_commands(roots, runner): + raise QualificationError("sanitized command evidence is invalid") + + +def _validate_file_identity(value: Any, expected_path: str) -> None: + _require_exact_keys(value, {"path", "bytes", "sha256"}) + if ( + value["path"] != expected_path + or not _is_positive_int(value["bytes"]) + or not _is_sha256(value["sha256"]) + ): + raise QualificationError("file identity evidence is invalid") + + +def _is_sha256(value: Any) -> bool: + return ( + isinstance(value, str) + and re.fullmatch(r"sha256:[0-9a-f]{64}", value) is not None + ) + + +def _validate_tcpip_events( + events: tuple[TcpipEvent, ...], + pre_cursor: EventLogCursor, + post_cursor: EventLogCursor, +) -> None: + if ( + pre_cursor.log_name != post_cursor.log_name + or post_cursor.last_record_id < pre_cursor.last_record_id + or len(events) > _MAX_TCPIP_EVENT_RECORDS + ): + raise QualificationError("Windows TCP/IP event evidence is invalid") + record_ids: set[int] = set() + for event in events: + if ( + event.event_id not in TCPIP_EVENT_IDS + or event.provider_name != "Tcpip" + or not isinstance(event.created_at_utc, str) + or not event.created_at_utc + or event.record_id <= pre_cursor.last_record_id + or event.record_id > post_cursor.last_record_id + or event.record_id in record_ids + ): + raise QualificationError("Windows TCP/IP event evidence is invalid") + record_ids.add(event.record_id) + + +async def _run_powershell_json(script: str) -> Any: + process = await asyncio.create_subprocess_exec( + "powershell.exe", + "-NoLogo", + "-NoProfile", + "-NonInteractive", + "-Command", + script, + env=_native_environment(), + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(), + timeout=_COLLECTOR_TIMEOUT_SECONDS, + ) + except TimeoutError as error: + process.kill() + await process.wait() + raise QualificationError("Windows evidence collector timed out") from error + if len(stdout) > _MAX_CAPTURE_BYTES or len(stderr) > _MAX_CAPTURE_BYTES: + raise QualificationError("Windows evidence collector output is oversized") + if process.returncode != 0: + raise QualificationError("Windows evidence collector exited nonzero") + if stderr: + raise QualificationError("Windows evidence collector emitted stderr") + if not stdout.endswith(b"\n") or stdout.count(b"\n") != 1: + raise QualificationError("Windows evidence collector output framing is invalid") + try: + return json.loads(stdout.decode("utf-8-sig")) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise QualificationError( + "Windows evidence collector output is invalid" + ) from error + + +async def _run_strict_command( + arguments: tuple[str, ...], + *, + cwd: Path, +) -> bytes: + process = await asyncio.create_subprocess_exec( + *arguments, + cwd=cwd, + env=_native_environment(), + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(), + timeout=_COLLECTOR_TIMEOUT_SECONDS, + ) + except TimeoutError as error: + process.kill() + await process.wait() + raise QualificationError("identity command timed out") from error + if len(stdout) > _MAX_CAPTURE_BYTES or len(stderr) > _MAX_CAPTURE_BYTES: + raise QualificationError("identity command output is oversized") + if process.returncode != 0: + raise QualificationError("identity command exited nonzero") + if stderr: + raise QualificationError("identity command emitted stderr") + return stdout + + +def _expected_sanitized_commands( + roots: BenchmarkRoots, + runner: IpcQualificationRunner, +) -> dict[str, Any]: + return { + **runner.sanitized_commands, + "product_commit": { + "executable": "git", + "argv": ["rev-parse", "HEAD"], + "working_directory": os.fspath(roots.product_root), + }, + "product_status": { + "executable": "git", + "argv": ["status", "--porcelain=v1", "-z"], + "working_directory": os.fspath(roots.product_root), + }, + "paper_commit": { + "executable": "git", + "argv": ["rev-parse", "HEAD"], + "working_directory": os.fspath(roots.test_repository_root), + }, + "paper_scoped_status": { + "executable": "git", + "argv": [ + "status", + "--porcelain=v1", + "-z", + "--", + *PAPER_FROZEN_SCOPE, + *BENCHMARK_GIT_EXCLUSIONS, + ], + "working_directory": os.fspath(roots.test_repository_root), + }, + "host_identity": { + "executable": "powershell.exe", + "script_id": "windows_cim_host_build_identity_v1", + }, + "event_cursor": { + "executable": "powershell.exe", + "script_id": "windows_system_event_log_cursor_v1", + }, + "tcpip_event_query": { + "executable": "powershell.exe", + "script_id": "windows_system_tcpip_4227_4231_cursor_query_v1", + }, + "gateway_process_sample": { + "executable": "powershell.exe", + "script_id": "windows_get_process_handle_private_rss_v1", + }, + "gateway_tcp_sample": { + "executable": "powershell.exe", + "script_id": "windows_get_net_tcp_connection_owner_pid_v1", + }, + } + + +def _file_identity(path: Path, parent: Path) -> dict[str, Any]: + _require_safe_file(path) + canonical = path.resolve(strict=True) + if not canonical.is_relative_to(parent): + raise QualificationError("qualification identity file escaped its root") + return { + "path": os.fspath(canonical), + "bytes": canonical.stat().st_size, + "sha256": _sha256_file(canonical), + } + + +def _require_safe_file(path: Path) -> None: + try: + canonical = path.resolve(strict=True) + except OSError as error: + raise QualificationError("qualification identity file is missing") from error + if path.is_symlink() or not path.is_file() or canonical != path: + raise QualificationError("qualification identity file is unsafe") + + +def _require_safe_directory(path: Path) -> None: + try: + canonical = path.resolve(strict=True) + except OSError as error: + raise QualificationError( + "qualification identity directory is missing" + ) from error + if path.is_symlink() or not path.is_dir() or canonical != path: + raise QualificationError("qualification identity directory is unsafe") + + +def _collector_error(phase: str, error: Exception) -> dict[str, str]: + return { + "phase": phase, + "error_type": type(error).__name__, + } + + +def _require_exact_keys(value: Any, keys: set[str]) -> None: + if not isinstance(value, dict) or set(value) != keys: + raise QualificationError("Windows evidence collector shape is invalid") + + +def _validate_phase(phase: str, allowed: set[str]) -> None: + if phase not in allowed: + raise QualificationError("qualification evidence phase is invalid") + + +def _validate_checkpoint( + phase: str, + completed_batches: int, + *, + include_cleanup: bool, +) -> None: + if not _is_nonnegative_int(completed_batches): + raise QualificationError("qualification checkpoint batch is invalid") + if phase == "readiness" and completed_batches == 0: + return + if ( + phase == "cadence" + and completed_batches > 0 + and completed_batches % PROCESS_SAMPLE_EVERY_BATCHES == 0 + ): + return + if phase == "pre_stop": + return + if include_cleanup and phase == "after_cleanup": + return + raise QualificationError("qualification evidence checkpoint is invalid") + + +def _is_positive_int(value: Any) -> bool: + return type(value) is int and value > 0 + + +def _is_nonnegative_int(value: Any) -> bool: + return type(value) is int and value >= 0 + + +def _invocation_record( + capture: InvocationCapture, + *, + batch_index: int, + slot: int, + invocation_index: int, +) -> dict[str, Any]: + validation = _validate_capture(capture) + return { + "schema_version": 1, + "qualification_only": QUALIFICATION_ONLY, + "performance_evidence": PERFORMANCE_EVIDENCE, + "invocation_index": invocation_index, + "batch_index": batch_index, + "slot": slot, + "request_id": capture.request_id, + "operation": "list_sandboxes", + "executable_role": "manager", + "started_monotonic_ns": capture.started_monotonic_ns, + "ended_monotonic_ns": capture.ended_monotonic_ns, + "elapsed_ns": max( + 0, + capture.ended_monotonic_ns - capture.started_monotonic_ns, + ), + "started_utc_ns": capture.started_utc_ns, + "ended_utc_ns": capture.ended_utc_ns, + "return_code": capture.return_code, + "stdout_bytes": len(capture.stdout), + "stderr_bytes": len(capture.stderr), + "stdout_sha256": _sha256_bytes(capture.stdout), + "stderr_sha256": _sha256_bytes(capture.stderr), + "result": "passed" if validation == "passed" else "failed", + "validation": validation, + "auth_token_recorded": False, + } + + +def _validate_capture(capture: InvocationCapture) -> str: + if capture.credential_exposed: + return "credential_echo" + if capture.execution_error is not None: + return capture.execution_error + if len(capture.stdout) > _MAX_CAPTURE_BYTES: + return "stdout_oversize" + if len(capture.stderr) > _MAX_CAPTURE_BYTES: + return "stderr_oversize" + if capture.return_code != 0: + return "nonzero_exit" + if capture.stderr: + return "unexpected_stderr" + if not capture.stdout.endswith(b"\n") or capture.stdout.count(b"\n") != 1: + return "stdout_framing" + try: + value = json.loads(capture.stdout) + except (UnicodeDecodeError, json.JSONDecodeError): + return "invalid_json" + if ( + not isinstance(value, dict) + or set(value) != {"sandboxes"} + or value["sandboxes"] != [] + ): + return "response_shape" + return "passed" + + +class _NdjsonWriter: + def __init__(self, path: Path) -> None: + self._path = path + self._descriptor = os.open( + path, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) + self._closed = False + + def append(self, records: list[dict[str, Any]]) -> None: + payload = b"".join( + json.dumps( + record, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ).encode() + + b"\n" + for record in records + ) + _write_all(self._descriptor, payload) + + def close(self) -> None: + if self._closed: + return + self._closed = True + try: + os.fsync(self._descriptor) + finally: + os.close(self._descriptor) + _sync_directory(self._path.parent) + + +def _create_evidence_root( + roots: BenchmarkRoots, + qualification_id: str, +) -> Path: + roots.validate_state() + _validate_identity(qualification_id) + parent = roots.results / "qualification-only" + parent.mkdir(mode=0o700, exist_ok=True) + evidence_root = parent / f"exp1-ipc-{qualification_id}" + evidence_root.mkdir(mode=0o700) + return evidence_root.resolve(strict=True) + + +def _request_id(qualification_id: str, batch_index: int, slot: int) -> str: + return f"exp1-ipc-{qualification_id}-batch-{batch_index:04d}-slot-{slot}" + + +def _validate_identity(value: str) -> None: + if _IDENTITY.fullmatch(value) is None: + raise QualificationError("qualification identity is invalid") + + +def _validate_workload(workload: QualificationWorkload) -> None: + if workload.batches < 1 or workload.concurrency < 1: + raise QualificationError("qualification workload must be positive") + + +def _validate_npipe_endpoint(endpoint: str) -> None: + name = endpoint.removeprefix(_NPIPE_PREFIX) + native_path = rf"\\.\pipe\{name.replace('/', '\\')}" + if ( + not endpoint.startswith(_NPIPE_PREFIX) + or not name + or _NPIPE_NAME.fullmatch(name) is None + or "\\" in endpoint + or "tcp://" in endpoint.lower() + or any(segment in {"", ".", ".."} for segment in name.split("/")) + or len(native_path.encode("utf-16-le")) // 2 > 256 + ): + raise QualificationError("qualification endpoint must be one safe npipe URI") + + +def _native_environment() -> dict[str, str]: + return { + name: os.environ[name] + for name in ("PATH", "SystemRoot", "WINDIR", "TEMP", "TMP") + if name in os.environ + } + + +async def _digest_stream( + stream: asyncio.StreamReader, +) -> dict[str, Any]: + digest = hashlib.sha256() + byte_count = 0 + while chunk := await stream.read(64 * 1024): + byte_count += len(chunk) + digest.update(chunk) + return { + "bytes": byte_count, + "sha256": f"sha256:{digest.hexdigest()}", + } + + +def _empty_stream_digest() -> dict[str, Any]: + return { + "bytes": 0, + "sha256": _sha256_bytes(b""), + } + + +def _sha256_bytes(value: bytes) -> str: + return f"sha256:{hashlib.sha256(value).hexdigest()}" + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return f"sha256:{digest.hexdigest()}" + + +def _write_new_json(path: Path, value: dict[str, Any]) -> None: + payload = ( + json.dumps( + value, + indent=2, + sort_keys=True, + ensure_ascii=False, + allow_nan=False, + ).encode() + + b"\n" + ) + descriptor = os.open( + path, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) + try: + _write_all(descriptor, payload) + os.fsync(descriptor) + finally: + os.close(descriptor) + _sync_directory(path.parent) + + +def _write_all(descriptor: int, payload: bytes) -> None: + view = memoryview(payload) + while view: + written = os.write(descriptor, view) + if written <= 0: + raise OSError("short qualification evidence write") + view = view[written:] diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/metadata.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/metadata.py index ac1216a..b63923d 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/metadata.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/metadata.py @@ -2,16 +2,55 @@ import asyncio import hashlib +import json import os import platform import shutil +from datetime import datetime, timezone from pathlib import Path from typing import Any +import yaml + +from .catalog import _prebuilt_executable +from .gateway import _container_daemon_executable from .paths import BenchmarkRoots +_EXP1_PAPER_PLANS = frozenset( + {"paper-env-smoke", "paper-pilot", "paper-good-pass"} +) +_EXP1_EXPECTED_HOST = { + "computer_name": "DESKTOP-OLP1ADS", + "operating_system": "windows", + "architecture": "x64", + "os_build_number": 26200, + "logical_processors": 48, + "total_memory_bytes": 137_438_953_472, + "filesystem": "NTFS", +} +_EXP1_EXPECTED_SANDBOX_LIMITS = { + "profile": "standard", + "nano_cpus": 1_000_000_000, + "vcpus": 1, + "memory_bytes": 536_870_912, + "pids_limit": 256, +} + + +class EnvironmentMetadataError(RuntimeError): + """Required run-start provenance could not be captured or validated.""" + + async def collect_environment(roots: BenchmarkRoots, plan: dict[str, Any]) -> dict[str, Any]: + paper_exp1 = _is_exp1_paper_plan(plan) + final_host: dict[str, Any] | None = None + sandbox_limits: dict[str, Any] | None = None + if paper_exp1: + final_host = await _capture_windows_final_host(roots.benchmark_state_root) + sandbox_limits = _configured_sandbox_limits(roots) + _validate_exp1_host(final_host) + _validate_exp1_sandbox_limits(sandbox_limits) commit = await _command(["git", "rev-parse", "HEAD"], roots.product_root) status = await _command(["git", "status", "--porcelain=v1", "-z"], roots.product_root) docker_version = await _command( @@ -21,10 +60,34 @@ async def collect_environment(roots: BenchmarkRoots, plan: dict[str, Any]) -> di image_digest = await _command( ["docker", "image", "inspect", "--format", "{{.Id}}", image], roots.product_root ) - daemon = _daemon_path(roots) - gateway = roots.product_bin_dir / "sandbox-gateway" + daemon = _container_daemon_executable(roots) + gateway = _prebuilt_executable(roots, "sandbox-gateway") + manager_cli = _prebuilt_executable(roots, "sandbox-manager-cli") + runtime_cli = _prebuilt_executable(roots, "sandbox-runtime-cli") + observability_cli = _prebuilt_executable(roots, "sandbox-observability-cli") usage = shutil.disk_usage(roots.benchmark_state_root) effective = plan["effective_environment"] + host = { + "operating_system": platform.system().lower(), + "architecture": platform.machine().lower(), + "kernel_release": platform.release() or None, + "docker_engine_version": docker_version or None, + "filesystem": effective.get("filesystem"), + "free_space_bytes": usage.free, + "monotonic_clock": "time.monotonic_ns", + } + if final_host is not None: + host.update(final_host) + windows_named_pipe = ( + os.name == "nt" and effective["client_cohort"] == "product_cli" + ) + gateway_transport = { + "transport": ( + "windows_named_pipe" if windows_named_pipe else "tcp_loopback" + ), + "scope": "local_only", + "rotation": "per_execution_block", + } return { "schema_version": 1, "treatment": { @@ -33,22 +96,213 @@ async def collect_environment(roots: BenchmarkRoots, plan: dict[str, Any]) -> di "source_diff_hash": _sha(status.encode()) if status else None, "daemon_binary_hash": _sha_file(daemon), "gateway_binary_hash": _sha_file(gateway), + "manager_cli_binary_hash": _sha_file(manager_cli), + "runtime_cli_binary_hash": _sha_file(runtime_cli), + "observability_cli_binary_hash": _sha_file(observability_cli), }, - "host": { - "operating_system": platform.system().lower(), - "architecture": platform.machine().lower(), - "kernel_release": platform.release() or None, - "docker_engine_version": docker_version or None, - "filesystem": effective.get("filesystem"), - "free_space_bytes": usage.free, - "monotonic_clock": "time.monotonic_ns", - }, + "host": host, + "sandbox_limits": sandbox_limits, "image_reference": image, "image_digest": image_digest or effective.get("image_digest"), "workspace_root_identity": effective["workspace_root_identity"], "client_cohort": effective["client_cohort"], - "gateway_endpoint_identity": "isolated_loopback_per_execution_block", + "gateway_endpoint_identity": ( + "isolated_windows_named_pipe_per_execution_block" + if windows_named_pipe + else "isolated_loopback_per_execution_block" + ), + "gateway_transport": gateway_transport, + } + + +def _is_exp1_paper_plan(plan: dict[str, Any]) -> bool: + canonical = plan.get("canonical_plan") + return ( + isinstance(canonical, dict) + and canonical.get("name") in _EXP1_PAPER_PLANS + ) + + +async def _capture_windows_final_host(target_path: Path) -> dict[str, Any]: + if os.name != "nt": + raise EnvironmentMetadataError( + "EXP1 final-host capture requires native Windows" + ) + script = ( + "$ErrorActionPreference='Stop'; " + "$target=(Get-Item -LiteralPath '.').FullName; " + "$os=Get-CimInstance Win32_OperatingSystem; " + "$computer=Get-CimInstance Win32_ComputerSystem; " + "$processors=@(Get-CimInstance Win32_Processor); " + "$names=@($processors | ForEach-Object { " + "([string]$_.Name).Trim() } | Sort-Object -Unique); " + "$processorLogical=[int](($processors | Measure-Object " + "-Property NumberOfLogicalProcessors -Sum).Sum); " + "$volumeRoot=[System.IO.Path]::GetPathRoot($target); " + "$volume=Get-Volume -DriveLetter $volumeRoot.Substring(0,1); " + "[ordered]@{" + "computer_name=[string]$computer.Name;" + "operating_system='windows';" + "os_caption=[string]$os.Caption;" + "os_version=[string]$os.Version;" + "os_build_number=[int]$os.BuildNumber;" + "architecture=[System.Runtime.InteropServices.RuntimeInformation]::" + "OSArchitecture.ToString().ToLowerInvariant();" + "cpu_model=($names -join ' + ');" + "logical_processors=[int]$computer.NumberOfLogicalProcessors;" + "processor_logical_processors=$processorLogical;" + "total_memory_bytes=[int64]$computer.TotalPhysicalMemory;" + "filesystem=[string]$volume.FileSystem;" + "volume_root=$volumeRoot" + "} | ConvertTo-Json -Compress" + ) + raw = await _command( + [ + "powershell.exe", + "-NoProfile", + "-NonInteractive", + "-ExecutionPolicy", + "Bypass", + "-Command", + script, + ], + target_path.resolve(strict=True), + ) + try: + value = json.loads(raw) + except (json.JSONDecodeError, TypeError) as error: + raise EnvironmentMetadataError( + "Windows final-host capture did not return valid JSON" + ) from error + if not isinstance(value, dict): + raise EnvironmentMetadataError( + "Windows final-host capture did not return an object" + ) + value.update( + { + "capture_boundary": "run_start_before_gateway_and_measurement", + "captured_at": datetime.now(timezone.utc) + .isoformat(timespec="microseconds") + .replace("+00:00", "Z"), + "capture_source": ( + "Windows CIM Win32_OperatingSystem, Win32_ComputerSystem, " + "Win32_Processor, and Get-Volume" + ), + } + ) + return value + + +def _configured_sandbox_limits(roots: BenchmarkRoots) -> dict[str, Any]: + template = ( + roots.product_bin_dir.parent / "config/windows-amd64.yml" + if os.name == "nt" + else roots.benchmark_source_root / "defaults/gateway.yml" + ) + return _configured_sandbox_limits_from_template(template) + + +def _configured_sandbox_limits_from_template( + template: Path, +) -> dict[str, Any]: + if template.is_symlink() or not template.is_file(): + raise EnvironmentMetadataError( + "authoritative gateway configuration is missing or unsafe" + ) + try: + config = yaml.safe_load(template.read_bytes()) + docker = config["manager"]["docker"] + profile_name = docker["resource_profile"] + profile = docker["resource_profiles"][profile_name] + nano_cpus = docker.get("nano_cpus", profile["nano_cpus"]) + memory_bytes = docker.get("memory_bytes", profile["memory_max_bytes"]) + pids_limit = profile["pids_max"] + except (KeyError, TypeError, yaml.YAMLError) as error: + raise EnvironmentMetadataError( + "authoritative gateway resource configuration is invalid" + ) from error + values = { + "profile": profile_name, + "nano_cpus": nano_cpus, + "vcpus": ( + nano_cpus // 1_000_000_000 + if isinstance(nano_cpus, int) + and not isinstance(nano_cpus, bool) + and nano_cpus % 1_000_000_000 == 0 + else None + ), + "memory_bytes": memory_bytes, + "pids_limit": pids_limit, + "authority": { + "kind": "released_gateway_configuration", + "path": os.fspath(template.resolve(strict=True)), + "sha256": _sha_file(template), + "selector": ( + f"manager.docker.resource_profiles.{profile_name} " + "with manager.docker nano_cpus/memory_bytes overrides" + ), + "effective_config_builder": ( + "benchmark_lab.gateway._effective_config preserves the " + "selected profile and resource override fields" + ), + "create_request_override": "none", + "capture_boundary": "run_start_before_gateway_and_measurement", + }, } + return values + + +def _validate_exp1_host(host: dict[str, Any]) -> None: + required_text = ("os_caption", "os_version", "cpu_model", "volume_root") + if any(not isinstance(host.get(field), str) or not host[field].strip() for field in required_text): + raise EnvironmentMetadataError( + "EXP1 final-host text fields are incomplete" + ) + if host.get("logical_processors") != host.get( + "processor_logical_processors" + ): + raise EnvironmentMetadataError( + "Windows processor and computer-system logical CPU counts disagree" + ) + mismatches = [ + field + for field, expected in _EXP1_EXPECTED_HOST.items() + if ( + str(host.get(field)).casefold() != expected.casefold() + if isinstance(expected, str) + else host.get(field) != expected + ) + ] + if mismatches: + raise EnvironmentMetadataError( + "EXP1 final-host identity drift: " + ", ".join(mismatches) + ) + if host.get("capture_boundary") != "run_start_before_gateway_and_measurement": + raise EnvironmentMetadataError("EXP1 host capture boundary is invalid") + + +def _validate_exp1_sandbox_limits(limits: dict[str, Any]) -> None: + mismatches = [ + field + for field, expected in _EXP1_EXPECTED_SANDBOX_LIMITS.items() + if limits.get(field) != expected + ] + authority = limits.get("authority") + if ( + mismatches + or not isinstance(authority, dict) + or authority.get("kind") != "released_gateway_configuration" + or not isinstance(authority.get("path"), str) + or not isinstance(authority.get("sha256"), str) + or not authority["sha256"].startswith("sha256:") + or authority.get("create_request_override") != "none" + or authority.get("capture_boundary") + != "run_start_before_gateway_and_measurement" + ): + detail = ", ".join(mismatches) or "authority" + raise EnvironmentMetadataError( + f"EXP1 effective sandbox limit drift: {detail}" + ) async def _command(args: list[str], cwd: Path) -> str: @@ -62,6 +316,9 @@ async def _command(args: list[str], cwd: Path) -> str: for name in ( "PATH", "HOME", + "USERPROFILE", + "SystemRoot", + "WINDIR", "DOCKER_HOST", "DOCKER_CONTEXT", "DOCKER_CONFIG", @@ -87,11 +344,6 @@ async def _command(args: list[str], cwd: Path) -> str: return stdout.decode(errors="replace").strip() -def _daemon_path(roots: BenchmarkRoots) -> Path: - suffix = "arm64" if platform.machine().lower() in {"arm64", "aarch64"} else "amd64" - return roots.product_root / "dist" / f"sandbox-daemon-linux-{suffix}" - - def _sha_file(path: Path) -> str | None: try: hasher = hashlib.sha256() diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/models.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/models.py index 9068d3b..906a5de 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/models.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/models.py @@ -75,7 +75,7 @@ class RunManifestCore(StrictModel): class BenchmarkReportV4(StrictModel): schema_version: Literal[4] - report_derivation_revision: Literal[3] + report_derivation_revision: Literal[3, 4] run_id: str = Field(min_length=1) state: Literal[ "planned", @@ -111,7 +111,7 @@ class BenchmarkReportV4(StrictModel): class RunDerivedSummaryV4(StrictModel): schema_version: Literal[4] - report_derivation_revision: Literal[3] + report_derivation_revision: Literal[3, 4] run_id: str = Field(min_length=1) plan_hash: str = Field(min_length=1) state: Literal[ diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/observability.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/observability.py index d524bcd..021c6cb 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/observability.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/observability.py @@ -2,7 +2,7 @@ from typing import Any, Literal -from pydantic import Field, ValidationError, model_validator +from pydantic import ConfigDict, Field, ValidationError, model_validator from .models import StrictModel @@ -36,10 +36,17 @@ class CgroupSample(StrictModel): class CgroupView(StrictModel): view: Literal["cgroup"] scope: Literal["sandbox"] - series: list[CgroupSample] = Field(min_length=1, max_length=4096) + availability: Literal["available", "partial"] + errors: list[str] + topology: dict[str, Any] + series: list[CgroupSample] = Field(max_length=4096) @model_validator(mode="after") def validate_series(self) -> "CgroupView": + if (self.availability == "available") != (not self.errors): + raise ValueError("cgroup availability and errors disagree") + if not self.series and self.availability != "partial": + raise ValueError("available cgroup view has no samples") previous: CgroupSample | None = None for sample in self.series: expected_interval = None if previous is None else sample.ts - previous.ts @@ -66,34 +73,61 @@ def validate_series(self) -> "CgroupView": return self -class SnapshotMetrics(StrictModel): +class SnapshotSandboxMetrics(StrictModel): + metrics_source: Literal["sandbox_cgroup"] + cgroup_path: str = Field(min_length=1) cpu_usec: int | None = Field(default=None, ge=0) mem_cur: int | None = Field(default=None, ge=0) mem_max: int | None = Field(default=None, ge=0) mem_max_unlimited: bool | None = None - cgroup_available: bool | None = None + cgroup_available: bool cgroup_error: str | None = None + io_rbytes: int | None = Field(default=None, ge=0) + io_wbytes: int | None = Field(default=None, ge=0) + pids_cur: int | None = Field(default=None, ge=0) + record_truncated_bytes: int | None = Field(default=None, alias="_truncated", ge=0) + + +class SnapshotWorkspaceMetrics(StrictModel): disk_bytes: int | None = Field(default=None, ge=0) disk_allocated_bytes: int | None = Field(default=None, ge=0) files: int | None = Field(default=None, ge=0) - disk_truncated: bool | None = None + disk_truncated: bool record_truncated_bytes: int | None = Field(default=None, alias="_truncated", ge=0) -class SnapshotDeltas(StrictModel): +class SnapshotSandboxDeltas(StrictModel): cpu_usec: int | None = Field(default=None, ge=0) + io_rbytes: int | None = Field(default=None, ge=0) + io_wbytes: int | None = Field(default=None, ge=0) + + +class SnapshotWorkspaceDeltas(StrictModel): + pass -class SnapshotSample(StrictModel): +class SnapshotSandboxSample(StrictModel): ts: int = Field(ge=0) sample_delta_ms: int | None = None - metrics: SnapshotMetrics - deltas: SnapshotDeltas + metrics: SnapshotSandboxMetrics + deltas: SnapshotSandboxDeltas + +class SnapshotWorkspaceSample(StrictModel): + ts: int = Field(ge=0) + sample_delta_ms: int | None = None + metrics: SnapshotWorkspaceMetrics + deltas: SnapshotWorkspaceDeltas -class SnapshotResources(StrictModel): - latest: SnapshotSample | None - history: list[SnapshotSample] + +class SnapshotSandboxResources(StrictModel): + latest: SnapshotSandboxSample | None + history: list[SnapshotSandboxSample] + + +class SnapshotWorkspaceResources(StrictModel): + latest: SnapshotWorkspaceSample | None + history: list[SnapshotWorkspaceSample] class SnapshotLayers(StrictModel): @@ -110,11 +144,12 @@ class NamespaceExecution(StrictModel): class SnapshotWorkspace(StrictModel): workspace_id: str = Field(min_length=1) lifecycle_state: Literal["active"] + finalization_state: Literal["active"] network_profile: str = Field(min_length=1) finalize_policy: str = Field(min_length=1) layers: SnapshotLayers namespace_fd_count: int | None = Field(ge=0) - resources: SnapshotResources + resources: SnapshotWorkspaceResources active_namespace_executions: list[NamespaceExecution] @@ -127,9 +162,16 @@ class SnapshotStack(StrictModel): active_leases: int = Field(ge=0) +class SnapshotEventStore(StrictModel): + dropped_storage: int = Field(ge=0) + dropped_oversized: int = Field(ge=0) + truncated_records: int = Field(ge=0) + + class SnapshotDaemon(StrictModel): daemon_pid: int = Field(gt=0) runtime_dir: str = Field(min_length=1) + event_store: SnapshotEventStore class SnapshotView(StrictModel): @@ -139,7 +181,7 @@ class SnapshotView(StrictModel): sampled_at_unix_ms: int = Field(ge=0) errors: list[str] daemon: SnapshotDaemon - resources: SnapshotResources + resources: SnapshotSandboxResources workspaces: list[SnapshotWorkspace] = Field(max_length=4096) stack: SnapshotStack | None @@ -230,6 +272,22 @@ class TraceView(StrictModel): spans: list[TraceNode] +class DaemonProcessMetrics(StrictModel): + model_config = ConfigDict(extra="allow", strict=True) + + available: bool + pid: int = Field(ge=0) + resident_memory_bytes: int | None = Field(ge=0) + peak_resident_memory_bytes: int | None = Field(ge=0) + cpu_time_us: int | None = Field(ge=0) + + +class DaemonView(StrictModel): + view: Literal["daemon"] + scope: Literal["sandbox"] + daemon: DaemonProcessMetrics + + def parse_cgroup(value: Any) -> CgroupView: return _parse(CgroupView, value, "cgroup") @@ -276,6 +334,18 @@ def visit(node: TraceNode, parent: str | None) -> None: return result +def parse_daemon(value: Any) -> DaemonView: + return _parse(DaemonView, value, "daemon") + + +def daemon_from_cgroup(value: CgroupView) -> DaemonProcessMetrics: + return _parse( + DaemonProcessMetrics, + value.topology.get("daemon"), + "cgroup topology daemon", + ) + + def _parse(model: type[StrictModel], value: Any, label: str) -> Any: try: return model.model_validate(value) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/paths.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/paths.py index ced04e1..db6b861 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/paths.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/paths.py @@ -10,6 +10,7 @@ MARKER_NAME = ".ownership.json" ROLE_NAMES = ("fixtures", "runs", "results", "runtime", "tmp") +_BINARY_FLAG = getattr(os, "O_BINARY", 0) class PathContractError(ValueError): @@ -127,7 +128,9 @@ def _overlap(left: Path, right: Path) -> bool: def _write_new_json(path: Path, value: dict[str, object]) -> None: payload = json.dumps(value, indent=2, sort_keys=True).encode() + b"\n" - descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + descriptor = os.open( + path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, 0o600 + ) try: os.write(descriptor, payload) os.fsync(descriptor) @@ -137,6 +140,8 @@ def _write_new_json(path: Path, value: dict[str, object]) -> None: def _sync_directory(path: Path) -> None: + if os.name == "nt": + return descriptor = os.open(path, os.O_RDONLY) try: os.fsync(descriptor) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/planning.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/planning.py index d4fecf8..126a1b1 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/planning.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/planning.py @@ -42,7 +42,7 @@ class ConfigurationBase(StrictModel): class PlanEnvironment(StrictModel): image: str = Field(min_length=1, max_length=512) - client_cohort: Literal["direct_client"] + client_cohort: Literal["direct_client", "product_cli"] class TrialCount(StrictModel): @@ -108,6 +108,7 @@ class RuntimeEnvironment: _FAMILIES = { + "create_sandbox": "sandbox_lifecycle", "exec_command": "command", "file_read": "files", "file_write": "files", @@ -116,15 +117,32 @@ class RuntimeEnvironment: "create_workspace": "workspace_lifecycle", "squash_layerstack": "layer_stack", } -_FAMILY_ORDER = ("command", "files", "workspace_lifecycle", "layer_stack") -_OPERATION_ORDER = ("exec_command", "file_read", "file_write", "file_edit", "file_blame", "create_workspace", "squash_layerstack") +_FAMILY_ORDER = ( + "sandbox_lifecycle", + "command", + "files", + "workspace_lifecycle", + "layer_stack", +) +_OPERATION_ORDER = ( + "create_sandbox", + "exec_command", + "file_read", + "file_write", + "file_edit", + "file_blame", + "create_workspace", + "squash_layerstack", +) _FAMILY_SEEDS = { + "sandbox_lifecycle": 0x53414E44424F5800, "command": 0x434F4D4D414E4401, "files": 0x46494C4553000002, "workspace_lifecycle": 0x574F524B53504303, "layer_stack": 0x4C41594552535404, } _ACCESS = { + "create_sandbox": ("public_gateway", "create_sandbox"), "exec_command": ("public_gateway", "exec_command"), "file_read": ("public_gateway", "file_read"), "file_write": ("public_gateway", "file_write"), @@ -134,6 +152,7 @@ class RuntimeEnvironment: "squash_layerstack": ("public_gateway", "squash_layerstacks"), } _COUNT = { + "create_sandbox": {"kind": "single_request"}, "exec_command": {"kind": "concurrent_requests", "factor": "concurrent_requests"}, "file_read": {"kind": "concurrent_requests", "factor": "concurrent_requests"}, "file_write": {"kind": "concurrent_requests", "factor": "concurrent_requests"}, @@ -143,6 +162,7 @@ class RuntimeEnvironment: "squash_layerstack": {"kind": "single_request_with_prepared_load", "load_factor": "live_sessions"}, } _CLEANUP = { + "create_sandbox": "destroy_sandbox_and_verify_registry", "exec_command": "resolve_from_isolation", "file_read": "verify_fixture_unchanged", "file_write": "resolve_from_isolation", @@ -152,6 +172,7 @@ class RuntimeEnvironment: "squash_layerstack": "destroy_topology_and_verify_baseline", } _EXPECTED_FACTORS = { + "create_sandbox": {"workspace_profile", "network_profile"}, "exec_command": {"concurrent_requests", "workspace_profile", "session_mode", "command_case"}, "file_read": {"concurrent_requests", "returned_bytes", "source", "target_mode"}, "file_write": {"concurrent_requests", "content_bytes", "destination", "target_mode"}, @@ -193,6 +214,7 @@ class RuntimeEnvironment: "create_sandbox", "destroy_sandbox", "inspect_sandbox", "cgroup", "snapshot" } _OPERATION_CATALOG_OPERATIONS = { + "create_sandbox": set(), "exec_command": {"exec_command"}, "file_read": {"file_read"}, "file_write": {"file_write"}, @@ -255,6 +277,12 @@ def expand_plan( for values in itertools.product(*(factors[name]["values"] for name in names)): raw = dict(zip(names, values, strict=True)) cell_body, isolation = _operation_cell(operation_id, raw) + if ( + canonical["name"] + in {"paper-env-smoke", "paper-pilot", "paper-good-pass"} + and "workspace_profile" not in cell_body + ): + cell_body["workspace_profile"] = "paper-100m" profile_name = cell_body.get("workspace_profile") profile = profiles.get(profile_name) if profile_name else None if profile_name: @@ -262,7 +290,11 @@ def expand_plan( findings.append(_finding("error", "unknown_workspace_profile", profile_name, "operations")) continue selected_names.add(profile_name) - destructive = operation_id in {"create_workspace", "squash_layerstack"} + destructive = operation_id in { + "create_sandbox", + "create_workspace", + "squash_layerstack", + } trial_kind = "destructive" if destructive else "fast" trial_counts = canonical["protocol"]["trial_defaults"][trial_kind] timeout_key = "squash_layerstack" if operation_id == "squash_layerstack" else "default" @@ -280,6 +312,14 @@ def expand_plan( canonical["environment"]["client_cohort"], protocol, profile_material, operation_cell, ]) access_kind, action = _ACCESS[operation_id] + if ( + operation_id == "create_workspace" + and canonical["environment"]["client_cohort"] == "product_cli" + ): + access_kind, action = ( + "public_gateway", + "create_workspace_session", + ) identity = {key: value for key, value in cell_body.items() if key not in {"command", "expected_exit_code", "output_limit_bytes", "resolved_isolation"}} comparison = { "operation": operation_id, @@ -308,7 +348,18 @@ def expand_plan( "gateway_mode": "isolated", } lifecycle = {"lifecycle_revision": 1, "failure_revision": 1, "stabilization_revision": 1, "automatic_retries": 0, "one_active_campaign": True, "sequential_families": True} - revisions = [{"operation_id": name, "semantic_revision": 1, "factor_schema_revision": 1, "comparison_projection_revision": 1} for name in ("exec_command", "file_read", "file_write", "file_edit", "file_blame", "create_workspace", "squash_layerstack")] + revision_names = [ + "exec_command", + "file_read", + "file_write", + "file_edit", + "file_blame", + "create_workspace", + "squash_layerstack", + ] + if any(cell["operation_id"] == "create_sandbox" for cell in cells): + revision_names.insert(0, "create_sandbox") + revisions = [{"operation_id": name, "semantic_revision": 1, "factor_schema_revision": 1, "comparison_projection_revision": 1} for name in revision_names] hash_environment = {key: value for key, value in effective.items() if key != "free_space_bytes"} plan_hash = _sha_json({"schema_version": 1, "plan_hash_revision": 2, "definition_schema_version": 2, "canonical_plan": canonical, "effective_environment": hash_environment, "fixed_lifecycle_policy": lifecycle, "definition_revisions": revisions, "selected_workspace_profiles": selected, "cells": cells, "execution_blocks": blocks}) trial_batches = sum(cell["protocol"]["warmups"] + cell["protocol"]["measured_trials"] for cell in cells) @@ -324,6 +375,18 @@ def _validate_plan( catalog_operations: frozenset[str] | set[str] | None, ) -> list[dict[str, Any]]: findings: list[dict[str, Any]] = [] + if ( + plan["name"] in {"paper-env-smoke", "paper-pilot", "paper-good-pass"} + and plan["environment"]["client_cohort"] != "product_cli" + ): + findings.append( + _finding( + "error", + "paper_plan_requires_product_cli", + "paper plans must select product_cli", + "environment.client_cohort", + ) + ) image = plan["environment"]["image"] if ( any(character.isspace() or ord(character) < 32 for character in image) @@ -453,7 +516,10 @@ def _value_key(value: Any) -> str: def _operation_cell(operation: str, raw: dict[str, Any]) -> tuple[dict[str, Any], str]: body = copy.deepcopy(raw) - if operation == "exec_command": + if operation == "create_sandbox": + isolation = "fresh_sandbox_per_trial" + body["resolved_isolation"] = isolation + elif operation == "exec_command": commands = { "noop": ("true", 0), "output64_kib": ("head -c 65536 /dev/zero | tr '\\000' x", 0), @@ -482,6 +548,14 @@ def _operation_cell(operation: str, raw: dict[str, Any]) -> tuple[dict[str, Any] def _order_cells(cells: list[dict[str, Any]], seed: int) -> list[dict[str, Any]]: + """Persist one seeded cell permutation per sequential family block. + + The runner exhausts each cell's warmup and measured repetitions before + advancing to the next cell in this recorded order. This is the benchmark's + existing interpretation of ``randomized_blocks``; repetition-level + re-randomization would change prepared-sandbox lifetimes and requires an + explicit protocol decision. + """ ordered: list[dict[str, Any]] = [] blocks: list[dict[str, Any]] = [] for family in _FAMILY_ORDER: @@ -497,7 +571,7 @@ def _order_cells(cells: list[dict[str, Any]], seed: int) -> list[dict[str, Any]] for index, (width, group) in enumerate(groups): _shuffle(group, seed ^ _FAMILY_SEEDS[family] ^ (width or 0)) cell_ids = [cell["cell_id"] for cell in group] - rust_family = {"command": "Command", "files": "Files", "workspace_lifecycle": "WorkspaceLifecycle", "layer_stack": "LayerStack"}[family] + rust_family = {"sandbox_lifecycle": "SandboxLifecycle", "command": "Command", "files": "Files", "workspace_lifecycle": "WorkspaceLifecycle", "layer_stack": "LayerStack"}[family] blocks.append({"block_id": _sha(f"v1:{rust_family}:{':'.join(cell_ids)}".encode()), "family_id": family, "cell_ids": cell_ids, "restart_reason": f"layerstack_remount_parallelism_changed:{width}" if index else None}) ordered.extend(group) cells[:] = ordered diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product.py index ddeafbe..ad1f5a1 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product.py @@ -10,10 +10,12 @@ from .models import StrictModel from .observability import ( CgroupView, + DaemonView, LayerstackView, SnapshotView, TraceView, parse_cgroup, + parse_daemon, parse_layerstack, parse_snapshot, parse_trace, @@ -40,13 +42,29 @@ class SharedBase(StrictModel): readonly: bool +class ResourceProfile(StrictModel): + name: str = Field(min_length=1) + nano_cpus: int = Field(gt=0) + memory_high_bytes: int = Field(gt=0) + memory_max_bytes: int = Field(gt=0) + pids_max: int = Field(gt=0) + workload_memory_high_bytes: int = Field(gt=0) + workload_memory_max_bytes: int = Field(gt=0) + workload_pids_max: int = Field(gt=0) + control_plane_pids_reserve: int = Field(ge=0) + daemon_runtime_profile: str = Field(min_length=1) + separate_workload_cgroup: bool + + class SandboxRecord(StrictModel): id: str workspace_root: str state: str + activity_revision: int = Field(ge=0) daemon: DaemonEndpoint | None daemon_http: DaemonEndpoint | None shared_base: SharedBase | None + resource_profile: ResourceProfile class ProductAccess: @@ -90,6 +108,40 @@ async def exec_command(self, sandbox_id: str, *, session_id: str | None, command args: dict[str, Any] = {"cmd": command, "timeout_ms": timeout_ms, "yield_time_ms": timeout_ms} return await self._sandbox_request("exec_command", sandbox_id, session_id, args, timeout_ms, request_id) + async def create_workspace_session( + self, + sandbox_id: str, + *, + network_profile: str, + timeout_ms: int, + request_id: str, + ) -> TimedGatewayResponse: + return await self._sandbox_request( + "create_workspace_session", + sandbox_id, + None, + {"network_profile": network_profile}, + timeout_ms, + request_id, + ) + + async def destroy_workspace_session( + self, + sandbox_id: str, + *, + session_id: str, + timeout_ms: int, + request_id: str, + ) -> TimedGatewayResponse: + return await self._sandbox_request( + "destroy_workspace_session", + sandbox_id, + None, + {"workspace_session_id": _identity(session_id)}, + timeout_ms, + request_id, + ) + async def file_read(self, sandbox_id: str, *, session_id: str | None, path: str, offset: int, limit: int, timeout_ms: int, request_id: str) -> TimedGatewayResponse: return await self._sandbox_request("file_read", sandbox_id, session_id, {"path": _product_path(path), "offset": offset, "limit": limit}, timeout_ms, request_id) @@ -131,6 +183,10 @@ async def observe_snapshot(self, sandbox_id: str, *, request_id: str) -> Snapsho response = await self._observe("snapshot", sandbox_id, {}, request_id) return parse_snapshot(response.value, sandbox_id) + async def observe_daemon(self, sandbox_id: str, *, request_id: str) -> DaemonView: + response = await self._observe("daemon", sandbox_id, {}, request_id) + return parse_daemon(response.value) + async def observe_layerstack(self, sandbox_id: str, *, request_id: str) -> LayerstackView: response = await self._observe("layerstack", sandbox_id, {}, request_id) return parse_layerstack(response.value) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product_cli.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product_cli.py new file mode 100644 index 0000000..c892d5f --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/product_cli.py @@ -0,0 +1,827 @@ +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import json +import os +import threading +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from .catalog import _prebuilt_executable +from .fixtures import same_native_path +from .paths import BenchmarkRoots, _sync_directory +from .product import ( + ProductAccess, + ProductAccessError, + SandboxRecord, + _identity, + _product_path, + _sandbox_record, +) +from .transport import ( + MAX_WIRE_BYTES, + GatewayEndpoint, + GatewayProductError, + GatewayTransportError, + TimedGatewayResponse, +) + +_EXECUTABLES = { + "manager": "sandbox-manager-cli", + "runtime": "sandbox-runtime-cli", + "observability": "sandbox-observability-cli", +} +_TRANSPORT_ERROR_KINDS = { + "config_error", + "connection_error", + "protocol_error", +} +_BINARY_FLAG = getattr(os, "O_BINARY", 0) +# A parseable schema-v2 metadata file is the sole commit marker. It embeds both +# redacted byte streams losslessly, then is written and flushed after the two +# closed projection files. Trial calls retain these complete marker bytes until +# their durability boundary; non-trial calls commit immediately. A crash can +# therefore leave immutable projections without a marker, or a self-contained +# marker from which projections can be reconstructed. +_EVIDENCE_COMMIT_PROTOCOL = "metadata-packed-payload-fsync-v1" +_EVIDENCE_FLUSH_CONCURRENCY = 8 + + +@dataclass +class _TrialEvidenceBuffer: + trial_id: str + pending: dict[Path, bytes] = field(default_factory=dict) + flushing: bool = False + + +class ProductCliAccess(ProductAccess): + """Closed product access implemented exclusively by released CLI subprocesses.""" + + def __init__( + self, + endpoint: GatewayEndpoint, + auth_token: str, + roots: BenchmarkRoots, + evidence_root: Path, + ) -> None: + self._runs_root = roots.runs.resolve(strict=True) + self._sandboxes: set[str] = set() + self._endpoint = endpoint + self._auth_token = auth_token + self._package_root = roots.product_bin_dir.parent.resolve(strict=True) + self._executables = { + role: _prebuilt_executable(roots, name) + for role, name in _EXECUTABLES.items() + } + self._executable_sha256 = { + role: _sha256_file(executable) + for role, executable in self._executables.items() + } + resolved_evidence_root = evidence_root.resolve(strict=True) + self._evidence_root = resolved_evidence_root / "cli-subprocesses" + self._evidence_root.mkdir(mode=0o700, exist_ok=True) + if self._evidence_root.is_symlink() or not self._evidence_root.is_dir(): + raise ProductAccessError("CLI evidence directory is unsafe") + self._evidence_root = self._evidence_root.resolve(strict=True) + if not self._evidence_root.is_relative_to(resolved_evidence_root): + raise ProductAccessError("CLI evidence directory escaped the run directory") + self._content_root = resolved_evidence_root / "cli-content" + self._content_root.mkdir(mode=0o700, exist_ok=True) + if self._content_root.is_symlink() or not self._content_root.is_dir(): + raise ProductAccessError("CLI content directory is unsafe") + self._content_root = self._content_root.resolve(strict=True) + if not self._content_root.is_relative_to(resolved_evidence_root): + raise ProductAccessError("CLI content directory escaped the run directory") + self._staged_file_writes: dict[str, str] = {} + self._evidence_buffer_lock = threading.Lock() + self._trial_evidence_buffer: _TrialEvidenceBuffer | None = None + + def begin_trial_evidence(self, trial_id: str) -> None: + trial_id = _identity(trial_id) + with self._evidence_buffer_lock: + if self._trial_evidence_buffer is not None: + raise ProductAccessError( + "CLI trial evidence transaction is already active" + ) + self._trial_evidence_buffer = _TrialEvidenceBuffer(trial_id) + + async def flush_trial_evidence(self, trial_id: str) -> None: + trial_id = _identity(trial_id) + with self._evidence_buffer_lock: + buffer = self._require_trial_evidence_buffer(trial_id) + if buffer.flushing: + raise ProductAccessError( + "CLI trial evidence transaction is already flushing" + ) + buffer.flushing = True + pending = tuple( + sorted(buffer.pending.items(), key=lambda item: item[0].name) + ) + + committed: list[Path] = [] + failures: list[BaseException] = [] + try: + semaphore = asyncio.Semaphore(_EVIDENCE_FLUSH_CONCURRENCY) + + async def commit(path: Path, content: bytes) -> Path: + async with semaphore: + await asyncio.to_thread( + _write_new, + path, + content, + discard_on_error=True, + ) + return path + + results = await asyncio.gather( + *(commit(path, content) for path, content in pending), + return_exceptions=True, + ) + for result in results: + if isinstance(result, BaseException): + failures.append(result) + else: + committed.append(result) + if committed: + try: + await asyncio.to_thread(_sync_directory, self._evidence_root) + except Exception as error: + failures.append(error) + finally: + with self._evidence_buffer_lock: + active = self._require_trial_evidence_buffer(trial_id) + for path in committed: + active.pending.pop(path, None) + active.flushing = False + + if failures: + raise BaseExceptionGroup("CLI trial evidence commit failed", failures) + + def end_trial_evidence(self, trial_id: str) -> None: + trial_id = _identity(trial_id) + with self._evidence_buffer_lock: + buffer = self._require_trial_evidence_buffer(trial_id) + if buffer.flushing: + raise ProductAccessError( + "cannot end a flushing CLI trial evidence transaction" + ) + pending = len(buffer.pending) + self._trial_evidence_buffer = None + if pending: + raise ProductAccessError( + "cannot end a CLI trial evidence transaction with " + f"{pending} pending commit marker(s)" + ) + + def _require_trial_evidence_buffer(self, trial_id: str) -> _TrialEvidenceBuffer: + buffer = self._trial_evidence_buffer + if buffer is None: + raise ProductAccessError("CLI trial evidence transaction is not active") + if buffer.trial_id != trial_id: + raise ProductAccessError( + "CLI trial evidence transaction identity does not match" + ) + return buffer + + async def stage_file_write_content(self, content: str, *, request_id: str) -> None: + request_id = _identity(request_id) + encoded = content.encode() + if len(encoded) > 4 * 1024 * 1024: + raise ProductAccessError("file content exceeds fixed bound") + if request_id in self._staged_file_writes: + raise ProductAccessError("file content was already staged") + path = self._content_path(request_id) + await asyncio.to_thread(_write_new, path, encoded) + self._staged_file_writes[request_id] = content + + def discard_file_write_content(self, request_id: str) -> None: + request_id = _identity(request_id) + self._staged_file_writes.pop(request_id, None) + self._remove_content_file(request_id) + + async def create_sandbox( + self, image: str, workspace_root: Path, *, request_id: str + ) -> tuple[SandboxRecord, TimedGatewayResponse]: + workspace = workspace_root.resolve(strict=True) + if workspace == self._runs_root or not workspace.is_relative_to( + self._runs_root + ): + raise ProductAccessError("sandbox workspace is not benchmark-owned") + response = await self._invoke( + "manager", + "create_sandbox", + [ + "--image", + image, + "--workspace-bind-root", + os.fspath(workspace), + "--count", + "1", + ], + timeout_seconds=600, + request_id=request_id, + ) + record = _sandbox_record(response.value) + if ( + record.state != "ready" + or not same_native_path(record.workspace_root, workspace) + or record.id in self._sandboxes + ): + raise ProductAccessError( + "create_sandbox response violated ownership or readiness" + ) + self._sandboxes.add(record.id) + return record, response + + async def inspect_sandbox( + self, sandbox_id: str, *, request_id: str + ) -> SandboxRecord: + self._require_owned(sandbox_id) + response = await self._invoke( + "manager", + "inspect_sandbox", + ["--sandbox-id", sandbox_id], + timeout_seconds=30, + request_id=request_id, + ) + record = _sandbox_record(response.value) + if record.id != sandbox_id or record.state != "ready": + raise ProductAccessError( + "inspect_sandbox response violated identity or readiness" + ) + return record + + async def destroy_sandbox( + self, sandbox_id: str, *, request_id: str + ) -> TimedGatewayResponse: + self._require_owned(sandbox_id) + response = await self._invoke( + "manager", + "destroy_sandbox", + ["--sandbox-id", sandbox_id], + timeout_seconds=600, + request_id=request_id, + ) + self._sandboxes.remove(sandbox_id) + return response + + async def squash_layerstacks( + self, + sandbox_id: str, + *, + timeout_ms: int, + request_id: str, + ) -> TimedGatewayResponse: + self._require_owned(sandbox_id) + return await self._invoke( + "manager", + "squash_layerstacks", + ["--sandbox-id", sandbox_id], + timeout_seconds=timeout_ms / 1000, + request_id=request_id, + ) + + async def cleanup_owned(self, *, request_prefix: str) -> None: + issues: list[BaseException] = [] + for index, sandbox_id in enumerate(sorted(self._sandboxes)): + try: + await self.destroy_sandbox( + sandbox_id, + request_id=f"{request_prefix}.destroy.{index}", + ) + except BaseException as error: + issues.append(error) + try: + await self.assert_no_sandboxes(request_id=f"{request_prefix}.list") + except BaseException as error: + issues.append(error) + if issues: + raise BaseExceptionGroup("product CLI cleanup failed", issues) + + async def assert_no_sandboxes(self, *, request_id: str) -> None: + response = await self._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=30, + request_id=request_id, + ) + if ( + not isinstance(response.value, dict) + or response.value.get("sandboxes") != [] + ): + raise ProductAccessError( + "isolated gateway retained sandbox records after CLI cleanup" + ) + + async def _observe( + self, + operation: str, + sandbox_id: str, + args: dict[str, Any], + request_id: str, + ) -> TimedGatewayResponse: + self._require_owned(sandbox_id) + operation_args = ["--sandbox-id", sandbox_id] + if operation == "cgroup": + operation_args.extend( + ["--scope", str(args["scope"]), "--window-ms", str(args["window_ms"])] + ) + elif operation == "trace": + operation_args.extend(["--trace-id", str(args["trace_id"])]) + elif args: + raise ProductAccessError("observability CLI arguments are not implemented") + return await self._invoke( + "observability", + operation, + operation_args, + timeout_seconds=30, + request_id=request_id, + ) + + async def _sandbox_request( + self, + operation: str, + sandbox_id: str, + session_id: str | None, + args: dict[str, Any], + timeout_ms: int, + request_id: str, + ) -> TimedGatewayResponse: + self._require_owned(sandbox_id) + operation_args: list[str] + if operation == "exec_command": + operation_args = [ + "--timeout-ms", + str(args["timeout_ms"]), + "--yield-time-ms", + str(args["yield_time_ms"]), + ] + if session_id is not None: + operation_args.extend(["--workspace-session-id", _identity(session_id)]) + operation_args.append(str(args["cmd"])) + elif operation == "create_workspace_session": + operation_args = ["--network-profile", str(args["network_profile"])] + elif operation == "destroy_workspace_session": + operation_args = [ + "--workspace-session-id", + _identity(str(args["workspace_session_id"])), + ] + elif operation == "file_read": + operation_args = [ + "--path", + _product_path(str(args["path"])), + "--offset", + str(args["offset"]), + "--limit", + str(args["limit"]), + ] + if session_id is not None: + operation_args.extend(["--workspace-session-id", _identity(session_id)]) + elif operation == "file_write": + operation_args = await self._file_write_arguments( + args, session_id, request_id + ) + elif operation == "file_edit": + operation_args = [ + "--path", + _product_path(str(args["path"])), + "--edits", + json.dumps( + args["edits"], + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ), + ] + if session_id is not None: + operation_args.extend(["--workspace-session-id", _identity(session_id)]) + elif operation == "file_blame": + operation_args = ["--path", _product_path(str(args["path"]))] + else: + raise ProductAccessError( + f"runtime CLI operation is not implemented: {operation}" + ) + try: + return await self._invoke( + "runtime", + operation, + operation_args, + timeout_seconds=timeout_ms / 1000, + request_id=request_id, + sandbox_id=sandbox_id, + ) + finally: + if operation == "file_write": + self._remove_content_file(request_id) + + async def _file_write_arguments( + self, + args: dict[str, Any], + session_id: str | None, + request_id: str, + ) -> list[str]: + content = args["content"] + if not isinstance(content, str): + raise ProductAccessError("file content exceeds fixed bound") + path = self._content_path(request_id) + if request_id not in self._staged_file_writes: + if len(content.encode()) > 4 * 1024 * 1024: + raise ProductAccessError("file content exceeds fixed bound") + await self.stage_file_write_content(content, request_id=request_id) + elif self._staged_file_writes[request_id] != content: + raise ProductAccessError("staged file content does not match request") + if path.is_symlink() or not path.is_file(): + raise ProductAccessError("staged file content is unsafe") + operation_args = [ + "--path", + _product_path(str(args["path"])), + "--content-file", + os.fspath(path), + ] + if session_id is not None: + operation_args.extend(["--workspace-session-id", _identity(session_id)]) + return operation_args + + def _content_path(self, request_id: str) -> Path: + digest = hashlib.sha256(_identity(request_id).encode()).hexdigest() + return self._content_root / f"{digest}.txt" + + def _remove_content_file(self, request_id: str) -> None: + path = self._content_path(request_id) + try: + path.unlink() + except FileNotFoundError: + pass + self._staged_file_writes.pop(request_id, None) + + async def _invoke( + self, + executable_role: str, + operation: str, + operation_args: list[str], + *, + timeout_seconds: float, + request_id: str, + sandbox_id: str | None = None, + ) -> TimedGatewayResponse: + request_id = _identity(request_id) + executable = self._executables[executable_role] + socket = self._endpoint.address + argv = [ + os.fspath(executable), + "--gateway-endpoint", + socket, + f"--gateway-auth-token={self._auth_token}", + "--request-id", + request_id, + ] + if sandbox_id is not None: + argv.extend(["--sandbox-id", _identity(sandbox_id)]) + argv.extend([operation, *operation_args]) + sanitized_argv = _sanitized_argv(argv, operation, self._auth_token) + started_ns = time.monotonic_ns() + process: asyncio.subprocess.Process | None = None + stdout = b"" + stderr = b"" + return_code: int | None = None + validation = "process_creation_failed" + try: + process = await asyncio.create_subprocess_exec( + *argv, + cwd=self._package_root, + env={ + name: os.environ[name] + for name in ( + "PATH", + "SystemRoot", + "WINDIR", + "TEMP", + "TMP", + ) + if name in os.environ + }, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + try: + stdout, stderr = await asyncio.wait_for( + process.communicate(), timeout=timeout_seconds + ) + except TimeoutError as error: + process.kill() + stdout, stderr = await process.communicate() + return_code = process.returncode + validation = "timeout" + raise GatewayTransportError("cli_timeout") from error + except asyncio.CancelledError: + process.kill() + stdout, stderr = await process.communicate() + return_code = process.returncode + validation = "cancelled" + raise + return_code = process.returncode + validation = "response_validation_failed" + value = _validate_response( + operation, + return_code, + stdout, + stderr, + self._auth_token, + ) + validation = "passed" + except BaseException as error: + ended_ns = time.monotonic_ns() + if validation == "response_validation_failed": + validation = _validation_failure(error) + try: + await asyncio.to_thread( + self._persist_invocation, + executable_role, + operation, + request_id, + sanitized_argv, + started_ns, + ended_ns, + return_code, + stdout, + stderr, + validation, + ) + except BaseException as persistence_error: + raise BaseExceptionGroup( + "CLI invocation and evidence persistence both failed", + [error, persistence_error], + ) + raise + ended_ns = time.monotonic_ns() + evidence = await asyncio.to_thread( + self._persist_invocation, + executable_role, + operation, + request_id, + sanitized_argv, + started_ns, + ended_ns, + return_code, + stdout, + stderr, + validation, + ) + return TimedGatewayResponse( + request_id=request_id, + latency_ns=ended_ns - started_ns, + response_bytes=len(stdout), + response_sha256=f"sha256:{hashlib.sha256(stdout).hexdigest()}", + value=value, + started_ns=started_ns, + transport_evidence=evidence, + ) + + def _persist_invocation( + self, + executable_role: str, + operation: str, + request_id: str, + sanitized_argv: list[str], + started_ns: int, + ended_ns: int, + return_code: int | None, + stdout: bytes, + stderr: bytes, + validation: str, + ) -> dict[str, Any]: + invocation_id = hashlib.sha256( + f"{executable_role}:{operation}:{request_id}".encode() + ).hexdigest() + stdout_path = self._evidence_root / f"{invocation_id}.stdout" + stderr_path = self._evidence_root / f"{invocation_id}.stderr" + metadata_path = self._evidence_root / f"{invocation_id}.json" + stdout = _redact_bytes(stdout, self._auth_token) + stderr = _redact_bytes(stderr, self._auth_token) + _write_new(stdout_path, stdout, durable=False) + _write_new(stderr_path, stderr, durable=False) + metadata = { + "schema_version": 2, + "invocation_id": invocation_id, + "request_id": request_id, + "operation": operation, + "executable_role": executable_role, + "executable_path": os.fspath(self._executables[executable_role]), + "executable_sha256": self._executable_sha256[executable_role], + "sanitized_argv": sanitized_argv, + "started_monotonic_ns": started_ns, + "ended_monotonic_ns": ended_ns, + "elapsed_ns": ended_ns - started_ns, + "return_code": return_code, + "stdout_path": stdout_path.relative_to( + self._evidence_root.parent + ).as_posix(), + "stderr_path": stderr_path.relative_to( + self._evidence_root.parent + ).as_posix(), + "stdout_bytes": len(stdout), + "stderr_bytes": len(stderr), + "stdout_sha256": f"sha256:{hashlib.sha256(stdout).hexdigest()}", + "stderr_sha256": f"sha256:{hashlib.sha256(stderr).hexdigest()}", + "stdout_base64": base64.b64encode(stdout).decode("ascii"), + "stderr_base64": base64.b64encode(stderr).decode("ascii"), + "response_validation": validation, + "evidence_commit": _EVIDENCE_COMMIT_PROTOCOL, + } + metadata_bytes = ( + json.dumps( + metadata, + indent=2, + sort_keys=True, + ensure_ascii=False, + allow_nan=False, + ).encode() + + b"\n" + ) + with self._evidence_buffer_lock: + buffer = self._trial_evidence_buffer + if buffer is not None: + if buffer.flushing: + raise ProductAccessError( + "cannot persist CLI evidence while its transaction is flushing" + ) + if metadata_path in buffer.pending: + raise ProductAccessError( + "CLI invocation evidence commit marker is duplicated" + ) + buffer.pending[metadata_path] = metadata_bytes + if buffer is None: + _write_new( + metadata_path, + metadata_bytes, + discard_on_error=True, + ) + _sync_directory(self._evidence_root) + transport_metadata = { + key: value + for key, value in metadata.items() + if key not in {"stdout_base64", "stderr_base64"} + } + return { + "kind": "product_cli_subprocess", + "metadata_path": metadata_path.relative_to( + self._evidence_root.parent + ).as_posix(), + **transport_metadata, + } + + +def _validate_response( + operation: str, + return_code: int | None, + stdout: bytes, + stderr: bytes, + auth_token: str, +) -> dict[str, Any]: + if auth_token.encode() in stdout or auth_token.encode() in stderr: + raise GatewayTransportError("credential_echo") + if len(stdout) > MAX_WIRE_BYTES or len(stderr) > MAX_WIRE_BYTES: + raise GatewayTransportError("cli_output_oversize") + if return_code != 0: + kind, message = _error_envelope(stderr) + if kind in _TRANSPORT_ERROR_KINDS: + raise GatewayTransportError(kind) + raise GatewayProductError(kind, message) + if stderr: + raise GatewayTransportError("cli_unexpected_stderr") + if not stdout.endswith(b"\n") or stdout.count(b"\n") != 1: + raise GatewayTransportError("cli_response_framing") + try: + value = json.loads(stdout) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise GatewayTransportError("invalid_json") from error + if not isinstance(value, dict): + raise GatewayTransportError("response_schema") + _validate_operation_shape(operation, value) + return value + + +def _validation_failure(error: BaseException) -> str: + if isinstance(error, GatewayTransportError): + return f"transport_error:{error}" + if isinstance(error, GatewayProductError): + return f"product_error:{error.kind}" + return f"validation_error:{type(error).__name__}" + + +def _validate_operation_shape(operation: str, value: dict[str, Any]) -> None: + required = { + "create_sandbox": {"id", "workspace_root", "state"}, + "inspect_sandbox": {"id", "workspace_root", "state"}, + "list_sandboxes": {"sandboxes"}, + "exec_command": {"status"}, + "create_workspace_session": {"workspace_session_id"}, + "destroy_workspace_session": {"workspace_session_id", "destroyed"}, + "file_read": {"path", "content", "bytes_read"}, + "file_write": {"path", "bytes_written"}, + "file_edit": {"path", "edits_applied"}, + "file_blame": {"path", "ranges"}, + "cgroup": {"view"}, + "daemon": {"view"}, + "layerstack": {"view"}, + "snapshot": {"sandbox_id", "lifecycle_state", "availability"}, + "trace": {"view"}, + }.get(operation, set()) + if not required.issubset(value): + raise GatewayTransportError("response_schema") + + +def _error_envelope(stderr: bytes) -> tuple[str, str]: + try: + value = json.loads(stderr) + error = value["error"] + kind = error["kind"] + message = error["message"] + except ( + UnicodeDecodeError, + json.JSONDecodeError, + KeyError, + TypeError, + ) as error: + raise GatewayTransportError("cli_error_schema") from error + if not isinstance(kind, str) or not kind or not isinstance(message, str): + raise GatewayTransportError("cli_error_schema") + return kind[:256], " ".join(message.split())[:1024] + + +def _sanitized_argv(argv: list[str], operation: str, auth_token: str) -> list[str]: + result: list[str] = [] + redact_next = False + for index, value in enumerate(argv): + if redact_next: + result.append("[REDACTED]") + redact_next = False + continue + if value == "--gateway-auth-token": + result.append(value) + redact_next = True + continue + if value.startswith("--gateway-auth-token="): + result.append("--gateway-auth-token=[REDACTED]") + continue + if index and argv[index - 1] == "--edits": + digest = hashlib.sha256(value.encode()).hexdigest() + result.append(f"[JSON sha256:{digest}]") + continue + if operation == "exec_command" and index == len(argv) - 1: + digest = hashlib.sha256(value.encode()).hexdigest() + result.append(f"[COMMAND sha256:{digest}]") + continue + result.append("[REDACTED]" if value == auth_token else value) + return result + + +def _redact_bytes(value: bytes, auth_token: str) -> bytes: + return value.replace(auth_token.encode(), b"[REDACTED]") + + +def _write_new( + path: Path, + content: bytes, + *, + durable: bool = True, + discard_on_error: bool = False, +) -> None: + created = False + try: + descriptor = os.open( + path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, 0o600 + ) + created = True + try: + view = memoryview(content) + while view: + written = os.write(descriptor, view) + if written <= 0: + raise OSError("short artifact write") + view = view[written:] + if durable: + os.fsync(descriptor) + finally: + os.close(descriptor) + except BaseException: + if discard_on_error and created: + try: + path.unlink() + except FileNotFoundError: + pass + raise + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return f"sha256:{digest.hexdigest()}" diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/recovery.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/recovery.py index bd99f68..d64616b 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/recovery.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/recovery.py @@ -7,7 +7,7 @@ from .artifacts import ArtifactError, ArtifactId, ArtifactStore, read_envelope_path from .models import OwnedPathMarker from .paths import BenchmarkRoots -from .safety import OwnershipError, OwnershipLedger +from .safety import OwnershipLedger TERMINAL_STATES = {"completed", "failed", "cancelled"} @@ -59,6 +59,17 @@ def scan(self) -> RecoveryResult: def recover_run(self, run_id: str) -> bool: return self._recover_if_needed(self._store.run_path(run_id)) + def cleanup_terminal_run(self, run_id: str) -> bool: + path = self._store.run_path(run_id) + manifest = read_envelope_path( + path / "run-manifest.json", ArtifactId.RUN_MANIFEST + ) + if manifest.get("run_id") != run_id: + raise ArtifactError("manifest does not prove the result directory identity") + if manifest.get("state") not in TERMINAL_STATES: + raise ArtifactError("explicit terminal cleanup requires a terminal run") + return self._cleanup_owned_paths(run_id) + def _recover_if_needed(self, path: Path) -> bool: if path.is_symlink() or not path.is_dir(): raise ArtifactError(f"unsafe result entry: {path}") @@ -73,6 +84,13 @@ def _recover_if_needed(self, path: Path) -> bool: if not isinstance(state, str): raise ArtifactError("manifest state is missing") if state in TERMINAL_STATES: + failure = manifest.get("failure") + if state in {"completed", "cancelled"} or ( + state == "failed" + and isinstance(failure, dict) + and failure.get("code") == "artifact_finalization_failed" + ): + return self._cleanup_owned_paths(run_id) return False journals = { @@ -81,26 +99,7 @@ def _recover_if_needed(self, path: Path) -> bool: ) for artifact_id in (ArtifactId.EVENTS, ArtifactId.OBSERVATIONS) } - ledger = OwnershipLedger(self._roots) - owned = [] - for role in ("runs", "runtime"): - marker = OwnedPathMarker(role=role, identity={"run_id": run_id}) - target = getattr(self._roots, role) / run_id - # A crash can happen before either disposable directory is created, - # or after one has already been removed. Absence is safe; presence - # must still carry the exact run marker before cleanup is attempted. - if target.exists() or target.is_symlink(): - ledger.adopt(target, marker) - owned.append((target, marker)) - - self._cleanup(run_id) - for target, marker in owned: - # The resource-specific cleanup may itself remove an owned runtime - # directory after proving its process identity. Absence is then the - # successful postcondition; any remaining directory must still be - # removed through the ledger and exact marker. - if target.exists() or target.is_symlink(): - ledger.remove(target, marker) + self._cleanup_owned_paths(run_id, always_cleanup=True) for artifact_id, journal in journals.items(): if journal.partial_tail_line is not None: self._store.quarantine_partial_tail(run_id, artifact_id) @@ -139,3 +138,30 @@ def _recover_if_needed(self, path: Path) -> bool: schema_version=manifest_envelope_version, ) return True + + def _cleanup_owned_paths( + self, run_id: str, *, always_cleanup: bool = False + ) -> bool: + ledger = OwnershipLedger(self._roots) + owned = [] + for role in ("runs", "runtime"): + marker = OwnedPathMarker(role=role, identity={"run_id": run_id}) + target = getattr(self._roots, role) / run_id + # A crash can happen before either disposable directory is created, + # or after one has already been removed. Absence is safe; presence + # must still carry the exact run marker before cleanup is attempted. + if target.exists() or target.is_symlink(): + ledger.adopt(target, marker) + owned.append((target, marker)) + + if not owned and not always_cleanup: + return False + self._cleanup(run_id) + for target, marker in owned: + # The resource-specific cleanup may itself remove an owned runtime + # directory after proving its process identity. Absence is then the + # successful postcondition; any remaining directory must still be + # removed through the ledger and exact marker. + if target.exists() or target.is_symlink(): + ledger.remove(target, marker) + return bool(owned) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/resource_sampling.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/resource_sampling.py index 4f24aca..1d9f27e 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/resource_sampling.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/resource_sampling.py @@ -7,72 +7,198 @@ import subprocess import time from collections.abc import Awaitable, Callable +from dataclasses import dataclass from pathlib import Path from typing import Any -from .observability import CgroupView, SnapshotView +from .fixtures import native_filesystem_path +from .observability import ( + CgroupView, + DaemonProcessMetrics, + ObservabilityError, + SnapshotView, + daemon_from_cgroup, +) from .product import ProductAccess +ResourceSink = Callable[[list[dict[str, Any]]], Awaitable[None]] +_MAX_IN_FLIGHT_SAMPLES = 1 +# The qualified Windows product samples its resource rings every 100 ms. A +# retry faster than that cadence can only launch duplicate CLI observations +# against the same ring generation. +_BOUNDARY_POLL_INTERVAL_SECONDS = 0.1 +_POST_BOUNDARY_FIRST_POLL_DELAY_SECONDS = 0.1 +_BOUNDARY_READINESS_TIMEOUT_SECONDS = 5.0 +_RESOURCE_RING_PENDING = "resource ring is not available yet" +_INAPPLICABLE_CREATE_COUNTERS = frozenset( + { + "daemon_cpu_time_ns", + "sandbox_cpu_time_ns", + "sandbox_block_read_bytes", + "sandbox_block_write_bytes", + } +) + + +class ResourceSamplingError(RuntimeError): + pass + + +@dataclass(frozen=True, slots=True) +class _ProductBoundary: + cgroup_timestamp_ms: int + snapshot_timestamp_ms: int + workspace_timestamps_ms: tuple[tuple[str, int], ...] -ResourceSink = Callable[[dict[str, Any]], Awaitable[None]] _METRICS = { - "runner_rss_bytes": ("bytes", "runner", "gauge", "maximum", "macos.ps"), + "runner_rss_bytes": ("bytes", "runner", "gauge", "maximum", "host_process_api"), "daemon_rss_bytes": ( - "bytes", "daemon", "gauge", "maximum", - "product_observability.snapshot.daemon.daemon_pid", + "bytes", + "daemon", + "gauge", + "maximum", + "product_observability.cgroup.topology.daemon.peak_resident_memory_bytes", ), "daemon_cpu_time_ns": ( - "nanoseconds", "daemon", "monotonic_counter", "delta", - "product_observability.snapshot.daemon.daemon_pid", + "nanoseconds", + "daemon", + "monotonic_counter", + "delta", + "product_observability.cgroup.topology.daemon.cpu_time_us", ), "sandbox_memory_current_bytes": ( - "bytes", "sandbox", "gauge", "maximum", + "bytes", + "sandbox", + "gauge", + "maximum", "product_observability.cgroup.docker_engine.memory.current", ), "sandbox_memory_peak_bytes": ( - "bytes", "sandbox", "gauge", "maximum", + "bytes", + "sandbox", + "gauge", + "maximum", "product_observability.cgroup.docker_engine.memory.current.sampled_peak", ), "sandbox_cpu_time_ns": ( - "nanoseconds", "sandbox", "monotonic_counter", "delta", + "nanoseconds", + "sandbox", + "monotonic_counter", + "delta", "product_observability.cgroup.docker_engine.cpu.usage_usec", ), "sandbox_block_read_bytes": ( - "bytes", "sandbox", "monotonic_counter", "delta", + "bytes", + "sandbox", + "monotonic_counter", + "delta", "product_observability.cgroup.docker_engine.io.read_bytes", ), "sandbox_block_write_bytes": ( - "bytes", "sandbox", "monotonic_counter", "delta", + "bytes", + "sandbox", + "monotonic_counter", + "delta", "product_observability.cgroup.docker_engine.io.write_bytes", ), "workspace_logical_bytes": ( - "bytes", "workspace", "gauge", "maximum", "filesystem.metadata", + "bytes", + "workspace", + "gauge", + "maximum", + "filesystem.metadata", ), "workspace_allocated_bytes": ( - "bytes", "workspace", "gauge", "maximum", "filesystem.metadata", + "bytes", + "workspace", + "gauge", + "maximum", + "filesystem.metadata", ), "workspace_file_count": ( - "count", "workspace", "gauge", "maximum", "filesystem.metadata", + "count", + "workspace", + "gauge", + "maximum", + "filesystem.metadata", ), "layerstack_bytes": ( - "bytes", "layerstack", "gauge", "maximum", + "bytes", + "layerstack", + "gauge", + "maximum", "product_observability.snapshot.stack.storage_allocated_bytes", ), "upperdir_bytes": ( - "bytes", "layerstack", "gauge", "maximum", + "bytes", + "workspace", + "gauge", + "delta", "product_observability.snapshot.workspaces.disk_allocated_bytes.sum", ), "host_free_bytes": ( - "bytes", "host_volume", "gauge", "minimum", "host_volume.df_posix", + "bytes", + "host_volume", + "gauge", + "minimum", + "filesystem.disk_usage", ), } +_METRIC_SEMANTIC_REVISIONS = { + "upperdir_bytes": 2, +} def resource_metric_source(metric_id: str) -> str: return _METRICS[metric_id][4] +class WorkspaceMetricCache: + """Cache immutable host-workspace tree metrics for one campaign runner.""" + + def __init__(self) -> None: + self._values: dict[str, dict[str, tuple[int | None, str | None]]] = {} + self._pending: dict[ + str, asyncio.Task[dict[str, tuple[int | None, str | None]]] + ] = {} + self._lock = asyncio.Lock() + + async def get(self, workspace: Path) -> dict[str, tuple[int | None, str | None]]: + resolved = workspace.resolve(strict=True) + key = os.path.normcase(os.fspath(resolved)) + async with self._lock: + cached = self._values.get(key) + if cached is not None: + return dict(cached) + task = self._pending.get(key) + if task is None: + task = asyncio.create_task( + asyncio.to_thread(_workspace_metrics, resolved), + name=f"workspace-metrics:{resolved.name}", + ) + self._pending[key] = task + try: + values = await asyncio.shield(task) + except BaseException: + async with self._lock: + if self._pending.get(key) is task: + del self._pending[key] + raise + async with self._lock: + self._values[key] = dict(values) + if self._pending.get(key) is task: + del self._pending[key] + return dict(values) + + async def invalidate(self, workspace: Path) -> None: + """Discard a path after an explicit host-workspace mutation.""" + resolved = workspace.resolve(strict=True) + key = os.path.normcase(os.fspath(resolved)) + async with self._lock: + self._values.pop(key, None) + + class TrialResourceSampler: def __init__( self, @@ -85,9 +211,17 @@ def __init__( interval_ms: int, campaign_started_ns: int, sink: ResourceSink, + workspace_cache: WorkspaceMetricCache | None = None, + counter_delta_applicable: bool = True, + expected_post_workspace_count: int | None = None, ) -> None: if not 10 <= interval_ms <= 60_000: raise ValueError("resource sample interval is invalid") + if ( + expected_post_workspace_count is not None + and expected_post_workspace_count < 1 + ): + raise ValueError("expected post-operation workspace count is invalid") self._product = product self._sandbox = sandbox self._workspace = workspace @@ -96,81 +230,515 @@ def __init__( self._interval = interval_ms / 1000 self._campaign_started_ns = campaign_started_ns self._sink = sink + self._workspace_cache = workspace_cache or WorkspaceMetricCache() + self._counter_delta_applicable = counter_delta_applicable + self._expected_post_workspace_count = expected_post_workspace_count + self._workspace_values: ( + dict[str, tuple[int | None, str | None]] | BaseException | None + ) = None self._stop = asyncio.Event() self._task: asyncio.Task[None] | None = None self._sample_index = 0 + self._samples: dict[int, asyncio.Task[list[dict[str, Any]]]] = {} + self._baseline_boundary: _ProductBoundary | None = None + self._post_response_threshold_ms: int | None = None + self._post_boundary_not_before_ns: int | None = None async def start(self) -> None: if self._task is not None: raise RuntimeError("resource sampler was already started") - await self._sample(sampled=False) - self._task = asyncio.create_task(self._loop(), name=f"resources:{self._trial_id}") + if self._workspace_values is None: + try: + self._workspace_values = await self._workspace_cache.get( + self._workspace + ) + except asyncio.CancelledError: + raise + except BaseException as error: + self._workspace_values = error + baseline = self._launch( + sampled=False, + scheduled_ns=time.monotonic_ns(), + force=True, + ) + # The operation barrier cannot be released until the counter baseline + # is an actual completed collection. + await baseline + self._task = asyncio.create_task( + self._loop(), name=f"resources:{self._trial_id}" + ) async def stop(self) -> None: if self._task is None: return + # Start the qualified product-ring wait at the post-response boundary. + # An already-admitted periodic collection must still drain before the + # mandatory boundary can launch, but its drain is independent of the + # ring's 100-ms freshness interval and can safely overlap that wait. + self._post_response_threshold_ms = time.time_ns() // 1_000_000 + self._post_boundary_not_before_ns = time.monotonic_ns() + int( + _POST_BOUNDARY_FIRST_POLL_DELAY_SECONDS * 1_000_000_000 + ) self._stop.set() await self._task self._task = None - await self._sample(sampled=False) + active = [task for task in self._samples.values() if not task.done()] + if active: + # A periodic collector admitted during the primary window keeps its + # evidence, but the mandatory boundary must not overlap it and + # exceed the one-expensive-collector perturbation cap. + await asyncio.gather(*active, return_exceptions=True) + # This boundary collection is mandatory even when a periodic collector + # was still completing at stop. It begins only after all measured + # responses have validated and gives counter metrics a real endpoint. + self._launch( + sampled=False, + scheduled_ns=time.monotonic_ns(), + force=True, + ) + results = await asyncio.gather( + *(self._samples[index] for index in sorted(self._samples)), + return_exceptions=True, + ) + errors: list[BaseException] = [] + for result in results: + if isinstance(result, BaseException): + errors.append(result) + continue + try: + # Samples may collect concurrently, but their complete record + # batches are persisted in scheduled order. The sink fsyncs + # each batch before stop returns, making the trial boundary + # the durability boundary. + await self._sink(result) + except BaseException as error: + errors.append(error) + self._samples.clear() + if errors: + raise BaseExceptionGroup("resource sampling failed", errors) async def _loop(self) -> None: + deadline = time.monotonic() + self._interval while True: + remaining = max(0.0, deadline - time.monotonic()) try: - await asyncio.wait_for(self._stop.wait(), timeout=self._interval) + await asyncio.wait_for(self._stop.wait(), timeout=remaining) except TimeoutError: - await self._sample(sampled=True) + self._launch( + sampled=True, + scheduled_ns=int(deadline * 1_000_000_000), + ) + deadline += self._interval continue return - async def _sample(self, *, sampled: bool) -> None: + def _launch( + self, + *, + sampled: bool, + scheduled_ns: int | None = None, + force: bool = False, + ) -> asyncio.Task[list[dict[str, Any]]]: index = self._sample_index self._sample_index += 1 - local, cgroup, snapshot = await asyncio.gather( - asyncio.to_thread(_local_metrics, self._workspace), - self._product.observe_cgroup( - self._sandbox, request_id=f"{self._trial_id}.observe.cgroup.{index}" - ), - self._product.observe_snapshot( - self._sandbox, request_id=f"{self._trial_id}.observe.snapshot.{index}" - ), + scheduled_ns = scheduled_ns or time.monotonic_ns() + active = sum(not task.done() for task in self._samples.values()) + if not force and active >= _MAX_IN_FLIGHT_SAMPLES: + task = asyncio.create_task( + self._unavailable_sample( + sampled=sampled, + scheduled_ns=scheduled_ns, + reason=( + "resource collector saturated at its fixed " + f"{_MAX_IN_FLIGHT_SAMPLES}-sample concurrency cap" + ), + ), + name=f"resources:{self._trial_id}:{index}:unavailable", + ) + else: + task = asyncio.create_task( + self._sample( + index=index, + sampled=sampled, + scheduled_ns=scheduled_ns, + ), + name=f"resources:{self._trial_id}:{index}", + ) + self._samples[index] = task + return task + + async def _sample( + self, *, index: int, sampled: bool, scheduled_ns: int + ) -> list[dict[str, Any]]: + started_ns = time.monotonic_ns() + boundary: _ProductBoundary | None = None + if not sampled and self._counter_delta_applicable: + boundary_result, dynamic = await asyncio.gather( + self._await_product_boundary( + index=index, + after=self._baseline_boundary, + post_response_threshold_ms=( + self._post_response_threshold_ms + if self._baseline_boundary is not None + else None + ), + ), + asyncio.to_thread(_dynamic_local_metrics, self._workspace), + return_exceptions=True, + ) + if isinstance(boundary_result, BaseException): + raise boundary_result + cgroup, snapshot, boundary = boundary_result + else: + dynamic, cgroup, snapshot = await asyncio.gather( + asyncio.to_thread(_dynamic_local_metrics, self._workspace), + self._product.observe_cgroup( + self._sandbox, + request_id=f"{self._trial_id}.observe.cgroup.{index}", + ), + self._product.observe_snapshot( + self._sandbox, + request_id=f"{self._trial_id}.observe.snapshot.{index}", + ), + return_exceptions=True, + ) + daemon = _daemon_metrics_from_cgroup(cgroup) + completed_ns = time.monotonic_ns() + observed_offset = max(0, started_ns - self._campaign_started_ns) + completed_offset = max(0, completed_ns - self._campaign_started_ns) + scheduled_offset = max(0, scheduled_ns - self._campaign_started_ns) + local = _combine_local_metrics(dynamic, self._workspace_values) + values = _resource_values(local, cgroup, snapshot, daemon) + if not self._counter_delta_applicable: + reason = ( + "counter delta is inapplicable because a sandbox-scoped " + "pre-create baseline cannot exist" + ) + for metric_id in _INAPPLICABLE_CREATE_COUNTERS: + values[metric_id] = (None, reason) + elif not sampled and self._baseline_boundary is None: + if boundary is None: + raise ResourceSamplingError( + "resource baseline boundary was not captured" + ) + if ( + self._expected_post_workspace_count is not None + and boundary.workspace_timestamps_ms + ): + raise ResourceSamplingError( + "workspace-create baseline unexpectedly had live workspaces" + ) + self._baseline_boundary = boundary + return self._records( + values, + offset=observed_offset, + scheduled_offset=scheduled_offset, + collection_started_offset=observed_offset, + collection_completed_offset=completed_offset, + sampled=sampled, + ) + + async def _await_product_boundary( + self, + *, + index: int, + after: _ProductBoundary | None, + post_response_threshold_ms: int | None, + ) -> tuple[CgroupView, SnapshotView, _ProductBoundary]: + post_boundary_not_before_ns = self._post_boundary_not_before_ns + if ( + after is None + and ( + post_response_threshold_ms is not None + or post_boundary_not_before_ns is not None + ) + ) or ( + after is not None + and ( + post_response_threshold_ms is None + or post_boundary_not_before_ns is None + ) + ): + raise ResourceSamplingError( + "post-response resource threshold does not match boundary phase" + ) + deadline = time.monotonic() + _BOUNDARY_READINESS_TIMEOUT_SECONDS + attempt = 0 + pending_reason = "product resource boundary was not ready" + if after is not None: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise ResourceSamplingError( + f"product resource boundary readiness timed out: {pending_reason}" + ) + freshness_wait = max( + 0.0, + (post_boundary_not_before_ns - time.monotonic_ns()) / 1_000_000_000, + ) + if freshness_wait: + await asyncio.sleep(min(freshness_wait, remaining)) + while True: + cgroup, snapshot = await asyncio.gather( + self._product.observe_cgroup( + self._sandbox, + request_id=( + f"{self._trial_id}.observe.cgroup.{index}.boundary.{attempt}" + ), + ), + self._product.observe_snapshot( + self._sandbox, + request_id=( + f"{self._trial_id}.observe.snapshot.{index}.boundary.{attempt}" + ), + ), + ) + boundary, pending_reason = _complete_product_boundary(cgroup, snapshot) + if boundary is not None: + if after is None or _boundary_is_fresh( + boundary, + after, + post_response_threshold_ms=post_response_threshold_ms, + expected_workspace_count=self._expected_post_workspace_count, + ): + return cgroup, snapshot, boundary + pending_reason = "product resource boundary was not newer than baseline" + remaining = deadline - time.monotonic() + if remaining <= 0: + raise ResourceSamplingError( + f"product resource boundary readiness timed out: {pending_reason}" + ) + attempt += 1 + await asyncio.sleep(min(_BOUNDARY_POLL_INTERVAL_SECONDS, remaining)) + + async def _unavailable_sample( + self, *, sampled: bool, scheduled_ns: int, reason: str + ) -> list[dict[str, Any]]: + values = {metric_id: (None, reason) for metric_id in _METRICS} + offset = max(0, scheduled_ns - self._campaign_started_ns) + return self._records( + values, + offset=offset, + scheduled_offset=offset, + collection_started_offset=None, + collection_completed_offset=None, + sampled=sampled, ) - offset = max(0, time.monotonic_ns() - self._campaign_started_ns) - values = _resource_values(local, cgroup, snapshot) + + def _records( + self, + values: dict[str, tuple[int | None, str | None]], + *, + offset: int, + scheduled_offset: int, + collection_started_offset: int | None, + collection_completed_offset: int | None, + sampled: bool, + ) -> list[dict[str, Any]]: + records: list[dict[str, Any]] = [] for metric_id in _METRICS: value, reason = values[metric_id] - await self._sink( + records.append( { "cell_id": self._cell_id, "trial_id": self._trial_id, "request_id": None, "reading": _reading( - metric_id, offset, value=value, reason=reason, sampled=sampled + metric_id, + offset, + scheduled_offset=scheduled_offset, + collection_started_offset=collection_started_offset, + collection_completed_offset=collection_completed_offset, + value=value, + reason=reason, + sampled=sampled, ), } ) + return records -def _resource_values( - local: dict[str, tuple[int | None, str | None]], +def _complete_product_boundary( cgroup: CgroupView, snapshot: SnapshotView, +) -> tuple[_ProductBoundary | None, str]: + if cgroup.availability != "available": + if not cgroup.series and cgroup.errors == [_RESOURCE_RING_PENDING]: + return None, _RESOURCE_RING_PENDING + raise ResourceSamplingError( + "product cgroup boundary was partial: " + + ("; ".join(cgroup.errors) or "no error was reported") + ) + latest_cgroup = cgroup.series[-1] + missing_cgroup = [ + name + for name in ("cpu_usec", "mem_cur", "io_rbytes", "io_wbytes") + if getattr(latest_cgroup.metrics, name) is None + ] + if missing_cgroup: + raise ResourceSamplingError( + "product cgroup boundary omitted required metrics: " + + ", ".join(missing_cgroup) + ) + if snapshot.availability != "available": + raise ResourceSamplingError( + "product snapshot boundary was partial: " + + ("; ".join(snapshot.errors) or "no error was reported") + ) + workspace_timestamps: list[tuple[str, int]] = [] + for workspace in snapshot.workspaces: + latest_workspace = workspace.resources.latest + if latest_workspace is None: + return ( + None, + f"workspace {workspace.workspace_id} resource sample is not available yet", + ) + metrics = latest_workspace.metrics + if ( + metrics.disk_allocated_bytes is None + or metrics.disk_truncated is True + or metrics.record_truncated_bytes is not None + ): + raise ResourceSamplingError( + f"workspace {workspace.workspace_id} upperdir allocation " + "was not completely reported" + ) + workspace_timestamps.append((workspace.workspace_id, latest_workspace.ts)) + return ( + _ProductBoundary( + cgroup_timestamp_ms=latest_cgroup.ts, + snapshot_timestamp_ms=snapshot.sampled_at_unix_ms, + workspace_timestamps_ms=tuple(sorted(workspace_timestamps)), + ), + "", + ) + + +def _boundary_is_fresh( + boundary: _ProductBoundary, + baseline: _ProductBoundary, + *, + post_response_threshold_ms: int, + expected_workspace_count: int | None, +) -> bool: + if boundary.cgroup_timestamp_ms <= max( + baseline.cgroup_timestamp_ms, + post_response_threshold_ms, + ): + return False + current_workspaces = dict(boundary.workspace_timestamps_ms) + baseline_workspaces = dict(baseline.workspace_timestamps_ms) + if ( + expected_workspace_count is not None + and len(current_workspaces) != expected_workspace_count + ): + return False + if not baseline_workspaces: + return not current_workspaces or all( + timestamp > max(baseline.snapshot_timestamp_ms, post_response_threshold_ms) + for timestamp in current_workspaces.values() + ) + if not baseline_workspaces.keys() <= current_workspaces.keys(): + return False + if any( + current_workspaces[workspace_id] <= max(timestamp, post_response_threshold_ms) + for workspace_id, timestamp in baseline_workspaces.items() + ): + return False + return all( + workspace_id in baseline_workspaces + or timestamp > max(baseline.snapshot_timestamp_ms, post_response_threshold_ms) + for workspace_id, timestamp in current_workspaces.items() + ) + + +def _daemon_metrics_from_cgroup( + cgroup: CgroupView | BaseException, +) -> DaemonProcessMetrics | BaseException: + if isinstance(cgroup, BaseException): + return cgroup + try: + return daemon_from_cgroup(cgroup) + except ObservabilityError as error: + return error + + +def _resource_values( + local: dict[str, tuple[int | None, str | None]] | BaseException, + cgroup: CgroupView | BaseException, + snapshot: SnapshotView | BaseException, + daemon: DaemonProcessMetrics | BaseException, ) -> dict[str, tuple[int | None, str | None]]: - latest = cgroup.series[-1].metrics - daemon_reason = ( - f"product snapshot exposed container PID {snapshot.daemon.daemon_pid} " - "without a host PID namespace and process start identity" + if isinstance(local, BaseException): + reason = _collector_failure("host/workspace", local) + local = { + metric_id: (None, reason) + for metric_id in ( + "runner_rss_bytes", + "workspace_logical_bytes", + "workspace_allocated_bytes", + "workspace_file_count", + "host_free_bytes", + ) + } + latest = ( + None + if isinstance(cgroup, BaseException) or not cgroup.series + else cgroup.series[-1].metrics + ) + cgroup_reason = ( + None + if latest is not None + else ( + _collector_failure("product cgroup", cgroup) + if isinstance(cgroup, BaseException) + else "; ".join(cgroup.errors) or "product cgroup sample was unavailable" + ) + ) + daemon_available = False if isinstance(daemon, BaseException) else daemon.available + daemon_reason = None + if not daemon_available: + if isinstance(daemon, BaseException): + daemon_reason = _collector_failure("product cgroup topology daemon", daemon) + else: + raw_reason = (daemon.model_extra or {}).get("error") + daemon_reason = ( + str(raw_reason) + if raw_reason + else "product daemon self metrics were unavailable" + ) + daemon_rss = ( + daemon.peak_resident_memory_bytes + if daemon_available and not isinstance(daemon, BaseException) + else None + ) + daemon_cpu = ( + daemon.cpu_time_us + if daemon_available and not isinstance(daemon, BaseException) + else None + ) + stack_value = ( + snapshot.stack.storage_allocated_bytes + if not isinstance(snapshot, BaseException) and snapshot.stack + else None ) - stack_value = snapshot.stack.storage_allocated_bytes if snapshot.stack else None stack_reason = ( - None if stack_value is not None else "LayerStack allocated storage was not reported by the product" + None + if stack_value is not None + else ( + _collector_failure("product snapshot", snapshot) + if isinstance(snapshot, BaseException) + else "LayerStack allocated storage was not reported by the product" + ) ) upperdir: int | None = 0 upperdir_reason: str | None = None - if snapshot.availability == "partial": + if isinstance(snapshot, BaseException): + upperdir = None + upperdir_reason = _collector_failure("product snapshot", snapshot) + elif snapshot.availability == "partial": upperdir = None upperdir_reason = "product snapshot was partial" - for workspace in snapshot.workspaces: + for workspace in () if isinstance(snapshot, BaseException) else snapshot.workspaces: sample = workspace.resources.latest if ( sample is None @@ -179,35 +747,65 @@ def _resource_values( or sample.metrics.record_truncated_bytes is not None ): upperdir = None - upperdir_reason = "workspace upperdir allocation was not completely reported" + upperdir_reason = ( + "workspace upperdir allocation was not completely reported" + ) break if upperdir is not None: upperdir += sample.metrics.disk_allocated_bytes return { **local, - "daemon_rss_bytes": (None, daemon_reason), - "daemon_cpu_time_ns": (None, daemon_reason), - "sandbox_memory_current_bytes": (latest.mem_cur, _missing("sandbox memory", latest.mem_cur)), - "sandbox_memory_peak_bytes": (latest.mem_cur, _missing("sandbox memory", latest.mem_cur)), + "daemon_rss_bytes": ( + daemon_rss, + daemon_reason or _missing("daemon peak resident memory", daemon_rss), + ), + "daemon_cpu_time_ns": ( + None if daemon_cpu is None else daemon_cpu * 1_000, + daemon_reason or _missing("daemon CPU counter", daemon_cpu), + ), + "sandbox_memory_current_bytes": ( + None if latest is None else latest.mem_cur, + cgroup_reason + if latest is None + else _missing("sandbox memory", latest.mem_cur), + ), + "sandbox_memory_peak_bytes": ( + None if latest is None else latest.mem_cur, + cgroup_reason + if latest is None + else _missing("sandbox memory", latest.mem_cur), + ), "sandbox_cpu_time_ns": ( - None if latest.cpu_usec is None else latest.cpu_usec * 1_000, - _missing("sandbox CPU counter", latest.cpu_usec), + None + if latest is None or latest.cpu_usec is None + else latest.cpu_usec * 1_000, + cgroup_reason + if latest is None + else _missing("sandbox CPU counter", latest.cpu_usec), ), "sandbox_block_read_bytes": ( - latest.io_rbytes, _missing("sandbox block-read counter", latest.io_rbytes) + None if latest is None else latest.io_rbytes, + cgroup_reason + if latest is None + else _missing("sandbox block-read counter", latest.io_rbytes), ), "sandbox_block_write_bytes": ( - latest.io_wbytes, _missing("sandbox block-write counter", latest.io_wbytes) + None if latest is None else latest.io_wbytes, + cgroup_reason + if latest is None + else _missing("sandbox block-write counter", latest.io_wbytes), ), "layerstack_bytes": (stack_value, stack_reason), "upperdir_bytes": (upperdir, upperdir_reason), } -def _local_metrics(workspace: Path) -> dict[str, tuple[int | None, str | None]]: - rss, rss_reason = _runner_rss() +def _workspace_metrics( + workspace: Path, +) -> dict[str, tuple[int | None, str | None]]: logical = allocated = files = 0 - pending = [workspace] + allocated_available = True + pending = [native_filesystem_path(workspace)] seen = 0 while pending: directory = pending.pop() @@ -215,24 +813,119 @@ def _local_metrics(workspace: Path) -> dict[str, tuple[int | None, str | None]]: for entry in entries: seen += 1 if seen > 1_000_000: - raise RuntimeError("workspace resource walk exceeded its fixed entry cap") + raise RuntimeError( + "workspace resource walk exceeded its fixed entry cap" + ) metadata = entry.stat(follow_symlinks=False) if stat.S_ISDIR(metadata.st_mode): pending.append(Path(entry.path)) elif stat.S_ISREG(metadata.st_mode): files += 1 logical += metadata.st_size - allocated += getattr(metadata, "st_blocks", 0) * 512 + blocks = getattr(metadata, "st_blocks", None) + if blocks is None: + allocated_available = False + elif allocated_available: + allocated += blocks * 512 return { - "runner_rss_bytes": (rss, rss_reason), "workspace_logical_bytes": (logical, None), - "workspace_allocated_bytes": (allocated, None), + "workspace_allocated_bytes": ( + allocated if allocated_available else None, + ( + None + if allocated_available + else "host filesystem metadata does not expose allocated block counts" + ), + ), "workspace_file_count": (files, None), - "host_free_bytes": (shutil.disk_usage(workspace).free, None), } +def _dynamic_local_metrics( + workspace: Path, +) -> dict[str, tuple[int | None, str | None]]: + rss, rss_reason = _runner_rss() + return { + "runner_rss_bytes": (rss, rss_reason), + "host_free_bytes": ( + shutil.disk_usage(native_filesystem_path(workspace)).free, + None, + ), + } + + +def _combine_local_metrics( + dynamic: dict[str, tuple[int | None, str | None]] | BaseException, + workspace: (dict[str, tuple[int | None, str | None]] | BaseException | None), +) -> dict[str, tuple[int | None, str | None]]: + if isinstance(dynamic, BaseException): + reason = _collector_failure("host dynamic", dynamic) + values = { + metric_id: (None, reason) + for metric_id in ("runner_rss_bytes", "host_free_bytes") + } + else: + values = dict(dynamic) + if workspace is None: + workspace = RuntimeError("workspace metrics were not initialized") + if isinstance(workspace, BaseException): + reason = _collector_failure("host workspace", workspace) + values.update( + { + metric_id: (None, reason) + for metric_id in ( + "workspace_logical_bytes", + "workspace_allocated_bytes", + "workspace_file_count", + ) + } + ) + else: + values.update(workspace) + return values + + def _runner_rss() -> tuple[int | None, str | None]: + if os.name == "nt": + try: + import ctypes + from ctypes import wintypes + + class ProcessMemoryCounters(ctypes.Structure): + _fields_ = [ + ("cb", wintypes.DWORD), + ("PageFaultCount", wintypes.DWORD), + ("PeakWorkingSetSize", ctypes.c_size_t), + ("WorkingSetSize", ctypes.c_size_t), + ("QuotaPeakPagedPoolUsage", ctypes.c_size_t), + ("QuotaPagedPoolUsage", ctypes.c_size_t), + ("QuotaPeakNonPagedPoolUsage", ctypes.c_size_t), + ("QuotaNonPagedPoolUsage", ctypes.c_size_t), + ("PagefileUsage", ctypes.c_size_t), + ("PeakPagefileUsage", ctypes.c_size_t), + ] + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + psapi = ctypes.WinDLL("psapi", use_last_error=True) + kernel32.GetCurrentProcess.argtypes = [] + kernel32.GetCurrentProcess.restype = wintypes.HANDLE + psapi.GetProcessMemoryInfo.argtypes = [ + wintypes.HANDLE, + ctypes.POINTER(ProcessMemoryCounters), + wintypes.DWORD, + ] + psapi.GetProcessMemoryInfo.restype = wintypes.BOOL + counters = ProcessMemoryCounters() + counters.cb = ctypes.sizeof(counters) + handle = kernel32.GetCurrentProcess() + succeeded = psapi.GetProcessMemoryInfo( + handle, ctypes.byref(counters), counters.cb + ) + if succeeded and counters.WorkingSetSize > 0: + return int(counters.WorkingSetSize), None + except (AttributeError, OSError, ValueError): + pass + return None, "Windows process resident memory observation was unavailable" try: completed = subprocess.run( ["/bin/ps", "-o", "rss=", "-p", str(os.getpid())], @@ -255,6 +948,9 @@ def _reading( metric_id: str, offset: int, *, + scheduled_offset: int, + collection_started_offset: int | None, + collection_completed_offset: int | None, value: int | None, reason: str | None, sampled: bool, @@ -263,13 +959,16 @@ def _reading( reading: dict[str, Any] = { "schema_version": 1, "metric_id": metric_id, - "metric_semantic_revision": 1, + "metric_semantic_revision": _METRIC_SEMANTIC_REVISIONS.get(metric_id, 1), "unit": unit, "scope": scope, "kind": kind, "aggregation": aggregation, "source": source, "monotonic_offset_ns": offset, + "scheduled_monotonic_offset_ns": scheduled_offset, + "collection_started_monotonic_offset_ns": collection_started_offset, + "collection_completed_monotonic_offset_ns": collection_completed_offset, "value": ( {"availability": "available", "value": float(value)} if value is not None @@ -287,3 +986,7 @@ def _reading( def _missing(label: str, value: int | None) -> str | None: return None if value is not None else f"{label} was not reported by the product" + + +def _collector_failure(label: str, error: BaseException) -> str: + return f"{label} resource collection failed: {type(error).__name__}" diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/runner.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/runner.py index f231496..14ee8ba 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/runner.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/runner.py @@ -17,17 +17,18 @@ ArtifactStore, ) from .derivation import build_report, utc_now -from .gateway import GatewayLauncher -from .fixtures import materialize_workspace +from .fixtures import materialize_workspace, materialize_workspaces, same_native_path +from .gateway import GatewayLauncher, cleanup_gateway_docker_resources from .metadata import collect_environment from .models import OwnedPathMarker from .paths import BenchmarkRoots from .product import ProductAccess -from .resource_sampling import TrialResourceSampler +from .product_cli import ProductCliAccess +from .reports import persist_report_bundle +from .resource_sampling import TrialResourceSampler, WorkspaceMetricCache from .safety import OwnershipLedger from .sessions import Session, SessionLifecycle -from .transport import GatewayProductError, GatewayTransportError, TimedGatewayResponse -from .reports import persist_report_bundle +from .transport import GatewayProductError, TimedGatewayResponse class CampaignError(RuntimeError): @@ -46,6 +47,12 @@ class CampaignResult: observations: int +@dataclass(slots=True) +class BatchTiming: + barrier_released_ns: int | None = None + batch_completed_ns: int | None = None + + @dataclass(frozen=True, slots=True) class TrialOutcome: responses: list[TimedGatewayResponse] @@ -85,7 +92,9 @@ class CampaignRunner: _CANCELLATION_GRACE_SECONDS = 5.0 _CLEANUP_TIMEOUT_SECONDS = 60.0 - def __init__(self, roots: BenchmarkRoots, *, event_sink: EventSink | None = None) -> None: + def __init__( + self, roots: BenchmarkRoots, *, event_sink: EventSink | None = None + ) -> None: self._roots = roots self._store = ArtifactStore(roots) self._event_sink = event_sink @@ -100,6 +109,11 @@ def __init__(self, roots: BenchmarkRoots, *, event_sink: EventSink | None = None self._manifest: dict[str, Any] = {} self._definitions: dict[str, Any] = {} self._started_at = "" + self._workspace_metric_cache = WorkspaceMetricCache() + self._prepared_cell_workspaces: dict[str, Path] = {} + self._trial_event_buffer: list[dict[str, Any]] | None = None + self._trial_observation_buffer: list[dict[str, Any]] | None = None + self._trial_event_sink_buffer: list[dict[str, Any]] | None = None def cancel(self) -> None: self._cancel.set() @@ -132,21 +146,26 @@ async def run( raise CampaignError("campaign ownership setup failed") from error self._store.write_immutable(run_id, ArtifactId.INTENT_PLAN, intent) self._store.write_immutable(run_id, ArtifactId.EXPANDED_PLAN, plan) - self._store.write_immutable(run_id, ArtifactId.DEFINITION_SNAPSHOT, definition_snapshot) + self._store.write_immutable( + run_id, ArtifactId.DEFINITION_SNAPSHOT, definition_snapshot + ) definition_reference = self._store.download_artifact( run_id, ArtifactId.DEFINITION_SNAPSHOT.value ).reference self._definitions = definition_snapshot self._image = _required_string(plan, "canonical_plan", "environment", "image") self._profiles = { - profile["id"]: profile for profile in plan.get("selected_workspace_profiles", []) + profile["id"]: profile + for profile in plan.get("selected_workspace_profiles", []) } self._seed = plan["canonical_plan"]["seed"] self._resource_interval_ms = plan["canonical_plan"]["protocol"][ "resource_interval_ms" ] environment = await collect_environment(self._roots, plan) - self._store.write_immutable(run_id, ArtifactId.ENVIRONMENT_METADATA, environment) + self._store.write_immutable( + run_id, ArtifactId.ENVIRONMENT_METADATA, environment + ) self._started_at = utc_now() self._manifest = self._new_manifest( run_id, @@ -164,26 +183,69 @@ async def run( state = "completed" failure: BaseException | None = None trial_batches = issued = 0 + gateway_instance_ids: list[str] = [] try: await self._event(run_id, {"kind": "run_state", "state": "running"}) cells_by_id = {cell["cell_id"]: cell for cell in plan["cells"]} + await asyncio.to_thread( + self._prepare_cell_workspaces, run_path, plan["cells"] + ) + client_cohort = plan["effective_environment"]["client_cohort"] for block in plan["execution_blocks"]: self._check_cancelled() family = block["family_id"] await self._event( run_id, - {"kind": "family_state", "family": _event_family(family), "state": "preparing"}, + { + "kind": "family_state", + "family": _event_family(family), + "state": "preparing", + }, ) gateway = await GatewayLauncher(self._roots).start( - run_id, remount_sweep_width=_block_width(block, plan) + run_id, + remount_sweep_width=_block_width(block, plan), + readiness_via_cli=client_cohort == "product_cli", + ) + gateway_instance_ids.append(gateway.identity.gateway_instance_id) + endpoint = gateway.identity.endpoint + self._manifest["gateway_execution_blocks"].append( + { + "block_id": block["block_id"], + "family_id": block["family_id"], + "gateway_instance_id": gateway.identity.gateway_instance_id, + "endpoint_uri": endpoint.uri, + "transport": endpoint.transport, + "scope": "local_only", + "rotation": "per_execution_block", + } + ) + self._store.replace_snapshot( + run_id, ArtifactId.RUN_MANIFEST, self._manifest ) - product = ProductAccess(gateway.client, self._roots.runs) + if client_cohort == "product_cli": + product = ProductCliAccess( + gateway.identity.endpoint, + gateway.auth_token, + self._roots, + self._store.run_path(run_id), + ) + elif client_cohort == "direct_client": + product = ProductAccess(gateway.client, self._roots.runs) + else: + raise CampaignError( + "expanded plan selected an unknown client cohort" + ) sessions = SessionLifecycle(product) block_error: BaseException | None = None try: await self._event( run_id, - {"kind": "family_state", "family": _event_family(family), "state": "running"}, + { + "kind": "family_state", + "family": _event_family(family), + "state": "running", + }, ) for cell_id in block["cell_ids"]: self._check_cancelled() @@ -194,14 +256,35 @@ async def run( issued += requests await self._event( run_id, - {"kind": "family_state", "family": _event_family(family), "state": "completed"}, + { + "kind": "family_state", + "family": _event_family(family), + "state": "completed", + }, ) except BaseException as error: block_error = error raise finally: + cleanup_errors: list[BaseException] = [] + if isinstance(product, ProductCliAccess): + try: + await self._shielded_cleanup( + product.cleanup_owned( + request_prefix=f"{run_id}.{family}.cleanup" + ) + ) + except BaseException as cleanup_error: + cleanup_errors.append(cleanup_error) try: - await self._shielded_cleanup(gateway.close()) + await self._shielded_cleanup( + gateway.close( + destroy_sandboxes_via_gateway=not isinstance( + product, ProductCliAccess + ), + retain_shared_base_volumes=True, + ) + ) await self._event( run_id, { @@ -213,10 +296,18 @@ async def run( }, ) except BaseException as cleanup_error: + cleanup_errors.append(cleanup_error) + if cleanup_errors: if block_error is None: - raise CampaignError("gateway cleanup failed") from cleanup_error - raise CampaignError("operation and gateway cleanup both failed") from BaseExceptionGroup( - "campaign cleanup", [block_error, cleanup_error] + raise CampaignError( + "gateway cleanup failed" + ) from BaseExceptionGroup( + "campaign cleanup", cleanup_errors + ) + raise CampaignError( + "operation and gateway cleanup both failed" + ) from BaseExceptionGroup( + "campaign cleanup", [block_error, *cleanup_errors] ) self._check_cancelled() await self._event(run_id, {"kind": "run_state", "state": "verifying"}) @@ -227,11 +318,37 @@ async def run( state = "cancelled" if self._cancel.is_set() else "failed" failure = error finally: + try: + await self._shielded_cleanup( + cleanup_gateway_docker_resources(tuple(gateway_instance_ids)) + ) + except BaseException as retained_cleanup_error: + state = "failed" + if failure is None: + failure = CampaignError( + "retained shared-base Docker cleanup failed" + ) + failure.__cause__ = retained_cleanup_error + else: + combined = CampaignError( + "campaign and retained shared-base Docker cleanup both failed" + ) + combined.__cause__ = BaseExceptionGroup( + "campaign final cleanup", + [failure, retained_cleanup_error], + ) + failure = combined try: ended_at = utc_now() - observations = self._store.read_records( - run_id, ArtifactId.OBSERVATIONS - ).records + observations_path = ( + self._store.run_path(run_id) + / ARTIFACT_SPECS[ArtifactId.OBSERVATIONS].file_name + ) + observations = ( + self._store.read_records(run_id, ArtifactId.OBSERVATIONS).records + if observations_path.exists() + else [] + ) report = build_report( run_id=run_id, state=state, @@ -277,7 +394,9 @@ async def run( failure = CampaignError("terminal artifact finalization failed") failure.__cause__ = finalization_error else: - failure = CampaignError("campaign and artifact finalization both failed") + failure = CampaignError( + "campaign and artifact finalization both failed" + ) failure.__cause__ = BaseExceptionGroup( "campaign finalization", [failure, finalization_error] ) @@ -296,7 +415,9 @@ async def run( pass if failure is not None: raise failure - return CampaignResult(run_id, state, trial_batches, issued, self._observation_sequence) + return CampaignResult( + run_id, state, trial_batches, issued, self._observation_sequence + ) async def _run_cell( self, @@ -307,9 +428,13 @@ async def _run_cell( cell: dict[str, Any], ) -> tuple[int, int]: cell_id = cell["cell_id"] - await self._event(run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "preparing"}) + await self._event( + run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "preparing"} + ) cell_context = await self._setup_cell(run_path, product, sessions, cell) - await self._event(run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "running"}) + await self._event( + run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "running"} + ) counts = [(True, index) for index in range(cell["protocol"]["warmups"])] + [ (False, index) for index in range(cell["protocol"]["measured_trials"]) ] @@ -318,14 +443,10 @@ async def _run_cell( try: for warmup, index in counts: self._check_cancelled() - trial_id = ( - f"trial-{cell_id[-16:]}-{'warmup' if warmup else 'measured'}-{index:06d}" - ) - await self._trial_state( - run_id, cell_id, trial_id, warmup, "preparing" - ) + trial_id = f"trial-{cell_id[-16:]}-{'warmup' if warmup else 'measured'}-{index:06d}" + await self._trial_state(run_id, cell_id, trial_id, warmup, "preparing") try: - outcome = await self._run_trial( + outcome = await self._run_trial_with_journal( run_id, run_path, product, @@ -347,9 +468,7 @@ async def _run_cell( else "failed", ) raise - await self._trial_state( - run_id, cell_id, trial_id, warmup, "completed" - ) + await self._trial_state(run_id, cell_id, trial_id, warmup, "completed") issued += len(outcome.responses) except BaseException as error: cell_error = error @@ -362,13 +481,74 @@ async def _run_cell( except BaseException as cleanup_error: if cell_error is None: raise CampaignError("cell cleanup failed") from cleanup_error - raise CampaignError("cell operation and cleanup both failed") from BaseExceptionGroup( - "cell cleanup", [cell_error, cleanup_error] - ) + raise CampaignError( + "cell operation and cleanup both failed" + ) from BaseExceptionGroup("cell cleanup", [cell_error, cleanup_error]) self._check_cancelled() - await self._event(run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "completed"}) + await self._event( + run_id, {"kind": "cell_state", "cell_id": cell_id, "state": "completed"} + ) return len(counts), issued + async def _run_trial_with_journal( + self, + run_id: str, + run_path: Path, + product: ProductAccess, + sessions: SessionLifecycle, + cell: dict[str, Any], + cell_context: CellContext | None, + trial_id: str, + warmup: bool, + sequence_in_cell: int, + ) -> TrialOutcome: + self._begin_trial_journal() + outcome: TrialOutcome | None = None + failure: BaseException | None = None + cli_evidence_started = False + try: + if isinstance(product, ProductCliAccess): + product.begin_trial_evidence(trial_id) + cli_evidence_started = True + outcome = await self._run_trial( + run_id, + run_path, + product, + sessions, + cell, + cell_context, + trial_id, + warmup, + sequence_in_cell, + ) + except BaseException as error: + failure = error + evidence_committed = True + if cli_evidence_started: + try: + await product.flush_trial_evidence(trial_id) + except BaseException as error: + evidence_committed = False + failure = _combine_failures(failure, error, "CLI trial evidence commit") + if evidence_committed: + try: + await self._flush_trial_journal(run_id) + except BaseException as error: + failure = _combine_failures(failure, error, "trial journal commit") + if cli_evidence_started: + try: + product.end_trial_evidence(trial_id) + except BaseException as error: + failure = _combine_failures( + failure, error, "CLI trial evidence boundary" + ) + self._end_trial_journal() + if failure is not None: + raise failure + if outcome is None: # pragma: no cover - defensive invariant + raise CampaignError("trial completed without an outcome") + return outcome + async def _run_trial( self, run_id: str, @@ -396,6 +576,8 @@ async def _run_trial( cleanup_restored = False infrastructure_failed = False sampler: TrialResourceSampler | None = None + sampler_stopped = False + batch_timing = BatchTiming() await self._trial_phase(run_id, cell_id, trial_id, warmup, "setup", "running") phase_started = time.monotonic_ns() @@ -405,35 +587,85 @@ async def _run_trial( run_path, product, sessions, cell, cell_context, trial_id ) setup_ns = time.monotonic_ns() - phase_started - await self._trial_phase(run_id, cell_id, trial_id, warmup, "setup", "completed") - - sampler = TrialResourceSampler( - product=product, - sandbox=context.sandbox, - workspace=context.workspace, - cell_id=cell_id, - trial_id=trial_id, - interval_ms=self._resource_interval_ms, - campaign_started_ns=self._started_ns, - sink=lambda data: self._resource_observation(run_id, data), - ) - await sampler.start() + await self._trial_phase( + run_id, cell_id, trial_id, warmup, "setup", "completed" + ) + + if operation_id != "create_sandbox": + sampler = TrialResourceSampler( + product=product, + sandbox=context.sandbox, + workspace=context.workspace, + cell_id=cell_id, + trial_id=trial_id, + interval_ms=self._resource_interval_ms, + campaign_started_ns=self._started_ns, + sink=lambda data: self._resource_observations(run_id, data), + workspace_cache=self._workspace_metric_cache, + expected_post_workspace_count=( + cell["operation"]["cell"]["workspace_count"] + if operation_id == "create_workspace" + else None + ), + ) + await sampler.start() failure_stage = "operation" - await self._trial_phase(run_id, cell_id, trial_id, warmup, "operation", "running") + await self._trial_phase( + run_id, cell_id, trial_id, warmup, "operation", "running" + ) self._check_cancelled() - batch_started_ns = time.monotonic_ns() responses = await self._operate( - product, sessions, cell, context, trial_id, run_id=run_id + product, + sessions, + cell, + context, + trial_id, + run_id=run_id, + batch_timing=batch_timing, ) - batch_makespan_ns = time.monotonic_ns() - batch_started_ns + if batch_timing.barrier_released_ns is None: + raise CampaignError("request batch did not record its barrier release") + validated_ended_ns = _validated_response_end_ns( + responses, fallback=batch_timing.batch_completed_ns + ) + if validated_ended_ns < batch_timing.barrier_released_ns: + raise CampaignError("request batch timing preceded its barrier release") + batch_makespan_ns = validated_ended_ns - batch_timing.barrier_released_ns operation_ns = batch_makespan_ns product_succeeded = True + if operation_id == "create_sandbox": + sampler = TrialResourceSampler( + product=product, + sandbox=context.sandbox, + workspace=context.workspace, + cell_id=cell_id, + trial_id=trial_id, + interval_ms=self._resource_interval_ms, + campaign_started_ns=self._started_ns, + sink=lambda data: self._resource_observations(run_id, data), + workspace_cache=self._workspace_metric_cache, + counter_delta_applicable=False, + ) + await sampler.start() + # Product responses have now exited and validated. End the resource + # window before journal persistence and correctness verification so + # those untimed phases cannot be attributed to the primary work. + failure_stage = "infrastructure" + await sampler.stop() + sampler_stopped = True + failure_stage = "operation" for response in responses: - await self._request_observation(run_id, cell, trial_id, warmup, response) - await self._trial_phase(run_id, cell_id, trial_id, warmup, "operation", "completed") + await self._request_observation( + run_id, cell, trial_id, warmup, response + ) + await self._trial_phase( + run_id, cell_id, trial_id, warmup, "operation", "completed" + ) failure_stage = "verify" - await self._trial_phase(run_id, cell_id, trial_id, warmup, "verify", "running") + await self._trial_phase( + run_id, cell_id, trial_id, warmup, "verify", "running" + ) self._check_cancelled() verify_started = time.monotonic_ns() await self._verify(product, sessions, cell, responses, context, trial_id) @@ -450,24 +682,33 @@ async def _run_trial( ) verify_ns = time.monotonic_ns() - verify_started checks_passed = True - await self._trial_phase(run_id, cell_id, trial_id, warmup, "verify", "completed") + await self._trial_phase( + run_id, cell_id, trial_id, warmup, "verify", "completed" + ) except BaseException as error: failure = error - infrastructure_failed = _trial_status(error, failure_stage) == "infrastructure_failed" + infrastructure_failed = ( + _trial_status(error, failure_stage) == "infrastructure_failed" + ) now = time.monotonic_ns() if failure_stage == "setup": setup_ns = now - phase_started - elif failure_stage == "operation" and "batch_started_ns" in locals(): - operation_ns = now - batch_started_ns + elif ( + failure_stage == "operation" + and batch_timing.barrier_released_ns is not None + ): + operation_ns = now - batch_timing.barrier_released_ns batch_makespan_ns = operation_ns elif failure_stage == "verify" and "verify_started" in locals(): verify_ns = now - verify_started finally: - if sampler is not None: + if sampler is not None and not sampler_stopped: try: await sampler.stop() except BaseException as sampler_error: - failure = _combine_failures(failure, sampler_error, "resource sampling") + failure = _combine_failures( + failure, sampler_error, "resource sampling" + ) failure_stage = "infrastructure" infrastructure_failed = True if context is not None: @@ -484,7 +725,9 @@ async def _run_trial( run_id, cell_id, trial_id, warmup, "teardown", "completed" ) except BaseException as teardown_error: - failure = _combine_failures(failure, teardown_error, "trial teardown") + failure = _combine_failures( + failure, teardown_error, "trial teardown" + ) failure_stage = "cleanup" teardown_ns = time.monotonic_ns() - teardown_started @@ -514,7 +757,9 @@ async def _run_trial( failure_stage = "infrastructure" infrastructure_failed = True - if failure is None and (artifact is None or not checks_passed or not cleanup_restored): + if failure is None and ( + artifact is None or not checks_passed or not cleanup_restored + ): failure = CampaignError("successful trial is missing required proof") failure_stage = "infrastructure" infrastructure_failed = True @@ -559,17 +804,23 @@ def _persist_operation_evidence( reference = self._store.write_trial_evidence( run_id, cell_id, trial_id, evidence ) - self._observation(run_id, {"record": "operation", "data": { - "operation_id": operation_id, - "cell_id": cell_id, - "trial_id": trial_id, - "request_id": ( - responses[0].request_id - if operation_id == "squash_layerstack" - else None - ), - "evidence": evidence, - }}) + self._observation( + run_id, + { + "record": "operation", + "data": { + "operation_id": operation_id, + "cell_id": cell_id, + "trial_id": trial_id, + "request_id": ( + responses[0].request_id + if operation_id == "squash_layerstack" + else None + ), + "evidence": evidence, + }, + }, + ) return { "artifact_id": reference.artifact_id, "media_type": reference.media_type, @@ -584,12 +835,21 @@ async def _setup_cell( sessions: SessionLifecycle, cell: dict[str, Any], ) -> CellContext | None: + if cell["operation"]["operation"] == "create_sandbox": + workspace = self._cell_workspace(run_path, cell) + return CellContext(workspace, "") isolation = _isolation(cell) - if isolation not in {"reusable_verified_fixture", "prepared_sandbox_per_cell", "fresh_sessions_per_trial"}: + if isolation not in { + "reusable_verified_fixture", + "prepared_sandbox_per_cell", + "fresh_sessions_per_trial", + }: return None - workspace = self._new_workspace(run_path, f"cell-{cell['cell_id'][-16:]}", cell) + workspace = self._cell_workspace(run_path, cell) record, _ = await product.create_sandbox( - self._image, workspace, request_id=f"cell-{cell['cell_id'][-16:]}.sandbox.create" + self._image, + workspace, + request_id=f"cell-{cell['cell_id'][-16:]}.sandbox.create", ) context = CellContext(workspace, record.id) try: @@ -600,7 +860,9 @@ async def _setup_cell( try: await self._teardown_cell(product, sessions, context, cell["cell_id"]) except BaseException as cleanup_error: - raise CampaignError("cell setup and cleanup both failed") from BaseExceptionGroup( + raise CampaignError( + "cell setup and cleanup both failed" + ) from BaseExceptionGroup( "cell setup cleanup", [setup_error, cleanup_error] ) raise @@ -615,6 +877,7 @@ async def _setup_trial( cell_context: CellContext | None, trial_id: str, ) -> TrialContext: + operation = cell["operation"]["operation"] if cell_context is not None: context = TrialContext( cell_context.workspace, @@ -626,26 +889,35 @@ async def _setup_trial( context.data["operation_session"] = cell_context.sessions[0] else: workspace = self._new_workspace(run_path, trial_id, cell) - record, _ = await product.create_sandbox( - self._image, workspace, request_id=f"{trial_id}.sandbox.create" - ) - context = TrialContext(workspace, record.id, True) + if operation == "create_sandbox": + context = TrialContext(workspace, "", False) + else: + record, _ = await product.create_sandbox( + self._image, workspace, request_id=f"{trial_id}.sandbox.create" + ) + context = TrialContext(workspace, record.id, True) try: - if cell_context is None: - await self._prepare_operation(product, sessions, cell, context, trial_id) - operation = cell["operation"]["operation"] + if cell_context is None and operation != "create_sandbox": + await self._prepare_operation( + product, sessions, cell, context, trial_id + ) body = cell["operation"]["cell"] - if operation in {"file_write", "file_edit"} and body["destination"] == "session": - await self._prepare_mutation(product, cell, context, trial_id) + if ( + operation in {"file_write", "file_edit"} + and body["destination"] == "session" + ): session, _ = await sessions.create_no_op( context.sandbox, "shared", request_id=f"{trial_id}.session.setup" ) context.sessions.append(session) + context.data["operation_id"] = operation except BaseException as setup_error: try: await self._teardown_trial(product, sessions, context, trial_id) except BaseException as cleanup_error: - raise CampaignError("trial setup and cleanup both failed") from BaseExceptionGroup( + raise CampaignError( + "trial setup and cleanup both failed" + ) from BaseExceptionGroup( "trial setup cleanup", [setup_error, cleanup_error] ) raise @@ -685,7 +957,7 @@ async def _prepare_operation( ) context.sessions.append(session) context.data["read_session"] = session - elif operation in {"file_write", "file_edit"} and body["destination"] == "publish": + elif operation in {"file_write", "file_edit"}: await self._prepare_mutation(product, cell, context, key) elif operation == "file_blame": await self._prepare_blame(product, cell, context, key) @@ -693,30 +965,55 @@ async def _prepare_operation( await self._prepare_squash(product, sessions, cell, context, key) async def _prepare_reads( - self, product: ProductAccess, cell: dict[str, Any], context: CellContext | TrialContext, key: str + self, + product: ProductAccess, + cell: dict[str, Any], + context: CellContext | TrialContext, + key: str, ) -> None: body = cell["operation"]["cell"] - count = body["concurrent_requests"] if body["target_mode"] == "independent" else 1 + count = ( + body["concurrent_requests"] if body["target_mode"] == "independent" else 1 + ) paths: list[str] = [] contents: list[str] = [] for index in range(count): path = f".eos-benchmark/{key}/read-{index}.txt" content = _content(body["returned_bytes"], f"read:{key}:{index}") - await product.file_write( - context.sandbox, session_id=None, path=path, content=content, - timeout_ms=cell["protocol"]["timeout_ms"], request_id=f"{key}.prepare.read.{index}" - ) paths.append(path) contents.append(content) + writes = [ + product.file_write( + context.sandbox, + session_id=None, + path=path, + content=contents[index], + timeout_ms=cell["protocol"]["timeout_ms"], + request_id=f"{key}.prepare.read.{index}", + ) + for index, path in enumerate(paths) + ] + if len(writes) == 1: + await writes[0] + else: + await _gather_complete(writes, "independent read fixture preparation") context.data.update(paths=paths, contents=contents) async def _prepare_mutation( - self, product: ProductAccess, cell: dict[str, Any], context: CellContext | TrialContext, key: str + self, + product: ProductAccess, + cell: dict[str, Any], + context: CellContext | TrialContext, + key: str, ) -> None: operation = cell["operation"]["operation"] body = cell["operation"]["cell"] - count = body["concurrent_requests"] if body["target_mode"] == "independent" else 1 - size = body["content_bytes"] if operation == "file_write" else body["file_bytes"] + count = ( + body["concurrent_requests"] if body["target_mode"] == "independent" else 1 + ) + size = ( + body["content_bytes"] if operation == "file_write" else body["file_bytes"] + ) paths: list[str] = [] before: list[str] = [] expected: list[str] = [] @@ -724,14 +1021,21 @@ async def _prepare_mutation( replacements_by_request = [0 for _ in range(body["concurrent_requests"])] baseline_request_ids: list[str] = [] if operation == "file_edit": - edits_by_request = [_edits(key, index, body["replacement_count"]) for index in range(body["concurrent_requests"])] + edits_by_request = [ + _edits(key, index, body["replacement_count"]) + for index in range(body["concurrent_requests"]) + ] for index in range(count): path = f".eos-benchmark/{key}/{operation}-{index}.txt" if operation == "file_write": original = _multiline_content(size, f"write-baseline:{key}:{index}") final = original else: - selected = edits_by_request[index:index + 1] if body["target_mode"] == "independent" else edits_by_request + selected = ( + edits_by_request[index : index + 1] + if body["target_mode"] == "independent" + else edits_by_request + ) path_edits = [edit for request in selected for edit in request] original, final, counts = _edit_content( size, body["match_density"], path_edits @@ -747,14 +1051,27 @@ async def _prepare_mutation( for edit in edits_by_request[request_index] ) baseline_request_id = f"{key}.prepare.{operation}.{index}" - await product.file_write( - context.sandbox, session_id=None, path=path, content=original, - timeout_ms=cell["protocol"]["timeout_ms"], request_id=baseline_request_id - ) paths.append(path) before.append(original) expected.append(final) baseline_request_ids.append(baseline_request_id) + writes = [ + product.file_write( + context.sandbox, + session_id=None, + path=path, + content=before[index], + timeout_ms=cell["protocol"]["timeout_ms"], + request_id=baseline_request_ids[index], + ) + for index, path in enumerate(paths) + ] + if len(writes) == 1: + await writes[0] + else: + await _gather_complete( + writes, f"independent {operation} baseline preparation" + ) context.data.update( paths=paths, before=before, @@ -765,7 +1082,11 @@ async def _prepare_mutation( ) async def _prepare_blame( - self, product: ProductAccess, cell: dict[str, Any], context: CellContext | TrialContext, key: str + self, + product: ProductAccess, + cell: dict[str, Any], + context: CellContext | TrialContext, + key: str, ) -> None: body = cell["operation"]["cell"] path = f".eos-benchmark/{key}/blame.txt" @@ -776,17 +1097,23 @@ async def _prepare_blame( for event in range(write_events): request_id = f"{key}.prepare.blame.{event}" await product.file_write( - context.sandbox, session_id=None, path=path, + context.sandbox, + session_id=None, + path=path, content=_blame_content(body["line_count"], segments, event), - timeout_ms=cell["protocol"]["timeout_ms"], request_id=request_id + timeout_ms=cell["protocol"]["timeout_ms"], + request_id=request_id, ) event_request_ids.append(request_id) if segments > 1: request_id = f"{key}.prepare.blame.{events - 1}" await product.file_edit( - context.sandbox, session_id=None, path=path, + context.sandbox, + session_id=None, + path=path, edits=[{"old_string": "A|", "new_string": "C|", "replace_all": True}], - timeout_ms=cell["protocol"]["timeout_ms"], request_id=request_id + timeout_ms=cell["protocol"]["timeout_ms"], + request_id=request_id, ) event_request_ids.append(request_id) context.data.update( @@ -838,26 +1165,40 @@ async def create_session() -> None: layer = 0 for block in range(body["squashable_blocks"]): for in_block in range(body["layers_per_block"]): - path = f".eos-benchmark/{key}/block-{block:04d}-layer-{in_block:04d}.txt" + path = ( + f".eos-benchmark/{key}/block-{block:04d}-layer-{in_block:04d}.txt" + ) content = _content(body["payload_bytes"], f"layer:{key}:{layer}") await product.file_write( - context.sandbox, session_id=None, path=path, content=content, - timeout_ms=timeout, request_id=f"{key}.layer.{layer}" + context.sandbox, + session_id=None, + path=path, + content=content, + timeout_ms=timeout, + request_id=f"{key}.layer.{layer}", ) - paths.append(path); contents.append(content); layer += 1 + paths.append(path) + contents.append(content) + layer += 1 if block + 1 < body["squashable_blocks"]: await product.file_write( - context.sandbox, session_id=None, + context.sandbox, + session_id=None, path=f".eos-benchmark/{key}/boundary-{block:04d}.txt", - content=f"boundary:{key}:{block}", timeout_ms=timeout, - request_id=f"{key}.boundary.{block}" + content=f"boundary:{key}:{block}", + timeout_ms=timeout, + request_id=f"{key}.boundary.{block}", ) await create_session() remaining_eligible = eligible - boundary_sessions if remaining_eligible: await product.file_write( - context.sandbox, session_id=None, path=f".eos-benchmark/{key}/boundary-top.txt", - content=f"boundary:{key}:top", timeout_ms=timeout, request_id=f"{key}.boundary.top" + context.sandbox, + session_id=None, + path=f".eos-benchmark/{key}/boundary-top.txt", + content=f"boundary:{key}:top", + timeout_ms=timeout, + request_id=f"{key}.boundary.top", ) for _ in range(remaining_eligible): await create_session() @@ -866,10 +1207,12 @@ async def create_session() -> None: for index, session in enumerate(context.sessions): if body["session_activity"] == "active": await product.file_write( - context.sandbox, session_id=session.session_id, + context.sandbox, + session_id=session.session_id, path=f".eos-benchmark/{key}/session-activity-{index}.txt", - content=f"active:{key}:{index}", timeout_ms=timeout, - request_id=f"{key}.activity.{index}" + content=f"active:{key}:{index}", + timeout_ms=timeout, + request_id=f"{key}.activity.{index}", ) baseline = await product.observe_layerstack( context.sandbox, request_id=f"{key}.observe.layerstack.s0" @@ -896,6 +1239,7 @@ async def _operate( trial_id: str, *, run_id: str | None = None, + batch_timing: BatchTiming | None = None, ) -> list[TimedGatewayResponse]: operation = cell["operation"]["operation"] body = cell["operation"]["cell"] @@ -908,7 +1252,7 @@ async def _operate( body["workspace_count"] if operation == "create_workspace" else 1 - if operation == "squash_layerstack" + if operation in {"create_sandbox", "squash_layerstack"} else body["concurrent_requests"] ) request_ids = ( @@ -916,77 +1260,190 @@ async def _operate( if operation == "squash_layerstack" else [f"{trial_id}.request.{index}" for index in range(request_count)] ) + if operation == "create_sandbox": + + async def create() -> TimedGatewayResponse: + record, response = await product.create_sandbox( + self._image, + context.workspace, + request_id=request_ids[0], + ) + context.sandbox = record.id + context.destroy_sandbox = True + context.data["created_sandbox"] = record + return response + + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [create], + batch_timing=batch_timing, + ) if operation == "exec_command": operation_session = context.data.get("operation_session") session_id = ( operation_session.session_id if operation_session is not None - else context.sessions[0].session_id if context.sessions else None - ) - return await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: product.exec_command( - sandbox, session_id=session_id, command=body["command"], timeout_ms=timeout, - request_id=request_ids[index] - ) for index in range(body["concurrent_requests"]) - ]) + else context.sessions[0].session_id + if context.sessions + else None + ) + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: product.exec_command( + sandbox, + session_id=session_id, + command=body["command"], + timeout_ms=timeout, + request_id=request_ids[index], + ) + for index in range(body["concurrent_requests"]) + ], + batch_timing=batch_timing, + ) if operation == "file_read": session = context.data.get("read_session") - return await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: product.file_read( - sandbox, session_id=session.session_id if session else None, - path=context.data["paths"][index if body["target_mode"] == "independent" else 0], - offset=1, limit=1, timeout_ms=timeout, request_id=request_ids[index] - ) for index in range(body["concurrent_requests"]) - ]) + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: product.file_read( + sandbox, + session_id=session.session_id if session else None, + path=context.data["paths"][ + index if body["target_mode"] == "independent" else 0 + ], + offset=1, + limit=1, + timeout_ms=timeout, + request_id=request_ids[index], + ) + for index in range(body["concurrent_requests"]) + ], + batch_timing=batch_timing, + ) if operation == "file_write": operation_session = context.data.get("operation_session") session_id = ( operation_session.session_id if operation_session is not None - else context.sessions[0].session_id if context.sessions else None + else context.sessions[0].session_id + if context.sessions + else None ) - contents = [_multiline_content(body["content_bytes"], f"write-request:{trial_id}:{index}") for index in range(body["concurrent_requests"])] + contents = [ + _multiline_content( + body["content_bytes"], f"write-request:{trial_id}:{index}" + ) + for index in range(body["concurrent_requests"]) + ] context.data["request_contents"] = contents - return await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: product.file_write( - sandbox, session_id=session_id, - path=context.data["paths"][index if body["target_mode"] == "independent" else 0], - content=contents[index], timeout_ms=timeout, request_id=request_ids[index] - ) for index in range(body["concurrent_requests"]) - ]) + try: + if isinstance(product, ProductCliAccess): + for index, content in enumerate(contents): + await product.stage_file_write_content( + content, request_id=request_ids[index] + ) + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: product.file_write( + sandbox, + session_id=session_id, + path=context.data["paths"][ + index if body["target_mode"] == "independent" else 0 + ], + content=contents[index], + timeout_ms=timeout, + request_id=request_ids[index], + ) + for index in range(body["concurrent_requests"]) + ], + batch_timing=batch_timing, + ) + finally: + if isinstance(product, ProductCliAccess): + for request_id in request_ids: + product.discard_file_write_content(request_id) if operation == "file_edit": operation_session = context.data.get("operation_session") session_id = ( operation_session.session_id if operation_session is not None - else context.sessions[0].session_id if context.sessions else None + else context.sessions[0].session_id + if context.sessions + else None ) context.data["operation_request_ids"] = request_ids - return await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: product.file_edit( - sandbox, session_id=session_id, - path=context.data["paths"][index if body["target_mode"] == "independent" else 0], - edits=context.data["edits"][index], timeout_ms=timeout, - request_id=request_ids[index] - ) for index in range(body["concurrent_requests"]) - ]) + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: product.file_edit( + sandbox, + session_id=session_id, + path=context.data["paths"][ + index if body["target_mode"] == "independent" else 0 + ], + edits=context.data["edits"][index], + timeout_ms=timeout, + request_id=request_ids[index], + ) + for index in range(body["concurrent_requests"]) + ], + batch_timing=batch_timing, + ) if operation == "file_blame": - return await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: product.file_blame( - sandbox, path=context.data["path"], timeout_ms=timeout, - request_id=request_ids[index] - ) for index in range(body["concurrent_requests"]) - ]) + return await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: product.file_blame( + sandbox, + path=context.data["path"], + timeout_ms=timeout, + request_id=request_ids[index], + ) + for index in range(body["concurrent_requests"]) + ], + batch_timing=batch_timing, + ) if operation == "create_workspace": - created = await self._run_request_batch(run_id, cell_id, trial_id, request_ids, [ - lambda index=index: sessions.create_no_op( - sandbox, body["network_profile"], request_id=request_ids[index], - timeout_ms=timeout - ) for index in range(body["workspace_count"]) - ]) + created = await self._run_request_batch( + run_id, + cell_id, + trial_id, + request_ids, + [ + lambda index=index: sessions.create_no_op( + sandbox, + body["network_profile"], + request_id=request_ids[index], + timeout_ms=timeout, + ) + for index in range(body["workspace_count"]) + ], + batch_timing=batch_timing, + ) context.sessions.extend(item[0] for item in created) return [item[1] for item in created] if operation == "squash_layerstack": + async def squash() -> TimedGatewayResponse: return await product.squash_layerstacks( sandbox, timeout_ms=timeout, request_id=request_ids[0] @@ -995,7 +1452,12 @@ async def squash() -> TimedGatewayResponse: if run_id is None: return [await squash()] return await self._run_request_batch( - run_id, cell_id, trial_id, request_ids, [squash] + run_id, + cell_id, + trial_id, + request_ids, + [squash], + batch_timing=batch_timing, ) raise CampaignError(f"operation {operation} is not implemented") @@ -1006,39 +1468,79 @@ async def _run_request_batch( trial_id: str, request_ids: list[str], operations: list[Callable[[], Awaitable[Any]]], + *, + batch_timing: BatchTiming | None = None, ) -> list[Any]: if run_id is None: - return await self._run_batch(operations) + return await self._run_batch(operations, batch_timing=batch_timing) terminal: set[str] = set() - for request_id in request_ids: - await self._request_state( - run_id, cell_id, trial_id, request_id, "waiting_at_barrier" - ) + pending_states: list[dict[str, Any]] = [] + + def state_record(request_id: str, state: str) -> dict[str, Any]: + self._event_sequence += 1 + return { + "sequence": self._event_sequence, + "run_id": run_id, + "monotonic_offset_ns": max(0, time.monotonic_ns() - self._started_ns), + "data": { + "kind": "request_state", + "cell_id": cell_id, + "trial_id": trial_id, + "request_id": request_id, + "state": state, + }, + } + + def defer(request_id: str, state: str) -> None: + pending_states.append(state_record(request_id, state)) + + def defer_all(state: str) -> None: + for request_id in request_ids: + defer(request_id, state) + + async def commit_pre_release() -> None: + # Setup/phase, waiting, and all-tasks-ready records form one ordered + # durability transaction before the gate opens. A failed commit + # prevents release; a successful commit proves every request was + # admitted and ready without paying three independent fsyncs. + defer_all("ready_at_barrier") + records = list(pending_states) + pending_states.clear() + event_buffer = getattr(self, "_trial_event_buffer", None) + sink_buffer = getattr(self, "_trial_event_sink_buffer", None) + if event_buffer is None and sink_buffer is None: + self._store.append_records(run_id, ArtifactId.EVENTS, records) + if self._event_sink is not None: + for record in records: + await self._event_sink(record) + return + if event_buffer is None or sink_buffer is None: + raise CampaignError("trial journal transaction is inconsistent") + event_buffer.extend(records) + if self._event_sink is not None: + sink_buffer.extend(records) + await self._flush_trial_journal(run_id) + + defer_all("waiting_at_barrier") async def observed( request_id: str, operation: Callable[[], Awaitable[Any]] ) -> Any: - await self._request_state( - run_id, cell_id, trial_id, request_id, "in_flight" - ) + # Record the transition in memory so journal fsync and live-event + # delivery cannot sit between barrier release and process creation. + defer(request_id, "in_flight") try: result = await operation() except asyncio.CancelledError: terminal.add(request_id) - await self._request_state( - run_id, cell_id, trial_id, request_id, "cancelled" - ) + defer(request_id, "cancelled") raise except BaseException: terminal.add(request_id) - await self._request_state( - run_id, cell_id, trial_id, request_id, "failed" - ) + defer(request_id, "failed") raise terminal.add(request_id) - await self._request_state( - run_id, cell_id, trial_id, request_id, "succeeded" - ) + defer(request_id, "succeeded") return result wrapped = [ @@ -1048,26 +1550,46 @@ async def observed( for request_id, operation in zip(request_ids, operations, strict=True) ] try: - return await self._run_batch(wrapped) + return await self._run_batch( + wrapped, + batch_timing=batch_timing, + before_release=commit_pre_release, + ) except asyncio.CancelledError: for request_id in request_ids: if request_id not in terminal: - await self._request_state( - run_id, cell_id, trial_id, request_id, "cancelled" - ) + defer(request_id, "cancelled") raise + finally: + if pending_states: + records = list(pending_states) + pending_states.clear() + self._store.append_records(run_id, ArtifactId.EVENTS, records) + if self._event_sink is not None: + for record in records: + await self._event_sink(record) async def _run_batch( - self, operations: list[Callable[[], Awaitable[Any]]] + self, + operations: list[Callable[[], Awaitable[Any]]], + *, + batch_timing: BatchTiming | None = None, + before_release: Callable[[], Awaitable[None]] | None = None, ) -> list[Any]: """Admit one synchronized request batch and give in-flight work bounded grace.""" self._check_cancelled() if not operations: raise CampaignError("request batch must not be empty") gate = asyncio.Event() + all_waiting = asyncio.Event() + waiting = 0 results: list[Any] = [None] * len(operations) async def run_one(index: int, operation: Callable[[], Awaitable[Any]]) -> None: + nonlocal waiting + waiting += 1 + if waiting == len(operations): + all_waiting.set() await gate.wait() results[index] = await operation() @@ -1075,6 +1597,11 @@ async def run_all() -> None: async with asyncio.TaskGroup() as group: for index, operation in enumerate(operations): group.create_task(run_one(index, operation)) + await all_waiting.wait() + if before_release is not None: + await before_release() + if batch_timing is not None: + batch_timing.barrier_released_ns = time.monotonic_ns() gate.set() batch = asyncio.create_task(run_all()) @@ -1085,6 +1612,8 @@ async def run_all() -> None: ) if batch in done: await batch + if batch_timing is not None: + batch_timing.batch_completed_ns = time.monotonic_ns() return results try: await asyncio.wait_for( @@ -1120,13 +1649,33 @@ async def _verify( value = response.value if not isinstance(value, dict): raise CampaignError("product response is not an object") - if operation == "exec_command": - expected = "4096\n" if body["command_case"] == "fixture_read" else ("x" * 65536 if body["command_case"] == "output64_kib" else "") - if value.get("status") != "ok" or value.get("exit_code") != body["expected_exit_code"] or value.get("output") != expected: + if operation == "create_sandbox": + record = await product.inspect_sandbox( + context.sandbox, + request_id=f"{trial_id}.verify.inspect", + ) + if ( + record.state != "ready" + or not same_native_path(record.workspace_root, context.workspace) + or body["network_profile"] != "shared" + ): + raise CampaignError("sandbox readiness check failed") + context.data["verified_sandbox"] = record + elif operation == "exec_command": + expected = _expected_command_output(body["command_case"]) + if ( + value.get("status") != "ok" + or value.get("exit_code") != body["expected_exit_code"] + or value.get("output") != expected + ): raise CampaignError("command correctness check failed") elif operation == "file_read": fixture = index if body["target_mode"] == "independent" else 0 - if not _read_matches(value, context.data["paths"][fixture], context.data["contents"][fixture]): + if not _read_matches( + value, + context.data["paths"][fixture], + context.data["contents"][fixture], + ): raise CampaignError("file read correctness check failed") elif operation == "file_write": fixture = index if body["target_mode"] == "independent" else 0 @@ -1156,43 +1705,45 @@ async def _verify( ): raise CampaignError("file blame correctness check failed") elif operation == "create_workspace": - if value.get("finalize_policy") != "no_op" or value.get("network_profile") != body["network_profile"] or not value.get("workspace_session_id"): + if ( + value.get("finalize_policy") != "no_op" + or value.get("network_profile") != body["network_profile"] + or not value.get("workspace_session_id") + ): raise CampaignError("workspace readiness check failed") if operation in {"file_write", "file_edit"}: - operation_session = context.data.get("operation_session") - session_id = ( - operation_session.session_id - if operation_session is not None - else context.sessions[0].session_id if context.sessions else None + expected = ( + context.data.get("request_contents") + if operation == "file_write" + else context.data["expected"] ) - expected = context.data.get("request_contents") if operation == "file_write" else context.data["expected"] - for index, path in enumerate(context.data["paths"]): - observed = await self._read_exact( - product, - context.sandbox, - session_id=session_id, - path=path, - expected_bytes=( - body["content_bytes"] if operation == "file_write" else body["file_bytes"] + await _gather_complete( + [ + self._verify_mutation_contents( + product, cell, context, trial_id, timeout, expected ), - timeout_ms=timeout, - request_id=f"{trial_id}.verify.{index}", - ) - allowed = expected if body["target_mode"] == "same_target" else [expected[index]] - if observed not in allowed: - raise CampaignError(f"{operation} content verification failed") - await self._verify_mutation_attribution( - product, cell, context, trial_id, timeout + self._verify_mutation_attribution( + product, cell, context, trial_id, timeout + ), + ], + "mutation correctness verification", ) context.data["observed_sha256"] = _sha_text("\n".join(expected)) if operation == "squash_layerstack": self._verify_squash_response(responses[0].value, context, sessions) for index, path in enumerate(context.data["paths"]): observed = await product.file_read( - context.sandbox, session_id=None, path=path, offset=1, limit=1, - timeout_ms=timeout, request_id=f"{trial_id}.verify.layer.{index}" + context.sandbox, + session_id=None, + path=path, + offset=1, + limit=1, + timeout_ms=timeout, + request_id=f"{trial_id}.verify.layer.{index}", ) - if not _read_matches(observed.value, path, context.data["contents"][index]): + if not _read_matches( + observed.value, path, context.data["contents"][index] + ): raise CampaignError("squash content equivalence check failed") context.data["content_equivalent"] = True @@ -1209,19 +1760,41 @@ async def _read_exact( ) -> str: if not 1 <= expected_bytes <= 4 * 1024 * 1024: raise CampaignError("verification read exceeds its fixed bound") - content = "" - offset = 1 - total_lines: int | None = None - for page in range(64): - response = await product.file_read( + page_limit = 2000 + + async def read_page(page: int, offset: int) -> Any: + return await product.file_read( sandbox, session_id=session_id, path=path, offset=offset, - limit=2000, + limit=page_limit, timeout_ms=timeout_ms, request_id=f"{request_id}.{page}", ) + + first = await read_page(0, 1) + first_value = first.value + if not isinstance(first_value, dict): + raise CampaignError("verification read response is invalid") + total_lines = first_value.get("total_lines") + if not isinstance(total_lines, int) or total_lines < 1: + raise CampaignError("verification read page contract failed") + page_count = (total_lines + page_limit - 1) // page_limit + if not 1 <= page_count <= 64: + raise CampaignError("verification read exceeded its page bound") + offsets = [1 + page * page_limit for page in range(page_count)] + responses = [first] + if page_count > 1: + responses.extend( + await _gather_complete( + [read_page(page, offsets[page]) for page in range(1, page_count)], + "verification read pages", + ) + ) + + content_pages: list[str] = [] + for page, (offset, response) in enumerate(zip(offsets, responses, strict=True)): value = response.value if not isinstance(value, dict): raise CampaignError("verification read response is invalid") @@ -1229,60 +1802,147 @@ async def _read_exact( num_lines = value.get("num_lines") known_total_lines = value.get("total_lines") next_offset = value.get("next_offset") + expected_num_lines = min(page_limit, total_lines - offset + 1) + expected_next = ( + offset + expected_num_lines if page + 1 < page_count else None + ) page_valid = ( value.get("path") == path and value.get("start_line") == offset and isinstance(page_content, str) - and isinstance(num_lines, int) - and 1 <= num_lines <= 2000 - and isinstance(known_total_lines, int) - and known_total_lines >= num_lines + and num_lines == expected_num_lines + and known_total_lines == total_lines and value.get("bytes_read") == len(page_content.encode()) and value.get("total_bytes") == expected_bytes - and value.get("truncated") == (next_offset is not None) - and (total_lines is None or total_lines == known_total_lines) + and next_offset == expected_next + and value.get("truncated") == (expected_next is not None) ) if not page_valid: raise CampaignError("verification read page contract failed") - total_lines = known_total_lines - if content: - content += "\n" - content += page_content - next_expected = offset + num_lines - if next_offset is None: - if next_expected != known_total_lines + 1 or len(content.encode()) != expected_bytes: - raise CampaignError("verification read completion contract failed") - return content - if next_offset != next_expected or next_offset > known_total_lines: - raise CampaignError("verification read continuation contract failed") - offset = next_offset - raise CampaignError("verification read exceeded its page bound") + content_pages.append(page_content) + content = "\n".join(content_pages) + if len(content.encode()) != expected_bytes: + raise CampaignError("verification read completion contract failed") + return content - async def _verify_mutation_attribution( + async def _verify_mutation_contents( self, product: ProductAccess, cell: dict[str, Any], context: TrialContext, trial_id: str, timeout_ms: int, + expected: list[str], ) -> None: operation = cell["operation"]["operation"] body = cell["operation"]["cell"] - if body["destination"] == "session": - for index, path in enumerate(context.data["paths"]): - snapshot = await self._read_exact( + operation_session = context.data.get("operation_session") + session_id = ( + operation_session.session_id + if operation_session is not None + else context.sessions[0].session_id + if context.sessions + else None + ) + paths = context.data["paths"] + if body["target_mode"] == "independent" and len(paths) > 1: + observed_contents = await _gather_complete( + [ + self._read_exact( + product, + context.sandbox, + session_id=session_id, + path=path, + expected_bytes=( + body["content_bytes"] + if operation == "file_write" + else body["file_bytes"] + ), + timeout_ms=timeout_ms, + request_id=f"{trial_id}.verify.{index}", + ) + for index, path in enumerate(paths) + ], + "independent content verification", + ) + else: + observed_contents = [ + await self._read_exact( product, context.sandbox, - session_id=None, + session_id=session_id, path=path, expected_bytes=( - body["content_bytes"] if operation == "file_write" else body["file_bytes"] + body["content_bytes"] + if operation == "file_write" + else body["file_bytes"] ), timeout_ms=timeout_ms, - request_id=f"{trial_id}.verify.snapshot.{index}", + request_id=f"{trial_id}.verify.{index}", ) + for index, path in enumerate(paths) + ] + for index, observed in enumerate(observed_contents): + allowed = ( + expected if body["target_mode"] == "same_target" else [expected[index]] + ) + if observed not in allowed: + raise CampaignError(f"{operation} content verification failed") + + async def _verify_mutation_attribution( + self, + product: ProductAccess, + cell: dict[str, Any], + context: TrialContext, + trial_id: str, + timeout_ms: int, + ) -> None: + operation = cell["operation"]["operation"] + body = cell["operation"]["cell"] + paths = context.data["paths"] + if body["destination"] == "session": + if body["target_mode"] == "independent" and len(paths) > 1: + snapshots = await _gather_complete( + [ + self._read_exact( + product, + context.sandbox, + session_id=None, + path=path, + expected_bytes=( + body["content_bytes"] + if operation == "file_write" + else body["file_bytes"] + ), + timeout_ms=timeout_ms, + request_id=f"{trial_id}.verify.snapshot.{index}", + ) + for index, path in enumerate(paths) + ], + "independent snapshot verification", + ) + else: + snapshots = [ + await self._read_exact( + product, + context.sandbox, + session_id=None, + path=path, + expected_bytes=( + body["content_bytes"] + if operation == "file_write" + else body["file_bytes"] + ), + timeout_ms=timeout_ms, + request_id=f"{trial_id}.verify.snapshot.{index}", + ) + for index, path in enumerate(paths) + ] + for index, snapshot in enumerate(snapshots): if snapshot != context.data["before"][index]: - raise CampaignError("session mutation escaped into the published snapshot") + raise CampaignError( + "session mutation escaped into the published snapshot" + ) context.data["attributed_layer_count"] = 0 return @@ -1290,13 +1950,32 @@ async def _verify_mutation_attribution( f"{trial_id}.request.{index}" for index in range(body["concurrent_requests"]) ] - for index, path in enumerate(context.data["paths"]): - response = await product.file_blame( - context.sandbox, - path=path, - timeout_ms=timeout_ms, - request_id=f"{trial_id}.verify.blame.{index}", + if body["target_mode"] == "independent" and len(paths) > 1: + blame_responses = await _gather_complete( + [ + product.file_blame( + context.sandbox, + path=path, + timeout_ms=timeout_ms, + request_id=f"{trial_id}.verify.blame.{index}", + ) + for index, path in enumerate(paths) + ], + "independent attribution verification", ) + else: + blame_responses = [ + await product.file_blame( + context.sandbox, + path=path, + timeout_ms=timeout_ms, + request_id=f"{trial_id}.verify.blame.{index}", + ) + for index, path in enumerate(paths) + ] + for index, (path, response) in enumerate( + zip(paths, blame_responses, strict=True) + ): value = response.value if not isinstance(value, dict) or value.get("path") != path: raise CampaignError("publish mutation attribution response is invalid") @@ -1313,9 +1992,7 @@ async def _verify_mutation_attribution( for request_index in request_indices } baseline_owner = f"operation:{context.data['baseline_request_ids'][index]}" - owners = { - item.get("owner") for item in ranges if isinstance(item, dict) - } + owners = {item.get("owner") for item in ranges if isinstance(item, dict)} if not owners.intersection(mutation_owners) or not owners.issubset( mutation_owners | {baseline_owner} ): @@ -1327,12 +2004,14 @@ def _verify_squash_response( ) -> None: if ( not isinstance(value, dict) - or not set(value).issubset({ - "manifest_version", - "squashed_blocks", - "swept_sessions", - "faulty_sessions", - }) + or not set(value).issubset( + { + "manifest_version", + "squashed_blocks", + "swept_sessions", + "faulty_sessions", + } + ) or type(value.get("manifest_version")) is not int or value["manifest_version"] <= 0 ): @@ -1344,14 +2023,13 @@ def _verify_squash_response( source_layer_ids: list[str] = [] squashed_layer_ids: set[str] = set() for block in blocks: - if ( - not isinstance(block, dict) - or not set(block).issubset({ + if not isinstance(block, dict) or not set(block).issubset( + { "squashed_layer_id", "replaced_layer_ids", "replaced_layers", "blocked_reasons", - }) + } ): raise CampaignError("squash block schema is invalid") squashed_id = block.get("squashed_layer_id") @@ -1372,17 +2050,19 @@ def _verify_squash_response( and ( not isinstance(reasons, list) or not reasons - or any(not isinstance(reason, str) or not reason for reason in reasons) + or any( + not isinstance(reason, str) or not reason + for reason in reasons + ) ) ) ): raise CampaignError("squash block values are invalid") squashed_layer_ids.add(squashed_id) source_layer_ids.extend(replaced) - if ( - len(source_layer_ids) != len(set(source_layer_ids)) - or squashed_layer_ids.intersection(source_layer_ids) - ): + if len(source_layer_ids) != len( + set(source_layer_ids) + ) or squashed_layer_ids.intersection(source_layer_ids): raise CampaignError("squash layer identities are inconsistent") expected = {session.session_id: session for session in context.sessions} observed: set[str] = set() @@ -1391,9 +2071,9 @@ def _verify_squash_response( for item in swept: if ( not isinstance(item, dict) - or not set(item).issubset({ - "session_id", "disposition", "reason", "class_detail" - }) + or not set(item).issubset( + {"session_id", "disposition", "reason", "class_detail"} + ) or item.get("session_id") not in expected or item.get("disposition") not in {"migrated", "identity", "leased", "faulty", "session_gone"} @@ -1449,7 +2129,9 @@ def _verify_squash_response( raise CampaignError("squash faulty session summary is invalid") response_faulty[item["session_id"]] = item["class_detail"] if response_faulty != faulty_details: - raise CampaignError("squash faulty session summary disagrees with dispositions") + raise CampaignError( + "squash faulty session summary disagrees with dispositions" + ) body = context.data["squash_cell"] expected_replaced = body["squashable_blocks"] * body["layers_per_block"] if ( @@ -1462,7 +2144,9 @@ def _verify_squash_response( ): raise CampaignError("squash manifest reduction shape is invalid") if dispositions["migrated"] != context.data["eligible_sessions"]: - raise CampaignError("squash migration count does not match the requested ratio") + raise CampaignError( + "squash migration count does not match the requested ratio" + ) context.data.update( expected_remount_spans=dispositions["migrated"], dispositions={ @@ -1481,14 +2165,38 @@ async def _teardown_trial( trial_id: str, ) -> None: issues: list[BaseException] = [] - for index, session in enumerate(reversed(context.sessions)): - try: - await sessions.destroy(session, request_id=f"{trial_id}.session.destroy.{index}") - except BaseException as error: - issues.append(error) + owned_sessions = list(reversed(context.sessions)) + if ( + context.data.get("operation_id") == "create_workspace" + and len(owned_sessions) > 1 + ): + destroyed = await asyncio.gather( + *( + sessions.destroy( + session, + request_id=f"{trial_id}.session.destroy.{index}", + ) + for index, session in enumerate(owned_sessions) + ), + return_exceptions=True, + ) + issues.extend( + result for result in destroyed if isinstance(result, BaseException) + ) + else: + for index, session in enumerate(owned_sessions): + try: + await sessions.destroy( + session, + request_id=f"{trial_id}.session.destroy.{index}", + ) + except BaseException as error: + issues.append(error) if context.destroy_sandbox: try: - await product.destroy_sandbox(context.sandbox, request_id=f"{trial_id}.sandbox.destroy") + await product.destroy_sandbox( + context.sandbox, request_id=f"{trial_id}.sandbox.destroy" + ) except BaseException as error: issues.append(error) if issues: @@ -1506,13 +2214,19 @@ async def _teardown_cell( issues: list[BaseException] = [] for index, session in enumerate(reversed(context.sessions)): try: - await sessions.destroy(session, request_id=f"cell-{cell_id[-16:]}.session.destroy.{index}") + await sessions.destroy( + session, request_id=f"cell-{cell_id[-16:]}.session.destroy.{index}" + ) + except BaseException as error: + issues.append(error) + if context.sandbox: + try: + await product.destroy_sandbox( + context.sandbox, + request_id=f"cell-{cell_id[-16:]}.sandbox.destroy", + ) except BaseException as error: issues.append(error) - try: - await product.destroy_sandbox(context.sandbox, request_id=f"cell-{cell_id[-16:]}.sandbox.destroy") - except BaseException as error: - issues.append(error) if issues: raise BaseExceptionGroup("aggregated cell cleanup", issues) @@ -1527,16 +2241,105 @@ def _new_workspace(self, run_path: Path, name: str, cell: dict[str, Any]) -> Pat try: profile = self._profiles[profile_id] except KeyError as error: - raise CampaignError("cell selected an unmaterialized workspace profile") from error - materialize_workspace( - self._roots.fixtures, workspace, profile, self._seed - ) + raise CampaignError( + "cell selected an unmaterialized workspace profile" + ) from error + materialize_workspace(self._roots.fixtures, workspace, profile, self._seed) workspace_manifest = workspace / "fixture-manifest.json" workspace_manifest.unlink() return workspace + def _prepare_cell_workspaces( + self, run_path: Path, cells: list[dict[str, Any]] + ) -> None: + if self._prepared_cell_workspaces: + raise CampaignError("cell workspaces were already prepared") + grouped: dict[str, list[tuple[str, Path]]] = {} + for cell in cells: + operation = cell["operation"]["operation"] + isolation = None if operation == "create_sandbox" else _isolation(cell) + if operation != "create_sandbox" and isolation not in { + "reusable_verified_fixture", + "prepared_sandbox_per_cell", + "fresh_sessions_per_trial", + }: + continue + profile_id = cell["operation"]["cell"].get("workspace_profile") + if profile_id is None: + continue + cell_id = cell["cell_id"] + workspace = _new_workspace(run_path, f"cell-{cell_id[-16:]}") + grouped.setdefault(profile_id, []).append((cell_id, workspace)) + for profile_id, prepared in grouped.items(): + try: + profile = self._profiles[profile_id] + except KeyError as error: + raise CampaignError( + "cell selected an unmaterialized workspace profile" + ) from error + workspaces = [workspace for _, workspace in prepared] + materialize_workspaces( + self._roots.fixtures, + workspaces, + profile, + self._seed, + max_workers=4, + ) + for cell_id, workspace in prepared: + (workspace / "fixture-manifest.json").unlink() + self._prepared_cell_workspaces[cell_id] = workspace + + def _cell_workspace(self, run_path: Path, cell: dict[str, Any]) -> Path: + prepared = self._prepared_cell_workspaces.get(cell["cell_id"]) + if prepared is not None: + return prepared + return self._new_workspace(run_path, f"cell-{cell['cell_id'][-16:]}", cell) + + def _begin_trial_journal(self) -> None: + if any( + buffer is not None + for buffer in ( + self._trial_event_buffer, + self._trial_observation_buffer, + self._trial_event_sink_buffer, + ) + ): + raise CampaignError("trial journal transaction is already active") + self._trial_event_buffer = [] + self._trial_observation_buffer = [] + self._trial_event_sink_buffer = [] + + def _end_trial_journal(self) -> None: + self._trial_event_buffer = None + self._trial_observation_buffer = None + self._trial_event_sink_buffer = None + + async def _flush_trial_journal(self, run_id: str) -> None: + observations = getattr(self, "_trial_observation_buffer", None) + events = getattr(self, "_trial_event_buffer", None) + sink_events = getattr(self, "_trial_event_sink_buffer", None) + if observations is None and events is None and sink_events is None: + return + if observations is None or events is None or sink_events is None: + raise CampaignError("trial journal transaction is inconsistent") + if observations: + self._store.append_records(run_id, ArtifactId.OBSERVATIONS, observations) + observations.clear() + if events: + self._store.append_records(run_id, ArtifactId.EVENTS, events) + events.clear() + if sink_events: + committed = list(sink_events) + sink_events.clear() + if self._event_sink is not None: + for record in committed: + await self._event_sink(record) + async def _event(self, run_id: str, data: dict[str, Any]) -> None: - if data.get("kind") == "run_state" and self._manifest: + force_durable = data.get("kind") == "run_state" + if force_durable: + await self._flush_trial_journal(run_id) + if force_durable and self._manifest: self._manifest["state"] = data["state"] if data["state"] == "failed" and self._manifest.get("failure") is None: self._manifest["failure"] = { @@ -1544,17 +2347,27 @@ async def _event(self, run_id: str, data: dict[str, Any]) -> None: "message": "campaign execution failed; inspect persisted events and logs", "infrastructure": True, } - self._store.replace_snapshot(run_id, ArtifactId.RUN_MANIFEST, self._manifest) + self._store.replace_snapshot( + run_id, ArtifactId.RUN_MANIFEST, self._manifest + ) self._event_sequence += 1 record = { "sequence": self._event_sequence, "run_id": run_id, - "monotonic_offset_ns": max(0, time.monotonic_ns() - self._started_ns) if self._started_ns else 0, + "monotonic_offset_ns": max(0, time.monotonic_ns() - self._started_ns) + if self._started_ns + else 0, "data": data, } - self._store.append_record(run_id, ArtifactId.EVENTS, record) - if self._event_sink is not None: - await self._event_sink(record) + if force_durable or getattr(self, "_trial_event_buffer", None) is None: + self._store.append_record(run_id, ArtifactId.EVENTS, record) + if self._event_sink is not None: + await self._event_sink(record) + else: + self._trial_event_buffer.append(record) + if self._event_sink is not None: + assert self._trial_event_sink_buffer is not None + self._trial_event_sink_buffer.append(record) def _new_manifest( self, @@ -1574,13 +2387,41 @@ def _new_manifest( for artifact_id, spec in ARTIFACT_SPECS.items() if artifact_id in PRODUCER_ARTIFACT_IDS and spec.schema_name is not None } + gateway_policy = { + "semantic_revision": 1, + "mode": "isolated", + "loopback_only": True, + "isolated_runtime_per_execution_block": True, + "remount_sweep_widths": sorted( + {_block_width(block, plan) for block in plan["execution_blocks"]} + ), + "maximum_connections": 256, + "readiness_timeout_ms": 60_000, + "readiness_probe_timeout_ms": 2_000, + "readiness_poll_interval_ms": 50, + } + if _is_v11_windows_named_pipe_treatment(environment): + gateway_policy.update( + { + "semantic_revision": 2, + "protocol_version": ( + "ephemeral-sandbox-v1-practical-performance-v1.1" + ), + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + "loopback_only": False, + } + ) return { "schema_version": 2, "run_id": run_id, "name": plan["canonical_plan"]["name"], "plan_hash": plan["plan_hash"], "starting_preset": starting_preset, - "configuration_scope": plan["canonical_plan"]["configuration_base"]["scope"], + "configuration_scope": plan["canonical_plan"]["configuration_base"][ + "scope" + ], "state": "planned", "failure": None, "created_at": self._started_at, @@ -1598,30 +2439,32 @@ def _new_manifest( "sha256": definition_sha256, }, "fixed_lifecycle_policy": plan["fixed_lifecycle_policy"], - "gateway_policy": { - "semantic_revision": 1, - "mode": "isolated", - "loopback_only": True, - "isolated_runtime_per_execution_block": True, - "remount_sweep_widths": sorted( - {_block_width(block, plan) for block in plan["execution_blocks"]} - ), - "maximum_connections": 256, - "readiness_timeout_ms": 60_000, - "readiness_probe_timeout_ms": 2_000, - "readiness_poll_interval_ms": 50, - }, + "gateway_policy": gateway_policy, + "gateway_execution_blocks": [], "started_at": self._started_at, "ended_at": None, "correctness": "pending", } async def _trial_phase( - self, run_id: str, cell_id: str, trial_id: str, warmup: bool, phase: str, state: str + self, + run_id: str, + cell_id: str, + trial_id: str, + warmup: bool, + phase: str, + state: str, ) -> None: await self._event( run_id, - {"kind": "trial_phase", "cell_id": cell_id, "trial_id": trial_id, "warmup": warmup, "phase": phase, "state": state}, + { + "kind": "trial_phase", + "cell_id": cell_id, + "trial_id": trial_id, + "warmup": warmup, + "phase": phase, + "state": state, + }, ) async def _trial_state( @@ -1657,47 +2500,94 @@ async def _request_state( }, ) - async def _resource_observation( - self, run_id: str, data: dict[str, Any] + async def _resource_observations( + self, run_id: str, samples: list[dict[str, Any]] ) -> None: - self._observation(run_id, {"record": "resource", "data": data}) - reading = data["reading"] - value = reading["value"] - available = value["availability"] == "available" - await self._event( - run_id, - { - "kind": "resource_window", - "cell_id": data["cell_id"], - "trial_id": data["trial_id"], - "metric_id": reading["metric_id"], - "value": value["value"] if available else None, - "unavailable_reason": None if available else value["reason"], - }, - ) + observations: list[dict[str, Any]] = [] + events: list[dict[str, Any]] = [] + for data in samples: + self._observation_sequence += 1 + observations.append( + { + "sequence": self._observation_sequence, + "record": {"record": "resource", "data": data}, + } + ) + reading = data["reading"] + value = reading["value"] + available = value["availability"] == "available" + self._event_sequence += 1 + events.append( + { + "sequence": self._event_sequence, + "run_id": run_id, + "monotonic_offset_ns": reading["monotonic_offset_ns"], + "data": { + "kind": "resource_window", + "cell_id": data["cell_id"], + "trial_id": data["trial_id"], + "metric_id": reading["metric_id"], + "value": value["value"] if available else None, + "unavailable_reason": (None if available else value["reason"]), + }, + } + ) + if getattr(self, "_trial_observation_buffer", None) is None: + # Non-trial callers retain one append and fsync per journal. + self._store.append_records(run_id, ArtifactId.OBSERVATIONS, observations) + self._store.append_records(run_id, ArtifactId.EVENTS, events) + if self._event_sink is not None: + for record in events: + await self._event_sink(record) + else: + assert self._trial_event_buffer is not None + self._trial_observation_buffer.extend(observations) + self._trial_event_buffer.extend(events) + if self._event_sink is not None: + assert self._trial_event_sink_buffer is not None + self._trial_event_sink_buffer.extend(events) def _observation(self, run_id: str, record: dict[str, Any]) -> None: self._observation_sequence += 1 - self._store.append_record( - run_id, ArtifactId.OBSERVATIONS, {"sequence": self._observation_sequence, "record": record} - ) + envelope = {"sequence": self._observation_sequence, "record": record} + if getattr(self, "_trial_observation_buffer", None) is None: + self._store.append_record(run_id, ArtifactId.OBSERVATIONS, envelope) + else: + self._trial_observation_buffer.append(envelope) async def _request_observation( - self, run_id: str, cell: dict[str, Any], trial_id: str, warmup: bool, response: TimedGatewayResponse + self, + run_id: str, + cell: dict[str, Any], + trial_id: str, + warmup: bool, + response: TimedGatewayResponse, ) -> None: - self._observation(run_id, {"record": "request", "data": { - "operation_id": cell["operation"]["operation"], - "cell_id": cell["cell_id"], "trial_id": trial_id, "request_id": response.request_id, - "warmup": warmup, - "start_offset_ns": max( - 0, - (response.started_ns - self._started_ns) - if response.started_ns is not None - else time.monotonic_ns() - self._started_ns - response.latency_ns, - ), - "latency_ns": response.latency_ns, "response_bytes": response.response_bytes, - "response_sha256": response.response_sha256, "status": "success", - }}) + self._observation( + run_id, + { + "record": "request", + "data": { + "operation_id": cell["operation"]["operation"], + "cell_id": cell["cell_id"], + "trial_id": trial_id, + "request_id": response.request_id, + "warmup": warmup, + "start_offset_ns": max( + 0, + (response.started_ns - self._started_ns) + if response.started_ns is not None + else time.monotonic_ns() + - self._started_ns + - response.latency_ns, + ), + "latency_ns": response.latency_ns, + "response_bytes": response.response_bytes, + "response_sha256": response.response_sha256, + "status": "success", + }, + }, + ) async def _registered_check_observations( self, @@ -1710,7 +2600,9 @@ async def _registered_check_observations( ) -> None: operation_id = cell["operation"]["operation"] operation = next( - item for item in self._definitions["operations"] if item["id"] == operation_id + item + for item in self._definitions["operations"] + if item["id"] == operation_id ) checks = operation["checks"] if not passed and failure_stage == "setup": @@ -1728,28 +2620,37 @@ async def _registered_check_observations( continue expected = check["help"] actual = "verified" if check_passed else f"failed during {failure_stage}" - self._observation(run_id, {"record": "check", "data": { - "operation_id": operation_id, - "cell_id": cell["cell_id"], - "trial_id": trial_id, - "request_id": None, - "check_id": check["id"], - "semantic_revision": check["semantic_revision"], - "passed": check_passed, - "expected": expected, - "actual": actual, - "artifact_id": None, - }}) - await self._event(run_id, { - "kind": "correctness", - "cell_id": cell["cell_id"], - "trial_id": trial_id, - "check_id": check["id"], - "passed": check_passed, - "expected": expected, - "actual": actual, - "artifact_id": None, - }) + self._observation( + run_id, + { + "record": "check", + "data": { + "operation_id": operation_id, + "cell_id": cell["cell_id"], + "trial_id": trial_id, + "request_id": None, + "check_id": check["id"], + "semantic_revision": check["semantic_revision"], + "passed": check_passed, + "expected": expected, + "actual": actual, + "artifact_id": None, + }, + }, + ) + await self._event( + run_id, + { + "kind": "correctness", + "cell_id": cell["cell_id"], + "trial_id": trial_id, + "check_id": check["id"], + "passed": check_passed, + "expected": expected, + "actual": actual, + "artifact_id": None, + }, + ) async def _phase_observations( self, @@ -1796,25 +2697,32 @@ def visit(nodes: list[Any]) -> None: commit_offset_ns = request_start + round( (node.offset_ms + node.span.dur_ms) * 1_000_000 ) - self._observation(run_id, {"record": "phase", "data": { - "id": definition["id"], - "semantic_revision": definition["semantic_revision"], - "unit": definition["unit"], - "cell_id": cell["cell_id"], - "trial_id": trial_id, - "request_id": response.request_id, - "source": definition["source"], - "correlation": definition["correlation"], - "trace_span_name": definition["trace_span_name"], - "start_offset_ns": request_start + round(node.offset_ms * 1_000_000), - "duration_ns": round(node.span.dur_ms * 1_000_000), - "status": { - "completed": "succeeded", - "error": "failed", - "cancelled": "cancelled", - "timed_out": "timed_out", - }[node.span.status], - }}) + self._observation( + run_id, + { + "record": "phase", + "data": { + "id": definition["id"], + "semantic_revision": definition["semantic_revision"], + "unit": definition["unit"], + "cell_id": cell["cell_id"], + "trial_id": trial_id, + "request_id": response.request_id, + "source": definition["source"], + "correlation": definition["correlation"], + "trace_span_name": definition["trace_span_name"], + "start_offset_ns": request_start + + round(node.offset_ms * 1_000_000), + "duration_ns": round(node.span.dur_ms * 1_000_000), + "status": { + "completed": "succeeded", + "error": "failed", + "cancelled": "cancelled", + "timed_out": "timed_out", + }[node.span.status], + }, + }, + ) visit(node.children) visit(trace.spans) @@ -1832,7 +2740,9 @@ def visit(nodes: list[Any]) -> None: f"observed {observed}" ) if any(failed.values()): - raise CampaignError(f"product trace contains failed squash phases: {failed}") + raise CampaignError( + f"product trace contains failed squash phases: {failed}" + ) if squash_node is None or commit_offset_ns is None: raise CampaignError("product trace omitted squash evidence boundaries") attrs = squash_node.span.attrs @@ -1844,14 +2754,16 @@ def visit(nodes: list[Any]) -> None: or attrs.get("sweep_width") != cell["operation"]["cell"]["remount_parallelism"] ): - raise CampaignError("product squash trace attributes disagree with the response") + raise CampaignError( + "product squash trace attributes disagree with the response" + ) s2 = _trace_layerstack_evidence(attrs, commit_offset_ns, sampled=False) if s2["manifest_version"]["value"] != value["manifest_version"]: - raise CampaignError("product squash trace manifest identity is inconsistent") + raise CampaignError( + "product squash trace manifest identity is inconsistent" + ) context.data["s2_post_commit"] = s2 - context.data["s1_sampled_peak"] = _sampled_peak( - context.data["s0_baseline"], s2 - ) + context.data["s1_sampled_peak"] = _sampled_peak(context.data["s0_baseline"], s2) settled, settled_offset = await self._settle_layerstack( product, context, trial_id @@ -1897,7 +2809,9 @@ async def _settle_layerstack( previous = signature poll += 1 await asyncio.sleep(0.1) - raise CampaignError("layerstack did not reach an exact three-sample quiet window") + raise CampaignError( + "layerstack did not reach an exact three-sample quiet window" + ) async def _shielded_cleanup(self, cleanup: Awaitable[None]) -> None: task = asyncio.create_task(cleanup) @@ -1947,27 +2861,35 @@ async def _trial_observation( and not outcome.infrastructure_failed and outcome.cleanup_baseline_restored ) - self._observation(run_id, {"record": "trial", "data": { - "operation_id": cell["operation"]["operation"], - "cell_id": cell["cell_id"], - "trial_id": trial_id, - "warmup": warmup, - "kind": "warmup" if warmup else "measured", - "sequence_in_cell": sequence_in_cell, - "reportable": reportable, - "latency_ns": outcome.batch_makespan_ns or None, - "request_count": len(outcome.responses), - "status": outcome.status, - "product_succeeded": outcome.product_succeeded, - "infrastructure_failed": outcome.infrastructure_failed, - "cleanup_baseline_restored": outcome.cleanup_baseline_restored, - "checks_passed": outcome.checks_passed, - "setup_ns": outcome.setup_ns, - "operation_ns": outcome.operation_ns, - "verify_ns": outcome.verify_ns, - "teardown_ns": outcome.teardown_ns, - "artifacts": [outcome.artifact] if outcome.artifact is not None else [], - }}) + self._observation( + run_id, + { + "record": "trial", + "data": { + "operation_id": cell["operation"]["operation"], + "cell_id": cell["cell_id"], + "trial_id": trial_id, + "warmup": warmup, + "kind": "warmup" if warmup else "measured", + "sequence_in_cell": sequence_in_cell, + "reportable": reportable, + "latency_ns": outcome.batch_makespan_ns or None, + "request_count": len(outcome.responses), + "status": outcome.status, + "product_succeeded": outcome.product_succeeded, + "infrastructure_failed": outcome.infrastructure_failed, + "cleanup_baseline_restored": outcome.cleanup_baseline_restored, + "checks_passed": outcome.checks_passed, + "setup_ns": outcome.setup_ns, + "operation_ns": outcome.operation_ns, + "verify_ns": outcome.verify_ns, + "teardown_ns": outcome.teardown_ns, + "artifacts": [outcome.artifact] + if outcome.artifact is not None + else [], + }, + }, + ) def _operation_evidence( @@ -1988,6 +2910,18 @@ def _operation_evidence( "stdout": _content_summary(output), "stderr": _content_summary(""), } + elif operation == "create_sandbox": + created = context.data["created_sandbox"] + verified = context.data["verified_sandbox"] + detail = { + "workspace_profile": body["workspace_profile"], + "network_profile": body["network_profile"], + "sandbox_id": created.id, + "create_state": created.state, + "verified_state": verified.state, + "workspace_root": str(context.workspace), + "ready": verified.state == "ready", + } elif operation == "file_read": content = context.data["contents"][0] detail = { @@ -2003,7 +2937,9 @@ def _operation_evidence( "observed_bytes": len(content.encode()), "expected_sha256": _sha_text(content), "observed_sha256": _sha_text(content), - "attribution": "workspace_session" if body["destination"] == "session" else "published_layer", + "attribution": "workspace_session" + if body["destination"] == "session" + else "published_layer", "attributed_layer_count": context.data["attributed_layer_count"], } elif operation == "file_edit": @@ -2015,7 +2951,9 @@ def _operation_evidence( "before_sha256": _sha_text("\n".join(context.data["before"])), "expected_sha256": _sha_text("\n".join(context.data["expected"])), "observed_sha256": context.data["observed_sha256"], - "attribution": "workspace_session" if body["destination"] == "session" else "published_layer", + "attribution": "workspace_session" + if body["destination"] == "session" + else "published_layer", "attributed_layer_count": context.data["attributed_layer_count"], } elif operation == "file_blame": @@ -2079,10 +3017,9 @@ def _operation_evidence( ) baseline = context.data["s0_view"] settled = context.data["s3_view"] - manifest_reduced = ( - settled.manifest_version > baseline.manifest_version - and len(settled.layers) < len(baseline.layers) - ) + manifest_reduced = settled.manifest_version > baseline.manifest_version and len( + settled.layers + ) < len(baseline.layers) if not manifest_reduced: raise CampaignError("squash did not produce a reduced settled manifest") detail = { @@ -2110,7 +3047,17 @@ def _operation_evidence( } else: # pragma: no cover - closed by planning and _operate raise CampaignError(f"operation {operation} has no evidence contract") - return {"operation": operation, "evidence": detail} + result = {"operation": operation, "evidence": detail} + transport = [ + response.transport_evidence + for response in responses + if response.transport_evidence is not None + ] + if transport: + if len(transport) != len(responses): + raise CampaignError("operation mixed product transport cohorts") + result["transport"] = transport + return result def _content_summary(value: Any) -> dict[str, Any]: @@ -2231,6 +3178,31 @@ def _sha_text(value: str) -> str: return f"sha256:{hashlib.sha256(value.encode()).hexdigest()}" +def _validated_response_end_ns( + responses: list[TimedGatewayResponse], *, fallback: int | None +) -> int: + ended = [ + response.started_ns + response.latency_ns + for response in responses + if response.started_ns is not None + ] + if len(ended) == len(responses) and ended: + return max(ended) + if fallback is None: + raise CampaignError("request batch did not record a validated completion") + return fallback + + +def _expected_command_output(command_case: str) -> str: + if command_case == "fixture_read": + # The released command API returns a line-window projection. Its + # `output` joins logical lines without retaining their delimiters. + return "4096" + if command_case == "output64_kib": + return "x" * 65536 + return "" + + def _required_string(value: dict[str, Any], *path: str) -> str: current: Any = value for component in path: @@ -2247,7 +3219,9 @@ def _isolation(cell: dict[str, Any]) -> str: comparison = cell["comparison_key"]["isolation"] resolved = cell["operation"]["cell"]["resolved_isolation"] except (KeyError, TypeError) as error: - raise CampaignError("expanded cell is missing its isolation contract") from error + raise CampaignError( + "expanded cell is missing its isolation contract" + ) from error if not isinstance(comparison, str) or comparison != resolved: raise CampaignError("expanded cell isolation authorities disagree") return comparison @@ -2272,11 +3246,18 @@ def _multiline_content(length: int, seed: str) -> str: def _edits(seed: str, request: int, count: int) -> list[dict[str, Any]]: - return [{ - "old_string": hashlib.sha256(f"old:{seed}:{request}:{index}".encode()).hexdigest(), - "new_string": hashlib.sha256(f"new:{seed}:{request}:{index}".encode()).hexdigest(), - "replace_all": True, - } for index in range(count)] + return [ + { + "old_string": hashlib.sha256( + f"old:{seed}:{request}:{index}".encode() + ).hexdigest(), + "new_string": hashlib.sha256( + f"new:{seed}:{request}:{index}".encode() + ).hexdigest(), + "replace_all": True, + } + for index in range(count) + ] def _edit_content( @@ -2320,7 +3301,12 @@ def _blame_content(lines: int, segments: int, event: int) -> str: def _read_matches(value: Any, path: str, content: str) -> bool: - return isinstance(value, dict) and value.get("path") == path and value.get("content") == content and value.get("bytes_read") == len(content.encode()) + return ( + isinstance(value, dict) + and value.get("path") == path + and value.get("content") == content + and value.get("bytes_read") == len(content.encode()) + ) def _expected_blame_ranges( @@ -2351,11 +3337,21 @@ def _expected_blame_ranges( def _blame_matches( value: Any, path: str, lines: int, expected_ranges: list[dict[str, Any]] ) -> bool: - if not isinstance(value, dict) or value.get("path") != path or not isinstance(value.get("ranges"), list): + if ( + not isinstance(value, dict) + or value.get("path") != path + or not isinstance(value.get("ranges"), list) + ): return False next_line = 1 for item in value["ranges"]: - if not isinstance(item, dict) or item.get("start_line") != next_line or not isinstance(item.get("line_count"), int) or item["line_count"] <= 0 or not isinstance(item.get("owner"), str): + if ( + not isinstance(item, dict) + or item.get("start_line") != next_line + or not isinstance(item.get("line_count"), int) + or item["line_count"] <= 0 + or not isinstance(item.get("owner"), str) + ): return False next_line += item["line_count"] return next_line == lines + 1 and value["ranges"] == expected_ranges @@ -2363,6 +3359,7 @@ def _blame_matches( def _event_family(family: str) -> str: return { + "sandbox_lifecycle": "sandbox_lifecycle", "command": "command", "files": "files", "workspace_lifecycle": "workspace_lifecycle", @@ -2391,10 +3388,28 @@ def _gateway_log_summary(records: tuple[Any, ...]) -> str: return "gateway logs retained as redacted digests: " + ",".join(parts) +def _is_v11_windows_named_pipe_treatment( + environment: dict[str, Any], +) -> bool: + return ( + environment.get("client_cohort") == "product_cli" + and environment.get("gateway_endpoint_identity") + == "isolated_windows_named_pipe_per_execution_block" + and environment.get("gateway_transport") + == { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + } + ) + + def _block_width(block: dict[str, Any], plan: dict[str, Any]) -> int: if block["family_id"] != "layer_stack": return 1 - cell = next(item for item in plan["cells"] if item["cell_id"] == block["cell_ids"][0]) + cell = next( + item for item in plan["cells"] if item["cell_id"] == block["cell_ids"][0] + ) return cell["operation"]["cell"]["remount_parallelism"] @@ -2410,6 +3425,20 @@ def _combine_failures( return error +async def _gather_complete(awaitables: list[Awaitable[Any]], label: str) -> list[Any]: + results = await asyncio.gather(*awaitables, return_exceptions=True) + cancellation = next( + (result for result in results if isinstance(result, asyncio.CancelledError)), + None, + ) + if cancellation is not None: + raise cancellation + errors = [result for result in results if isinstance(result, BaseException)] + if errors: + raise BaseExceptionGroup(label, errors) + return results + + def _trial_status(failure: BaseException | None, stage: str) -> str: if failure is None: return "success" diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/safety.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/safety.py index c2080a8..c09135d 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/safety.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/safety.py @@ -5,10 +5,14 @@ from pydantic import ValidationError +from .fixtures import native_filesystem_path from .models import OwnedPathMarker from .paths import BenchmarkRoots, MARKER_NAME, PathContractError, _sync_directory +_BINARY_FLAG = getattr(os, "O_BINARY", 0) + + class OwnershipError(ValueError): pass @@ -23,7 +27,11 @@ def register(self, target: Path, marker: OwnedPathMarker) -> Path: marker_path = canonical / MARKER_NAME payload = json.dumps(marker.model_dump(mode="json"), indent=2, sort_keys=True).encode() + b"\n" try: - descriptor = os.open(marker_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + descriptor = os.open( + marker_path, + os.O_WRONLY | os.O_CREAT | os.O_EXCL | _BINARY_FLAG, + 0o600, + ) except OSError as error: raise OwnershipError(f"ownership marker already exists or cannot be created: {marker_path}") from error try: @@ -48,7 +56,7 @@ def remove(self, target: Path, expected: OwnedPathMarker) -> None: raise OwnershipError("ownership identity mismatch") if self._entries.get(canonical) != expected: raise OwnershipError("target is absent from the active ownership ledger") - shutil.rmtree(canonical) + shutil.rmtree(native_filesystem_path(canonical)) self._entries.pop(canonical) _sync_directory(canonical.parent) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/service.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/service.py index 5ccb013..7718ee1 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/service.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/service.py @@ -7,6 +7,7 @@ import json import secrets import shutil +import subprocess import time import uuid from dataclasses import dataclass @@ -17,7 +18,7 @@ from pydantic import ValidationError from .artifacts import ArtifactError, ArtifactId, ArtifactStore -from .catalog import CatalogError, export_catalog +from .catalog import CatalogError, export_catalog, probe_released_cli_operations from .comparison import compare_runs from .gateway import recover_stale_gateway from .planning import ( @@ -96,9 +97,15 @@ def refresh_product_catalog(self) -> None: result = export_catalog(self.roots) self._catalog_operations = result.catalog.operation_names() self._catalog_error = None - except CatalogError as error: - self._catalog_operations = None - self._catalog_error = str(error) + except CatalogError as export_error: + try: + self._catalog_operations = probe_released_cli_operations(self.roots) + self._catalog_error = None + except (CatalogError, OSError, subprocess.SubprocessError) as probe_error: + self._catalog_operations = None + self._catalog_error = ( + f"{export_error}; released CLI probe failed: {probe_error}" + ) @property def catalog_error(self) -> str | None: @@ -515,9 +522,14 @@ async def cleanup(self, run_id: str) -> dict[str, Any]: # unless the requested run is the only nonterminal result. manifest = self.store.read_envelope(run_id, ArtifactId.RUN_MANIFEST) if manifest["state"] in TERMINAL_STATES: - runtime = self.roots.runtime / run_id - if runtime.exists() or runtime.is_symlink(): - await recover_stale_gateway(self.roots, run_id) + + def cleanup_gateway(_: str) -> None: + runtime = self.roots.runtime / run_id + if runtime.exists() or runtime.is_symlink(): + asyncio.run(recover_stale_gateway(self.roots, run_id)) + + scanner = RecoveryScanner(self.roots, cleanup_gateway) + await asyncio.to_thread(scanner.cleanup_terminal_run, run_id) return {"schema_version": 1, "run_id": run_id, "cleaned": True, "terminalized": False} def cleanup_gateway(_: str) -> None: diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/sessions.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/sessions.py index fb3cbe0..1a75884 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/sessions.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/sessions.py @@ -1,22 +1,12 @@ from __future__ import annotations -import asyncio -import hashlib -import json -import os -import time from dataclasses import dataclass -from typing import Any from pydantic import ValidationError from .models import StrictModel from .product import ProductAccess, ProductAccessError, _identity -from .transport import MAX_WIRE_BYTES, TimedGatewayResponse - - -DAEMON_AUTH_FIELD = "_sandbox_daemon_auth_token" -DOCKER_AUTH_LABEL = "eos.auth_token" +from .transport import TimedGatewayResponse class SessionError(RuntimeError): @@ -60,11 +50,12 @@ def retire_product_destroyed(self, session: Session) -> None: async def create_no_op(self, sandbox_id: str, network_profile: str, *, request_id: str, timeout_ms: int = 120000) -> tuple[Session, TimedGatewayResponse]: if network_profile not in {"shared", "isolated"}: raise SessionError("network profile is not allowlisted") - record = await self._product.inspect_sandbox(sandbox_id, request_id=f"{request_id}.inspect") - if record.daemon is None: - raise SessionError("sandbox daemon endpoint is unavailable") - token = await _lookup_auth(sandbox_id) - response = await _daemon_request(record.daemon.host, record.daemon.port, token, "create_workspace_session", sandbox_id, {"network_profile": network_profile}, request_id, timeout_ms) + response = await self._product.create_workspace_session( + sandbox_id, + network_profile=network_profile, + timeout_ms=timeout_ms, + request_id=request_id, + ) try: created = CreatedSession.model_validate(response.value) session_id = _identity(created.workspace_session_id) @@ -78,11 +69,12 @@ async def create_no_op(self, sandbox_id: str, network_profile: str, *, request_i async def destroy(self, session: Session, *, request_id: str, timeout_ms: int = 120000) -> TimedGatewayResponse: if self._sessions.get(session.session_id) != session.sandbox_id: raise SessionError("session is not owned by this lifecycle") - record = await self._product.inspect_sandbox(session.sandbox_id, request_id=f"{request_id}.inspect") - if record.daemon is None: - raise SessionError("sandbox daemon endpoint is unavailable") - token = await _lookup_auth(session.sandbox_id) - response = await _daemon_request(record.daemon.host, record.daemon.port, token, "destroy_workspace_session", session.sandbox_id, {"workspace_session_id": session.session_id}, request_id, timeout_ms) + response = await self._product.destroy_workspace_session( + session.sandbox_id, + session_id=session.session_id, + timeout_ms=timeout_ms, + request_id=request_id, + ) try: destroyed = DestroyedSession.model_validate(response.value) except ValidationError as error: @@ -91,53 +83,3 @@ async def destroy(self, session: Session, *, request_id: str, timeout_ms: int = raise SessionError("destroy session response violated lifecycle contract") del self._sessions[session.session_id] return response - - -async def _lookup_auth(sandbox_id: str) -> str: - process = await asyncio.create_subprocess_exec( - "docker", "inspect", "--format", f'{{{{ index .Config.Labels "{DOCKER_AUTH_LABEL}" }}}}', sandbox_id, - env={name: os.environ[name] for name in ("PATH", "HOME", "DOCKER_HOST", "DOCKER_CONTEXT", "DOCKER_CONFIG") if name in os.environ}, - stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, - ) - try: - stdout, stderr = await asyncio.wait_for(process.communicate(), 30) - except TimeoutError as error: - process.kill(); await process.wait() - raise SessionError("daemon credential lookup timed out") from error - token = stdout.decode(errors="strict").strip() if process.returncode == 0 and len(stdout) <= 4096 and len(stderr) <= 16384 else "" - if not token or token == "" or any(character.isspace() for character in token): - raise SessionError("daemon credential lookup failed") - return token - - -async def _daemon_request(host: str, port: int, token: str, operation: str, sandbox_id: str, args: dict[str, Any], request_id: str, timeout_ms: int) -> TimedGatewayResponse: - request = {"op": operation, "request_id": _identity(request_id), "scope": {"kind": "sandbox", "sandbox_id": _identity(sandbox_id)}, "args": args, DAEMON_AUTH_FIELD: token} - payload = json.dumps(request, ensure_ascii=False, separators=(",", ":"), allow_nan=False).encode() + b"\n" - if len(payload) > MAX_WIRE_BYTES: - raise SessionError("daemon request exceeds wire bound") - writer: asyncio.StreamWriter | None = None - try: - reader, writer = await asyncio.wait_for(asyncio.open_connection(host, port, limit=MAX_WIRE_BYTES + 1), timeout_ms / 1000) - started = time.monotonic_ns() - async with asyncio.timeout(timeout_ms / 1000): - writer.write(payload); await writer.drain() - if writer.can_write_eof(): writer.write_eof() - raw = await reader.readuntil(b"\n") - ended = time.monotonic_ns() - trailing = await reader.read(1) - except (OSError, TimeoutError, asyncio.IncompleteReadError, asyncio.LimitOverrunError) as error: - raise SessionError("daemon transport failed") from error - finally: - if writer is not None: - writer.close() - try: await writer.wait_closed() - except OSError: pass - if trailing or token.encode() in raw or len(raw) > MAX_WIRE_BYTES: - raise SessionError("daemon response framing or credential contract failed") - try: - value = json.loads(raw) - except (UnicodeDecodeError, json.JSONDecodeError) as error: - raise SessionError("daemon response JSON is invalid") from error - if isinstance(value, dict) and "error" in value: - raise SessionError("daemon product operation failed") - return TimedGatewayResponse(request_id, ended - started, len(raw), f"sha256:{hashlib.sha256(raw).hexdigest()}", value) diff --git a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/transport.py b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/transport.py index f07253e..727a7c4 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/transport.py +++ b/ephemeral-sandbox-v1/benchmark/backend/benchmark_lab/transport.py @@ -2,6 +2,7 @@ import hashlib import ipaddress import json +import os import re import secrets import time @@ -18,6 +19,8 @@ MAX_ID_BYTES = 256 _SAFE_ID = re.compile(r"^[A-Za-z0-9_.:-]+$") _SENSITIVE_WORD = re.compile(r"(?i)\b(token|secret|password|credential|authorization)\b") +_NPIPE_URI_PREFIX = "npipe://./pipe/" +_SAFE_NPIPE_SEGMENT = re.compile(r"^[A-Za-z0-9_.-]+$") class GatewayError(RuntimeError): @@ -62,10 +65,18 @@ class ProductErrorEnvelope(StrictModel): @dataclass(frozen=True, slots=True) class GatewayEndpoint: - host: str - port: int + host: str | None = None + port: int | None = None + named_pipe_uri: str | None = None def __post_init__(self) -> None: + if self.named_pipe_uri is not None: + if self.host is not None or self.port is not None: + raise ValueError("gateway endpoint must select exactly one transport") + _validated_named_pipe_uri(self.named_pipe_uri) + return + if self.host is None or self.port is None or isinstance(self.port, bool): + raise ValueError("gateway TCP endpoint is incomplete") try: address = ipaddress.ip_address(self.host) except ValueError as error: @@ -73,6 +84,60 @@ def __post_init__(self) -> None: if not address.is_loopback or not 1 <= self.port <= 65535: raise ValueError("gateway endpoint must be a valid loopback socket") + @classmethod + def windows_named_pipe(cls, uri: str) -> "GatewayEndpoint": + return cls(named_pipe_uri=uri) + + @classmethod + def parse(cls, value: str) -> "GatewayEndpoint": + if value.startswith(_NPIPE_URI_PREFIX): + return cls.windows_named_pipe(value) + address = value.removeprefix("tcp://") + if address.startswith("["): + end = address.find("]") + if end < 0 or address[end + 1 : end + 2] != ":": + raise ValueError("gateway TCP endpoint is invalid") + host = address[1:end] + port_text = address[end + 2 :] + else: + host, separator, port_text = address.rpartition(":") + if not separator: + raise ValueError("gateway TCP endpoint is invalid") + try: + port = int(port_text) + except ValueError as error: + raise ValueError("gateway TCP endpoint is invalid") from error + return cls(host, port) + + @property + def transport(self) -> Literal["tcp_loopback", "windows_named_pipe"]: + return ( + "windows_named_pipe" + if self.named_pipe_uri is not None + else "tcp_loopback" + ) + + @property + def address(self) -> str: + if self.named_pipe_uri is not None: + return self.named_pipe_uri + assert self.host is not None and self.port is not None + host = f"[{self.host}]" if ":" in self.host else self.host + return f"{host}:{self.port}" + + @property + def uri(self) -> str: + if self.named_pipe_uri is not None: + return self.named_pipe_uri + return f"tcp://{self.address}" + + @property + def native_named_pipe_path(self) -> str: + if self.named_pipe_uri is None: + raise ValueError("gateway endpoint is not a Windows named pipe") + name = self.named_pipe_uri.removeprefix(_NPIPE_URI_PREFIX) + return "\\\\.\\pipe\\" + name.replace("/", "\\") + @dataclass(frozen=True, slots=True) class TimedGatewayResponse: @@ -82,6 +147,7 @@ class TimedGatewayResponse: response_sha256: str value: Any started_ns: int | None = None + transport_evidence: dict[str, Any] | None = None class GatewayClient: @@ -152,10 +218,8 @@ async def request( try: try: reader, writer = await asyncio.wait_for( - asyncio.open_connection( - self._endpoint.host, - self._endpoint.port, - limit=self._max_wire_bytes + 1, + _open_gateway_connection( + self._endpoint, limit=self._max_wire_bytes + 1 ), timeout=timeout_seconds, ) @@ -212,6 +276,60 @@ async def request( ) +async def _open_gateway_connection( + endpoint: GatewayEndpoint, + *, + limit: int, +) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]: + if endpoint.transport == "tcp_loopback": + assert endpoint.host is not None and endpoint.port is not None + return await asyncio.open_connection( + endpoint.host, + endpoint.port, + limit=limit, + ) + return await _open_windows_named_pipe_connection(endpoint, limit=limit) + + +async def _open_windows_named_pipe_connection( + endpoint: GatewayEndpoint, + *, + limit: int, +) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]: + if os.name != "nt": + raise OSError("Windows named-pipe transport is unavailable") + loop = asyncio.get_running_loop() + create_pipe_connection = getattr(loop, "create_pipe_connection", None) + if create_pipe_connection is None: + raise OSError("the active event loop does not support Windows named pipes") + reader = asyncio.StreamReader(limit=limit) + protocol = asyncio.StreamReaderProtocol(reader) + transport, _ = await create_pipe_connection( + lambda: protocol, + endpoint.native_named_pipe_path, + ) + writer = asyncio.StreamWriter(transport, protocol, reader, loop) + return reader, writer + + +def _validated_named_pipe_uri(uri: str) -> str: + if not isinstance(uri, str) or not uri.startswith(_NPIPE_URI_PREFIX): + raise ValueError("gateway named-pipe URI is invalid") + name = uri.removeprefix(_NPIPE_URI_PREFIX) + segments = name.split("/") + if ( + not name + or len(name.encode()) > 240 + or any( + segment in {"", ".", ".."} + or _SAFE_NPIPE_SEGMENT.fullmatch(segment) is None + for segment in segments + ) + ): + raise ValueError("gateway named-pipe URI is invalid") + return uri + + def _validated_id(field: str, value: str) -> str: if ( not isinstance(value, str) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_artifacts.py b/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_artifacts.py index b6f7a7f..0fa4d41 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_artifacts.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_artifacts.py @@ -1,8 +1,10 @@ +import hashlib import json from pathlib import Path import pytest +import benchmark_lab.artifacts as artifacts_module from benchmark_lab.artifacts import ( ArtifactError, ArtifactId, @@ -14,6 +16,9 @@ GOLDEN = Path(__file__).parents[3] / "tests/fixtures/golden" +FROZEN_SQUASH_EVIDENCE = ( + GOLDEN / "artifacts/operation-evidence-v1-squash.json" +) def roots(tmp_path: Path) -> BenchmarkRoots: @@ -156,12 +161,50 @@ def test_store_writes_atomically_and_quarantines_partial_tail(tmp_path: Path) -> assert quarantine.read_bytes() == b'{"partial":' +def test_journal_batch_append_uses_one_trial_boundary_fsync( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + store = ArtifactStore(roots(tmp_path)) + store.create_run("run-batch") + fsync_calls = 0 + real_fsync = artifacts_module.os.fsync + + def counted_fsync(descriptor: int) -> None: + nonlocal fsync_calls + fsync_calls += 1 + real_fsync(descriptor) + + monkeypatch.setattr(artifacts_module.os, "fsync", counted_fsync) + events = [ + { + "sequence": sequence, + "run_id": "run-batch", + "monotonic_offset_ns": sequence, + "data": {"kind": "log", "level": "info", "message": str(sequence)}, + } + for sequence in (1, 2, 3) + ] + + store.append_records("run-batch", ArtifactId.EVENTS, events) + + assert fsync_calls == 1 + assert store.read_records("run-batch", ArtifactId.EVENTS).records == events + + def test_bounded_evidence_is_content_addressed_and_paths_are_closed(tmp_path: Path) -> None: store = ArtifactStore(roots(tmp_path)) store.create_run("run-1") first = store.write_trial_evidence("run-1", "cell:1", "trial-1", {"ok": True}) second = store.write_trial_evidence("run-1", "cell:1", "trial-1", {"ok": True}) assert first == second + assert ":" not in first.label + assert first.label.startswith("bounded-evidence/operation-evidence-") + content = store.download_artifact("run-1", first.artifact_id).content + assert json.loads(content)["data"] == { + "cell_id": "cell:1", + "trial_id": "trial-1", + "ok": True, + } with pytest.raises(ArtifactError, match="path component"): store.write_trial_evidence("run-1", "../escape", "trial-1", {}) with pytest.raises(ArtifactError, match="runtime secret"): @@ -189,7 +232,9 @@ def test_artifact_index_and_download_use_only_opaque_ids(tmp_path: Path) -> None with pytest.raises(ArtifactError, match="unknown artifact id"): store.download_artifact("run-1", unsafe) - evidence_path = next((run / "cells").rglob("operation-evidence-*.json")) + evidence_path = next( + (run / "bounded-evidence").glob("operation-evidence-*.json") + ) evidence_path.rename(evidence_path.with_name("operation-evidence-" + "0" * 64 + ".json")) with pytest.raises(ArtifactError, match="digest mismatch"): store.list_artifacts("run-1") @@ -197,15 +242,36 @@ def test_artifact_index_and_download_use_only_opaque_ids(tmp_path: Path) -> None def test_frozen_bounded_evidence_is_indexable_without_product_access(tmp_path: Path) -> None: benchmark_roots = roots(tmp_path) - destination = benchmark_roots.results / "run-1" - source = GOLDEN / "rust" / "quick-smoke-completed" - import shutil + destination = benchmark_roots.results / "r" + payload = FROZEN_SQUASH_EVIDENCE.read_bytes() + digest = hashlib.sha256(payload).hexdigest() + relative = Path( + "cells/c/trials/t/bounded-evidence" + ) / f"operation-evidence-{digest}.json" + evidence_path = destination / relative + evidence_path.parent.mkdir(parents=True) + evidence_path.write_bytes(payload) + + frozen = read_envelope_path(FROZEN_SQUASH_EVIDENCE, ArtifactId.BOUNDED_EVIDENCE) + assert frozen["operation"] == "squash_layerstack" + assert frozen["evidence"]["manifest_reduced"] is True - shutil.copytree(source, destination) - index = ArtifactStore(benchmark_roots).list_artifacts("run-1") + store = ArtifactStore(benchmark_roots) + index = store.list_artifacts("r") evidence = [item for item in index if item.artifact_id.startswith("bounded_evidence_")] - assert len(evidence) == 48 - assert all(item.size_bytes <= 1024 * 1024 for item in evidence) + assert len(evidence) == 1 + reference = evidence[0] + expected_label = relative.as_posix() + expected_id = hashlib.sha256(expected_label.encode()).hexdigest() + assert reference.artifact_id == f"bounded_evidence_{expected_id}" + assert reference.label == expected_label + assert reference.sha256 == f"sha256:{digest}" + assert reference.size_bytes == len(payload) + assert reference.size_bytes <= 1024 * 1024 + + download = store.download_artifact("r", reference.artifact_id) + assert download.reference == reference + assert download.content == payload def test_safe_incomplete_removal_refuses_unknown_entries(tmp_path: Path) -> None: diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_reports.py b/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_reports.py index 87e26ee..30d5e93 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_reports.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/compatibility/test_reports.py @@ -56,7 +56,12 @@ def test_report_bundle_persists_compatible_artifacts_atomically(tmp_path: Path) regenerated = RunCorpus.open(store.run_path(corpus.report.run_id)) assert regenerated.report == corpus.report assert regenerated.summary == corpus.summary - assert (regenerated.path / "export.csv").read_bytes() == (source / "export.csv").read_bytes() + assert (regenerated.path / "export.csv").read_bytes() == render_csv_export( + corpus.report + ).encode() + assert (regenerated.path / "export.csv").read_text() == ( + source / "export.csv" + ).read_text() assert json.loads((regenerated.path / "export.json").read_bytes()) == json.loads( (source / "export.json").read_bytes() ) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/conftest.py b/ephemeral-sandbox-v1/benchmark/backend/tests/conftest.py new file mode 100644 index 0000000..97d651c --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/conftest.py @@ -0,0 +1,23 @@ +import os +from collections.abc import Callable +from pathlib import Path + +import pytest + + +@pytest.fixture +def symlink_or_skip() -> Callable[..., None]: + def create( + link: Path, + target: Path, + *, + target_is_directory: bool = False, + ) -> None: + try: + link.symlink_to(target, target_is_directory=target_is_directory) + except OSError as error: + if os.name == "nt" and getattr(error, "winerror", None) == 1314: + pytest.skip("Windows symlink privilege is unavailable") + raise + + return create diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_api.py b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_api.py index bc54ed9..47dbcbd 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_api.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_api.py @@ -7,9 +7,11 @@ import pytest from benchmark_lab.api import create_app -from benchmark_lab.artifacts import ArtifactId, ArtifactStore +from benchmark_lab.artifacts import ArtifactStore +from benchmark_lab.models import OwnedPathMarker from benchmark_lab.paths import BenchmarkRoots from benchmark_lab.runner import CampaignRunner +from benchmark_lab.safety import OwnershipLedger from benchmark_lab.service import CampaignService, _ActiveRun @@ -65,7 +67,7 @@ async def test_api_security_and_definition_contract(tmp_path: Path) -> None: definitions = (await client.get("/api/v1/definitions")).json() assert definitions["schema_version"] == 1 - assert len(definitions["catalog"]["operations"]) == 7 + assert len(definitions["catalog"]["operations"]) == 8 assert {item["configuration_base"]["scope"] for item in definitions["defaults"]} == { "all", "command", "files", "workspace", "layerstack" } @@ -193,3 +195,51 @@ async def test_event_sink_never_blocks_campaign_on_slow_browser(tmp_path: Path) await sink({"sequence": 1, "data": {"kind": "log"}}) await sink({"sequence": 2, "data": {"kind": "log"}}) assert service._subscribers["run"] == set() + + +@pytest.mark.asyncio +async def test_recover_and_cleanup_remove_owned_terminal_workspace_only( + tmp_path: Path, +) -> None: + service = CampaignService(_roots(tmp_path)) + _copy_run(service.store) + result_path = service.store.run_path(HISTORICAL_RUN_ID) + manifest_before = (result_path / "run-manifest.json").read_bytes() + report_before = (result_path / "report.json").read_bytes() + unrelated = service.roots.runs / "unrelated-run" + unrelated.mkdir() + OwnershipLedger(service.roots).register( + unrelated, + OwnedPathMarker(role="runs", identity={"run_id": "unrelated-run"}), + ) + + def create_terminal_residue() -> Path: + target = service.roots.runs / HISTORICAL_RUN_ID + target.mkdir() + OwnershipLedger(service.roots).register( + target, + OwnedPathMarker(role="runs", identity={"run_id": HISTORICAL_RUN_ID}), + ) + return target + + terminal_residue = create_terminal_residue() + recovered = await service.recover() + + assert recovered["execution_available"] + assert recovered["recovered_run_ids"] == [HISTORICAL_RUN_ID] + assert not terminal_residue.exists() + assert unrelated.is_dir() + + terminal_residue = create_terminal_residue() + cleaned = await service.cleanup(HISTORICAL_RUN_ID) + + assert cleaned == { + "schema_version": 1, + "run_id": HISTORICAL_RUN_ID, + "cleaned": True, + "terminalized": False, + } + assert not terminal_residue.exists() + assert unrelated.is_dir() + assert (result_path / "run-manifest.json").read_bytes() == manifest_before + assert (result_path / "report.json").read_bytes() == report_before diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_catalog.py b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_catalog.py index bb6e978..8d0ee87 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_catalog.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_catalog.py @@ -1,8 +1,12 @@ +import hashlib import json +import os +import subprocess from pathlib import Path import pytest +import benchmark_lab.catalog as catalog_module from benchmark_lab.catalog import CatalogError, export_catalog, read_catalog from benchmark_lab.paths import BenchmarkRoots @@ -19,9 +23,10 @@ def roots(tmp_path: Path) -> BenchmarkRoots: return BenchmarkRoots.resolve(test, product, binaries, initialize=True) -def fake_exporter(benchmark_roots: BenchmarkRoots, script: str) -> Path: - executable = benchmark_roots.product_bin_dir / "sandbox-catalog-export" - executable.write_text("#!/bin/sh\n" + script) +def fake_exporter(benchmark_roots: BenchmarkRoots) -> Path: + suffix = ".exe" if os.name == "nt" else "" + executable = benchmark_roots.product_bin_dir / f"sandbox-catalog-export{suffix}" + executable.write_bytes(b"fixed prebuilt exporter fixture\n") executable.chmod(0o700) return executable @@ -31,28 +36,86 @@ def test_reads_strict_frozen_catalog_and_required_operations() -> None: names = exported.operation_names() assert len(names) == 20 assert {"create_sandbox", "exec_command", "file_read", "file_write", "squash_layerstacks"} <= names + value = json.loads(GOLDEN.read_bytes()) + argument = next( + argument + for operation in value["domains"]["runtime"]["operations"] + for argument in operation["args"] + ) + argument["kind"] = "float" + read_catalog(json.dumps(value).encode()) with pytest.raises(CatalogError, match="schema"): read_catalog(GOLDEN.read_bytes().replace(b'"schema_version": 1', b'"schema_version": 2', 1)) -def test_invokes_only_canonical_prebuilt_exporter_and_hashes_exact_bytes(tmp_path: Path) -> None: +def test_invokes_only_canonical_prebuilt_exporter_and_hashes_exact_bytes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: benchmark_roots = roots(tmp_path) - fake_exporter(benchmark_roots, f'exec /bin/cat "{GOLDEN}"\n') + executable = fake_exporter(benchmark_roots) + invocations: list[tuple[list[str], dict[str, object]]] = [] + + def run(args: list[str], **kwargs: object) -> subprocess.CompletedProcess[bytes]: + invocations.append((args, kwargs)) + return subprocess.CompletedProcess(args, 0, GOLDEN.read_bytes(), b"") + + monkeypatch.setattr(catalog_module.subprocess, "run", run) exported = export_catalog(benchmark_roots) + assert invocations == [ + ( + [os.fspath(executable)], + { + "cwd": benchmark_roots.product_root, + "env": {"PATH": "/usr/bin:/bin", "LANG": "C", "LC_ALL": "C"}, + "stdin": subprocess.DEVNULL, + "stdout": subprocess.PIPE, + "stderr": subprocess.PIPE, + "timeout": 10.0, + "check": False, + }, + ) + ] assert exported.content == GOLDEN.read_bytes() - assert exported.sha256.startswith("sha256:") + assert exported.sha256 == f"sha256:{hashlib.sha256(GOLDEN.read_bytes()).hexdigest()}" + assert exported.executable_sha256 == ( + f"sha256:{hashlib.sha256(executable.read_bytes()).hexdigest()}" + ) exported.require_operations({"exec_command", "file_read"}) with pytest.raises(CatalogError, match="missing required"): exported.require_operations({"not_a_product_operation"}) -def test_exporter_failure_and_unsafe_binary_fail_closed(tmp_path: Path) -> None: +def test_exporter_failure_is_bounded_and_fails_closed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: benchmark_roots = roots(tmp_path) - executable = fake_exporter(benchmark_roots, 'printf "bad exporter" >&2\nexit 7\n') + fake_exporter(benchmark_roots) + monkeypatch.setattr( + catalog_module.subprocess, + "run", + lambda args, **kwargs: subprocess.CompletedProcess( + args, 7, b"", b"bad exporter" + ), + ) with pytest.raises(CatalogError, match="bad exporter"): export_catalog(benchmark_roots) + + +def test_unsafe_exporter_symlink_fails_before_process_execution( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, symlink_or_skip +) -> None: + benchmark_roots = roots(tmp_path) + executable = fake_exporter(benchmark_roots) executable.unlink() - executable.symlink_to("/bin/true") + target = benchmark_roots.product_root / "outside-exporter" + target.write_bytes(b"outside") + target.chmod(0o700) + symlink_or_skip(executable, target) + monkeypatch.setattr( + catalog_module.subprocess, + "run", + lambda *args, **kwargs: pytest.fail("unsafe exporter was executed"), + ) with pytest.raises(CatalogError, match="unsafe"): export_catalog(benchmark_roots) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_planning.py b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_planning.py index b001493..65f0ba1 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_planning.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_planning.py @@ -60,6 +60,72 @@ def test_expansion_is_deterministic_and_counts_requests() -> None: assert first["estimates"]["issued_operation_request_count"] == 96 +def test_seeded_family_block_order_is_persisted_and_reproducible() -> None: + preset = load_preset(ROOT / "presets/paper-good-pass.yml") + profiles = load_workspace_profiles(ROOT / "defaults/workspace-profiles") + environment = RuntimeEnvironment( + "C:\\benchmark-fixture\\test-repository", None, None, None + ) + first = expand_plan( + preset.plan, + environment=environment, + profiles=profiles, + catalog_operations=_catalog_operations(), + ) + repeated = expand_plan( + preset.plan, + environment=environment, + profiles=profiles, + catalog_operations=_catalog_operations(), + ) + + assert first["execution_blocks"] == repeated["execution_blocks"] + assert [ + cell_id + for block in first["execution_blocks"] + for cell_id in block["cell_ids"] + ] == [cell["cell_id"] for cell in first["cells"]] + cells = {cell["cell_id"]: cell for cell in first["cells"]} + assert all( + all(cells[cell_id]["family_id"] == block["family_id"] for cell_id in block["cell_ids"]) + for block in first["execution_blocks"] + ) + + +@pytest.mark.parametrize( + ("preset_name", "expected_batches", "expected_requests"), + [ + ("paper-env-smoke.yml", 19, 55), + ("paper-pilot.yml", 133, 385), + ("paper-good-pass.yml", 1938, 5610), + ], +) +def test_exp1_presets_preserve_frozen_matrix_and_exact_counts( + preset_name: str, expected_batches: int, expected_requests: int +) -> None: + preset = load_preset(ROOT / "presets" / preset_name) + plan = expand_plan( + preset.plan, + environment=RuntimeEnvironment( + "C:\\benchmark-fixture\\test-repository", None, None, None + ), + profiles=load_workspace_profiles(ROOT / "defaults/workspace-profiles"), + catalog_operations=_catalog_operations(), + ) + + assert plan["runnable"] is True + assert plan["effective_environment"]["client_cohort"] == "product_cli" + assert plan["estimates"]["cell_count"] == 19 + assert plan["estimates"]["trial_batch_count"] == expected_batches + assert ( + plan["estimates"]["issued_operation_request_count"] == expected_requests + ) + assert { + cell["operation"]["cell"].get("workspace_profile") + for cell in plan["cells"] + } == {"paper-100m"} + + def test_unknown_profile_prevents_run() -> None: preset = load_preset(ROOT / "presets/quick-smoke.yml") preset.plan.operations[0].configuration.factors["workspace_profile"].values = ["missing"] diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_zero_rust_guard.py b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_zero_rust_guard.py index 335c0fc..b797962 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_zero_rust_guard.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/contract/test_zero_rust_guard.py @@ -9,7 +9,13 @@ TOOL_PATTERN = re.compile(r"\b(?:ca" r"rgo|ru" r"stc|ru" r"stup|cl" r"ippy|mi" r"ri)\b", re.IGNORECASE) PRODUCT_IMPORTS = {"ephemeral_sandbox", "sandbox_gateway", "sandbox_manager", "sandbox_runtime"} GENERATED_DIRECTORIES = { - ".pytest_cache", "__pycache__", "dist", "node_modules", "playwright-report", "test-results" + ".pytest_cache", + ".venv", + "__pycache__", + "dist", + "node_modules", + "playwright-report", + "test-results", } diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_lifecycle.py b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_lifecycle.py index c2607f4..e36e5a7 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_lifecycle.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_lifecycle.py @@ -1,21 +1,27 @@ import asyncio +import json import os import signal from pathlib import Path from typing import Any +import benchmark_lab.gateway as gateway_module +import benchmark_lab.product_cli as product_cli_module import pytest import yaml - from benchmark_lab.gateway import ( GatewayLauncher, GatewayLifecycleError, + _cleanup_docker_resources, + _cli_readiness_request_id, + _wait_until_ready, + cleanup_gateway_docker_resources, recover_stale_gateway, ) from benchmark_lab.models import OwnedPathMarker from benchmark_lab.paths import BenchmarkRoots from benchmark_lab.safety import OwnershipLedger -from benchmark_lab.transport import TimedGatewayResponse +from benchmark_lab.transport import GatewayProductError, TimedGatewayResponse class FakeProcess: @@ -53,10 +59,119 @@ async def request( return TimedGatewayResponse("request-1", 1, 3, "sha256:abc", value) -async def no_orphans(_: str) -> None: +async def no_orphans(_: str, __: bool) -> None: pass +def test_cli_readiness_request_id_is_unique_per_gateway_instance() -> None: + first = _cli_readiness_request_id("run-1", "benchmark-gateway-first", 0) + second = _cli_readiness_request_id("run-1", "benchmark-gateway-second", 0) + + assert first != second + assert first == "run-1.benchmark-gateway-first.ready.0" + + +@pytest.mark.skipif(os.name != "nt", reason="Windows named-pipe policy") +def test_product_cli_execution_blocks_get_unique_named_pipe_endpoints() -> None: + first = gateway_module._execution_block_endpoint(True) + second = gateway_module._execution_block_endpoint(True) + explicit_tcp = gateway_module._execution_block_endpoint(False) + + assert first.transport == second.transport == "windows_named_pipe" + assert first.uri.startswith("npipe://./pipe/ephemeral-sandbox-benchmark-") + assert second.uri.startswith("npipe://./pipe/ephemeral-sandbox-benchmark-") + assert first.uri != second.uri + assert explicit_tcp.transport == "tcp_loopback" + + +async def test_cli_readiness_product_rejection_fails_without_retry( + tmp_path: Path, +) -> None: + process = FakeProcess() + pid_path = tmp_path / "gateway.pid" + pid_path.write_text(str(process.pid)) + attempts = 0 + + async def rejected() -> None: + nonlocal attempts + attempts += 1 + raise GatewayProductError("invalid_request", "safe rejection") + + with pytest.raises( + GatewayLifecycleError, match="readiness CLI request was rejected" + ): + await _wait_until_ready( + process, + pid_path, + FakeClient(), + readiness_timeout_seconds=1, + readiness_probe=rejected, + ) + + assert attempts == 1 + + +@pytest.mark.skipif(os.name != "nt", reason="Windows named-pipe policy") +async def test_product_cli_gateway_uses_exact_named_pipe_for_config_and_readiness( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + benchmark_roots = roots(tmp_path) + process = FakeProcess() + fake_client = FakeClient() + config_bind_addresses: list[str] = [] + client_endpoints = [] + cli_endpoints = [] + + async def process_factory(*args: str, **_: Any) -> FakeProcess: + config = yaml.safe_load(Path(args[-1]).read_bytes()) + config_bind_addresses.append(config["gateway"]["bind_addr"]) + Path(config["gateway"]["pid_path"]).write_text(str(process.pid)) + return process + + def client_factory(endpoint, _token: str) -> FakeClient: + client_endpoints.append(endpoint) + return fake_client + + class FakeCli: + def __init__(self, endpoint, _token, _roots, _evidence_root) -> None: + cli_endpoints.append(endpoint) + + async def assert_no_sandboxes(self, *, request_id: str) -> None: + assert request_id.endswith(".ready.0") + + def kill_group(_: int, __: signal.Signals) -> None: + process.stop() + + monkeypatch.setattr(product_cli_module, "ProductCliAccess", FakeCli) + gateway = await GatewayLauncher( + benchmark_roots, + process_factory=process_factory, + client_factory=client_factory, + kill_group=kill_group, + process_identity=lambda _: "process-identity-1", + orphan_cleanup=no_orphans, + ).start( + "run-1", + readiness_timeout_seconds=1, + readiness_via_cli=True, + ) + runtime = benchmark_roots.runtime / "run-1" + owner = json.loads((runtime / "owner-process.json").read_text()) + endpoint = gateway.identity.endpoint + + assert endpoint.transport == "windows_named_pipe" + assert config_bind_addresses == [endpoint.uri] + assert client_endpoints == [endpoint] + assert cli_endpoints == [endpoint] + assert owner["schema_version"] == 2 + assert owner["endpoint_uri"] == endpoint.uri + assert "endpoint_host" not in owner + assert "endpoint_port" not in owner + + await gateway.close(destroy_sandboxes_via_gateway=False) + + def roots(tmp_path: Path) -> BenchmarkRoots: test = tmp_path / "test" product = tmp_path / "product" @@ -65,17 +180,18 @@ def roots(tmp_path: Path) -> BenchmarkRoots: defaults = Path(__file__).parents[3] / "defaults/gateway.yml" (source / "defaults").mkdir() (source / "defaults/gateway.yml").write_bytes(defaults.read_bytes()) + if os.name == "nt": + (product / "config").mkdir(parents=True) + (product / "config/windows-amd64.yml").write_bytes(defaults.read_bytes()) binaries = product / "bin" binaries.mkdir(parents=True) + suffix = ".exe" if os.name == "nt" else "" for name in ("sandbox-gateway", "sandbox-daemon"): - path = binaries / name + path = binaries / f"{name}{suffix}" path.write_text("prebuilt executable") path.chmod(0o700) - tools = product / "dist/git" - tools.mkdir(parents=True) - for name in ("linux-arm64.tar", "linux-amd64.tar"): - (tools / name).write_bytes(b"archive") daemon = product / "dist" / "sandbox-daemon-linux-arm64" + daemon.parent.mkdir(parents=True) daemon.write_bytes(b"\x7fELFfake container executable") daemon.chmod(0o700) amd64_daemon = product / "dist" / "sandbox-daemon-linux-amd64" @@ -90,7 +206,9 @@ def roots(tmp_path: Path) -> BenchmarkRoots: return benchmark_roots -async def test_launches_prebuilt_gateway_with_private_state_and_cleans_up(tmp_path: Path) -> None: +async def test_launches_prebuilt_gateway_with_private_state_and_cleans_up( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() fake_client = FakeClient() @@ -118,20 +236,76 @@ def kill_group(pid: int, sent_signal: signal.Signals) -> None: ).start("run-1", readiness_timeout_seconds=1) runtime = benchmark_roots.runtime / "run-1" assert launch["args"][1:5] == ("serve", "--backend", "docker", "--config-yaml") - assert launch["start_new_session"] is True - assert launch["cwd"] == benchmark_roots.product_root - token = (runtime / "gateway.token").read_text().strip() - assert token and oct((runtime / "gateway.token").stat().st_mode & 0o777) == "0o600" + if os.name == "nt": + assert launch["creationflags"] != 0 + assert "start_new_session" not in launch + assert launch["cwd"] == benchmark_roots.product_bin_dir.parent + else: + assert launch["start_new_session"] is True + assert "creationflags" not in launch + assert launch["cwd"] == benchmark_roots.product_root + token_path = runtime / "gateway.token" + token = token_path.read_text().strip() + assert token and token_path.is_file() and not token_path.is_symlink() + if os.name != "nt": + assert oct(token_path.stat().st_mode & 0o777) == "0o600" assert token not in repr(gateway.client) assert "must-redact" not in (runtime / "gateway.log.jsonl").read_text() + shared_cache = benchmark_roots.runs / "run-1" / "shared-base-cache" + assert Path(launch["env"]["EOS_SHARED_BASE_CACHE"]) == shared_cache.resolve() assert fake_client.calls == [("list_sandboxes", {})] await gateway.close() assert fake_client.calls == [("list_sandboxes", {}), ("list_sandboxes", {})] assert not runtime.exists() + assert shared_cache.is_dir() assert all("must-redact" not in record.text for record in gateway.logs) -async def test_shutdown_sweeps_every_sandbox_before_process_exit(tmp_path: Path) -> None: +async def test_successive_gateways_reuse_the_run_scoped_shared_base_cache( + tmp_path: Path, +) -> None: + benchmark_roots = roots(tmp_path) + processes = [FakeProcess(43210), FakeProcess(43211)] + launch_environments: list[dict[str, str]] = [] + launch_index = 0 + + async def process_factory(*args: str, **kwargs: Any) -> FakeProcess: + nonlocal launch_index + process = processes[launch_index] + launch_index += 1 + launch_environments.append(kwargs["env"]) + config = yaml.safe_load(Path(args[-1]).read_bytes()) + Path(config["gateway"]["pid_path"]).write_text(str(process.pid)) + return process + + def kill_group(pid: int, _: signal.Signals) -> None: + next(process for process in processes if process.pid == pid).stop() + + launcher = GatewayLauncher( + benchmark_roots, + process_factory=process_factory, + client_factory=lambda _endpoint, _token: FakeClient(), + kill_group=kill_group, + process_identity=lambda pid: f"process-identity-{pid}", + orphan_cleanup=no_orphans, + ) + first = await launcher.start("run-1", readiness_timeout_seconds=1) + await first.close(retain_shared_base_volumes=True) + second = await launcher.start("run-1", readiness_timeout_seconds=1) + await second.close(retain_shared_base_volumes=True) + + expected = (benchmark_roots.runs / "run-1" / "shared-base-cache").resolve() + assert [Path(item["EOS_SHARED_BASE_CACHE"]) for item in launch_environments] == [ + expected, + expected, + ] + assert first.identity.gateway_instance_id != second.identity.gateway_instance_id + assert expected.is_dir() + + +async def test_shutdown_sweeps_every_sandbox_before_process_exit( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() fake_client = FakeClient( @@ -166,7 +340,9 @@ def kill_group(_: int, __: signal.Signals) -> None: ] -async def test_readiness_failure_terminates_and_removes_private_runtime(tmp_path: Path) -> None: +async def test_readiness_failure_terminates_and_removes_private_runtime( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() @@ -219,7 +395,118 @@ def kill_group(_: int, __: signal.Signals) -> None: assert not (runtime / "gateway.token").exists() -async def test_stale_recovery_proves_process_identity_before_signalling(tmp_path: Path) -> None: +async def test_cross_gateway_final_cleanup_removes_first_owner_shared_base_volume( + monkeypatch: pytest.MonkeyPatch, +) -> None: + first_gateway_id = "benchmark-gateway-first" + second_gateway_id = "benchmark-gateway-second" + root_hash = "a" * 64 + shared = f"eos-shared-base-{root_hash}" + ordinary = "eos-sandbox-layer-stack" + calls: list[tuple[str, ...]] = [] + finalizing = False + + async def docker_output(*arguments: str) -> bytes: + calls.append(arguments) + owner = arguments[-1].removeprefix("label=eos.gateway_instance_id=") + if arguments[:2] == ("ps", "-aq"): + return ( + b"0123456789ab\n" + if owner == first_gateway_id and not finalizing + else b"" + ) + if arguments[:3] == ("volume", "ls", "--quiet"): + if owner != first_gateway_id: + return b"" + if finalizing: + return f"{shared}\n".encode() + return f"{shared}\n{ordinary}\n".encode() + if arguments[:2] == ("volume", "inspect"): + name = arguments[2] + labels = { + "eos.gateway_instance_id": first_gateway_id, + } + if name == shared: + labels.update( + { + "eos.shared_base.root_hash": root_hash, + "eos.shared_base.target": "/eos/layer-stack/base", + "eos.shared_base.readonly": "true", + } + ) + else: + labels.update( + { + "eos.sandbox_id": "sandbox-1", + "eos.cleanup_policy": "remove-on-destroy", + } + ) + return json.dumps([{"Name": name, "Labels": labels}]).encode() + if arguments[:2] in {("rm", "--force"), ("volume", "rm")}: + return b"" + raise AssertionError(arguments) + + monkeypatch.setattr(gateway_module, "_docker_output", docker_output) + + await _cleanup_docker_resources(first_gateway_id, retain_shared_base_volumes=True) + assert ("rm", "--force", "0123456789ab") in calls + assert ("volume", "rm", ordinary) in calls + assert ("volume", "rm", shared) not in calls + + finalizing = True + await cleanup_gateway_docker_resources((second_gateway_id, first_gateway_id)) + assert ( + "volume", + "ls", + "--quiet", + "--filter", + f"label=eos.gateway_instance_id={second_gateway_id}", + ) in calls + assert ("volume", "rm", shared) in calls + + +async def test_cleanup_preserves_ambiguous_shared_base_volume( + monkeypatch: pytest.MonkeyPatch, +) -> None: + gateway_instance_id = "benchmark-gateway-owned" + root_hash = "b" * 64 + shared = f"eos-shared-base-{root_hash}" + calls: list[tuple[str, ...]] = [] + + async def docker_output(*arguments: str) -> bytes: + calls.append(arguments) + if arguments[:2] == ("ps", "-aq"): + return b"" + if arguments[:3] == ("volume", "ls", "--quiet"): + return f"{shared}\n".encode() + if arguments[:2] == ("volume", "inspect"): + return json.dumps( + [ + { + "Name": shared, + "Labels": { + "eos.gateway_instance_id": gateway_instance_id, + "eos.shared_base.root_hash": "c" * 64, + "eos.shared_base.target": "/eos/layer-stack/base", + "eos.shared_base.readonly": "true", + }, + } + ] + ).encode() + raise AssertionError(arguments) + + monkeypatch.setattr(gateway_module, "_docker_output", docker_output) + + with pytest.raises(GatewayLifecycleError, match="cleanup was incomplete"): + await _cleanup_docker_resources( + gateway_instance_id, retain_shared_base_volumes=False + ) + assert ("volume", "rm", shared) not in calls + + +async def test_stale_recovery_proves_process_identity_before_signalling( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() fake_client = FakeClient([{"sandboxes": []}, {"sandboxes": []}]) @@ -235,8 +522,8 @@ def kill_group(_: int, __: signal.Signals) -> None: current_identity[0] = None process.stop() - async def cleanup(identity: str) -> None: - cleaned.append(identity) + async def cleanup(identity: str, retain_shared_base_volumes: bool) -> None: + cleaned.append(f"{identity}:{retain_shared_base_volumes}") abandoned = await GatewayLauncher( benchmark_roots, @@ -249,6 +536,18 @@ async def cleanup(identity: str) -> None: await asyncio.sleep(0.01) os.close(abandoned._log_descriptor) abandoned._log_descriptor = None + + original_lstat = Path.lstat + + def private_token_lstat(path: Path) -> os.stat_result: + metadata = original_lstat(path) + if path.name != "gateway.token": + return metadata + values = list(metadata) + values[0] &= ~0o077 + return os.stat_result(values) + + monkeypatch.setattr(Path, "lstat", private_token_lstat) await recover_stale_gateway( benchmark_roots, "run-1", @@ -257,11 +556,13 @@ async def cleanup(identity: str) -> None: process_identity=lambda _: current_identity[0], orphan_cleanup=cleanup, ) - assert cleaned + assert cleaned and cleaned[-1].endswith(":False") assert not (benchmark_roots.runtime / "run-1").exists() -async def test_stale_recovery_refuses_reused_pid_and_preserves_evidence(tmp_path: Path) -> None: +async def test_stale_recovery_refuses_reused_pid_and_preserves_evidence( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() @@ -289,7 +590,9 @@ async def process_factory(*args: str, **_: Any) -> FakeProcess: await gateway.close() -async def test_stale_recovery_aggregates_orphan_cleanup_and_deletes_token(tmp_path: Path) -> None: +async def test_stale_recovery_aggregates_orphan_cleanup_and_deletes_token( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) process = FakeProcess() @@ -310,8 +613,10 @@ async def process_factory(*args: str, **_: Any) -> FakeProcess: process.stop() os.close(abandoned._log_descriptor) abandoned._log_descriptor = None + retention_modes: list[bool] = [] - async def fails(_: str) -> None: + async def fails(_: str, retain_shared_base_volumes: bool) -> None: + retention_modes.append(retain_shared_base_volumes) raise RuntimeError("injected cleanup failure") with pytest.raises(GatewayLifecycleError, match="Docker resource cleanup"): @@ -322,19 +627,26 @@ async def fails(_: str) -> None: orphan_cleanup=fails, ) runtime = benchmark_roots.runtime / "run-1" + assert retention_modes == [False] assert runtime.exists() assert not (runtime / "gateway.token").exists() -def test_unowned_run_and_binary_symlink_fail_before_launch(tmp_path: Path) -> None: +def test_unowned_run_fails_before_launch(tmp_path: Path) -> None: benchmark_roots = roots(tmp_path) (benchmark_roots.runs / "run-1" / ".ownership.json").write_text("{}") with pytest.raises(GatewayLifecycleError, match="ownership"): asyncio.run(GatewayLauncher(benchmark_roots).start("run-1")) - benchmark_roots = roots(tmp_path / "second") - gateway_binary = benchmark_roots.product_bin_dir / "sandbox-gateway" + +def test_binary_symlink_fails_before_launch(tmp_path: Path, symlink_or_skip) -> None: + benchmark_roots = roots(tmp_path) + suffix = ".exe" if os.name == "nt" else "" + gateway_binary = benchmark_roots.product_bin_dir / f"sandbox-gateway{suffix}" gateway_binary.unlink() - gateway_binary.symlink_to("/bin/true") + target = benchmark_roots.product_root / "outside-gateway" + target.write_bytes(b"outside") + target.chmod(0o700) + symlink_or_skip(gateway_binary, target) with pytest.raises(GatewayLifecycleError, match="preflight"): asyncio.run(GatewayLauncher(benchmark_roots).start("run-1")) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_transport.py b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_transport.py index 49a8a73..5bff67b 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_transport.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_gateway_transport.py @@ -4,6 +4,7 @@ import pytest +import benchmark_lab.transport as transport_module from benchmark_lab.transport import ( AUTH_FIELD, GatewayClient, @@ -196,3 +197,69 @@ def test_endpoint_scope_and_request_validation_are_strict() -> None: asyncio.run(client.request("op", {"kind": "system", "extra": 1}, {}, timeout_seconds=1)) with pytest.raises(ValueError, match="operation"): asyncio.run(client.request("bad operation", {"kind": "system"}, {}, timeout_seconds=1)) + + +def test_named_pipe_endpoint_is_canonical_and_strict() -> None: + uri = "npipe://./pipe/ephemeral-sandbox-benchmark-0123abcd" + endpoint = GatewayEndpoint.windows_named_pipe(uri) + + assert endpoint.transport == "windows_named_pipe" + assert endpoint.address == uri + assert endpoint.uri == uri + assert endpoint.native_named_pipe_path == ( + r"\\.\pipe\ephemeral-sandbox-benchmark-0123abcd" + ) + assert GatewayEndpoint.parse(uri) == endpoint + assert GatewayEndpoint.parse("tcp://127.0.0.1:47621") == GatewayEndpoint( + "127.0.0.1", 47621 + ) + + for invalid in ( + "npipe://./pipe/", + "npipe://./pipe/../escape", + "npipe://./pipe/name with spaces", + "unix:///tmp/gateway.sock", + ): + with pytest.raises(ValueError): + GatewayEndpoint.parse(invalid) + + +@pytest.mark.asyncio +async def test_named_pipe_connection_failure_never_falls_back_to_tcp( + monkeypatch: pytest.MonkeyPatch, +) -> None: + named_pipe_attempts = 0 + tcp_attempts = 0 + + async def named_pipe(*_args, **_kwargs): + nonlocal named_pipe_attempts + named_pipe_attempts += 1 + raise OSError("named pipe unavailable") + + async def tcp(*_args, **_kwargs): + nonlocal tcp_attempts + tcp_attempts += 1 + raise AssertionError("TCP fallback is prohibited") + + monkeypatch.setattr( + transport_module, "_open_windows_named_pipe_connection", named_pipe + ) + monkeypatch.setattr(transport_module.asyncio, "open_connection", tcp) + client = GatewayClient( + GatewayEndpoint.windows_named_pipe( + "npipe://./pipe/ephemeral-sandbox-benchmark-no-fallback" + ), + TOKEN, + ) + + with pytest.raises(GatewayTransportError) as captured: + await client.request( + "list_sandboxes", + {"kind": "system"}, + {}, + timeout_seconds=1, + ) + + assert captured.value.kind == "connection_error" + assert named_pipe_attempts == 1 + assert tcp_attempts == 0 diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_recovery.py b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_recovery.py index 77de6f1..90e6f1b 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_recovery.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_recovery.py @@ -176,14 +176,108 @@ def test_interior_corruption_is_fatal_read_only_and_other_runs_are_scanned(tmp_p assert cleaned == ["run-2"] -def test_terminal_runs_are_never_recovered(tmp_path: Path) -> None: +def test_intentionally_retained_failed_run_is_not_automatically_recovered( + tmp_path: Path, +) -> None: benchmark_roots = roots(tmp_path) store = interrupted_run(benchmark_roots) manifest = store.read_envelope("run-1", ArtifactId.RUN_MANIFEST) manifest["state"] = "failed" + manifest["failure"] = { + "code": "campaign_failed", + "message": "retained for diagnosis", + "infrastructure": True, + } store.replace_snapshot("run-1", ArtifactId.RUN_MANIFEST, manifest) called: list[str] = [] result = RecoveryScanner(benchmark_roots, called.append).scan() assert result.execution_available assert result.recovered_run_ids == () assert called == [] + assert (benchmark_roots.runs / "run-1").exists() + assert (benchmark_roots.runtime / "run-1").exists() + + +@pytest.mark.parametrize( + ("state", "failure_code"), + [ + ("completed", None), + ("cancelled", None), + ("failed", "artifact_finalization_failed"), + ], +) +def test_terminal_run_with_cleanup_expected_removes_only_exact_owned_residue( + tmp_path: Path, state: str, failure_code: str | None +) -> None: + benchmark_roots = roots(tmp_path) + store = interrupted_run(benchmark_roots, state=state, owned_roles=("runs",)) + if failure_code is not None: + manifest = store.read_envelope("run-1", ArtifactId.RUN_MANIFEST) + manifest["failure"] = { + "code": failure_code, + "message": "terminal workspace removal was interrupted", + "infrastructure": True, + } + store.replace_snapshot("run-1", ArtifactId.RUN_MANIFEST, manifest) + unrelated = benchmark_roots.runs / "unrelated-run" + unrelated.mkdir() + OwnershipLedger(benchmark_roots).register( + unrelated, + OwnedPathMarker(role="runs", identity={"run_id": "unrelated-run"}), + ) + manifest_path = store.run_path("run-1") / "run-manifest.json" + manifest_before = manifest_path.read_bytes() + called: list[str] = [] + + result = RecoveryScanner(benchmark_roots, called.append).scan() + + assert result.execution_available + assert result.recovered_run_ids == ("run-1",) + assert called == ["run-1"] + assert not (benchmark_roots.runs / "run-1").exists() + assert unrelated.is_dir() + assert manifest_path.read_bytes() == manifest_before + + +def test_terminal_residue_marker_mismatch_blocks_recovery_without_mutation( + tmp_path: Path, +) -> None: + benchmark_roots = roots(tmp_path) + store = interrupted_run( + benchmark_roots, state="completed", owned_roles=("runs",) + ) + marker = benchmark_roots.runs / "run-1" / ".ownership.json" + marker.write_text(marker.read_text().replace('"run-1"', '"other-run"')) + manifest_path = store.run_path("run-1") / "run-manifest.json" + manifest_before = manifest_path.read_bytes() + called: list[str] = [] + + result = RecoveryScanner(benchmark_roots, called.append).scan() + + assert not result.execution_available + assert result.recovered_run_ids == () + assert len(result.issues) == 1 + assert called == [] + assert (benchmark_roots.runs / "run-1").is_dir() + assert manifest_path.read_bytes() == manifest_before + + +def test_explicit_cleanup_removes_owned_workspace_from_terminal_failed_run( + tmp_path: Path, +) -> None: + benchmark_roots = roots(tmp_path) + store = interrupted_run( + benchmark_roots, state="failed", owned_roles=("runs",) + ) + manifest_path = store.run_path("run-1") / "run-manifest.json" + manifest_before = manifest_path.read_bytes() + called: list[str] = [] + + cleaned = RecoveryScanner( + benchmark_roots, called.append + ).cleanup_terminal_run("run-1") + + assert cleaned + assert called == ["run-1"] + assert not (benchmark_roots.runs / "run-1").exists() + assert manifest_path.read_bytes() == manifest_before diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_runner.py b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_runner.py index 5db157f..cbfbb23 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_runner.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/integration/test_runner.py @@ -9,12 +9,20 @@ import benchmark_lab.runner as runner_module from benchmark_lab.artifacts import ArtifactId, ArtifactStore +from benchmark_lab.observability import parse_cgroup, parse_daemon, parse_snapshot from benchmark_lab.paths import BenchmarkRoots -from benchmark_lab.observability import parse_cgroup, parse_snapshot from benchmark_lab.reports import RunCorpus -from benchmark_lab.runner import CampaignError, CampaignRunner, TrialContext -from benchmark_lab.transport import GatewayTransportError, TimedGatewayResponse - +from benchmark_lab.runner import ( + BatchTiming, + CampaignError, + CampaignRunner, + TrialContext, +) +from benchmark_lab.transport import ( + GatewayEndpoint, + GatewayTransportError, + TimedGatewayResponse, +) ROOT = Path(__file__).resolve().parents[3] GOLDEN = ROOT / "tests/fixtures/golden/rust/quick-smoke-completed" @@ -33,6 +41,60 @@ def _artifact(name: str) -> dict: return json.loads((GOLDEN / name).read_text())["data"] +def test_cell_workspaces_are_materialized_in_one_distinct_bounded_batch( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + runner._profiles = {"paper": {"id": "paper"}} + runner._seed = 7 + run_path = roots.runs / "run-batch-workspaces" + run_path.mkdir() + cells = [ + { + "cell_id": f"sha256:{index:064x}", + "operation": { + "operation": "create_sandbox", + "cell": {"workspace_profile": "paper"}, + }, + } + for index in range(3) + ] + calls: list[tuple[list[Path], int]] = [] + + def materialize( + fixtures_root: Path, + workspaces: list[Path], + profile: dict, + seed: int, + *, + max_workers: int, + ) -> dict: + assert fixtures_root == roots.fixtures + assert profile == runner._profiles["paper"] + assert seed == runner._seed + calls.append((list(workspaces), max_workers)) + for workspace in workspaces: + (workspace / "fixture-manifest.json").write_text("{}") + return {} + + monkeypatch.setattr(runner_module, "materialize_workspaces", materialize) + + runner._prepare_cell_workspaces(run_path, cells) + + assert len(calls) == 1 + workspaces, max_workers = calls[0] + assert max_workers == 4 + assert len(workspaces) == len(cells) + assert len({workspace.resolve() for workspace in workspaces}) == len(cells) + assert runner._prepared_cell_workspaces == { + cell["cell_id"]: run_path / f"cell-{cell['cell_id'][-16:]}" for cell in cells + } + assert all( + not (workspace / "fixture-manifest.json").exists() for workspace in workspaces + ) + + def _single_file_read_plan() -> dict: plan = copy.deepcopy(_artifact("expanded-plan.json")) cell = next(item for item in plan["cells"] if item["operation_id"] == "file_read") @@ -52,10 +114,31 @@ def _single_file_read_plan() -> dict: return plan +def _two_block_file_read_plan() -> dict: + plan = _single_file_read_plan() + first = plan["cells"][0] + second = copy.deepcopy(first) + second["cell_id"] = f"sha256:{'f' * 64}" + second_block = copy.deepcopy(plan["execution_blocks"][0]) + second_block["block_id"] = f"sha256:{'e' * 64}" + second_block["cell_ids"] = [second["cell_id"]] + plan["cells"] = [first, second] + plan["execution_blocks"] = [plan["execution_blocks"][0], second_block] + plan["estimates"] = { + "cell_count": 2, + "trial_batch_count": 2, + "issued_operation_request_count": 2, + } + return plan + + class FakeProduct: def __init__(self, *, fail_destroy: bool = False) -> None: self.files: dict[str, str] = {} self.fail_destroy = fail_destroy + self.cgroup_timestamp = 0 + self.snapshot_timestamp = 0 + self.resource_timestamp_base_ms = time.time_ns() // 1_000_000 + 10_000 async def create_sandbox(self, image, workspace, *, request_id): return SimpleNamespace(id="sandbox-1"), self._response(request_id, {}) @@ -88,36 +171,73 @@ async def file_read( ) async def observe_cgroup(self, sandbox, *, request_id): - return parse_cgroup({ - "view": "cgroup", - "scope": "sandbox", - "series": [{ - "ts": 1, - "sample_delta_ms": None, - "metrics": { - "metrics_source": "docker_engine", - "cpu_usec": 1, - "mem_cur": 1024, - "mem_max": 2048, - "io_rbytes": 0, - "io_wbytes": 0, - }, - "deltas": {}, - }], - }) + self.cgroup_timestamp += 1 + return parse_cgroup( + { + "view": "cgroup", + "scope": "sandbox", + "availability": "available", + "errors": [], + "topology": {}, + "series": [ + { + "ts": self.resource_timestamp_base_ms + self.cgroup_timestamp, + "sample_delta_ms": None, + "metrics": { + "metrics_source": "docker_engine", + "cpu_usec": 1, + "mem_cur": 1024, + "mem_max": 2048, + "io_rbytes": 0, + "io_wbytes": 0, + }, + "deltas": {}, + } + ], + } + ) async def observe_snapshot(self, sandbox, *, request_id): - return parse_snapshot({ - "sandbox_id": sandbox, - "lifecycle_state": "ready", - "availability": "available", - "sampled_at_unix_ms": 1, - "errors": [], - "daemon": {"daemon_pid": 7, "runtime_dir": "/run/fake"}, - "resources": {"latest": None, "history": []}, - "workspaces": [], - "stack": None, - }, sandbox) + self.snapshot_timestamp += 1 + return parse_snapshot( + { + "sandbox_id": sandbox, + "lifecycle_state": "ready", + "availability": "available", + "sampled_at_unix_ms": ( + self.resource_timestamp_base_ms + self.snapshot_timestamp + ), + "errors": [], + "daemon": { + "daemon_pid": 7, + "runtime_dir": "/run/fake", + "event_store": { + "dropped_storage": 0, + "dropped_oversized": 0, + "truncated_records": 0, + }, + }, + "resources": {"latest": None, "history": []}, + "workspaces": [], + "stack": None, + }, + sandbox, + ) + + async def observe_daemon(self, sandbox, *, request_id): + return parse_daemon( + { + "view": "daemon", + "scope": "sandbox", + "daemon": { + "available": True, + "pid": 7, + "resident_memory_bytes": 1024, + "peak_resident_memory_bytes": 2048, + "cpu_time_us": 1, + }, + } + ) @staticmethod def _response(request_id: str, value: dict) -> TimedGatewayResponse: @@ -127,17 +247,45 @@ def _response(request_id: str, value: dict) -> TimedGatewayResponse: class FakeGateway: client = object() - def __init__(self) -> None: + def __init__(self, gateway_instance_id: str = "benchmark-gateway-fake") -> None: self.closed = False + self.retained_shared_base_volumes = False + self.finalized = False + self.identity = SimpleNamespace( + gateway_instance_id=gateway_instance_id, + endpoint=GatewayEndpoint("127.0.0.1", 47621), + ) - async def close(self) -> None: + async def close( + self, + *, + destroy_sandboxes_via_gateway: bool, + retain_shared_base_volumes: bool, + ) -> None: + assert destroy_sandboxes_via_gateway is True + self.retained_shared_base_volumes = retain_shared_base_volumes self.closed = True -def _install_fakes(monkeypatch, fake: FakeProduct, gateway: FakeGateway) -> None: +def _install_fakes( + monkeypatch, fake: FakeProduct, gateway: FakeGateway | list[FakeGateway] +) -> None: + gateways = gateway if isinstance(gateway, list) else [gateway] + launched = 0 + class Launcher: - async def start(self, run_id, *, remount_sweep_width): - return gateway + async def start( + self, + run_id, + *, + remount_sweep_width, + readiness_via_cli, + ): + nonlocal launched + assert readiness_via_cli is False + selected = gateways[launched] + launched += 1 + return selected async def environment(roots, plan): return { @@ -155,9 +303,28 @@ async def environment(roots, plan): "workspace_root_identity": None, "client_cohort": "direct_client", "gateway_endpoint_identity": "fake", + "gateway_transport": { + "transport": "tcp_loopback", + "scope": "local_only", + "rotation": "per_execution_block", + }, } + async def cleanup_gateway_docker_resources( + gateway_instance_ids: tuple[str, ...], + ) -> None: + assert gateway_instance_ids == tuple( + item.identity.gateway_instance_id for item in gateways + ) + for item in gateways: + item.finalized = True + monkeypatch.setattr(runner_module, "GatewayLauncher", lambda roots: Launcher()) + monkeypatch.setattr( + runner_module, + "cleanup_gateway_docker_resources", + cleanup_gateway_docker_resources, + ) monkeypatch.setattr(runner_module, "ProductAccess", lambda client, runs: fake) monkeypatch.setattr(runner_module, "collect_environment", environment) @@ -184,6 +351,8 @@ def test_runner_persists_verified_terminal_corpus_and_removes_workspace( assert result.trial_batches == 1 assert result.issued_requests == 1 assert gateway.closed + assert gateway.retained_shared_base_volumes + assert gateway.finalized assert not (roots.runs / "run-success").exists() corpus = RunCorpus.open(ArtifactStore(roots).run_path("run-success")) manifest_envelope = json.loads( @@ -198,10 +367,36 @@ def test_runner_persists_verified_terminal_corpus_and_removes_workspace( } assert corpus.manifest["treatment"] == corpus.environment["treatment"] assert corpus.manifest["correctness"] == "pass" - definition_reference = ArtifactStore(roots).download_artifact( - "run-success", ArtifactId.DEFINITION_SNAPSHOT.value - ).reference - assert corpus.manifest["definition_snapshot"]["sha256"] == definition_reference.sha256 + assert { + "semantic_revision": 1, + "mode": "isolated", + "loopback_only": True, + }.items() <= corpus.manifest["gateway_policy"].items() + assert { + "protocol_version", + "transport", + "scope", + "rotation", + }.isdisjoint(corpus.manifest["gateway_policy"]) + assert corpus.manifest["gateway_execution_blocks"] == [ + { + "block_id": plan["execution_blocks"][0]["block_id"], + "family_id": plan["execution_blocks"][0]["family_id"], + "gateway_instance_id": gateway.identity.gateway_instance_id, + "endpoint_uri": "tcp://127.0.0.1:47621", + "transport": "tcp_loopback", + "scope": "local_only", + "rotation": "per_execution_block", + } + ] + definition_reference = ( + ArtifactStore(roots) + .download_artifact("run-success", ArtifactId.DEFINITION_SNAPSHOT.value) + .reference + ) + assert ( + corpus.manifest["definition_snapshot"]["sha256"] == definition_reference.sha256 + ) assert corpus.report is not None assert corpus.report.definition_snapshot_sha256 == definition_reference.sha256 assert set(corpus.manifest["artifact_schemas"]) == { @@ -229,6 +424,7 @@ def test_runner_persists_verified_terminal_corpus_and_removes_workspace( ] assert [event["state"] for event in request_states] == [ "waiting_at_barrier", + "ready_at_barrier", "in_flight", "succeeded", ] @@ -285,6 +481,157 @@ def test_runner_persists_verified_terminal_corpus_and_removes_workspace( assert trials[0]["artifacts"][0]["artifact_id"].startswith("bounded_evidence_") +def test_runner_finalizes_retained_volumes_from_every_execution_block( + tmp_path: Path, monkeypatch +) -> None: + roots = _roots(tmp_path) + fake = FakeProduct() + gateways = [ + FakeGateway("benchmark-gateway-first"), + FakeGateway("benchmark-gateway-second"), + ] + _install_fakes(monkeypatch, fake, gateways) + + result = asyncio.run( + CampaignRunner(roots).run( + "run-two-blocks", + _two_block_file_read_plan(), + intent=_artifact("intent-plan.json"), + definition_snapshot=_artifact("definition-snapshot.json"), + ) + ) + + assert result.state == "completed" + assert result.trial_batches == 2 + assert all(gateway.closed for gateway in gateways) + assert all(gateway.retained_shared_base_volumes for gateway in gateways) + assert all(gateway.finalized for gateway in gateways) + assert not (roots.runs / "run-two-blocks").exists() + + +def test_gateway_execution_block_prefix_is_durable_before_product_and_crash( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + fake = FakeProduct() + gateway = FakeGateway("benchmark-gateway-first") + _install_fakes(monkeypatch, fake, gateway) + plan = _two_block_file_read_plan() + durable_before_product: list[list[dict]] = [] + durable_before_crash: list[dict] = [] + launches = 0 + + class SimulatedCrash(BaseException): + pass + + class Launcher: + async def start( + self, + run_id, + *, + remount_sweep_width, + readiness_via_cli, + ): + nonlocal launches + assert remount_sweep_width == 1 + assert readiness_via_cli is False + if launches: + snapshot = ArtifactStore(roots).read_envelope( + run_id, ArtifactId.RUN_MANIFEST + ) + durable_before_crash.extend( + copy.deepcopy(snapshot["gateway_execution_blocks"]) + ) + raise SimulatedCrash("simulated process interruption") + launches += 1 + return gateway + + def observed_product_access(client, runs): + assert client is gateway.client + snapshot = ArtifactStore(roots).read_envelope( + "run-crash-prefix", ArtifactId.RUN_MANIFEST + ) + durable_before_product.append( + copy.deepcopy(snapshot["gateway_execution_blocks"]) + ) + return fake + + monkeypatch.setattr(runner_module, "GatewayLauncher", lambda roots: Launcher()) + monkeypatch.setattr(runner_module, "ProductAccess", observed_product_access) + + with pytest.raises(SimulatedCrash, match="simulated process interruption"): + asyncio.run( + CampaignRunner(roots).run( + "run-crash-prefix", + plan, + intent=_artifact("intent-plan.json"), + definition_snapshot=_artifact("definition-snapshot.json"), + ) + ) + + expected_prefix = [ + { + "block_id": plan["execution_blocks"][0]["block_id"], + "family_id": plan["execution_blocks"][0]["family_id"], + "gateway_instance_id": gateway.identity.gateway_instance_id, + "endpoint_uri": "tcp://127.0.0.1:47621", + "transport": "tcp_loopback", + "scope": "local_only", + "rotation": "per_execution_block", + } + ] + assert durable_before_product == [expected_prefix] + assert durable_before_crash == expected_prefix + assert ( + ArtifactStore(roots) + .read_envelope("run-crash-prefix", ArtifactId.RUN_MANIFEST)[ + "gateway_execution_blocks" + ] + == expected_prefix + ) + + +def test_manifest_v11_policy_is_scoped_to_windows_product_cli_named_pipe( + tmp_path: Path, +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + runner._started_at = "2026-07-31T00:00:00.000000Z" + plan = _single_file_read_plan() + environment = { + "treatment": {"source_commit": "fake"}, + "client_cohort": "product_cli", + "gateway_endpoint_identity": ( + "isolated_windows_named_pipe_per_execution_block" + ), + "gateway_transport": { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + }, + } + + manifest = runner._new_manifest( + "run-v11-policy", + plan, + _artifact("definition-snapshot.json"), + "sha256:" + "1" * 64, + environment, + None, + ) + + assert { + "semantic_revision": 2, + "protocol_version": ( + "ephemeral-sandbox-v1-practical-performance-v1.1" + ), + "mode": "isolated", + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + "loopback_only": False, + }.items() <= manifest["gateway_policy"].items() + + def test_runner_cleanup_failure_forces_failed_retained_run( tmp_path: Path, monkeypatch ) -> None: @@ -304,15 +651,55 @@ def test_runner_cleanup_failure_forces_failed_retained_run( ) assert gateway.closed + assert gateway.finalized assert (roots.runs / "run-cleanup-failed").exists() store = ArtifactStore(roots) - assert store.read_envelope("run-cleanup-failed", ArtifactId.RUN_MANIFEST)["state"] == "failed" + assert ( + store.read_envelope("run-cleanup-failed", ArtifactId.RUN_MANIFEST)["state"] + == "failed" + ) corpus = RunCorpus.open(store.run_path("run-cleanup-failed")) assert corpus.report is not None assert corpus.report.state == "failed" assert corpus.report.correctness_verdict == "fail" +def test_retained_volume_cleanup_failure_forces_failed_retained_run( + tmp_path: Path, monkeypatch +) -> None: + roots = _roots(tmp_path) + fake = FakeProduct() + gateway = FakeGateway() + _install_fakes(monkeypatch, fake, gateway) + cleanup_calls: list[tuple[str, ...]] = [] + + async def fails(gateway_instance_ids: tuple[str, ...]) -> None: + cleanup_calls.append(gateway_instance_ids) + raise RuntimeError("simulated retained volume cleanup failure") + + monkeypatch.setattr(runner_module, "cleanup_gateway_docker_resources", fails) + + with pytest.raises( + CampaignError, match="retained shared-base Docker cleanup failed" + ): + asyncio.run( + CampaignRunner(roots).run( + "run-retained-cleanup-failed", + _single_file_read_plan(), + intent=_artifact("intent-plan.json"), + definition_snapshot=_artifact("definition-snapshot.json"), + ) + ) + + assert cleanup_calls == [(gateway.identity.gateway_instance_id,)] + assert gateway.closed + assert (roots.runs / "run-retained-cleanup-failed").exists() + manifest = ArtifactStore(roots).read_envelope( + "run-retained-cleanup-failed", ArtifactId.RUN_MANIFEST + ) + assert manifest["state"] == "failed" + + def test_runner_clean_cancellation_removes_owned_workspace( tmp_path: Path, monkeypatch ) -> None: @@ -343,6 +730,7 @@ async def cancel_during_operation(record: dict) -> None: assert result.state == "cancelled" assert gateway.closed + assert gateway.finalized assert not (roots.runs / "run-cancelled").exists() corpus = RunCorpus.open(ArtifactStore(roots).run_path("run-cancelled")) assert corpus.report is not None @@ -395,6 +783,1163 @@ async def operation() -> None: assert admitted is False +@pytest.mark.asyncio +async def test_request_batch_records_release_after_every_task_reaches_barrier() -> None: + runner = CampaignRunner.__new__(CampaignRunner) + runner._cancel = asyncio.Event() + timing = BatchTiming() + started: list[int] = [] + + async def operation() -> str: + started.append(time.monotonic_ns()) + await asyncio.sleep(0) + return "ok" + + result = await runner._run_batch( + [operation, operation, operation], batch_timing=timing + ) + + assert result == ["ok", "ok", "ok"] + assert timing.barrier_released_ns is not None + assert timing.batch_completed_ns is not None + assert all(value >= timing.barrier_released_ns for value in started) + assert timing.batch_completed_ns >= max(started) + + +@pytest.mark.asyncio +async def test_in_flight_event_fsync_is_deferred_until_after_batch_completion( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + store = runner._store + store.create_run("run-deferred-states") + runner._started_ns = time.monotonic_ns() + append_sizes: list[int] = [] + real_append = store.append_records + + def observed_append(run_id, artifact_id, records): + append_sizes.append(len(records)) + real_append(run_id, artifact_id, records) + + monkeypatch.setattr(store, "append_records", observed_append) + launch_append_counts: list[int] = [] + + async def operation() -> TimedGatewayResponse: + launch_append_counts.append(len(append_sizes)) + started_ns = time.monotonic_ns() + return TimedGatewayResponse( + "request", + 1, + 1, + "sha256:response", + {}, + started_ns=started_ns, + ) + + request_ids = [f"request-{index}" for index in range(3)] + await runner._run_request_batch( + "run-deferred-states", + "cell-1", + "trial-1", + request_ids, + [operation, operation, operation], + batch_timing=BatchTiming(), + ) + + # Waiting intent and the all-tasks-ready barrier share one ordered durable + # commit before release. No in-flight fsync occurs before any operation + # begins; six in-flight/terminal records append after all operations return. + assert launch_append_counts == [1, 1, 1] + assert append_sizes == [6, 6] + + +@pytest.mark.asyncio +async def test_pre_release_event_commit_failure_keeps_operations_behind_barrier( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + store = runner._store + store.create_run("run-pre-release-failure") + runner._started_ns = time.monotonic_ns() + runner._begin_trial_journal() + await runner._event( + "run-pre-release-failure", + { + "kind": "trial_phase", + "cell_id": "cell-1", + "trial_id": "trial-1", + "warmup": False, + "phase": "operation", + "state": "running", + }, + ) + launched = False + + async def operation() -> str: + nonlocal launched + launched = True + return "unexpected" + + def fail_append(*_args, **_kwargs) -> None: + raise OSError("simulated pre-release durability failure") + + monkeypatch.setattr(store, "append_records", fail_append) + with pytest.raises(BaseExceptionGroup) as captured: + await runner._run_request_batch( + "run-pre-release-failure", + "cell-1", + "trial-1", + ["request-1"], + [operation], + batch_timing=BatchTiming(), + ) + + durability_failures = captured.value.subgroup(OSError) + assert durability_failures is not None + assert any( + str(error) == "simulated pre-release durability failure" + for error in durability_failures.exceptions + ) + assert launched is False + assert [ + record["data"]["state"] + for record in runner._trial_event_buffer + if record["data"]["kind"] == "request_state" + ] == ["waiting_at_barrier", "ready_at_barrier"] + runner._end_trial_journal() + + +@pytest.mark.asyncio +async def test_trial_journal_groups_non_barrier_records_by_artifact( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + delivered: list[dict] = [] + + async def event_sink(record: dict) -> None: + delivered.append(record) + + runner = CampaignRunner(roots, event_sink=event_sink) + store = runner._store + store.create_run("run-grouped-journal") + runner._started_ns = time.monotonic_ns() + appends: list[tuple[ArtifactId, int]] = [] + real_append = store.append_records + + def observed_append(run_id, artifact_id, records): + appends.append((artifact_id, len(records))) + real_append(run_id, artifact_id, records) + + monkeypatch.setattr(store, "append_records", observed_append) + runner._begin_trial_journal() + await runner._event( + "run-grouped-journal", + { + "kind": "trial_phase", + "cell_id": "cell-1", + "trial_id": "trial-1", + "warmup": False, + "phase": "verify", + "state": "completed", + }, + ) + runner._observation( + "run-grouped-journal", + { + "record": "operation", + "data": { + "operation_id": "file_read", + "cell_id": "cell-1", + "trial_id": "trial-1", + "request_id": None, + "evidence": {}, + }, + }, + ) + await runner._resource_observations( + "run-grouped-journal", + [ + { + "cell_id": "cell-1", + "trial_id": "trial-1", + "request_id": None, + "reading": { + "metric_id": "sandbox_cpu_time_us", + "monotonic_offset_ns": 1, + "value": {"availability": "available", "value": 1}, + }, + } + ], + ) + + assert appends == [] + assert delivered == [] + await runner._flush_trial_journal("run-grouped-journal") + runner._end_trial_journal() + + assert appends == [ + (ArtifactId.OBSERVATIONS, 2), + (ArtifactId.EVENTS, 2), + ] + assert [record["data"]["kind"] for record in delivered] == [ + "trial_phase", + "resource_window", + ] + assert ( + len(store.read_records("run-grouped-journal", ArtifactId.OBSERVATIONS).records) + == 2 + ) + + +@pytest.mark.asyncio +async def test_request_barriers_flush_prior_trial_records_in_sequence( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + store = runner._store + store.create_run("run-grouped-barrier") + runner._started_ns = time.monotonic_ns() + event_batches: list[list[str]] = [] + real_append = store.append_records + + def observed_append(run_id, artifact_id, records): + if artifact_id is ArtifactId.EVENTS: + event_batches.append([record["data"]["state"] for record in records]) + real_append(run_id, artifact_id, records) + + monkeypatch.setattr(store, "append_records", observed_append) + runner._begin_trial_journal() + await runner._event( + "run-grouped-barrier", + { + "kind": "trial_phase", + "cell_id": "cell-1", + "trial_id": "trial-1", + "warmup": False, + "phase": "setup", + "state": "completed", + }, + ) + + async def operation() -> str: + return "ok" + + result = await runner._run_request_batch( + "run-grouped-barrier", + "cell-1", + "trial-1", + ["request-1", "request-2"], + [operation, operation], + batch_timing=BatchTiming(), + ) + runner._end_trial_journal() + + assert result == ["ok", "ok"] + assert event_batches == [ + [ + "completed", + "waiting_at_barrier", + "waiting_at_barrier", + "ready_at_barrier", + "ready_at_barrier", + ], + ["in_flight", "succeeded", "in_flight", "succeeded"], + ] + + +@pytest.mark.asyncio +async def test_failed_trial_commits_buffer_before_propagating( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + store = runner._store + store.create_run("run-failed-grouped-journal") + runner._started_ns = time.monotonic_ns() + + async def failed_trial(*_args, **_kwargs): + await runner._event( + "run-failed-grouped-journal", + { + "kind": "trial_phase", + "cell_id": "cell-1", + "trial_id": "trial-1", + "warmup": False, + "phase": "setup", + "state": "running", + }, + ) + raise RuntimeError("simulated trial failure") + + monkeypatch.setattr(runner, "_run_trial", failed_trial) + with pytest.raises(RuntimeError, match="simulated trial failure"): + await runner._run_trial_with_journal( + "run-failed-grouped-journal", + tmp_path, + object(), + object(), + {}, + None, + "trial-1", + False, + 0, + ) + + assert ( + len(store.read_records("run-failed-grouped-journal", ArtifactId.EVENTS).records) + == 1 + ) + assert runner._trial_event_buffer is None + assert runner._trial_observation_buffer is None + + +@pytest.mark.asyncio +async def test_cli_evidence_commits_before_trial_journal( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + runner._store.create_run("run-cli-evidence-order") + runner._started_ns = time.monotonic_ns() + order: list[str] = [] + + class FakeCli(runner_module.ProductCliAccess): + def __init__(self) -> None: + pass + + def begin_trial_evidence(self, trial_id: str) -> None: + assert trial_id == "trial-1" + order.append("evidence-begin") + + async def flush_trial_evidence(self, trial_id: str) -> None: + assert trial_id == "trial-1" + order.append("evidence-flush") + + def end_trial_evidence(self, trial_id: str) -> None: + assert trial_id == "trial-1" + order.append("evidence-end") + + async def trial(*_args, **_kwargs): + order.append("trial") + await runner._event( + "run-cli-evidence-order", + { + "kind": "trial_phase", + "cell_id": "cell-1", + "trial_id": "trial-1", + "warmup": False, + "phase": "teardown", + "state": "completed", + }, + ) + return object() + + real_flush = runner._flush_trial_journal + + async def flush_journal(run_id: str) -> None: + order.append("journal-flush") + await real_flush(run_id) + + monkeypatch.setattr(runner, "_run_trial", trial) + monkeypatch.setattr(runner, "_flush_trial_journal", flush_journal) + + await runner._run_trial_with_journal( + "run-cli-evidence-order", + tmp_path, + FakeCli(), + object(), + {}, + None, + "trial-1", + False, + 0, + ) + + assert order == [ + "evidence-begin", + "trial", + "evidence-flush", + "journal-flush", + "evidence-end", + ] + assert runner._trial_event_buffer is None + assert ( + len( + runner._store.read_records( + "run-cli-evidence-order", ArtifactId.EVENTS + ).records + ) + == 1 + ) + + +@pytest.mark.asyncio +async def test_file_write_content_is_staged_before_batch_release() -> None: + runner = CampaignRunner.__new__(CampaignRunner) + runner._cancel = asyncio.Event() + staged: set[str] = set() + launch_stage_counts: list[int] = [] + + class FakeCli(runner_module.ProductCliAccess): + def __init__(self) -> None: + pass + + async def stage_file_write_content( + self, content: str, *, request_id: str + ) -> None: + assert content + staged.add(request_id) + + def discard_file_write_content(self, request_id: str) -> None: + staged.discard(request_id) + + async def file_write(self, *args, request_id: str, **kwargs): + launch_stage_counts.append(len(staged)) + assert request_id in staged + started_ns = time.monotonic_ns() + return TimedGatewayResponse( + request_id, + 1, + 1, + "sha256:response", + {"path": kwargs["path"], "bytes_written": 4}, + started_ns=started_ns, + ) + + cell = { + "cell_id": "sha256:" + "4" * 64, + "protocol": {"timeout_ms": 1000}, + "operation": { + "operation": "file_write", + "cell": { + "concurrent_requests": 5, + "content_bytes": 4, + "target_mode": "independent", + }, + }, + } + context = TrialContext( + Path("workspace"), + "sandbox", + False, + data={"paths": [f"file-{index}.txt" for index in range(5)]}, + ) + + responses = await runner._operate( + FakeCli(), + object(), + cell, + context, + "trial-1", + batch_timing=BatchTiming(), + ) + + assert len(responses) == 5 + assert launch_stage_counts == [5] * 5 + assert staged == set() + + +@pytest.mark.asyncio +async def test_trial_makespan_uses_validated_response_end_not_persistence_end( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + runner._started_ns = 1 + runner._definitions = {"operations": [{"id": "file_read", "checks": []}]} + ordering: list[str] = [] + + class Sampler: + def __init__(self, **kwargs) -> None: + pass + + async def start(self) -> None: + ordering.append("resource-start") + + async def stop(self) -> None: + ordering.append("resource-stop") + + async def no_op(*args, **kwargs) -> None: + pass + + async def setup(*args, **kwargs): + return TrialContext(tmp_path / "workspace", "sandbox", True) + + async def operate(*args, **kwargs): + ordering.append("operate") + timing = kwargs["batch_timing"] + timing.barrier_released_ns = 100 + # This models request-state/evidence persistence completing long after + # the validated responses. It must not enter the primary makespan. + timing.batch_completed_ns = 10_000 + return [ + TimedGatewayResponse( + "request-1", + 50, + 1, + "sha256:response-1", + {}, + started_ns=150, + ), + TimedGatewayResponse( + "request-2", + 100, + 1, + "sha256:response-2", + {}, + started_ns=200, + ), + ] + + async def observe_request(*args, **kwargs) -> None: + ordering.append("request-observation") + + async def verify(*args, **kwargs) -> None: + ordering.append("verify") + + async def teardown(*args, **kwargs) -> None: + pass + + monkeypatch.setattr(runner_module, "TrialResourceSampler", Sampler) + monkeypatch.setattr(runner_module, "_operation_evidence", lambda *args: {}) + monkeypatch.setattr(runner, "_trial_phase", no_op) + monkeypatch.setattr(runner, "_setup_trial", setup) + monkeypatch.setattr(runner, "_operate", operate) + monkeypatch.setattr(runner, "_verify", verify) + monkeypatch.setattr(runner, "_teardown_trial", teardown) + monkeypatch.setattr(runner, "_request_observation", observe_request) + monkeypatch.setattr(runner, "_registered_check_observations", no_op) + monkeypatch.setattr(runner, "_trial_observation", no_op) + monkeypatch.setattr( + runner, + "_persist_operation_evidence", + lambda *args, **kwargs: {"artifact_id": "bounded"}, + ) + cell = { + "cell_id": "sha256:" + "3" * 64, + "protocol": {"timeout_ms": 1}, + "operation": { + "operation": "file_read", + "cell": { + "concurrent_requests": 2, + "target_mode": "independent", + }, + }, + } + + outcome = await runner._run_trial( + "run-timing", + tmp_path, + object(), + object(), + cell, + None, + "trial-1", + False, + 0, + ) + + assert outcome.batch_makespan_ns == 200 + assert outcome.operation_ns == 200 + assert ordering == [ + "resource-start", + "operate", + "resource-stop", + "request-observation", + "request-observation", + "verify", + ] + + +@pytest.mark.asyncio +async def test_create_workspace_sampler_requires_planned_post_workspace_count( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + runner._started_ns = 1 + runner._definitions = {"operations": [{"id": "create_workspace", "checks": []}]} + sampler_arguments: list[dict] = [] + + class Sampler: + def __init__(self, **kwargs) -> None: + sampler_arguments.append(kwargs) + + async def start(self) -> None: + pass + + async def stop(self) -> None: + pass + + async def no_op(*args, **kwargs) -> None: + pass + + async def setup(*args, **kwargs): + return TrialContext(tmp_path / "workspace", "sandbox", True) + + async def operate(*args, **kwargs): + timing = kwargs["batch_timing"] + timing.barrier_released_ns = 100 + timing.batch_completed_ns = 200 + return [ + TimedGatewayResponse( + f"request-{index}", + 100, + 1, + f"sha256:response-{index}", + {}, + started_ns=100, + ) + for index in range(5) + ] + + monkeypatch.setattr(runner_module, "TrialResourceSampler", Sampler) + monkeypatch.setattr( + runner_module, + "_operation_evidence", + lambda *args: {"evidence": {}}, + ) + monkeypatch.setattr(runner, "_trial_phase", no_op) + monkeypatch.setattr(runner, "_setup_trial", setup) + monkeypatch.setattr(runner, "_operate", operate) + monkeypatch.setattr(runner, "_verify", no_op) + monkeypatch.setattr(runner, "_teardown_trial", no_op) + monkeypatch.setattr(runner, "_request_observation", no_op) + monkeypatch.setattr(runner, "_registered_check_observations", no_op) + monkeypatch.setattr(runner, "_trial_observation", no_op) + monkeypatch.setattr( + runner, + "_persist_operation_evidence", + lambda *args, **kwargs: {"artifact_id": "bounded"}, + ) + cell = { + "cell_id": "sha256:" + "4" * 64, + "protocol": {"timeout_ms": 1}, + "operation": { + "operation": "create_workspace", + "cell": { + "workspace_count": 5, + "network_profile": "shared", + }, + }, + } + + outcome = await runner._run_trial( + "run-create-workspace-boundary", + tmp_path, + object(), + object(), + cell, + None, + "trial-1", + False, + 0, + ) + + assert outcome.status == "success" + assert len(sampler_arguments) == 1 + assert sampler_arguments[0]["expected_post_workspace_count"] == 5 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["file_write", "file_edit"]) +@pytest.mark.parametrize("target_mode", ["independent", "same_target"]) +async def test_session_mutation_publishes_one_cell_baseline_and_reuses_it( + tmp_path: Path, + operation: str, + target_mode: str, +) -> None: + roots = _roots(tmp_path) + runner = CampaignRunner(roots) + run_path = roots.runs / "run-session-baseline" + run_path.mkdir() + concurrent_requests = 3 + baseline_writes: list[tuple[str, str]] = [] + + class Product: + def __init__(self) -> None: + self.published: dict[str, str] = {} + self.session_views: dict[str, dict[str, str]] = {} + + async def create_sandbox(self, *args, **kwargs): + return SimpleNamespace(id="sandbox-cell"), object() + + async def destroy_sandbox(self, *args, **kwargs): + return object() + + async def file_write( + self, + sandbox, + *, + session_id, + path, + content, + request_id, + **kwargs, + ): + target = ( + self.published if session_id is None else self.session_views[session_id] + ) + target[path] = content + if session_id is None: + baseline_writes.append((path, request_id)) + return TimedGatewayResponse( + request_id, + 1, + 1, + "sha256:write", + { + "type": "update", + "path": path, + "bytes_written": len(content.encode()), + }, + ) + + async def file_edit( + self, + sandbox, + *, + session_id, + path, + edits, + request_id, + **kwargs, + ): + assert session_id is not None + content = self.session_views[session_id][path] + replacements = 0 + for edit in edits: + count = content.count(edit["old_string"]) + replacements += count + content = content.replace( + edit["old_string"], + edit["new_string"], + -1 if edit["replace_all"] else 1, + ) + self.session_views[session_id][path] = content + return TimedGatewayResponse( + request_id, + 1, + 1, + "sha256:edit", + { + "type": "edit", + "path": path, + "edits_applied": len(edits), + "replacements": replacements, + "bytes_written": len(content.encode()), + }, + ) + + product = Product() + + class Sessions: + def __init__(self) -> None: + self.created: list[str] = [] + self.destroyed: list[str] = [] + + async def create_no_op(self, sandbox, network_profile, *, request_id, **kwargs): + assert network_profile == "shared" + session_id = f"session-{len(self.created)}" + self.created.append(session_id) + product.session_views[session_id] = dict(product.published) + return ( + SimpleNamespace( + sandbox_id=sandbox, + session_id=session_id, + network_profile=network_profile, + ), + object(), + ) + + async def destroy(self, session, *, request_id, **kwargs): + self.destroyed.append(session.session_id) + del product.session_views[session.session_id] + return object() + + sessions = Sessions() + body: dict[str, object] = { + "destination": "session", + "target_mode": target_mode, + "concurrent_requests": concurrent_requests, + "resolved_isolation": "fresh_sessions_per_trial", + } + if operation == "file_write": + body["content_bytes"] = 4096 + else: + body.update( + file_bytes=4096, + replacement_count=1, + match_density=1.0, + ) + cell = { + "cell_id": "sha256:" + "7" * 64, + "comparison_key": {"isolation": "fresh_sessions_per_trial"}, + "protocol": {"timeout_ms": 1000}, + "operation": {"operation": operation, "cell": body}, + } + + cell_context = await runner._setup_cell(run_path, product, sessions, cell) + assert cell_context is not None + published_baseline = dict(product.published) + expected_paths = concurrent_requests if target_mode == "independent" else 1 + assert len(baseline_writes) == expected_paths + assert all( + request_id.startswith("cell-7777777777777777.prepare.") + for _, request_id in baseline_writes + ) + + for index in range(2): + trial_id = f"trial-{index}" + context = await runner._setup_trial( + run_path, product, sessions, cell, cell_context, trial_id + ) + assert context.data["paths"] == cell_context.data["paths"] + assert len(context.sessions) == 1 + session_id = context.sessions[0].session_id + assert product.session_views[session_id] == published_baseline + + await runner._operate(product, sessions, cell, context, trial_id) + + assert product.published == published_baseline + assert product.session_views[session_id] != published_baseline + assert len(baseline_writes) == expected_paths + await runner._teardown_trial(product, sessions, context, trial_id) + + assert sessions.created == ["session-0", "session-1"] + assert sessions.destroyed == sessions.created + await runner._teardown_cell(product, sessions, cell_context, cell["cell_id"]) + + +@pytest.mark.asyncio +async def test_independent_verification_reads_run_concurrently( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + active = 0 + maximum_active = 0 + all_started = asyncio.Event() + attribution_started = asyncio.Event() + release = asyncio.Event() + paths = [f"file-{index}.txt" for index in range(5)] + + async def read_exact(*args, path: str, **kwargs) -> str: + nonlocal active, maximum_active + active += 1 + maximum_active = max(maximum_active, active) + if active == len(paths): + all_started.set() + await release.wait() + active -= 1 + return f"content:{path}" + + async def verify_attribution(*args, **kwargs) -> None: + attribution_started.set() + await release.wait() + + monkeypatch.setattr(runner, "_read_exact", read_exact) + monkeypatch.setattr(runner, "_verify_mutation_attribution", verify_attribution) + body = { + "destination": "session", + "target_mode": "independent", + "concurrent_requests": 5, + "content_bytes": 16, + } + cell = { + "operation": {"operation": "file_write", "cell": body}, + "protocol": {"timeout_ms": 1000}, + } + responses = [ + TimedGatewayResponse( + f"request-{index}", + 1, + 1, + f"sha256:{index}", + { + "type": "update", + "path": path, + "bytes_written": body["content_bytes"], + }, + ) + for index, path in enumerate(paths) + ] + context = TrialContext( + tmp_path, + "sandbox", + False, + data={ + "paths": paths, + "request_contents": [f"content:{path}" for path in paths], + "operation_session": SimpleNamespace(session_id="session"), + }, + ) + + verifying = asyncio.create_task( + runner._verify( + object(), object(), cell, responses, context, "trial-independent" + ) + ) + await asyncio.wait_for(all_started.wait(), timeout=1) + await asyncio.wait_for(attribution_started.wait(), timeout=1) + release.set() + await verifying + + assert maximum_active == 5 + + +@pytest.mark.asyncio +async def test_mutation_verification_preserves_both_branch_failures( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + both_started = asyncio.Event() + started = 0 + + async def fail_after_both(message: str) -> None: + nonlocal started + started += 1 + if started == 2: + both_started.set() + await both_started.wait() + raise CampaignError(message) + + async def verify_contents(*args, **kwargs) -> None: + await fail_after_both("content branch failed") + + async def verify_attribution(*args, **kwargs) -> None: + await fail_after_both("attribution branch failed") + + monkeypatch.setattr(runner, "_verify_mutation_contents", verify_contents) + monkeypatch.setattr(runner, "_verify_mutation_attribution", verify_attribution) + body = { + "destination": "session", + "target_mode": "independent", + "concurrent_requests": 1, + "content_bytes": 16, + } + cell = { + "operation": {"operation": "file_write", "cell": body}, + "protocol": {"timeout_ms": 1000}, + } + response = TimedGatewayResponse( + "request-0", + 1, + 1, + "sha256:0", + { + "type": "update", + "path": "file.txt", + "bytes_written": body["content_bytes"], + }, + ) + context = TrialContext( + tmp_path, + "sandbox", + False, + data={ + "paths": ["file.txt"], + "request_contents": ["content"], + "operation_session": SimpleNamespace(session_id="session"), + }, + ) + + with pytest.raises(BaseExceptionGroup) as captured: + await runner._verify( + object(), object(), cell, [response], context, "trial-both-fail" + ) + + messages = {str(error) for error in captured.value.exceptions} + assert messages == {"content branch failed", "attribution branch failed"} + + +@pytest.mark.asyncio +async def test_verification_read_fetches_remaining_pages_concurrently( + tmp_path: Path, +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + lines = [f"{index:04d}" for index in range(4500)] + content = "\n".join(lines) + active = 0 + maximum_active = 0 + remaining_started = asyncio.Event() + release = asyncio.Event() + calls: list[tuple[int, int, str]] = [] + + class Product: + async def file_read( + self, + *args, + offset: int, + limit: int, + request_id: str, + **kwargs, + ): + nonlocal active, maximum_active + calls.append((offset, limit, request_id)) + if offset > 1: + active += 1 + maximum_active = max(maximum_active, active) + if active == 2: + remaining_started.set() + await release.wait() + active -= 1 + selected = lines[offset - 1 : offset - 1 + limit] + next_offset = ( + offset + len(selected) + if offset - 1 + len(selected) < len(lines) + else None + ) + page_content = "\n".join(selected) + return SimpleNamespace( + value={ + "path": "fixture.txt", + "content": page_content, + "start_line": offset, + "num_lines": len(selected), + "total_lines": len(lines), + "bytes_read": len(page_content.encode()), + "total_bytes": len(content.encode()), + "next_offset": next_offset, + "truncated": next_offset is not None, + } + ) + + reading = asyncio.create_task( + runner._read_exact( + Product(), + "sandbox", + session_id=None, + path="fixture.txt", + expected_bytes=len(content.encode()), + timeout_ms=1000, + request_id="trial.verify", + ) + ) + await asyncio.wait_for(remaining_started.wait(), timeout=1) + assert maximum_active == 2 + release.set() + + assert await reading == content + assert calls == [ + (1, 2000, "trial.verify.0"), + (2001, 2000, "trial.verify.1"), + (4001, 2000, "trial.verify.2"), + ] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["file_read", "file_write", "file_edit"]) +async def test_independent_fixture_writes_run_concurrently( + tmp_path: Path, + operation: str, +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + active = 0 + maximum_active = 0 + all_started = asyncio.Event() + release = asyncio.Event() + request_ids: list[str] = [] + + class Product: + async def file_write(self, *args, request_id: str, **kwargs): + nonlocal active, maximum_active + request_ids.append(request_id) + active += 1 + maximum_active = max(maximum_active, active) + if active == 5: + all_started.set() + await release.wait() + active -= 1 + return object() + + body: dict[str, object] = { + "target_mode": "independent", + "concurrent_requests": 5, + } + if operation == "file_read": + body["returned_bytes"] = 4096 + elif operation == "file_write": + body["content_bytes"] = 4096 + else: + body.update( + file_bytes=4096, + replacement_count=1, + match_density=1.0, + ) + cell = { + "operation": {"operation": operation, "cell": body}, + "protocol": {"timeout_ms": 1000}, + } + context = TrialContext(tmp_path, "sandbox", False) + + if operation == "file_read": + preparing = asyncio.create_task( + runner._prepare_reads(Product(), cell, context, "trial-independent") + ) + else: + preparing = asyncio.create_task( + runner._prepare_mutation(Product(), cell, context, "trial-independent") + ) + await asyncio.wait_for(all_started.wait(), timeout=1) + assert maximum_active == 5 + release.set() + await preparing + + assert len(context.data["paths"]) == 5 + assert len(request_ids) == 5 + assert len(set(request_ids)) == 5 + + +@pytest.mark.asyncio +async def test_create_workspace_teardown_destroys_sessions_in_parallel_first( + tmp_path: Path, +) -> None: + runner = CampaignRunner(_roots(tmp_path)) + sessions_to_destroy = [ + SimpleNamespace(session_id=f"session-{index}", sandbox_id="sandbox") + for index in range(5) + ] + all_started = asyncio.Event() + release = asyncio.Event() + started: list[str] = [] + sandbox_destroyed = False + + class Lifecycle: + async def destroy(self, session, *, request_id: str): + started.append(session.session_id) + if len(started) == len(sessions_to_destroy): + all_started.set() + await release.wait() + + class Product: + async def destroy_sandbox(self, sandbox: str, *, request_id: str): + nonlocal sandbox_destroyed + assert len(started) == len(sessions_to_destroy) + sandbox_destroyed = True + + context = TrialContext( + tmp_path, + "sandbox", + True, + sessions=sessions_to_destroy, + data={"operation_id": "create_workspace"}, + ) + teardown = asyncio.create_task( + runner._teardown_trial( + Product(), Lifecycle(), context, "trial-create-workspace" + ) + ) + await asyncio.wait_for(all_started.wait(), timeout=1) + assert not sandbox_destroyed + release.set() + await teardown + + assert sandbox_destroyed + assert set(started) == {session.session_id for session in sessions_to_destroy} + + @pytest.mark.asyncio async def test_cancellation_request_persists_exactly_one_public_transition( tmp_path: Path, @@ -451,6 +1996,9 @@ async def setup(*args, **kwargs): return TrialContext(tmp_path / "workspace", "sandbox", True) async def operate(*args, **kwargs): + timing = kwargs["batch_timing"] + timing.barrier_released_ns = time.monotonic_ns() + timing.batch_completed_ns = time.monotonic_ns() return [TimedGatewayResponse("request-1", 1, 1, "sha256:response", {})] async def verify(*args, **kwargs) -> None: diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/__init__.cpython-313.pyc b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/__init__.cpython-313.pyc deleted file mode 100644 index 66be0df..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/__init__.cpython-313.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/test_fixtures.cpython-313-pytest-9.1.1.pyc b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/test_fixtures.cpython-313-pytest-9.1.1.pyc deleted file mode 100644 index 9602c5e..0000000 Binary files a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/__pycache__/test_fixtures.cpython-313-pytest-9.1.1.pyc and /dev/null differ diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_derivation.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_derivation.py index 84fc9b5..e6fb9f0 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_derivation.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_derivation.py @@ -2,7 +2,7 @@ import json from pathlib import Path -from benchmark_lab.derivation import build_report +from benchmark_lab.derivation import _aggregate_resource, build_report from benchmark_lab.resource_sampling import resource_metric_source @@ -200,6 +200,61 @@ def test_derives_all_registered_metrics_lifecycle_and_resource_semantics() -> No assert [item["identity"]["id"] for item in result["metrics"]] == sorted(metrics) +def test_resource_delta_allows_signed_gauge_but_rejects_counter_reset() -> None: + readings = [ + {"value": {"availability": "available", "value": 25.0}}, + {"value": {"availability": "available", "value": 10.0}}, + ] + + assert _aggregate_resource(readings, "delta", "gauge") == (-15.0, None) + assert _aggregate_resource(readings, "delta", "monotonic_counter") == ( + None, + "monotonic counter reset during the trial window", + ) + + +def test_product_cli_report_names_the_cli_timing_and_evidence_boundary() -> None: + plan = _plan_for("file_read") + cell = plan["cells"][0] + definitions = _artifact("definition-snapshot.json") + trial_id = "trial-product-cli" + observations = [_trial(cell, trial_id, 1_000), _request(cell, trial_id, 900)] + observations.extend(_checks(definitions, cell, trial_id)) + environment = _artifact("environment-metadata.json") + environment["client_cohort"] = "product_cli" + environment["treatment"].update( + { + "manager_cli_binary_hash": "sha256:manager", + "runtime_cli_binary_hash": "sha256:runtime", + "observability_cli_binary_hash": "sha256:observability", + } + ) + + report = build_report( + run_id="unit-product-cli", + state="completed", + plan=plan, + definitions=definitions, + definition_snapshot_sha256="sha256:fixture-definition-snapshot", + environment=environment, + observations=observations, + started_at="2026-01-01T00:00:00Z", + ended_at="2026-01-01T00:00:01Z", + ) + metrics = { + item["identity"]["id"]: item["identity"] for item in report.cells[0]["metrics"] + } + + assert metrics["request_latency_ns"]["source"] == ( + "product_cli_subprocess_monotonic_spawn_to_validated_json" + ) + assert report.methods["primary_timing_boundary"]["client_cohort"] == "product_cli" + assert report.methods["cli_evidence"]["directory"] == "cli-subprocesses" + assert report.methods["executable_identities"]["runtime_cli_binary_hash"] == ( + "sha256:runtime" + ) + + def test_completed_report_fails_when_a_registered_check_is_missing() -> None: plan = _plan_for("file_read") cell = plan["cells"][0] diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_archive.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_archive.py new file mode 100644 index 0000000..13a99f3 --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_archive.py @@ -0,0 +1,969 @@ +import importlib.util +import base64 +import json +import subprocess +from pathlib import Path + +import pytest + +PAPER_ROOT = Path(__file__).resolve().parents[4] +SCRIPT = PAPER_ROOT / "experiments/scripts/archive_exp1_run.py" +SPEC = importlib.util.spec_from_file_location("archive_exp1_run_test", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +archive_exp1_run = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(archive_exp1_run) + + +def _environment() -> dict: + host = { + "computer_name": "DESKTOP-OLP1ADS", + "operating_system": "windows", + "architecture": "x64", + "os_caption": "Microsoft Windows 11", + "os_version": "10.0.26200", + "os_build_number": 26200, + "cpu_model": "AMD Ryzen Threadripper 7960X 24-Cores", + "logical_processors": 48, + "processor_logical_processors": 48, + "total_memory_bytes": 137_438_953_472, + "filesystem": "NTFS", + "volume_root": "C:\\", + "capture_boundary": "run_start_before_gateway_and_measurement", + "captured_at": "2026-07-30T00:00:00.000000Z", + "capture_source": "Windows CIM and Get-Volume", + } + limits = { + "profile": "standard", + "nano_cpus": 1_000_000_000, + "vcpus": 1, + "memory_bytes": 536_870_912, + "pids_limit": 256, + "authority": { + "kind": "released_gateway_configuration", + "path": "C:\\package\\config\\windows-amd64.yml", + "sha256": "sha256:" + "a" * 64, + "selector": "manager.docker.resource_profiles.standard", + "effective_config_builder": "benchmark_lab.gateway._effective_config", + "create_request_override": "none", + "capture_boundary": "run_start_before_gateway_and_measurement", + }, + } + return { + "schema_version": 1, + "host": host, + "sandbox_limits": limits, + } + + +def _envelope(name: str, data: dict) -> dict: + return { + "schema_name": name, + "schema_version": 1, + "data": data, + } + + +def _refresh_inventory( + root: Path, + run_id: str, + disposition: str = "exploratory", + run_status: str = "completed", +) -> None: + entries, total_bytes, tree_hash = archive_exp1_run.archive_inventory(root) + archive_exp1_run.write_json( + root / "archive-manifest.json", + { + "schema_version": 1, + "run_id": run_id, + "disposition": disposition, + "run_status": run_status, + "eligibility": archive_exp1_run.archive_eligibility( + disposition, run_status + ), + "archive_file_count": len(entries), + "archive_bytes": total_bytes, + "content_tree_sha256": tree_hash, + "files": entries, + }, + ) + + +def _archive( + root: Path, + disposition: str = "exploratory", + run_status: str = "completed", +) -> tuple[Path, dict]: + run_id = "019fb000-0000-7000-8000-000000000000" + environment = _environment() + correctness = "pass" if run_status == "completed" else "fail" + manifest = { + "run_id": run_id, + "environment": environment, + "state": run_status, + "correctness": correctness, + } + if run_status == "failed": + manifest["failure"] = { + "code": "campaign_failed", + "message": "campaign execution failed; inspect raw evidence", + "infrastructure": True, + } + report = { + "run_id": run_id, + "state": run_status, + "correctness_verdict": correctness, + "warnings": ( + [] + if run_status == "completed" + else [ + { + "code": "missing_correctness_observations", + "message": "A measured-trial verdict is missing.", + } + ] + ), + } + product_commit = "b" * 40 + treatment = { + "source_commit": product_commit, + "source_dirty": False, + "source_diff_hash": None, + } + manifest["treatment"] = treatment + post_run_commit = product_commit if run_status == "completed" else "e" * 40 + post_run_status = "" if run_status == "completed" else " M crates/fix.rs" + cleanup = { + "run_workspace_exists": False, + "runtime_exists": False, + "matching_product_processes": [], + "run_labeled_containers": [], + "gateway_labeled_containers": [], + "run_labeled_volumes": [], + "gateway_labeled_volumes": [], + "product_branch": "main", + "product_commit": post_run_commit, + "product_status_porcelain": post_run_status, + "product_checkout_policy": ( + "clean_exact_recorded_treatment" + if run_status == "completed" + else "post_run_drift_recorded_failed_ineligible" + ), + } + freeze_tag = ( + { + "availability": "available", + "name": "paper-v1-freeze", + "reference": "refs/tags/paper-v1-freeze", + "object_type": "tag", + "tag_object": "c" * 40, + "peeled_commit": product_commit, + } + if disposition == "final" + else { + "availability": "unavailable", + "reason": "pre-freeze smoke/exploratory archive", + "required_final_tag": "paper-v1-freeze", + } + ) + product = { + "commit": product_commit, + "dirty": False, + "recorded_treatment": treatment, + "post_run_checkout": { + "branch": "main", + "commit": post_run_commit, + "status_porcelain": post_run_status, + "dirty": bool(post_run_status), + "capture_boundary": ( + archive_exp1_run.product_checkout_capture_boundary(run_status) + ), + }, + "freeze_tag": freeze_tag, + } + (root / "raw").mkdir(parents=True) + archive_exp1_run.write_json( + root / "raw/run-manifest.json", + _envelope("eos_benchmark_run_manifest", manifest), + ) + archive_exp1_run.write_json( + root / "raw/environment-metadata.json", + _envelope("eos_benchmark_environment_metadata", environment), + ) + archive_exp1_run.write_json( + root / "raw/report.json", + _envelope("eos_benchmark_report", report), + ) + archive_exp1_run.write_json( + root / "run-manifest.json", + _envelope("eos_benchmark_run_manifest", manifest), + ) + archive_exp1_run.write_json( + root / "report.json", + _envelope("eos_benchmark_report", report), + ) + (root / "failures.md").write_text( + ( + "# Failures\n\n- None.\n" + if run_status == "completed" + else "# Failures\n\n- Terminal campaign failure; raw evidence retained.\n" + ), + encoding="utf-8", + ) + (root / "cleanup").mkdir() + archive_exp1_run.write_json( + root / "cleanup/cleanup-proof.json", cleanup + ) + archive_exp1_run.write_json( + root / "environment-preflight.txt", + { + "recorded_run_environment": environment, + "final_host": environment["host"], + "sandbox_limits": environment["sandbox_limits"], + "product": product, + "cleanup": cleanup, + }, + ) + archive_exp1_run.write_json( + root / "campaign-manifest.json", + { + "run_id": run_id, + "disposition": disposition, + "run_status": run_status, + "eligibility": archive_exp1_run.archive_eligibility( + disposition, run_status + ), + "state": run_status, + "correctness": correctness, + "benchmark_source": { + "capture_boundary": ( + archive_exp1_run.benchmark_source_capture_boundary( + run_status + ) + ), + }, + "final_host": environment["host"], + "sandbox_limits": environment["sandbox_limits"], + "product": product, + "cleanup": cleanup, + "protocol": { + "version": ("v1.0" if disposition == "final" else "pre-freeze-exp1"), + "freeze_state": ("frozen" if disposition == "final" else "pre_freeze"), + }, + "analysis_and_archiving_code": { + "files": [ + {"path": ("experiments/analysis/scripts/generate_exp1_tables.py")} + ] + }, + "paper_git": { + "commit": "d" * 40, + "dirty": False, + "status_porcelain": [], + "generated_exclusions": (archive_exp1_run.BENCHMARK_GIT_EXCLUSIONS), + "freeze_state": ( + "clean_frozen_commit" + if disposition == "final" + else "pre_freeze_worktree" + ), + }, + }, + ) + _refresh_inventory(root, run_id, disposition, run_status) + return root, environment + + +def _source_run( + root: Path, + *, + disposition: str = "smoke", + run_status: str = "failed", +) -> tuple[str, str]: + expected = archive_exp1_run.DISPOSITIONS[disposition] + run_id = "019fb000-0000-7000-8000-000000000001" + plan_hash = "sha256:" + "e" * 64 + correctness = "pass" if run_status == "completed" else "fail" + manifest = { + "run_id": run_id, + "name": expected["name"], + "state": run_status, + "correctness": correctness, + "plan_hash": plan_hash, + "environment": _environment(), + } + if run_status == "failed": + manifest["failure"] = { + "code": "campaign_failed", + "message": "campaign execution failed; inspect persisted evidence", + "infrastructure": True, + } + plan = { + "plan_hash": plan_hash, + "estimates": { + "cell_count": expected["cells"], + "trial_batch_count": expected["batches"], + "issued_operation_request_count": expected["requests"], + }, + "effective_environment": {"client_cohort": "product_cli"}, + "cells": [ + { + "protocol": { + "warmups": expected["warmups"], + "measured_trials": expected["measured"], + }, + "operation": {"cell": {"workspace_profile": "paper-100m"}}, + } + for _ in range(expected["cells"]) + ], + } + report = { + "run_id": run_id, + "state": run_status, + "correctness_verdict": correctness, + "cells": [ + { + "counts": { + "warmup": expected["warmups"], + "measured_attempted": expected["measured"], + "successful": ( + expected["measured"] if run_status == "completed" else 0 + ), + "product_failed": 0, + "correctness_failed": 0, + "infrastructure_failed": (0 if run_status == "completed" else 1), + "cleanup_invalid": 0, + "missing_primary_latency": (0 if run_status == "completed" else 1), + }, + "checks": [], + } + for _ in range(expected["cells"]) + ], + "warnings": ( + [] + if run_status == "completed" + else [ + { + "code": "missing_correctness_observations", + "message": "A measured-trial verdict is missing.", + } + ] + ), + } + intent = {"name": expected["name"]} + root.mkdir(parents=True) + for name, schema, data in ( + ("run-manifest.json", "eos_benchmark_run_manifest", manifest), + ("expanded-plan.json", "eos_benchmark_expanded_plan", plan), + ("report.json", "eos_benchmark_report", report), + ("intent-plan.json", "eos_benchmark_intent_plan", intent), + ): + archive_exp1_run.write_json(root / name, _envelope(schema, data)) + return run_id, plan_hash + + +def test_archive_verifier_preserves_run_start_provenance_exactly( + tmp_path: Path, +) -> None: + root, _ = _archive(tmp_path / "archive") + + result = archive_exp1_run.verify_archive(root) + + assert result["verified"] is True + + +def test_failed_archive_is_explicitly_ineligible_and_preserves_report( + tmp_path: Path, +) -> None: + root, _ = _archive( + tmp_path / "archive", + disposition="smoke", + run_status="failed", + ) + + result = archive_exp1_run.verify_archive(root) + + assert result["verified"] is True + manifest = archive_exp1_run.load_json(root / "archive-manifest.json") + campaign = archive_exp1_run.load_json(root / "campaign-manifest.json") + assert manifest["run_status"] == "failed" + assert manifest["eligibility"] == "failed_ineligible" + assert campaign["eligibility"] == "failed_ineligible" + assert campaign["benchmark_source"]["capture_boundary"] == ( + "captured after terminal cleanup and after the failed-corpus " + "archival-tool amendment; archived benchmark source is post-run " + "preservation code and must not be interpreted as byte-identical " + "run-time source" + ) + assert campaign["product"]["commit"] == "b" * 40 + assert campaign["product"]["recorded_treatment"]["source_dirty"] is False + assert campaign["product"]["post_run_checkout"] == { + "branch": "main", + "capture_boundary": ( + "post-run checkout HEAD and status were captured after terminal " + "cleanup; they may reflect a subsequent corrective amendment and " + "are not the at-run treatment identity" + ), + "commit": "e" * 40, + "dirty": True, + "status_porcelain": " M crates/fix.rs", + } + assert (root / "raw/report.json").read_bytes() == ( + root / "report.json" + ).read_bytes() + + +def test_failed_archive_cannot_be_promoted_by_manifest_edit( + tmp_path: Path, +) -> None: + run_id = "019fb000-0000-7000-8000-000000000000" + root, _ = _archive( + tmp_path / "archive", + disposition="final", + run_status="failed", + ) + archive_manifest = archive_exp1_run.load_json(root / "archive-manifest.json") + archive_manifest["run_status"] = "completed" + archive_manifest["eligibility"] = "frozen_final_candidate" + archive_exp1_run.write_json(root / "archive-manifest.json", archive_manifest) + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="terminal status or eligibility provenance is invalid", + ): + archive_exp1_run.verify_archive(root) + + _refresh_inventory(root, run_id, "final", "failed") + assert archive_exp1_run.verify_archive(root)["verified"] is True + + +def test_failed_archive_verifier_still_rejects_cleanup_leak( + tmp_path: Path, +) -> None: + run_id = "019fb000-0000-7000-8000-000000000000" + root, _ = _archive( + tmp_path / "archive", + disposition="smoke", + run_status="failed", + ) + cleanup = archive_exp1_run.load_json( + root / "cleanup/cleanup-proof.json" + ) + cleanup["run_labeled_containers"] = ["leaked-container"] + archive_exp1_run.write_json( + root / "cleanup/cleanup-proof.json", cleanup + ) + preflight = archive_exp1_run.load_json( + root / "environment-preflight.txt" + ) + preflight["cleanup"] = cleanup + archive_exp1_run.write_json( + root / "environment-preflight.txt", preflight + ) + campaign = archive_exp1_run.load_json(root / "campaign-manifest.json") + campaign["cleanup"] = cleanup + archive_exp1_run.write_json(root / "campaign-manifest.json", campaign) + _refresh_inventory(root, run_id, "smoke", "failed") + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="post-run cleanup proof failed", + ): + archive_exp1_run.verify_archive(root) + + +def test_completed_archive_verifier_rejects_post_run_product_drift( + tmp_path: Path, +) -> None: + run_id = "019fb000-0000-7000-8000-000000000000" + root, _ = _archive(tmp_path / "archive") + cleanup = archive_exp1_run.load_json( + root / "cleanup/cleanup-proof.json" + ) + cleanup["product_commit"] = "e" * 40 + cleanup["product_status_porcelain"] = " M crates/fix.rs" + archive_exp1_run.write_json( + root / "cleanup/cleanup-proof.json", cleanup + ) + preflight = archive_exp1_run.load_json( + root / "environment-preflight.txt" + ) + preflight["cleanup"] = cleanup + preflight["product"]["post_run_checkout"].update( + { + "commit": cleanup["product_commit"], + "status_porcelain": cleanup["product_status_porcelain"], + "dirty": True, + } + ) + archive_exp1_run.write_json( + root / "environment-preflight.txt", preflight + ) + campaign = archive_exp1_run.load_json(root / "campaign-manifest.json") + campaign["cleanup"] = cleanup + campaign["product"] = preflight["product"] + archive_exp1_run.write_json(root / "campaign-manifest.json", campaign) + _refresh_inventory(root, run_id, "exploratory", "completed") + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="completed archive product checkout drift", + ): + archive_exp1_run.verify_archive(root) + + +def test_validate_source_requires_explicit_failed_status( + tmp_path: Path, +) -> None: + source = tmp_path / "source" + run_id, plan_hash = _source_run(source) + + manifest, _, report, _ = archive_exp1_run.validate_source( + source, + run_id=run_id, + disposition="smoke", + run_status="failed", + expected_plan_hash=plan_hash, + ) + + assert manifest["failure"]["code"] == "campaign_failed" + assert report["warnings"] + with pytest.raises( + archive_exp1_run.ArchiveError, + match="does not match requested archive status", + ): + archive_exp1_run.validate_source( + source, + run_id=run_id, + disposition="smoke", + run_status="completed", + expected_plan_hash=plan_hash, + ) + + +def _write_committed_cli_evidence(source: Path) -> tuple[Path, Path]: + invocation_id = "a" * 64 + cli = source / "cli-subprocesses" + cli.mkdir(parents=True) + stdout = b'{"sandboxes":[]}\n' + stderr = b"" + stdout_path = cli / f"{invocation_id}.stdout" + stderr_path = cli / f"{invocation_id}.stderr" + metadata_path = cli / f"{invocation_id}.json" + stdout_path.write_bytes(stdout) + stderr_path.write_bytes(stderr) + archive_exp1_run.write_json( + metadata_path, + { + "schema_version": 2, + "invocation_id": invocation_id, + "request_id": "request-1", + "return_code": 0, + "response_validation": "passed", + "sanitized_argv": ["--gateway-auth-token=[REDACTED]"], + "stdout_path": f"cli-subprocesses/{invocation_id}.stdout", + "stderr_path": f"cli-subprocesses/{invocation_id}.stderr", + "stdout_bytes": len(stdout), + "stderr_bytes": len(stderr), + "stdout_sha256": f"sha256:{archive_exp1_run.hashlib.sha256(stdout).hexdigest()}", + "stderr_sha256": f"sha256:{archive_exp1_run.hashlib.sha256(stderr).hexdigest()}", + "stdout_base64": base64.b64encode(stdout).decode("ascii"), + "stderr_base64": base64.b64encode(stderr).decode("ascii"), + "evidence_commit": "metadata-packed-payload-fsync-v1", + }, + ) + return metadata_path, stdout_path + + +def test_cli_evidence_v2_commit_validates_payload_integrity( + tmp_path: Path, +) -> None: + source = tmp_path / "source" + metadata_path, _ = _write_committed_cli_evidence(source) + + summary = archive_exp1_run.inspect_cli_evidence(source) + metadata = archive_exp1_run.load_json(metadata_path) + + assert summary["invocation_count"] == 1 + assert summary["return_codes"] == {"0": 1} + assert summary["auth_redaction"] == "passed" + assert archive_exp1_run.decode_cli_evidence_payload( + metadata, "stdout" + ) == b'{"sandboxes":[]}\n' + assert archive_exp1_run.decode_cli_evidence_payload(metadata, "stderr") == b"" + + +@pytest.mark.parametrize( + "mutation", + ["missing_payload", "changed_payload", "changed_packed_payload", "bad_marker"], +) +def test_cli_evidence_v2_rejects_uncommitted_or_corrupt_payloads( + tmp_path: Path, + mutation: str, +) -> None: + source = tmp_path / "source" + metadata_path, stdout_path = _write_committed_cli_evidence(source) + if mutation == "missing_payload": + stdout_path.unlink() + expected = "payload" + elif mutation == "changed_payload": + stdout_path.write_bytes(b"changed\n") + expected = "integrity" + elif mutation == "changed_packed_payload": + metadata = archive_exp1_run.load_json(metadata_path) + metadata["stdout_base64"] = base64.b64encode(b"changed\n").decode("ascii") + archive_exp1_run.write_json(metadata_path, metadata) + expected = "integrity" + else: + metadata = archive_exp1_run.load_json(metadata_path) + metadata["evidence_commit"] = "unknown" + archive_exp1_run.write_json(metadata_path, metadata) + expected = "commit marker" + + with pytest.raises(archive_exp1_run.ArchiveError, match=expected): + archive_exp1_run.inspect_cli_evidence(source) + + +def test_failed_observation_and_cli_evidence_may_be_partial( + tmp_path: Path, +) -> None: + source = tmp_path / "source" + cli = source / "cli-subprocesses" + cli.mkdir(parents=True) + records = [ + { + "record": "request", + "data": {"request_id": "request-1"}, + }, + {"record": "trial", "data": {}}, + {"record": "operation", "data": {}}, + ] + with (source / "observations.ndjson").open( + "w", encoding="utf-8", newline="\n" + ) as stream: + for sequence, record in enumerate(records, 1): + stream.write( + json.dumps( + {"data": {"sequence": sequence, "record": record}}, + sort_keys=True, + ) + + "\n" + ) + archive_exp1_run.write_json( + cli / "request-1.json", + { + "request_id": "request-1", + "return_code": 1, + "response_validation": "failed", + "stderr_bytes": 17, + "sanitized_argv": ["--gateway-auth-token=[REDACTED]"], + }, + ) + + observations, resources = archive_exp1_run.inspect_observations( + source, + expected_requests=55, + expected_batches=19, + allow_partial=True, + ) + cli_summary = archive_exp1_run.inspect_cli_evidence(source, allow_failures=True) + + assert observations["request_id_count"] == 1 + assert resources == [] + assert cli_summary["return_codes"] == {"1": 1} + assert cli_summary["response_validation"] == {"failed": 1} + with pytest.raises(archive_exp1_run.ArchiveError): + archive_exp1_run.inspect_observations( + source, + expected_requests=55, + expected_batches=19, + ) + with pytest.raises(archive_exp1_run.ArchiveError): + archive_exp1_run.inspect_cli_evidence(source) + + +def test_archive_verifier_rejects_post_run_limit_substitution( + tmp_path: Path, +) -> None: + root, environment = _archive(tmp_path / "archive") + preflight = archive_exp1_run.load_json(root / "environment-preflight.txt") + preflight["sandbox_limits"] = { + **environment["sandbox_limits"], + "memory_bytes": 0, + } + archive_exp1_run.write_json(root / "environment-preflight.txt", preflight) + _refresh_inventory(root, "019fb000-0000-7000-8000-000000000000") + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="did not preserve run-start provenance exactly", + ): + archive_exp1_run.verify_archive(root) + + +def test_archive_validation_rejects_missing_final_host_fields() -> None: + environment = _environment() + del environment["host"]["cpu_model"] + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="final-host text evidence is incomplete", + ): + archive_exp1_run.validate_exp1_environment(environment) + + +def test_final_archive_requires_v1_protocol_and_annotated_product_tag( + tmp_path: Path, +) -> None: + root, _ = _archive(tmp_path / "archive", disposition="final") + + result = archive_exp1_run.verify_archive(root) + + assert result["verified"] is True + + +def test_product_freeze_tag_requires_annotated_tag_peeling_to_commit( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + commit = "a" * 40 + tag_object = "b" * 40 + + def git_value(root: Path, *args: str) -> str: + if args[0] == "rev-parse" and args[1].endswith("^{tag}"): + return tag_object + if args[:2] == ("cat-file", "-t"): + return "tag" + if args[0] == "rev-parse" and args[1].endswith("^{}"): + return commit + raise AssertionError(args) + + monkeypatch.setattr(archive_exp1_run, "git_value", git_value) + + value = archive_exp1_run.product_freeze_tag( + tmp_path, disposition="final", product_commit=commit + ) + + assert value["name"] == "paper-v1-freeze" + assert value["tag_object"] == tag_object + assert value["peeled_commit"] == commit + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="does not peel to the measured product commit", + ): + archive_exp1_run.product_freeze_tag( + tmp_path, disposition="final", product_commit="c" * 40 + ) + + +def test_v11_requires_safe_unique_named_pipe_per_execution_block() -> None: + environment = _environment() + environment.update( + gateway_endpoint_identity=( + "isolated_windows_named_pipe_per_execution_block" + ), + gateway_transport=dict(archive_exp1_run.V11_GATEWAY_TRANSPORT), + ) + plan = { + "execution_blocks": [ + {"block_id": "block-1", "family_id": "create"}, + {"block_id": "block-2", "family_id": "runtime"}, + ] + } + manifest = { + "gateway_policy": { + "protocol_version": archive_exp1_run.PROTOCOLS["v1.1"]["id"], + "mode": "isolated", + "isolated_runtime_per_execution_block": True, + "loopback_only": False, + **archive_exp1_run.V11_GATEWAY_TRANSPORT, + }, + "gateway_execution_blocks": [ + { + "block_id": "block-1", + "family_id": "create", + "gateway_instance_id": "gateway-1", + "endpoint_uri": "npipe://./pipe/eos-exp1-block-1", + **archive_exp1_run.V11_GATEWAY_TRANSPORT, + }, + { + "block_id": "block-2", + "family_id": "runtime", + "gateway_instance_id": "gateway-2", + "endpoint_uri": "npipe://./pipe/eos-exp1-block-2", + **archive_exp1_run.V11_GATEWAY_TRANSPORT, + }, + ], + } + + archive_exp1_run.validate_protocol_transport( + protocol_version="v1.1", + environment=environment, + manifest=manifest, + plan=plan, + completed=True, + ) + + manifest["gateway_execution_blocks"][1]["endpoint_uri"] = ( + "npipe://./pipe/eos-exp1-block-1" + ) + with pytest.raises( + archive_exp1_run.ArchiveError, + match="execution-block endpoint evidence is unsafe", + ): + archive_exp1_run.validate_protocol_transport( + protocol_version="v1.1", + environment=environment, + manifest=manifest, + plan=plan, + completed=True, + ) + + +def test_new_archive_requires_explicit_protocol_version( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + monkeypatch.setattr( + "sys.argv", + [ + "archive_exp1_run.py", + "--run-id", + "run", + "--disposition", + "smoke", + "--expected-plan-hash", + "sha256:" + "a" * 64, + "--paper-root", + ".", + "--product-root", + ".", + "--product-bin-dir", + ".", + "--product-archive", + "package.zip", + "--image", + "image", + ], + ) + + assert archive_exp1_run.main() == 1 + assert "protocol_version" in capsys.readouterr().err + with pytest.raises(SystemExit): + archive_exp1_run.parser().parse_args(["--protocol-version", "v1.0"]) + + +def test_v11_final_freeze_uses_annotated_v11_tag( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + commit = "a" * 40 + + def git_value(root: Path, *args: str) -> str: + if args[0] == "rev-parse" and args[1].endswith("^{tag}"): + assert "paper-v1.1-freeze" in args[1] + return "b" * 40 + if args[:2] == ("cat-file", "-t"): + return "tag" + assert "paper-v1.1-freeze" in args[1] + return commit + + monkeypatch.setattr(archive_exp1_run, "git_value", git_value) + value = archive_exp1_run.product_freeze_tag( + tmp_path, + disposition="final", + product_commit=commit, + protocol_version="v1.1", + ) + assert value["name"] == "paper-v1.1-freeze" + + +def test_final_paper_git_provenance_requires_clean_frozen_scope( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + def git_value(root: Path, *args: str) -> str: + if args[:2] == ("rev-parse", "--show-toplevel"): + return str(tmp_path) + if args[:2] == ("status", "--porcelain=v1"): + assert "benchmark" in args + assert "progress.md" in args + assert "experiment_inventory.md" in args + assert "experiments/exp1-v1.1-protocol-amendment.md" in args + assert "experiments/environment_setup.md" in args + assert "experiments/experiment_log.md" in args + assert "paper_state.json" in args + assert "plan/progress.md" in args + assert "experiments/scripts/project_exp1_final_runtime.py" in args + assert "experiments/analysis/scripts/generate_exp1_tables.py" in args + assert ":(exclude,glob)benchmark/**/*.pyc" in args + return " M benchmark/backend/benchmark_lab/metadata.py" + raise AssertionError(args) + + monkeypatch.setattr(archive_exp1_run, "git_value", git_value) + + with pytest.raises( + archive_exp1_run.ArchiveError, + match="not a clean frozen commit", + ): + archive_exp1_run.paper_git_provenance(tmp_path, disposition="final") + + +def test_v11_archive_binds_protocol_live_state_and_analysis_sources() -> None: + assert archive_exp1_run.PAPER_PROTOCOL_PATHS == ( + "progress.md", + "plan/task-packets/exp1-cli-performance-campaign.md", + "experiment_inventory.md", + "experiments/exp1-v1.1-protocol-amendment.md", + "experiments/environment_setup.md", + "experiments/expected_tables.md", + "experiments/experiment_log.md", + "benchmark/PAPER_ARTIFACT.md", + "paper_state.json", + "plan/progress.md", + ) + assert archive_exp1_run.PAPER_ANALYSIS_PATHS == ( + "benchmark/backend/benchmark_lab/derivation.py", + "benchmark/backend/benchmark_lab/reports.py", + "experiments/scripts/archive_exp1_run.py", + "experiments/scripts/project_exp1_final_runtime.py", + "experiments/analysis/scripts/generate_exp1_tables.py", + ) + assert set(archive_exp1_run.PAPER_PROTOCOL_PATHS) <= { + "benchmark/PAPER_ARTIFACT.md", + *archive_exp1_run.PAPER_FROZEN_SCOPE, + } + assert set(archive_exp1_run.PAPER_ANALYSIS_PATHS) <= { + "benchmark/backend/benchmark_lab/derivation.py", + "benchmark/backend/benchmark_lab/reports.py", + *archive_exp1_run.PAPER_FROZEN_SCOPE, + } + + +def test_final_paper_git_ignores_generated_pycache_but_blocks_source( + tmp_path: Path, +) -> None: + def git(*args: str) -> None: + subprocess.run( + ["git", *args], + cwd=tmp_path, + check=True, + capture_output=True, + ) + + (tmp_path / "benchmark/pkg/__pycache__").mkdir(parents=True) + source = tmp_path / "benchmark/pkg/source.py" + bytecode = tmp_path / "benchmark/pkg/__pycache__/source.pyc" + source.write_text("VALUE = 1\n", encoding="utf-8") + bytecode.write_bytes(b"initial bytecode") + git("init", "-q") + git("config", "user.name", "EXP1 Test") + git("config", "user.email", "exp1@example.invalid") + git("add", ".") + git("commit", "-q", "-m", "fixture") + + bytecode.write_bytes(b"generated bytecode drift") + provenance = archive_exp1_run.paper_git_provenance(tmp_path, disposition="final") + assert provenance["dirty"] is False + assert ":(exclude,glob)benchmark/**/*.pyc" in provenance["generated_exclusions"] + + source.write_text("VALUE = 2\n", encoding="utf-8") + with pytest.raises( + archive_exp1_run.ArchiveError, + match="not a clean frozen commit", + ): + archive_exp1_run.paper_git_provenance(tmp_path, disposition="final") diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_runtime_projection.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_runtime_projection.py new file mode 100644 index 0000000..8a738b9 --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_exp1_runtime_projection.py @@ -0,0 +1,463 @@ +import copy +import importlib.util +import json +import sys +from pathlib import Path + +import pytest + +PAPER_ROOT = Path(__file__).resolve().parents[4] +SCRIPT = PAPER_ROOT / "experiments/scripts/project_exp1_final_runtime.py" +SPEC = importlib.util.spec_from_file_location("project_exp1_final_runtime_test", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +projection = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = projection +SPEC.loader.exec_module(projection) + +SMOKE_ARCHIVE = PAPER_ROOT / "experiments/runs/019fb5e4-3f62-7760-bc3f-e7501502ec74" +PILOT_ARCHIVE = PAPER_ROOT / "experiments/runs/019fb5f1-d73a-7128-9bab-d75dd229c020" +V11_SMOKE_ARCHIVE = PAPER_ROOT / "experiments/runs/019fb83a-54bc-79db-b6ac-6189fb28f5f2" +V11_PILOT_ARCHIVE = PAPER_ROOT / "experiments/runs/019fb84e-aef1-7fdc-9a56-1adbe712f30d" +V11_FINAL_PLAN = PAPER_ROOT / "tmp/validate-paper-good-pass-v11-20260731T1327Z.json" + + +def test_v11_projection_rejects_unsafe_named_pipe_endpoint() -> None: + campaign = { + "protocol": { + "version": "v1.1", + "id": projection.PROTOCOLS["v1.1"]["id"], + } + } + manifest = { + "environment": { + "gateway_endpoint_identity": ( + "isolated_windows_named_pipe_per_execution_block" + ), + "gateway_transport": dict(projection.V11_GATEWAY_TRANSPORT), + }, + "gateway_policy": { + "protocol_version": projection.PROTOCOLS["v1.1"]["id"], + "mode": "isolated", + "isolated_runtime_per_execution_block": True, + "loopback_only": False, + **projection.V11_GATEWAY_TRANSPORT, + }, + "gateway_execution_blocks": [ + { + "block_id": "block-1", + "family_id": "runtime", + "gateway_instance_id": "gateway-1", + "endpoint_uri": "npipe://./pipe/../unsafe", + **projection.V11_GATEWAY_TRANSPORT, + } + ], + } + plan = {"execution_blocks": [{"block_id": "block-1", "family_id": "runtime"}]} + + with pytest.raises( + projection.ProjectionError, + match="execution-block endpoint evidence is unsafe", + ): + projection._validate_protocol_transport(campaign, manifest, plan, "v1.1") + + +def _plan(role: str) -> dict: + repetitions = { + "smoke": (0, 1, 19, 55, "paper-env-smoke"), + "pilot": (2, 5, 133, 385, "paper-pilot"), + "final": (2, 100, 1_938, 5_610, "paper-good-pass"), + } + warmups, measured, batches, requests, name = repetitions[role] + cells = [] + cell_ids = [] + for index in range(19): + cell_id = f"cell-{index:02d}" + cell_ids.append(cell_id) + cells.append( + { + "cell_id": cell_id, + "family_id": "family", + "operation_id": f"operation-{index:02d}", + "operation_semantic_revision": 1, + "factor_schema_revision": 1, + "comparison_key": {"index": index}, + "operation": {"kind": "synthetic", "index": index}, + "protocol": { + "destructive": False, + "warmups": warmups, + "measured_trials": measured, + "timeout_ms": 120_000, + "cleanup": "synthetic-cleanup", + }, + } + ) + return { + "schema_version": 1, + "runnable": True, + "is_customized": False, + "plan_hash": f"sha256:{role}", + "canonical_plan": { + "name": name, + "configuration_base": {}, + "seed": 1, + "environment": {}, + "operations": [], + "protocol": { + "order": "randomized_blocks", + "resource_interval_ms": 100, + "timeout_ms": {"default": 120_000}, + "trial_defaults": { + "fast": { + "warmups": warmups, + "measured_trials": measured, + }, + "destructive": { + "warmups": warmups, + "measured_trials": measured, + }, + }, + }, + }, + "effective_environment": {"client_cohort": "product_cli"}, + "fixed_lifecycle_policy": {"scope": "synthetic"}, + "selected_workspace_profiles": [], + "cells": cells, + "execution_blocks": [ + { + "block_id": "block-family", + "family_id": "family", + "cell_ids": cell_ids, + "restart_reason": None, + } + ], + "estimates": { + "cell_count": 19, + "trial_batch_count": batches, + "issued_operation_request_count": requests, + }, + "validation": [], + } + + +def _trials( + prefix: str, + active_ns: list[int], + warmups: int, + gaps_ns: list[int], +) -> tuple: + trials = [] + cursor = 0 + for index, active in enumerate(active_ns): + trials.append( + projection.TrialSpan( + trial_id=f"{prefix}-{index}", + warmup=index < warmups, + start_ns=cursor, + end_ns=cursor + active, + phase_ns={phase: 0 for phase in projection.PHASES}, + ) + ) + if index < len(gaps_ns): + cursor += active + gaps_ns[index] + return tuple(trials) + + +def _protocol_file(path: str, marker: str) -> dict: + return { + "path": path, + "bytes": len(marker), + "sha256": f"sha256:{marker * 64}", + } + + +def _v11_provenance(marker: str = "a") -> dict: + return { + "same": True, + "freeze_state": { + "protocol": "pre_freeze", + "paper_git": "pre_freeze_worktree", + }, + "protocol_files": [ + _protocol_file(path, marker) + for path in sorted(projection.V11_PROTOCOL_FILE_PATHS) + ], + } + + +def _profile(role: str) -> projection.RunProfile: + raw_plan = _plan(role) + semantics = projection._validate_plan(raw_plan, role=role) + cells = {} + for semantic in semantics["cells"]: + cell = semantics["cell_data"][semantic] + if role == "smoke": + active, warmups, gaps = [5], 0, [] + leading, trailing = 10, 5 + else: + active, warmups, gaps = ( + [7, 11, 13, 17, 19, 23, 29], + 2, + [ + 2, + 3, + 5, + 7, + 11, + 14, + ], + ) + leading, trailing = 12, 4 + cells[semantic] = projection.CellProfile( + semantic_key=semantic, + semantic_sha256=projection.canonical_sha256(json.loads(semantic)), + family_id="family", + operation_id=cell["operation_id"], + leading_ns=leading, + trailing_ns=trailing, + trials=_trials(cell["operation_id"], active, warmups, gaps), + gaps_ns=tuple(gaps), + ) + return projection.RunProfile( + role=role, + identity={ + "run_id": role, + "protocol_version": "v1.1", + "provenance": _v11_provenance(), + }, + plan=semantics, + elapsed_ns=1_000 if role == "smoke" else 900, + run_residual_ns=100 if role == "smoke" else 101, + family_residual_ns={"family": 50 if role == "smoke" else 53}, + cells=cells, + ) + + +def _reviewed_final_plan_from_pilot() -> dict: + final = copy.deepcopy( + projection.envelope_data( + PILOT_ARCHIVE / "raw/expanded-plan.json", + "eos_benchmark_expanded_plan", + ) + ) + final["canonical_plan"]["name"] = "paper-good-pass" + defaults = final["canonical_plan"]["protocol"]["trial_defaults"] + for trial_class in defaults.values(): + trial_class["warmups"] = 2 + trial_class["measured_trials"] = 100 + for cell in final["cells"]: + cell["protocol"]["warmups"] = 2 + cell["protocol"]["measured_trials"] = 100 + final["estimates"]["cell_count"] = 19 + final["estimates"]["trial_batch_count"] = 1_938 + final["estimates"]["issued_operation_request_count"] = 5_610 + final["plan_hash"] = projection.canonical_sha256(final) + return final + + +def test_cross_run_allows_only_prefreeze_status_hash_evolution(): + smoke = _profile("smoke") + pilot = _profile("pilot") + for entry in pilot.identity["provenance"]["protocol_files"]: + if entry["path"] in projection.V11_PREFREEZE_MUTABLE_STATUS_PATHS: + entry.update(_protocol_file(entry["path"], "b")) + + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +def test_cross_run_rejects_scientific_protocol_hash_drift(): + smoke = _profile("smoke") + pilot = _profile("pilot") + path = "experiments/exp1-v1.1-protocol-amendment.md" + for entry in pilot.identity["provenance"]["protocol_files"]: + if entry["path"] == path: + entry.update(_protocol_file(path, "b")) + + with pytest.raises( + projection.ProjectionError, + match="provenance identities drifted", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +def test_cross_run_rejects_invalid_or_duplicate_protocol_identity(): + smoke = _profile("smoke") + pilot = _profile("pilot") + ledger = smoke.identity["provenance"]["protocol_files"][0] + smoke.identity["provenance"]["protocol_files"].append(copy.deepcopy(ledger)) + pilot.identity["provenance"] = copy.deepcopy(smoke.identity["provenance"]) + + with pytest.raises( + projection.ProjectionError, + match="protocol-file identity is invalid", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +@pytest.mark.parametrize("invalid", [None, [], "not-a-list"]) +def test_cross_run_rejects_missing_protocol_file_identity(invalid): + smoke = _profile("smoke") + pilot = _profile("pilot") + smoke.identity["provenance"]["protocol_files"] = invalid + pilot.identity["provenance"]["protocol_files"] = copy.deepcopy(invalid) + + with pytest.raises( + projection.ProjectionError, + match="protocol-file identity", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +def test_cross_run_rejects_extra_protocol_file_identity(): + smoke = _profile("smoke") + pilot = _profile("pilot") + extra = _protocol_file("unexpected.md", "a") + smoke.identity["provenance"]["protocol_files"].append(extra) + pilot.identity["provenance"]["protocol_files"].append(copy.deepcopy(extra)) + + with pytest.raises( + projection.ProjectionError, + match="protocol-file identity set is invalid", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +def test_cross_run_rejects_non_prefreeze_status(): + smoke = _profile("smoke") + pilot = _profile("pilot") + smoke.identity["provenance"]["freeze_state"]["protocol"] = "frozen" + pilot.identity["provenance"] = copy.deepcopy(smoke.identity["provenance"]) + + with pytest.raises( + projection.ProjectionError, + match="freeze-state identity is invalid", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +@pytest.mark.parametrize( + "category", + [ + "benchmark_source", + "analysis_and_archiving_code", + "artifact_schemas", + "definition_snapshot", + "docker", + "fixture", + "gateway", + "host", + "image", + "lifecycle", + "product", + "sandbox_limits", + "treatment", + ], +) +def test_cross_run_rejects_non_status_provenance_drift(category): + smoke = _profile("smoke") + pilot = _profile("pilot") + smoke.identity["provenance"][category] = {"identity": "same"} + pilot.identity["provenance"][category] = {"identity": "different"} + + with pytest.raises( + projection.ProjectionError, + match="provenance identities drifted", + ): + projection.validate_cross_run(smoke, pilot, _plan("final")) + + +def test_projection_archive_rejects_non_prefreeze_campaign(): + campaign = { + "protocol": { + "freeze_state": "frozen", + "files": [ + _protocol_file(path, "a") + for path in sorted(projection.V11_PROTOCOL_FILE_PATHS) + ], + }, + "paper_git": {"freeze_state": "frozen_worktree"}, + } + + with pytest.raises( + projection.ProjectionError, + match="not in the pre-freeze state", + ): + projection._validate_prefreeze_campaign(campaign, "v1.1") + + +def test_structural_projection_uses_exact_arithmetic_and_fixed_units_once(): + result = projection.project_structural( + _profile("smoke"), + _profile("pilot"), + _plan("final"), + final_plan_sha256="sha256:final-file", + script_sha256="sha256:script", + ) + + assert result["decomposition"]["run_fixed_ns"] == 101 + assert result["decomposition"]["family_fixed_ns"] == {"family": 53} + assert {cell["fixed"]["total_ns"] for cell in result["decomposition"]["cells"]} == { + 17 + } + assert result["models"]["central_structural"] == { + "numerator_ns": 226_773, + "denominator": 4, + "ceil_ns": 56_694, + } + assert result["models"]["observed_envelope_ns"] == 82_861 + assert result["gate_3_runtime_pass"] is True + assert projection.render_json(result) == projection.render_json(result) + + +def test_v11_smoke_and_pilot_archives_pass_reviewed_runtime_gate(): + smoke = projection.load_run_profile( + V11_SMOKE_ARCHIVE, + "smoke", + protocol_version="v1.1", + ) + pilot = projection.load_run_profile( + V11_PILOT_ARCHIVE, + "pilot", + protocol_version="v1.1", + ) + result = projection.project_structural( + smoke, + pilot, + projection.load_json(V11_FINAL_PLAN), + final_plan_sha256=projection.sha256_file(V11_FINAL_PLAN), + script_sha256="sha256:test-script", + ) + + assert result["display_seconds"] == { + "pilot_elapsed": "276.094047000", + "central_structural": "1179.784426150", + "observed_envelope": "1303.732241600", + "limit": "1400.000000000", + } + assert result["gate_3_runtime_pass"] is True + assert result["decision"] == "pass_runtime_projection" + + +def test_accepted_smoke_and_pilot_archives_still_fail_runtime_gate(): + smoke = projection.load_run_profile(SMOKE_ARCHIVE, "smoke") + pilot = projection.load_run_profile(PILOT_ARCHIVE, "pilot") + result = projection.project_structural( + smoke, + pilot, + _reviewed_final_plan_from_pilot(), + final_plan_sha256="sha256:reviewed-final-fixture", + script_sha256="sha256:test-script", + ) + + assert result["display_seconds"] == { + "pilot_elapsed": "609.215958000", + "central_structural": "2460.664462900", + "observed_envelope": "2745.098529500", + "limit": "1400.000000000", + } + assert result["pass_conditions"] == { + "pilot_elapsed_within_limit": True, + "central_structural_within_limit": False, + "observed_envelope_within_limit": False, + } + assert result["gate_3_runtime_pass"] is False + assert result["decision"].startswith("block_freeze") diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_fixtures.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_fixtures.py index 91a169f..d574014 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_fixtures.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_fixtures.py @@ -1,10 +1,18 @@ import json +import os +import shutil +import subprocess from pathlib import Path +from threading import Event, Lock import pytest import benchmark_lab.fixtures as fixtures -from benchmark_lab.fixtures import FixtureError, materialize_workspace +from benchmark_lab.fixtures import ( + FixtureError, + materialize_workspace, + materialize_workspaces, +) def _profile() -> dict[str, object]: @@ -28,16 +36,252 @@ def test_materialized_fixture_is_deterministic_and_exact(tmp_path: Path) -> None reused = materialize_workspace(cache, second, _profile(), 41) assert manifest == reused - first_files = sorted(path.relative_to(first) for path in first.rglob("*") if path.is_file()) - second_files = sorted(path.relative_to(second) for path in second.rglob("*") if path.is_file()) + first_files = sorted( + path.relative_to(first) for path in first.rglob("*") if path.is_file() + ) + second_files = sorted( + path.relative_to(second) for path in second.rglob("*") if path.is_file() + ) assert first_files == second_files assert len(first_files) == 8 # seven payloads plus the versioned manifest - assert sum( - (first / path).stat().st_size + assert ( + sum( + (first / path).stat().st_size + for path in first_files + if path.name != "fixture-manifest.json" + ) + == 101 + ) + assert all( + (first / path).read_bytes() == (second / path).read_bytes() for path in first_files - if path.name != "fixture-manifest.json" - ) == 101 - assert all((first / path).read_bytes() == (second / path).read_bytes() for path in first_files) + ) + + +def test_stable_cache_content_hash_is_validated_once( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + cache = tmp_path / "fixtures" + first = tmp_path / "first" + second = tmp_path / "second" + first.mkdir() + second.mkdir() + calls = 0 + hash_tree = fixtures._fixture_tree_hash + + def count_hashes(root: Path, payloads: list[tuple[Path, int]]) -> str: + nonlocal calls + calls += 1 + return hash_tree(root, payloads) + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr(fixtures, "_fixture_tree_hash", count_hashes) + + materialize_workspace(cache, first, _profile(), 41) + materialize_workspace(cache, second, _profile(), 41) + + assert calls == 1 + + +def test_batch_materialization_is_exact_independent_and_validates_cache_once( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + cache = tmp_path / "fixtures" + destinations = tuple(tmp_path / f"workspace-{index}" for index in range(3)) + for destination in destinations: + destination.mkdir() + validations = 0 + validate_cache = fixtures._validated_fixture_cache + + def count_validations( + source: Path, + profile: dict[str, object], + manifest: dict[str, object], + ) -> fixtures._TreeInventory: + nonlocal validations + validations += 1 + return validate_cache(source, profile, manifest) + + monkeypatch.setattr(fixtures, "_validated_fixture_cache", count_validations) + + manifest = materialize_workspaces( + cache, destinations, _profile(), 41, max_workers=2 + ) + + assert manifest["actual_file_count"] == 7 + assert validations == 1 + source = next(cache.rglob("fixture-manifest.json")).parent + relative_files = sorted( + path.relative_to(destinations[0]) + for path in destinations[0].rglob("*") + if path.is_file() + ) + assert len(relative_files) == 8 + for relative in relative_files: + paths = [ + source / relative, + *(destination / relative for destination in destinations), + ] + assert len({(path.stat().st_dev, path.stat().st_ino) for path in paths}) == len( + paths + ) + assert len({path.stat().st_size for path in paths}) == 1 + assert len({path.read_bytes() for path in paths}) == 1 + + +def test_batch_materialization_bounds_concurrent_copies( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + cache = tmp_path / "fixtures" + destinations = tuple(tmp_path / f"workspace-{index}" for index in range(5)) + for destination in destinations: + destination.mkdir() + active = 0 + maximum_active = 0 + lock = Lock() + two_workers_started = Event() + materialize = fixtures._materialize_validated_workspace + + def count_concurrency( + source: Path, + destination: Path, + inventory: fixtures._TreeInventory, + ) -> None: + nonlocal active, maximum_active + with lock: + active += 1 + maximum_active = max(maximum_active, active) + if active == 2: + two_workers_started.set() + try: + assert two_workers_started.wait(timeout=5) + materialize(source, destination, inventory) + finally: + with lock: + active -= 1 + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr(fixtures, "_materialize_validated_workspace", count_concurrency) + + materialize_workspaces(cache, destinations, _profile(), 41, max_workers=2) + + assert maximum_active == 2 + + +def test_batch_materialization_rejects_extra_destination_entry( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workspace = tmp_path / "workspace" + workspace.mkdir() + copy_tree = fixtures._copy_tree_with_python + + def copy_with_extra_entry( + source: Path, + destination: Path, + inventory: fixtures._TreeInventory, + ) -> None: + copy_tree(source, destination, inventory) + (destination / "unexpected.bin").write_bytes(b"unexpected") + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr(fixtures, "_copy_tree_with_python", copy_with_extra_entry) + + with pytest.raises(FixtureError, match="drifted"): + materialize_workspaces( + tmp_path / "fixtures", + (workspace,), + _profile(), + 41, + ) + + +def test_batch_materialization_rejects_hardlinked_destination_files( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workspace = tmp_path / "workspace" + workspace.mkdir() + + def hardlink_tree( + source: Path, + destination: Path, + inventory: fixtures._TreeInventory, + ) -> None: + for relative in sorted( + inventory.directories, + key=lambda value: (value.count("/"), value), + ): + if relative != ".": + (destination / Path(relative)).mkdir(parents=True) + for relative, _ in inventory.files: + os.link(source / Path(relative), destination / Path(relative)) + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr(fixtures, "_copy_tree_with_python", hardlink_tree) + + with pytest.raises(FixtureError, match="not independent"): + materialize_workspaces( + tmp_path / "fixtures", + (workspace,), + _profile(), + 41, + ) + + +@pytest.mark.parametrize("max_workers", [True, 0, 9]) +def test_batch_materialization_rejects_invalid_worker_limit( + tmp_path: Path, max_workers: int +) -> None: + workspace = tmp_path / "workspace" + workspace.mkdir() + + with pytest.raises(FixtureError, match="worker limit"): + materialize_workspaces( + tmp_path / "fixtures", + (workspace,), + _profile(), + 41, + max_workers=max_workers, + ) + + +def test_batch_materialization_rejects_duplicate_workspaces( + tmp_path: Path, +) -> None: + workspace = tmp_path / "workspace" + workspace.mkdir() + + with pytest.raises(FixtureError, match="distinct"): + materialize_workspaces( + tmp_path / "fixtures", + (workspace, workspace), + _profile(), + 41, + ) + + +def test_batch_materialization_revalidates_source_after_worker_failure( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + cache = tmp_path / "fixtures" + workspace = tmp_path / "workspace" + workspace.mkdir() + materialize = fixtures._materialize_validated_workspace + + def drift_source( + source: Path, + destination: Path, + inventory: fixtures._TreeInventory, + ) -> None: + materialize(source, destination, inventory) + payload = next(source.rglob("file-*.bin")) + payload.write_bytes(b"x" * payload.stat().st_size) + raise FixtureError("simulated worker failure") + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr(fixtures, "_materialize_validated_workspace", drift_source) + + with pytest.raises(FixtureError, match="batch materialization"): + materialize_workspaces(cache, (workspace,), _profile(), 41, max_workers=1) def test_fixture_cache_rejects_identity_corruption(tmp_path: Path) -> None: @@ -65,16 +309,227 @@ def test_fixture_copy_rejects_nonempty_workspace(tmp_path: Path) -> None: assert (workspace / "user-file").read_text() == "preserve" -def test_depth_100_is_admitted_and_generates_100_parent_components() -> None: +@pytest.mark.parametrize("returncode", [0, 7]) +def test_windows_copy_accepts_success_codes_and_creates_independent_files( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, returncode: int +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + (source / "nested").mkdir(parents=True) + (source / "nested" / "payload.bin").write_bytes(b"original") + destination.mkdir() + executable = Path("C:/Windows/System32/robocopy.exe") + invocations: list[tuple[list[str], dict[str, object]]] = [] + + def run( + arguments: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[bytes]: + invocations.append((arguments, kwargs)) + shutil.copytree(source, destination, dirs_exist_ok=True) + return subprocess.CompletedProcess(arguments, returncode, b"copy summary", b"") + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: True) + monkeypatch.setattr(fixtures, "_robocopy_executable", lambda: executable) + monkeypatch.setattr(fixtures.subprocess, "run", run) + + fixtures._copy_tree(source, destination) + + assert invocations == [ + ( + [ + os.fspath(executable), + os.fspath(source.resolve(strict=True)), + os.fspath(destination.resolve(strict=True)), + *fixtures._ROBOCOPY_OPTIONS, + ], + { + "shell": False, + "stdin": subprocess.DEVNULL, + "stdout": subprocess.PIPE, + "stderr": subprocess.PIPE, + "timeout": fixtures._ROBOCOPY_TIMEOUT_SECONDS, + "check": False, + }, + ) + ] + assert fixtures._ROBOCOPY_OPTIONS[:7] == ( + "/E", + "/COPY:D", + "/DCOPY:D", + "/R:0", + "/W:0", + "/MT:4", + "/XJ", + ) + copied = destination / "nested" / "payload.bin" + assert not os.path.samefile(source / "nested" / "payload.bin", copied) + copied.write_bytes(b"modified") + assert (source / "nested" / "payload.bin").read_bytes() == b"original" + + +@pytest.mark.parametrize("returncode", [8, 16]) +def test_windows_copy_rejects_robocopy_failure_codes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, returncode: int +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + source.mkdir() + (source / "payload.bin").write_bytes(b"content") + destination.mkdir() + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: True) + monkeypatch.setattr(fixtures, "_robocopy_executable", lambda: Path("robocopy.exe")) + monkeypatch.setattr( + fixtures.subprocess, + "run", + lambda arguments, **kwargs: subprocess.CompletedProcess( + arguments, returncode, b"copy summary", b"copy failure" + ), + ) + + with pytest.raises(FixtureError, match=f"exit code {returncode}"): + fixtures._copy_tree(source, destination) + + +@pytest.mark.parametrize( + ("stdout", "message"), + [(None, "output is missing"), (b"", "summary is missing")], +) +def test_windows_copy_rejects_missing_process_output( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + stdout: bytes | None, + message: str, +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + source.mkdir() + (source / "payload.bin").write_bytes(b"content") + destination.mkdir() + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: True) + monkeypatch.setattr(fixtures, "_robocopy_executable", lambda: Path("robocopy.exe")) + monkeypatch.setattr( + fixtures.subprocess, + "run", + lambda arguments, **kwargs: subprocess.CompletedProcess( + arguments, 1, stdout, b"" + ), + ) + + with pytest.raises(FixtureError, match=message): + fixtures._copy_tree(source, destination) + + +def test_windows_copy_rejects_missing_destination_entries( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + source.mkdir() + (source / "first.bin").write_bytes(b"first") + (source / "second.bin").write_bytes(b"second") + destination.mkdir() + + def run( + arguments: list[str], **kwargs: object + ) -> subprocess.CompletedProcess[bytes]: + shutil.copyfile(source / "first.bin", destination / "first.bin") + return subprocess.CompletedProcess(arguments, 1, b"copy summary", b"") + + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: True) + monkeypatch.setattr(fixtures, "_robocopy_executable", lambda: Path("robocopy.exe")) + monkeypatch.setattr(fixtures.subprocess, "run", run) + + with pytest.raises(FixtureError, match="drifted"): + fixtures._copy_tree(source, destination) + + +def test_validated_cache_rejects_same_size_content_drift( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + fixtures_root = tmp_path / "fixtures" + _, source = fixtures.prepare_workspace_fixture(fixtures_root, _profile(), 41) + first = tmp_path / "first" + second = tmp_path / "second" + first.mkdir() + second.mkdir() + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + materialize_workspace(fixtures_root, first, _profile(), 41) + payload = next(path for path in source.rglob("file-*.bin") if path.is_file()) + state = payload.stat() + payload.write_bytes(b"x" * state.st_size) + os.utime( + payload, + ns=(state.st_atime_ns, state.st_mtime_ns + 1_000_000), + ) + + with pytest.raises(FixtureError, match="drifted after content validation"): + materialize_workspace(fixtures_root, second, _profile(), 41) + + +def test_python_copy_fallback_preserves_identity_without_links( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + (source / "nested").mkdir(parents=True) + (source / "nested" / "payload.bin").write_bytes(b"original") + destination.mkdir() + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: False) + monkeypatch.setattr( + fixtures.subprocess, + "run", + lambda *args, **kwargs: pytest.fail("robocopy must not run off Windows"), + ) + + fixtures._copy_tree(source, destination) + + copied = destination / "nested" / "payload.bin" + assert copied.read_bytes() == b"original" + assert not os.path.samefile(source / "nested" / "payload.bin", copied) + + +def test_copy_rejects_source_symlink_before_process_execution( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + symlink_or_skip, +) -> None: + source = tmp_path / "source" + destination = tmp_path / "destination" + source.mkdir() + target = source / "target.bin" + target.write_bytes(b"content") + symlink_or_skip(source / "linked.bin", target) + destination.mkdir() + monkeypatch.setattr(fixtures, "_uses_native_windows_copy", lambda: True) + monkeypatch.setattr( + fixtures.subprocess, + "run", + lambda *args, **kwargs: pytest.fail( + "unsafe fixture must fail before process execution" + ), + ) + + with pytest.raises(FixtureError, match="symbolic link"): + fixtures._copy_tree(source, destination) + + +def test_depth_100_is_admitted_and_materializes_on_native_host( + tmp_path: Path, +) -> None: profile = _profile() profile["fixture"] = { "file_count": 100, "logical_bytes": 100, "maximum_depth": 100, } + workspace = tmp_path / "workspace" + workspace.mkdir() assert fixtures._validated_fixture_dimensions(profile["fixture"]) == (100, 100, 100) - assert len(fixtures._relative_path(99, 100).parent.parts) == 100 + deepest = fixtures._relative_path(99, 100) + assert len(deepest.parent.parts) == 100 + materialize_workspace(tmp_path / "fixtures", workspace, profile, 41) + assert fixtures.native_filesystem_path(workspace / deepest).is_file() def test_depth_above_supported_limit_is_rejected() -> None: diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_ipc_qualification.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_ipc_qualification.py new file mode 100644 index 0000000..8ad0d1d --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_ipc_qualification.py @@ -0,0 +1,1188 @@ +import argparse +import asyncio +import json +import os +from pathlib import Path +from typing import Any + +import pytest +from benchmark_lab import cli +from benchmark_lab import ipc_qualification as ipc_module +from benchmark_lab.ipc_qualification import ( + BENCHMARK_GIT_EXCLUSIONS, + EXP1_IPC_BATCH_COUNT, + EXP1_IPC_CONCURRENCY, + EXP1_IPC_INVOCATION_COUNT, + MAX_GATEWAY_HANDLE_GROWTH, + MAX_GATEWAY_PRIVATE_BYTES_GROWTH, + MAX_GATEWAY_RSS_BYTES_GROWTH, + PAPER_FROZEN_SCOPE, + PERFORMANCE_EVIDENCE, + PREREGISTERED_ARTIFACT_SHA256, + PREREGISTERED_PACKAGE_SHA256, + PREREGISTERED_PRODUCT_BRANCH, + PREREGISTERED_PRODUCT_COMMIT, + PROCESS_SAMPLE_EVERY_BATCHES, + PRODUCTION_WORKLOAD, + QUALIFICATION_ONLY, + RESOURCE_GROWTH_POLICY_SOURCE, + TCPIP_EVENT_IDS, + EventLogCursor, + GatewayProcessSample, + GatewayTcpSample, + InvocationCapture, + OwnedTcpConnection, + QualificationError, + QualificationWorkload, + TcpipEvent, + _create_evidence_root, + _execute_qualification, + _expected_sanitized_commands, + _validate_capture, + _validate_npipe_endpoint, +) +from benchmark_lab.paths import BenchmarkRoots + + +class FakeRunner: + endpoint = "npipe://./pipe/ephemeral-sandbox-test-qualification" + + def __init__( + self, + *, + failure_index: int | None = None, + raise_index: int | None = None, + stop_override: dict[str, Any] | None = None, + ) -> None: + self.failure_index = failure_index + self.raise_index = raise_index + self.stop_override = stop_override + self.started = 0 + self.stopped = 0 + self.active = 0 + self.peak_active = 0 + self.request_ids: list[str] = [] + + @property + def gateway_pid(self) -> int: + return 4242 + + @property + def executable_paths(self) -> dict[str, str]: + return { + "gateway": "C:\\fake-package\\bin\\sandbox-gateway.exe", + "manager_cli": "C:\\fake-package\\bin\\sandbox-manager-cli.exe", + "runtime_cli": "C:\\fake-package\\bin\\sandbox-runtime-cli.exe", + "observability_cli": ( + "C:\\fake-package\\bin\\sandbox-observability-cli.exe" + ), + } + + @property + def sanitized_commands(self) -> dict[str, Any]: + return { + "gateway_serve": { + "executable_path": self.executable_paths["gateway"], + "argv": [ + "serve", + "--backend", + "none", + "--gateway-endpoint", + self.endpoint, + "--auth-token=", + "--pid-file", + "C:\\fake-evidence\\gateway.pid", + ], + "working_directory": "C:\\fake-package", + "stdin": "null", + "stdout": "pipe_digest_only", + "stderr": "pipe_digest_only", + }, + "manager_list_sandboxes": { + "executable_path": self.executable_paths["manager_cli"], + "argv_template": [ + "--gateway-endpoint", + self.endpoint, + "--gateway-auth-token=", + "--request-id", + "", + "list_sandboxes", + ], + "working_directory": "C:\\fake-package", + "stdin": "null", + "stdout": "strict_single_json_line_hash_only", + "stderr": "must_be_empty_hash_only", + }, + } + + async def start(self) -> None: + self.started += 1 + + async def invoke(self, request_id: str) -> InvocationCapture: + index = len(self.request_ids) + self.request_ids.append(request_id) + self.active += 1 + self.peak_active = max(self.peak_active, self.active) + try: + await asyncio.sleep(0) + if index == self.raise_index: + raise OSError("injected invocation failure") + failed = index == self.failure_index + return InvocationCapture( + request_id=request_id, + started_monotonic_ns=1000 + index * 10, + ended_monotonic_ns=1005 + index * 10, + started_utc_ns=2000 + index * 10, + ended_utc_ns=2005 + index * 10, + return_code=0, + stdout=( + b'{"sandboxes":[],"secret":"test-secret-token"}\n' + if failed + else b'{"sandboxes":[]}\n' + ), + stderr=b"", + credential_exposed=failed, + ) + finally: + self.active -= 1 + + async def stop(self) -> dict[str, Any]: + self.stopped += 1 + evidence = { + "schema_version": 1, + "gateway_pid": self.gateway_pid, + "executable_path": self.executable_paths["gateway"], + "process_exited": True, + "pid_file_removed": True, + "sanitized_command": self.sanitized_commands["gateway_serve"], + "termination": "terminate", + "return_code": 0, + "stdout": { + "bytes": 0, + "sha256": ( + "sha256:e3b0c44298fc1c149afbf4c8996fb924" + "27ae41e4649b934ca495991b7852b855" + ), + }, + "stderr": { + "bytes": 0, + "sha256": ( + "sha256:e3b0c44298fc1c149afbf4c8996fb924" + "27ae41e4649b934ca495991b7852b855" + ), + }, + "auth_token_recorded": False, + } + if self.stop_override: + evidence.update(self.stop_override) + return evidence + + +class FakeCollector: + def __init__( + self, + *, + events: tuple[TcpipEvent, ...] = (), + readiness_handles: int = 100, + post_handles: int = 100, + readiness_private_bytes: int = 8 * 1024 * 1024, + post_private_bytes: int = 8 * 1024 * 1024, + readiness_rss_bytes: int = 12 * 1024 * 1024, + post_rss_bytes: int = 12 * 1024 * 1024, + fail_phase: str | None = None, + process_overrides: dict[tuple[str, int], tuple[int, int, int]] | None = None, + tcp_connections: ( + dict[tuple[str, int], tuple[OwnedTcpConnection, ...]] | None + ) = None, + identity_case: str | None = None, + ) -> None: + self.events = events + self.readiness_handles = readiness_handles + self.post_handles = post_handles + self.readiness_private_bytes = readiness_private_bytes + self.post_private_bytes = post_private_bytes + self.readiness_rss_bytes = readiness_rss_bytes + self.post_rss_bytes = post_rss_bytes + self.fail_phase = fail_phase + self.process_overrides = process_overrides or {} + self.tcp_connections = tcp_connections or {} + self.identity_case = identity_case + self.calls: list[str] = [] + + async def qualification_identity( + self, + roots: BenchmarkRoots, + runner: FakeRunner, + ) -> dict[str, Any]: + self.calls.append("identity") + if self.fail_phase == "identity": + raise OSError("injected identity failure") + commit = PREREGISTERED_PRODUCT_COMMIT + package = os.fspath(roots.product_bin_dir.parent) + digest = "sha256:" + "0" * 64 + + def file_identity(path: str, sha256: str = digest) -> dict[str, Any]: + return {"path": path, "bytes": 1, "sha256": sha256} + + value = { + "schema_version": 1, + "captured_monotonic_ns": 10, + "captured_utc_ns": 20, + "product": { + "commit": commit, + "branch": PREREGISTERED_PRODUCT_BRANCH, + "status_clean": True, + "status_bytes": 0, + "status_sha256": ( + "sha256:e3b0c44298fc1c149afbf4c8996fb924" + "27ae41e4649b934ca495991b7852b855" + ), + "package_directory": package, + "package_directory_name": f"windows-exp1-{commit[:8]}", + "expected_package_directory_name": f"windows-exp1-{commit[:8]}", + "package_name_matches_commit": True, + "package_zip_path": f"{package}.zip", + "package_zip_bytes": 1, + "package_zip_sha256": PREREGISTERED_PACKAGE_SHA256, + }, + "paper": { + "commit": "b" * 40, + "scoped_status_clean": True, + "scoped_status_bytes": 0, + "scoped_status_sha256": ( + "sha256:e3b0c44298fc1c149afbf4c8996fb924" + "27ae41e4649b934ca495991b7852b855" + ), + "paper_root": os.fspath(roots.test_repository_root), + "frozen_scope": list(PAPER_FROZEN_SCOPE), + "generated_exclusions": list(BENCHMARK_GIT_EXCLUSIONS), + "policy_source": ( + "experiments/scripts/archive_exp1_run.py:" + "PAPER_FROZEN_SCOPE+BENCHMARK_GIT_EXCLUSIONS" + ), + }, + "executables": { + role: file_identity(path, PREREGISTERED_ARTIFACT_SHA256[role]) + for role, path in runner.executable_paths.items() + }, + "packaged_support": { + "linux_daemon": file_identity( + os.fspath( + roots.product_bin_dir.parent + / "dist" + / "sandbox-daemon-linux-amd64" + ), + PREREGISTERED_ARTIFACT_SHA256["linux_daemon"], + ), + "windows_config": file_identity( + os.fspath( + roots.product_bin_dir.parent / "config" / "windows-amd64.yml" + ), + PREREGISTERED_ARTIFACT_SHA256["windows_config"], + ), + }, + "qualifier_sources": { + "ipc_qualification": file_identity( + os.fspath(Path(ipc_module.__file__).resolve()) + ), + "benchmark_cli": file_identity( + os.fspath(Path(ipc_module.__file__).with_name("cli.py").resolve()) + ), + "packaged_gateway_launcher": file_identity( + os.fspath( + roots.product_bin_dir + / "start-sandbox-windows-docker-gateway.ps1" + ) + ), + "ipc_qualification_test": file_identity( + os.fspath( + roots.benchmark_source_root + / "backend" + / "tests" + / "unit" + / "test_ipc_qualification.py" + ) + ), + "protocol_amendment": file_identity( + os.fspath( + roots.test_repository_root + / "experiments" + / "exp1-v1.1-protocol-amendment.md" + ) + ), + }, + "host": { + "computer_name": "DESKTOP-OLP1ADS", + "os_caption": "Windows", + "os_version": "10.0.26200", + "os_build_number": 26200, + "architecture": "x64", + "logical_processors": 48, + "total_memory_bytes": 137_438_953_472, + }, + "build": { + "package_identity": f"windows-exp1-{commit[:8]}", + "product_commit": commit, + "python_executable": "C:\\Python313\\python.exe", + "python_implementation": "CPython", + "python_version": "3.13.0", + "python_architecture": "amd64", + "package_build_command": [ + "powershell.exe", + "-NoProfile", + "-ExecutionPolicy", + "Bypass", + "-File", + ".\\bin\\package-windows-amd64-release.ps1", + "-PackageName", + f"windows-exp1-{commit[:8]}", + "-OutDir", + "target", + "-Profile", + "release", + ], + }, + "sanitized_commands": _expected_sanitized_commands(roots, runner), + } + if self.identity_case == "dirty": + value["product"]["status_clean"] = False + value["product"]["status_bytes"] = 1 + elif self.identity_case == "package_mismatch": + value["product"]["package_name_matches_commit"] = False + elif self.identity_case == "command_shape": + value["sanitized_commands"]["gateway_serve"]["argv"].append("--secret") + elif self.identity_case == "paper_dirty": + value["paper"]["scoped_status_clean"] = False + value["paper"]["scoped_status_bytes"] = 1 + elif self.identity_case == "branch": + value["product"]["branch"] = "feature" + elif self.identity_case == "artifact_hash": + value["executables"]["gateway"]["sha256"] = digest + elif self.identity_case == "host": + value["host"]["computer_name"] = "OTHER-HOST" + return value + + async def event_cursor(self, phase: str) -> EventLogCursor: + self.calls.append(f"cursor:{phase}") + if self.fail_phase == f"cursor:{phase}": + raise OSError("injected cursor failure") + is_pre = phase == "pre_readiness" + return EventLogCursor( + phase=phase, + captured_monotonic_ns=100 if is_pre else 400, + captured_utc_ns=200 if is_pre else 500, + log_name="System", + last_record_id=1000 if is_pre else 1100, + ) + + async def tcpip_events( + self, + after_record_id: int, + through_record_id: int, + ) -> tuple[TcpipEvent, ...]: + self.calls.append(f"events:{after_record_id}:{through_record_id}") + if self.fail_phase == "events": + raise OSError("injected event query failure") + return self.events + + async def gateway_process_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayProcessSample: + call = f"process:{phase}:{completed_batches}" + self.calls.append(call) + if self.fail_phase == call: + raise OSError("injected process sample failure") + readiness = phase == "readiness" + default = ( + ( + self.readiness_handles, + self.readiness_private_bytes, + self.readiness_rss_bytes, + ) + if readiness + else ( + self.post_handles, + self.post_private_bytes, + self.post_rss_bytes, + ) + ) + handles, private_bytes, rss_bytes = self.process_overrides.get( + (phase, completed_batches), + default, + ) + return GatewayProcessSample( + phase=phase, + captured_monotonic_ns=250 + completed_batches, + captured_utc_ns=300 + completed_batches, + pid=pid, + completed_batches=completed_batches, + handle_count=handles, + private_bytes=private_bytes, + rss_bytes=rss_bytes, + ) + + async def gateway_tcp_sample( + self, + pid: int, + phase: str, + completed_batches: int, + ) -> GatewayTcpSample: + call = f"tcp:{phase}:{completed_batches}" + self.calls.append(call) + if self.fail_phase == call: + raise OSError("injected TCP sample failure") + return GatewayTcpSample( + phase=phase, + captured_monotonic_ns=260 + completed_batches, + captured_utc_ns=310 + completed_batches, + pid=pid, + completed_batches=completed_batches, + connections=self.tcp_connections.get((phase, completed_batches), ()), + ) + + +def _roots(tmp_path: Path) -> BenchmarkRoots: + test_root = tmp_path / "paper" + product_root = tmp_path / "product" + product_bin = ( + product_root + / "target" + / f"windows-exp1-{PREREGISTERED_PRODUCT_COMMIT[:8]}" + / "bin" + ) + (test_root / "benchmark").mkdir(parents=True) + product_bin.mkdir(parents=True) + return BenchmarkRoots.resolve( + test_root, + product_root, + product_bin, + initialize=True, + ) + + +@pytest.mark.asyncio +async def test_fixed_workload_runs_concurrency_five_without_retry_or_pacing( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-success" + evidence_root = _create_evidence_root(roots, qualification_id) + runner = FakeRunner() + collector = FakeCollector() + + summary = await _execute_qualification( + roots, + runner, + collector, + evidence_root, + qualification_id, + QualificationWorkload(batches=2, concurrency=5), + ) + + assert summary["status"] == "passed" + assert summary["qualification_only"] is True + assert summary["performance_evidence"] is False + assert summary["transport"] == { + "kind": "windows_named_pipe", + "endpoint": runner.endpoint, + "tcp_used": False, + "fallback_allowed": False, + "retry_allowed": False, + "pacing_allowed": False, + "gateway_count": 1, + } + assert summary["workload"]["attempted_invocations"] == 10 + assert summary["workload"]["successful_invocations"] == 10 + assert summary["workload"]["planned_batches"] == 2 + assert summary["workload"]["concurrency"] == 5 + assert runner.started == 1 + assert runner.stopped == 1 + assert runner.peak_active == 5 + assert len(runner.request_ids) == len(set(runner.request_ids)) == 10 + assert collector.calls == [ + "identity", + "cursor:pre_readiness", + "process:readiness:0", + "tcp:readiness:0", + "process:pre_stop:2", + "tcp:pre_stop:2", + "tcp:after_cleanup:2", + "cursor:post_cleanup", + "events:1000:1100", + ] + assert summary["host_evidence"]["status"] == "passed" + assert summary["host_evidence"]["event_log"]["new_event_count"] == 0 + growth = summary["host_evidence"]["gateway_process"]["growth"] + assert growth["peak_over_readiness"]["passed"] is True + assert growth["final_pre_stop_over_readiness"]["passed"] is True + assert summary["host_evidence"]["gateway_tcp"]["passed"] is True + assert summary["host_evidence"]["gateway_stop"]["validation_passed"] is True + assert summary["host_evidence"]["policy"]["maximum_post_readiness_growth"] == { + "handle_count": 32, + "private_bytes": 16 * 1024 * 1024, + "rss_bytes": 16 * 1024 * 1024, + } + assert summary["host_evidence"]["policy"]["source"] == ( + RESOURCE_GROWTH_POLICY_SOURCE + ) + + records = [ + json.loads(line) + for line in (evidence_root / "invocations.ndjson") + .read_text(encoding="utf-8") + .splitlines() + ] + assert len(records) == 10 + assert all(record["result"] == "passed" for record in records) + assert all(record["qualification_only"] is True for record in records) + assert all(record["performance_evidence"] is False for record in records) + assert all(record["stdout_bytes"] == 17 for record in records) + assert all(record["stderr_bytes"] == 0 for record in records) + assert len({record["request_id"] for record in records}) == 10 + persisted_summary = json.loads( + (evidence_root / "summary.json").read_text(encoding="utf-8") + ) + assert persisted_summary == summary + persisted_host_evidence = json.loads( + (evidence_root / "host-evidence.json").read_text(encoding="utf-8") + ) + assert persisted_host_evidence == summary["host_evidence"] + manifest = json.loads( + (evidence_root / "qualification-manifest.json").read_text(encoding="utf-8") + ) + assert manifest["qualification_only"] is True + assert manifest["performance_evidence"] is False + assert manifest["identity"] == summary["host_evidence"]["identity"] + assert manifest["gateway_stop"] == summary["gateway"] + assert summary["evidence"]["qualification_manifest_json_sha256"].startswith( + "sha256:" + ) + + +@pytest.mark.asyncio +async def test_failure_retains_complete_batch_evidence_and_stops( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-failure" + evidence_root = _create_evidence_root(roots, qualification_id) + runner = FakeRunner(failure_index=2) + + summary = await _execute_qualification( + roots, + runner, + FakeCollector(), + evidence_root, + qualification_id, + QualificationWorkload(batches=3, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == "credential_echo" + assert summary["workload"]["attempted_invocations"] == 5 + assert summary["workload"]["successful_invocations"] == 4 + assert summary["workload"]["failed_invocations"] == 1 + assert summary["workload"]["batches_started"] == 1 + assert summary["workload"]["batches_completed"] == 0 + assert summary["first_failure"]["invocation_index"] == 2 + assert runner.stopped == 1 + evidence = b"".join(path.read_bytes() for path in sorted(evidence_root.iterdir())) + assert b"test-secret-token" not in evidence + assert ( + len( + (evidence_root / "invocations.ndjson") + .read_text(encoding="utf-8") + .splitlines() + ) + == 5 + ) + + +@pytest.mark.asyncio +async def test_runner_exception_becomes_partial_failure_record( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-runner-error" + evidence_root = _create_evidence_root(roots, qualification_id) + runner = FakeRunner(raise_index=1) + + summary = await _execute_qualification( + roots, + runner, + FakeCollector(), + evidence_root, + qualification_id, + QualificationWorkload(batches=2, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == "runner_error:OSError" + assert summary["workload"]["attempted_invocations"] == 5 + assert summary["first_failure"]["request_id"] == runner.request_ids[1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("event_id", TCPIP_EVENT_IDS) +async def test_new_tcpip_event_fails_qualification_with_cursor_evidence( + tmp_path: Path, + event_id: int, +) -> None: + roots = _roots(tmp_path) + qualification_id = f"qualification-tcpip-event-{event_id}" + evidence_root = _create_evidence_root(roots, qualification_id) + event = TcpipEvent( + event_id=event_id, + record_id=1050, + created_at_utc="2026-07-31T08:00:00.0000000Z", + ) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector(events=(event,)), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == "tcpip_event_detected" + assert summary["gate_failures"] == ["tcpip_event_detected"] + assert summary["workload"]["successful_invocations"] == 5 + event_evidence = summary["host_evidence"]["event_log"] + assert event_evidence["pre_readiness_cursor"]["last_record_id"] == 1000 + assert event_evidence["post_cleanup_cursor"]["last_record_id"] == 1100 + assert event_evidence["query_interval"] == ( + "(pre_readiness_record_id, post_cleanup_record_id]" + ) + assert event_evidence["new_event_count"] == 1 + assert event_evidence["events"] == [ + { + "event_id": event_id, + "record_id": 1050, + "created_at_utc": "2026-07-31T08:00:00.0000000Z", + "provider_name": "Tcpip", + } + ] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("collector_kwargs", "failure", "metric"), + [ + ( + {"post_handles": 100 + MAX_GATEWAY_HANDLE_GROWTH + 1}, + "gateway_handle_peak_growth_exceeded", + "handle_count", + ), + ( + { + "post_private_bytes": ( + 8 * 1024 * 1024 + MAX_GATEWAY_PRIVATE_BYTES_GROWTH + 1 + ) + }, + "gateway_private_bytes_peak_growth_exceeded", + "private_bytes", + ), + ( + {"post_rss_bytes": (12 * 1024 * 1024 + MAX_GATEWAY_RSS_BYTES_GROWTH + 1)}, + "gateway_rss_peak_growth_exceeded", + "rss_bytes", + ), + ], +) +async def test_gateway_process_growth_policy_is_fail_closed( + tmp_path: Path, + collector_kwargs: dict[str, int], + failure: str, + metric: str, +) -> None: + roots = _roots(tmp_path) + qualification_id = f"qualification-{metric.replace('_', '-')}" + evidence_root = _create_evidence_root(roots, qualification_id) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector(**collector_kwargs), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == failure + decision = summary["host_evidence"]["gateway_process"]["growth"][ + "peak_over_readiness" + ]["decisions"][metric] + assert decision["within_bound"] is False + assert decision["growth"] == decision["maximum_allowed_growth"] + 1 + assert summary["workload"]["successful_invocations"] == 5 + + +@pytest.mark.asyncio +async def test_gateway_process_growth_at_exact_bounds_passes( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-exact-growth-bounds" + evidence_root = _create_evidence_root(roots, qualification_id) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector( + post_handles=100 + MAX_GATEWAY_HANDLE_GROWTH, + post_private_bytes=(8 * 1024 * 1024 + MAX_GATEWAY_PRIVATE_BYTES_GROWTH), + post_rss_bytes=12 * 1024 * 1024 + MAX_GATEWAY_RSS_BYTES_GROWTH, + ), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "passed" + growth = summary["host_evidence"]["gateway_process"]["growth"] + for gate in ("peak_over_readiness", "final_pre_stop_over_readiness"): + decisions = growth[gate]["decisions"] + assert all(decision["within_bound"] for decision in decisions.values()) + assert all( + decision["growth"] == decision["maximum_allowed_growth"] + for decision in decisions.values() + ) + + +@pytest.mark.asyncio +async def test_cadence_peak_growth_fails_even_when_final_returns_to_baseline( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-cadence-peak" + evidence_root = _create_evidence_root(roots, qualification_id) + collector = FakeCollector( + process_overrides={ + ("cadence", 100): ( + 100 + MAX_GATEWAY_HANDLE_GROWTH + 1, + 8 * 1024 * 1024, + 12 * 1024 * 1024, + ), + ("pre_stop", 100): ( + 100, + 8 * 1024 * 1024, + 12 * 1024 * 1024, + ), + } + ) + + summary = await _execute_qualification( + roots, + FakeRunner(), + collector, + evidence_root, + qualification_id, + QualificationWorkload(batches=100, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == "gateway_handle_peak_growth_exceeded" + growth = summary["host_evidence"]["gateway_process"]["growth"] + assert growth["peak_over_readiness"]["passed"] is False + assert growth["final_pre_stop_over_readiness"]["passed"] is True + assert "process:cadence:100" in collector.calls + assert "tcp:cadence:100" in collector.calls + assert summary["workload"]["successful_invocations"] == 500 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("phase", "completed_batches", "workload_batches"), + [ + ("readiness", 0, 1), + ("cadence", 100, 100), + ("pre_stop", 1, 1), + ("after_cleanup", 1, 1), + ], +) +async def test_owned_tcp_at_any_required_checkpoint_fails( + tmp_path: Path, + phase: str, + completed_batches: int, + workload_batches: int, +) -> None: + roots = _roots(tmp_path) + qualification_id = f"qualification-tcp-{phase.replace('_', '-')}" + evidence_root = _create_evidence_root(roots, qualification_id) + connection = OwnedTcpConnection( + state="Listen", + local_address="127.0.0.1", + local_port=7878, + remote_address="0.0.0.0", + remote_port=0, + ) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector( + tcp_connections={(phase, completed_batches): (connection,)}, + ), + evidence_root, + qualification_id, + QualificationWorkload(batches=workload_batches, concurrency=5), + ) + + assert summary["status"] == "failed" + assert "gateway_owned_tcp_detected" in summary["gate_failures"] + tcp = summary["host_evidence"]["gateway_tcp"] + assert tcp["owned_connection_count"] == 1 + matching = [ + sample + for sample in tcp["samples"] + if sample["phase"] == phase and sample["completed_batches"] == completed_batches + ] + assert matching[0]["connections"][0]["local_port"] == 7878 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("fail_phase", "workload_batches"), + [ + ("tcp:readiness:0", 1), + ("process:cadence:100", 100), + ("tcp:cadence:100", 100), + ("tcp:pre_stop:1", 1), + ("tcp:after_cleanup:1", 1), + ], +) +async def test_missing_checkpoint_evidence_fails_closed( + tmp_path: Path, + fail_phase: str, + workload_batches: int, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-missing-" + fail_phase.replace(":", "-").replace( + "_", "-" + ) + evidence_root = _create_evidence_root(roots, qualification_id) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector(fail_phase=fail_phase), + evidence_root, + qualification_id, + QualificationWorkload(batches=workload_batches, concurrency=5), + ) + + assert summary["status"] == "failed" + assert "host_evidence_collection_error" in summary["gate_failures"] + assert any( + failure in summary["gate_failures"] + for failure in ( + "gateway_process_evidence_incomplete", + "gateway_tcp_evidence_incomplete", + ) + ) + assert summary["host_evidence"]["collector_errors"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("override", "field"), + [ + ({"gateway_pid": 9999}, "gateway_pid"), + ({"executable_path": "C:\\wrong.exe"}, "executable_path"), + ({"process_exited": False}, "process_exited"), + ({"pid_file_removed": False}, "pid_file_removed"), + ({"sanitized_command": {"redacted": False}}, "sanitized_command"), + ({"termination": "kill_after_terminate_timeout"}, "termination"), + ({"return_code": None}, "return_code"), + ({"stdout": {"bytes": 0, "sha256": "invalid"}}, "stdout"), + ({"auth_token_recorded": True}, "auth_token_recorded"), + ], +) +async def test_stop_evidence_fields_are_fail_closed( + tmp_path: Path, + override: dict[str, Any], + field: str, +) -> None: + roots = _roots(tmp_path) + qualification_id = f"qualification-stop-{field.replace('_', '-')}" + evidence_root = _create_evidence_root(roots, qualification_id) + + summary = await _execute_qualification( + roots, + FakeRunner(stop_override=override), + FakeCollector(), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "failed" + assert "gateway_stop_evidence_invalid" in summary["gate_failures"] + assert summary["host_evidence"]["gateway_stop"]["validation_passed"] is False + assert summary["host_evidence"]["gateway_stop"]["validation_errors"] == [ + {"phase": "gateway_stop_evidence", "error_type": "QualificationError"} + ] + assert summary["host_evidence"]["event_log"]["query_completed"] is True + assert summary["host_evidence"]["gateway_tcp"]["samples"][-1]["phase"] == ( + "after_cleanup" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "identity_case", + [ + "dirty", + "package_mismatch", + "command_shape", + "paper_dirty", + "branch", + "artifact_hash", + "host", + ], +) +async def test_identity_drift_blocks_gateway_start_and_preserves_manifest( + tmp_path: Path, + identity_case: str, +) -> None: + roots = _roots(tmp_path) + qualification_id = f"qualification-identity-{identity_case.replace('_', '-')}" + evidence_root = _create_evidence_root(roots, qualification_id) + runner = FakeRunner() + + summary = await _execute_qualification( + roots, + runner, + FakeCollector(identity_case=identity_case), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "failed" + assert runner.started == 0 + assert "qualification_identity_invalid" in summary["gate_failures"] + assert summary["workload"]["attempted_invocations"] == 0 + manifest = json.loads( + (evidence_root / "qualification-manifest.json").read_text(encoding="utf-8") + ) + assert manifest["status"] == "failed" + assert manifest["identity"] is not None + + +@pytest.mark.asyncio +async def test_collector_failure_preserves_partial_host_evidence( + tmp_path: Path, +) -> None: + roots = _roots(tmp_path) + qualification_id = "qualification-collector-failure" + evidence_root = _create_evidence_root(roots, qualification_id) + + summary = await _execute_qualification( + roots, + FakeRunner(), + FakeCollector(fail_phase="cursor:post_cleanup"), + evidence_root, + qualification_id, + QualificationWorkload(batches=1, concurrency=5), + ) + + assert summary["status"] == "failed" + assert summary["failure_kind"] == "host_evidence_collection_error" + assert summary["gate_failures"] == [ + "host_evidence_collection_error", + "tcpip_event_evidence_incomplete", + ] + host = summary["host_evidence"] + assert host["event_log"]["pre_readiness_cursor"] is not None + assert host["event_log"]["post_cleanup_cursor"] is None + assert host["event_log"]["query_completed"] is False + assert host["gateway_process"]["passed"] is True + assert host["collector_errors"] == [ + {"phase": "post_cleanup_cursor", "error_type": "OSError"} + ] + assert (evidence_root / "host-evidence.json").is_file() + + +@pytest.mark.parametrize( + ("capture", "expected"), + [ + ( + InvocationCapture( + "request-1", + 1, + 2, + 3, + 4, + 1, + b"", + b"", + ), + "nonzero_exit", + ), + ( + InvocationCapture( + "request-1", + 1, + 2, + 3, + 4, + 0, + b'{"sandboxes":[]}\n', + b"warning\n", + ), + "unexpected_stderr", + ), + ( + InvocationCapture( + "request-1", + 1, + 2, + 3, + 4, + 0, + b'{"sandboxes":[]}\nextra\n', + b"", + ), + "stdout_framing", + ), + ( + InvocationCapture( + "request-1", + 1, + 2, + 3, + 4, + 0, + b"not-json\n", + b"", + ), + "invalid_json", + ), + ( + InvocationCapture( + "request-1", + 1, + 2, + 3, + 4, + 0, + b'{"sandboxes":[],"extra":true}\n', + b"", + ), + "response_shape", + ), + ], +) +def test_invocation_validation_is_strict( + capture: InvocationCapture, + expected: str, +) -> None: + assert _validate_capture(capture) == expected + + +def test_production_workload_is_fixed() -> None: + assert EXP1_IPC_INVOCATION_COUNT == 25_000 + assert EXP1_IPC_CONCURRENCY == 5 + assert EXP1_IPC_BATCH_COUNT == 5_000 + assert PRODUCTION_WORKLOAD == QualificationWorkload( + batches=5_000, + concurrency=5, + ) + assert PRODUCTION_WORKLOAD.invocation_count == 25_000 + assert QUALIFICATION_ONLY is True + assert PERFORMANCE_EVIDENCE is False + assert TCPIP_EVENT_IDS == (4227, 4231) + assert MAX_GATEWAY_HANDLE_GROWTH == 32 + assert MAX_GATEWAY_PRIVATE_BYTES_GROWTH == 16 * 1024 * 1024 + assert MAX_GATEWAY_RSS_BYTES_GROWTH == 16 * 1024 * 1024 + assert RESOURCE_GROWTH_POLICY_SOURCE == ( + "EXP1 v1.1 IPC qualification policy preregistration" + ) + assert PROCESS_SAMPLE_EVERY_BATCHES == 100 + + +def test_paper_scope_matches_archive_contract() -> None: + assert PAPER_FROZEN_SCOPE == ( + "benchmark", + "progress.md", + "plan/task-packets/exp1-cli-performance-campaign.md", + "experiment_inventory.md", + "experiments/exp1-v1.1-protocol-amendment.md", + "experiments/environment_setup.md", + "experiments/expected_tables.md", + "experiments/experiment_log.md", + "paper_state.json", + "plan/progress.md", + "experiments/scripts/archive_exp1_run.py", + "experiments/scripts/project_exp1_final_runtime.py", + "experiments/analysis/scripts/generate_exp1_tables.py", + ) + assert BENCHMARK_GIT_EXCLUSIONS == ( + ":(exclude,glob)benchmark/**/.pytest_cache/**", + ":(exclude,glob)benchmark/**/.venv/**", + ":(exclude,glob)benchmark/**/__pycache__/**", + ":(exclude,glob)benchmark/**/dist/**", + ":(exclude,glob)benchmark/**/node_modules/**", + ":(exclude,glob)benchmark/**/playwright-report/**", + ":(exclude,glob)benchmark/**/test-results/**", + ":(exclude,glob)benchmark/**/*.pyc", + ) + + +def test_only_safe_npipe_endpoint_is_accepted() -> None: + _validate_npipe_endpoint("npipe://./pipe/ephemeral-sandbox-exp1-ipc-1234") + for endpoint in ( + "127.0.0.1:7878", + "tcp://127.0.0.1:7878", + "unix:///tmp/gateway.sock", + "npipe://./pipe/../escape", + "npipe://server/pipe/gateway", + "npipe://./pipe/name with spaces", + "npipe://./pipe/name?query", + f"npipe://./pipe/{'x' * 249}", + ): + with pytest.raises(QualificationError): + _validate_npipe_endpoint(endpoint) + + +def test_cli_command_has_no_workload_override_flags(tmp_path: Path) -> None: + roots = _roots(tmp_path) + arguments = cli.parser().parse_args( + [ + "qualify-exp1-ipc", + "--test-repository-root", + str(roots.test_repository_root), + "--product-root", + str(roots.product_root), + "--product-bin-dir", + str(roots.product_bin_dir), + ] + ) + + assert arguments.command == "qualify-exp1-ipc" + assert not hasattr(arguments, "batches") + assert not hasattr(arguments, "concurrency") + assert not hasattr(arguments, "invocations") + + +def test_cli_returns_two_for_failed_qualification( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + async def failed(_roots: object) -> dict[str, str]: + return {"status": "failed"} + + monkeypatch.setattr(cli, "qualify_exp1_ipc", failed) + service = argparse.Namespace(roots=object()) + arguments = argparse.Namespace(command="qualify-exp1-ipc") + + assert cli._dispatch(arguments, service) == 2 + assert json.loads(capsys.readouterr().out) == {"status": "failed"} diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_metadata.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_metadata.py new file mode 100644 index 0000000..a612b1d --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_metadata.py @@ -0,0 +1,163 @@ +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from benchmark_lab import metadata + + +def _host() -> dict: + return { + "computer_name": "DESKTOP-OLP1ADS", + "operating_system": "windows", + "architecture": "x64", + "os_caption": "Microsoft Windows 11", + "os_version": "10.0.26200", + "os_build_number": 26200, + "cpu_model": "AMD Ryzen Threadripper 7960X 24-Cores", + "logical_processors": 48, + "processor_logical_processors": 48, + "total_memory_bytes": 137_438_953_472, + "filesystem": "NTFS", + "volume_root": "C:\\", + "capture_boundary": "run_start_before_gateway_and_measurement", + "captured_at": "2026-07-30T00:00:00.000000Z", + "capture_source": "Windows CIM and Get-Volume", + } + + +def _limits() -> dict: + return { + "profile": "standard", + "nano_cpus": 1_000_000_000, + "vcpus": 1, + "memory_bytes": 536_870_912, + "pids_limit": 256, + "authority": { + "kind": "released_gateway_configuration", + "path": "C:\\package\\config\\windows-amd64.yml", + "sha256": "sha256:" + "a" * 64, + "selector": "manager.docker.resource_profiles.standard", + "effective_config_builder": "benchmark_lab.gateway._effective_config", + "create_request_override": "none", + "capture_boundary": "run_start_before_gateway_and_measurement", + }, + } + + +def test_released_configuration_resolves_effective_standard_limits( + tmp_path: Path, +) -> None: + config = tmp_path / "windows-amd64.yml" + config.write_text( + """ +manager: + docker: + resource_profile: standard + resource_profiles: + standard: + nano_cpus: 2000000000 + memory_max_bytes: 1073741824 + pids_max: 256 + nano_cpus: 1000000000 + memory_bytes: 536870912 +""".lstrip(), + encoding="utf-8", + ) + + limits = metadata._configured_sandbox_limits_from_template(config) + + assert limits["profile"] == "standard" + assert limits["nano_cpus"] == 1_000_000_000 + assert limits["vcpus"] == 1 + assert limits["memory_bytes"] == 536_870_912 + assert limits["pids_limit"] == 256 + assert limits["authority"]["sha256"].startswith("sha256:") + assert limits["authority"]["create_request_override"] == "none" + + +def test_exp1_host_and_limit_validation_fail_closed_on_drift() -> None: + host = _host() + host["logical_processors"] = 47 + with pytest.raises( + metadata.EnvironmentMetadataError, match="logical CPU counts disagree" + ): + metadata._validate_exp1_host(host) + + limits = _limits() + limits["memory_bytes"] = 0 + with pytest.raises( + metadata.EnvironmentMetadataError, + match="effective sandbox limit drift", + ): + metadata._validate_exp1_sandbox_limits(limits) + + +@pytest.mark.asyncio +async def test_paper_environment_records_final_host_and_limits_at_run_start( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + binary = tmp_path / "binary" + binary.write_bytes(b"released") + host = _host() + limits = _limits() + + async def capture(target: Path) -> dict: + assert target == tmp_path + return dict(host) + + async def command(args: list[str], cwd: Path) -> str: + if args[:3] == ["git", "rev-parse", "HEAD"]: + return "1" * 40 + if args[:3] == ["git", "status", "--porcelain=v1"]: + return "" + if args[:2] == ["docker", "version"]: + return "29.0.1" + if args[:3] == ["docker", "image", "inspect"]: + return "sha256:" + "2" * 64 + raise AssertionError(args) + + monkeypatch.setattr(metadata, "_capture_windows_final_host", capture) + monkeypatch.setattr( + metadata, "_configured_sandbox_limits", lambda roots: dict(limits) + ) + monkeypatch.setattr(metadata, "_command", command) + monkeypatch.setattr( + metadata, "_container_daemon_executable", lambda roots: binary + ) + monkeypatch.setattr( + metadata, "_prebuilt_executable", lambda roots, name: binary + ) + roots = SimpleNamespace( + product_root=tmp_path, + benchmark_state_root=tmp_path, + ) + plan = { + "canonical_plan": { + "name": "paper-good-pass", + "environment": {"image": "ubuntu@sha256:fixed"}, + }, + "effective_environment": { + "filesystem": None, + "workspace_root_identity": "sha256:workspace", + "client_cohort": "product_cli", + "image_digest": None, + }, + } + + environment = await metadata.collect_environment(roots, plan) + + assert environment["host"]["capture_boundary"] == ( + "run_start_before_gateway_and_measurement" + ) + assert environment["host"]["os_build_number"] == 26200 + assert environment["host"]["filesystem"] == "NTFS" + assert environment["sandbox_limits"] == limits + assert environment["gateway_endpoint_identity"] == ( + "isolated_windows_named_pipe_per_execution_block" + ) + assert environment["gateway_transport"] == { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + } diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_observability.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_observability.py index b31b06b..e5bf317 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_observability.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_observability.py @@ -1,6 +1,14 @@ -import pytest +from copy import deepcopy -from benchmark_lab.observability import ObservabilityError, parse_trace +import pytest +from benchmark_lab.observability import ( + ObservabilityError, + daemon_from_cgroup, + parse_cgroup, + parse_daemon, + parse_snapshot, + parse_trace, +) def _trace() -> dict: @@ -36,3 +44,288 @@ def test_trace_still_rejects_an_explicit_wrong_root_parent() -> None: with pytest.raises(ObservabilityError, match="tree contract"): parse_trace(value, "request-1") + + +def test_cgroup_accepts_partial_view_before_resource_ring_is_ready() -> None: + parsed = parse_cgroup( + { + "view": "cgroup", + "scope": "sandbox", + "availability": "partial", + "errors": ["resource ring is not available yet"], + "topology": {"schema_version": 2, "available": True}, + "series": [], + } + ) + + assert parsed.series == [] + + +def _daemon_metrics() -> dict: + return { + "available": True, + "pid": 7, + "resident_memory_bytes": 1024, + "peak_resident_memory_bytes": 2048, + "cpu_time_us": 3, + } + + +def _cgroup_with_daemon(daemon: object) -> dict: + return { + "view": "cgroup", + "scope": "sandbox", + "availability": "partial", + "errors": ["resource ring is not available yet"], + "topology": {"daemon": daemon}, + "series": [], + } + + +def test_cgroup_daemon_uses_the_standalone_daemon_payload_model() -> None: + daemon = _daemon_metrics() + daemon["future_diagnostic"] = {"value": 1} + + embedded = daemon_from_cgroup(parse_cgroup(_cgroup_with_daemon(daemon))) + standalone = parse_daemon( + {"view": "daemon", "scope": "sandbox", "daemon": daemon} + ).daemon + + assert embedded == standalone + assert embedded.peak_resident_memory_bytes == 2048 + + +@pytest.mark.parametrize( + "daemon", + [ + None, + {}, + {**_daemon_metrics(), "available": 1}, + {**_daemon_metrics(), "pid": True}, + {**_daemon_metrics(), "resident_memory_bytes": -1}, + {key: value for key, value in _daemon_metrics().items() if key != "cpu_time_us"}, + ], +) +def test_cgroup_daemon_rejects_missing_or_malformed_payload( + daemon: object, +) -> None: + cgroup = parse_cgroup(_cgroup_with_daemon(daemon)) + + with pytest.raises( + ObservabilityError, + match="product cgroup topology daemon response schema is invalid", + ): + daemon_from_cgroup(cgroup) + + +def test_cgroup_daemon_requires_the_embedded_object() -> None: + cgroup = parse_cgroup(_cgroup_with_daemon(_daemon_metrics())) + del cgroup.topology["daemon"] + + with pytest.raises( + ObservabilityError, + match="product cgroup topology daemon response schema is invalid", + ): + daemon_from_cgroup(cgroup) + + +def test_snapshot_accepts_current_daemon_event_store() -> None: + parsed = parse_snapshot( + { + "sandbox_id": "sandbox-1", + "lifecycle_state": "ready", + "availability": "available", + "sampled_at_unix_ms": 1, + "errors": [], + "daemon": { + "daemon_pid": 7, + "runtime_dir": "/eos/runtime/daemon", + "event_store": { + "dropped_storage": 0, + "dropped_oversized": 0, + "truncated_records": 0, + }, + }, + "resources": {"latest": None, "history": []}, + "workspaces": [ + { + "workspace_id": "workspace-1", + "lifecycle_state": "active", + "finalization_state": "active", + "network_profile": "shared", + "finalize_policy": "no_op", + "layers": {"base_root_hash": None, "layer_count": 1}, + "namespace_fd_count": 0, + "resources": {"latest": None, "history": []}, + "active_namespace_executions": [], + } + ], + "stack": None, + }, + "sandbox-1", + ) + + assert parsed.daemon.event_store.truncated_records == 0 + assert parsed.workspaces[0].finalization_state == "active" + + +def _snapshot_with_bc1e6ee0_resources() -> dict: + return { + "sandbox_id": "eos-e173422b-df20-4e1d-a954-c6ebfd659675", + "lifecycle_state": "ready", + "availability": "available", + "sampled_at_unix_ms": 1785457778803, + "errors": [], + "daemon": { + "daemon_pid": 7, + "runtime_dir": "/eos/runtime/daemon", + "event_store": { + "dropped_storage": 0, + "dropped_oversized": 0, + "truncated_records": 0, + }, + }, + "resources": { + "latest": { + "ts": 1785457778639, + "sample_delta_ms": 250, + "metrics": { + "metrics_source": "sandbox_cgroup", + "cgroup_path": "/sys/fs/cgroup/", + "cgroup_available": True, + "cpu_usec": 57196, + "mem_cur": 5091328, + "mem_max": 536870912, + "mem_max_unlimited": False, + "io_rbytes": 98304, + "io_wbytes": 16384, + "pids_cur": 12, + }, + "deltas": { + "cpu_usec": 610, + "io_rbytes": 0, + "io_wbytes": 0, + }, + }, + "history": [], + }, + "workspaces": [ + { + "workspace_id": "00000118c73863b675fec5", + "lifecycle_state": "active", + "finalization_state": "active", + "network_profile": "shared", + "finalize_policy": "no_op", + "layers": { + "base_root_hash": ( + "6158805bdb0976490e45c9c206d73f474d7b449a55ba489a5af80fa4f6103070" + ), + "layer_count": 1, + }, + "namespace_fd_count": 3, + "resources": { + "latest": { + "ts": 1785457778639, + "sample_delta_ms": 250, + "metrics": { + "disk_bytes": 0, + "disk_allocated_bytes": 4096, + "files": 0, + "disk_truncated": False, + }, + "deltas": {}, + }, + "history": [], + }, + "active_namespace_executions": [], + } + ], + "stack": { + "layer_count": 1, + "layers_bytes": None, + "layers_allocated_bytes": None, + "storage_allocated_bytes": None, + "staging_entry_count": 0, + "active_leases": 1, + }, + } + + +def test_snapshot_accepts_exact_bc1e6ee0_resource_shape() -> None: + value = _snapshot_with_bc1e6ee0_resources() + + parsed = parse_snapshot(value, value["sandbox_id"]) + + assert parsed.resources.latest is not None + assert parsed.resources.latest.metrics.metrics_source == "sandbox_cgroup" + assert parsed.resources.latest.metrics.io_rbytes == 98304 + assert parsed.resources.latest.metrics.pids_cur == 12 + assert parsed.resources.latest.deltas.io_wbytes == 0 + assert parsed.workspaces[0].resources.latest is not None + assert parsed.workspaces[0].resources.latest.metrics.disk_allocated_bytes == 4096 + + +@pytest.mark.parametrize( + ("target", "field", "value"), + [ + (("resources", "latest", "metrics"), "disk_bytes", 0), + (("resources", "latest", "metrics"), "future_counter", 1), + (("resources", "latest", "deltas"), "future_counter", 1), + ( + ("workspaces", 0, "resources", "latest", "metrics"), + "metrics_source", + "sandbox_cgroup", + ), + (("workspaces", 0, "resources", "latest", "metrics"), "cpu_usec", 1), + (("workspaces", 0, "resources", "latest", "deltas"), "io_rbytes", 1), + ], +) +def test_snapshot_rejects_unknown_or_mixed_scope_resource_keys( + target: tuple[str | int, ...], + field: str, + value: object, +) -> None: + snapshot = deepcopy(_snapshot_with_bc1e6ee0_resources()) + selected = snapshot + for part in target: + selected = selected[part] + selected[field] = value + + with pytest.raises( + ObservabilityError, match="product snapshot response schema is invalid" + ): + parse_snapshot(snapshot, snapshot["sandbox_id"]) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("metrics_source", None), + ("metrics_source", "docker_engine"), + ("cgroup_path", None), + ("cgroup_available", None), + ], +) +def test_snapshot_requires_sandbox_cgroup_identity( + field: str, + value: object, +) -> None: + snapshot = deepcopy(_snapshot_with_bc1e6ee0_resources()) + snapshot["resources"]["latest"]["metrics"][field] = value + + with pytest.raises( + ObservabilityError, match="product snapshot response schema is invalid" + ): + parse_snapshot(snapshot, snapshot["sandbox_id"]) + + +def test_snapshot_requires_workspace_disk_identity() -> None: + snapshot = deepcopy(_snapshot_with_bc1e6ee0_resources()) + del snapshot["workspaces"][0]["resources"]["latest"]["metrics"][ + "disk_truncated" + ] + + with pytest.raises( + ObservabilityError, match="product snapshot response schema is invalid" + ): + parse_snapshot(snapshot, snapshot["sandbox_id"]) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product.py new file mode 100644 index 0000000..fb59fb9 --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product.py @@ -0,0 +1,46 @@ +import pytest + +from benchmark_lab.product import ProductAccessError, _sandbox_record + + +def current_sandbox_record() -> dict[str, object]: + return { + "id": "sandbox-1", + "workspace_root": "/benchmark/runs/run-1/workspace", + "state": "ready", + "activity_revision": 0, + "daemon": {"host": "127.0.0.1", "port": 32768}, + "daemon_http": {"host": "127.0.0.1", "port": 32769}, + "shared_base": { + "source": "/cache/base", + "target": "/eos/layer-stack/base", + "root_hash": "abc", + "readonly": True, + }, + "resource_profile": { + "name": "standard", + "nano_cpus": 1_000_000_000, + "memory_high_bytes": 402_653_184, + "memory_max_bytes": 536_870_912, + "pids_max": 256, + "workload_memory_high_bytes": 402_653_184, + "workload_memory_max_bytes": 402_653_184, + "workload_pids_max": 224, + "control_plane_pids_reserve": 32, + "daemon_runtime_profile": "standard", + "separate_workload_cgroup": True, + }, + } + + +def test_accepts_current_sandbox_resource_profile() -> None: + record = _sandbox_record(current_sandbox_record()) + assert record.activity_revision == 0 + assert record.resource_profile.memory_max_bytes == 536_870_912 + + +def test_rejects_unknown_sandbox_response_fields() -> None: + value = current_sandbox_record() + value["unknown"] = True + with pytest.raises(ProductAccessError, match="schema"): + _sandbox_record(value) diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product_cli.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product_cli.py new file mode 100644 index 0000000..235b92a --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_product_cli.py @@ -0,0 +1,843 @@ +import asyncio +import base64 +import hashlib +import json +import os +import threading +import time +from pathlib import Path +from typing import Any + +import pytest + +import benchmark_lab.product_cli as product_cli_module +from benchmark_lab.paths import BenchmarkRoots +from benchmark_lab.product import ProductAccessError +from benchmark_lab.product_cli import ProductCliAccess, _validate_operation_shape +from benchmark_lab.transport import ( + GatewayEndpoint, + GatewayProductError, + GatewayTransportError, +) + + +class FakeProcess: + def __init__( + self, + stdout: bytes, + stderr: bytes = b"", + returncode: int = 0, + *, + block_until_killed: bool = False, + ) -> None: + self.stdout = stdout + self.stderr = stderr + self.returncode = returncode + self.block_until_killed = block_until_killed + self.killed = False + self.started = asyncio.Event() + self._released = asyncio.Event() + + async def communicate(self) -> tuple[bytes, bytes]: + self.started.set() + if self.block_until_killed and not self.killed: + await self._released.wait() + return self.stdout, self.stderr + + def kill(self) -> None: + self.killed = True + self.returncode = -9 + self._released.set() + + +def test_snapshot_cli_shape_matches_released_top_level_response() -> None: + _validate_operation_shape( + "snapshot", + { + "sandbox_id": "sandbox-1", + "lifecycle_state": "ready", + "availability": "available", + "daemon": {}, + "resources": {}, + "workspaces": [], + "stack": {}, + }, + ) + + +def _roots(tmp_path: Path) -> BenchmarkRoots: + test_root = tmp_path / "paper" + product_root = tmp_path / "product" + bin_root = product_root / "bin" + (test_root / "benchmark").mkdir(parents=True) + bin_root.mkdir(parents=True) + return BenchmarkRoots.resolve( + test_root, + product_root, + bin_root, + initialize=True, + ) + + +def _access( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + process_factory: Any, + endpoint: GatewayEndpoint | None = None, +) -> tuple[ProductCliAccess, Path, list[str]]: + roots = _roots(tmp_path) + selected: list[str] = [] + + def executable(_roots: BenchmarkRoots, name: str) -> Path: + selected.append(name) + path = roots.product_bin_dir / f"{name}.exe" + path.write_bytes(name.encode()) + return path.resolve(strict=True) + + monkeypatch.setattr("benchmark_lab.product_cli._prebuilt_executable", executable) + monkeypatch.setattr( + "benchmark_lab.product_cli.asyncio.create_subprocess_exec", + process_factory, + ) + run_root = roots.runs / "run-1" + run_root.mkdir() + return ( + ProductCliAccess( + endpoint or GatewayEndpoint("127.0.0.1", 47621), + "-secret-token", + roots, + run_root, + ), + run_root, + selected, + ) + + +@pytest.mark.asyncio +async def test_invocation_uses_exact_cli_and_persists_redacted_evidence( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + calls: list[tuple[tuple[str, ...], dict[str, Any]]] = [] + + async def create_process(*argv: str, **kwargs: Any) -> FakeProcess: + calls.append((argv, kwargs)) + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, selected = _access(tmp_path, monkeypatch, create_process) + response = await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="run-1.ready.0", + ) + + assert selected == [ + "sandbox-manager-cli", + "sandbox-runtime-cli", + "sandbox-observability-cli", + ] + argv, kwargs = calls[0] + assert Path(argv[0]).name == "sandbox-manager-cli.exe" + assert argv[1:] == ( + "--gateway-endpoint", + "127.0.0.1:47621", + "--gateway-auth-token=-secret-token", + "--request-id", + "run-1.ready.0", + "list_sandboxes", + ) + assert kwargs["cwd"] == access._package_root + assert response.request_id == "run-1.ready.0" + assert response.value == {"sandboxes": []} + assert response.latency_ns > 0 + metadata_files = list((run_root / "cli-subprocesses").glob("*.json")) + assert len(metadata_files) == 1 + metadata_text = metadata_files[0].read_text(encoding="utf-8") + metadata = json.loads(metadata_text) + assert "-secret-token" not in metadata_text + assert metadata["sanitized_argv"][3] == "--gateway-auth-token=[REDACTED]" + assert metadata["response_validation"] == "passed" + assert metadata["request_id"] == response.request_id + assert metadata["schema_version"] == 2 + assert metadata["evidence_commit"] == "metadata-packed-payload-fsync-v1" + assert response.transport_evidence["metadata_path"].endswith(".json") + assert "stdout_base64" not in response.transport_evidence + assert "stderr_base64" not in response.transport_evidence + + +@pytest.mark.asyncio +async def test_named_pipe_uri_is_passed_exactly_and_token_is_redacted( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + calls: list[tuple[str, ...]] = [] + uri = "npipe://./pipe/ephemeral-sandbox-benchmark-exact-uri" + + async def create_process(*argv: str, **_kwargs: Any) -> FakeProcess: + calls.append(argv) + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access( + tmp_path, + monkeypatch, + create_process, + GatewayEndpoint.windows_named_pipe(uri), + ) + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="run-1.named-pipe", + ) + + assert calls[0][1:4] == ( + "--gateway-endpoint", + uri, + "--gateway-auth-token=-secret-token", + ) + metadata_text = next( + (run_root / "cli-subprocesses").glob("*.json") + ).read_text(encoding="utf-8") + metadata = json.loads(metadata_text) + assert "-secret-token" not in metadata_text + assert metadata["sanitized_argv"][2] == uri + assert metadata["sanitized_argv"][3] == ( + "--gateway-auth-token=[REDACTED]" + ) + + +@pytest.mark.asyncio +async def test_executable_digests_are_cached_once_per_access_instance( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + digest_calls: list[Path] = [] + real_sha256_file = product_cli_module._sha256_file + + def observed_sha256_file(path: Path) -> str: + digest_calls.append(path) + return real_sha256_file(path) + + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(b'{"sandboxes":[]}\n') + + monkeypatch.setattr(product_cli_module, "_sha256_file", observed_sha256_file) + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + + for index in range(2): + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id=f"run-1.digest.{index}", + ) + + assert sorted(path.name for path in digest_calls) == [ + "sandbox-manager-cli.exe", + "sandbox-observability-cli.exe", + "sandbox-runtime-cli.exe", + ] + manager_digests = { + json.loads(path.read_text(encoding="utf-8"))["executable_sha256"] + for path in (run_root / "cli-subprocesses").glob("*.json") + } + assert manager_digests == {real_sha256_file(access._executables["manager"])} + + +@pytest.mark.asyncio +async def test_invocation_evidence_uses_metadata_as_the_only_durable_commit( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + fsync_calls: list[int] = [] + monkeypatch.setattr( + product_cli_module.os, + "fsync", + lambda descriptor: fsync_calls.append(descriptor), + ) + + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="run-1.commit", + ) + + evidence = run_root / "cli-subprocesses" + assert len(fsync_calls) == 1 + (os.name != "nt") + assert sorted(path.suffix for path in evidence.iterdir()) == [ + ".json", + ".stderr", + ".stdout", + ] + metadata = json.loads(next(evidence.glob("*.json")).read_text(encoding="utf-8")) + assert metadata["evidence_commit"] == "metadata-packed-payload-fsync-v1" + for stream in ("stdout", "stderr"): + payload = (run_root / metadata[f"{stream}_path"]).read_bytes() + assert base64.b64decode(metadata[f"{stream}_base64"], validate=True) == payload + assert metadata[f"{stream}_bytes"] == len(payload) + assert metadata[f"{stream}_sha256"] == ( + f"sha256:{hashlib.sha256(payload).hexdigest()}" + ) + + +@pytest.mark.asyncio +async def test_trial_evidence_defers_markers_until_one_bounded_flush( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + fsync_calls: list[int] = [] + monkeypatch.setattr( + product_cli_module.os, + "fsync", + lambda descriptor: fsync_calls.append(descriptor), + ) + access.begin_trial_evidence("trial-1") + + responses = await asyncio.gather( + *( + access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id=f"trial-1.request.{index}", + ) + for index in range(3) + ) + ) + + evidence = run_root / "cli-subprocesses" + assert not list(evidence.glob("*.json")) + assert len(list(evidence.glob("*.stdout"))) == 3 + assert len(list(evidence.glob("*.stderr"))) == 3 + assert fsync_calls == [] + assert all( + response.transport_evidence is not None + and response.transport_evidence["metadata_path"].endswith(".json") + for response in responses + ) + + await access.flush_trial_evidence("trial-1") + access.end_trial_evidence("trial-1") + + metadata_paths = sorted(evidence.glob("*.json")) + assert len(metadata_paths) == 3 + assert len(fsync_calls) == 3 + (os.name != "nt") + for path in metadata_paths: + metadata = json.loads(path.read_text(encoding="utf-8")) + assert metadata["schema_version"] == 2 + assert metadata["evidence_commit"] == "metadata-packed-payload-fsync-v1" + for stream in ("stdout", "stderr"): + payload = (run_root / metadata[f"{stream}_path"]).read_bytes() + assert ( + base64.b64decode(metadata[f"{stream}_base64"], validate=True) == payload + ) + + +@pytest.mark.asyncio +async def test_concurrent_trial_evidence_preserves_every_unique_invocation( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*argv: str, **_kwargs: Any) -> FakeProcess: + request_id = argv[argv.index("--request-id") + 1] + index = int(request_id.rsplit(".", 1)[1]) + await asyncio.sleep((7 - index) * 0.001) + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + access.begin_trial_evidence("trial-concurrent") + request_ids = [f"trial-concurrent.request.{index}" for index in range(8)] + + responses = await asyncio.gather( + *( + access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id=request_id, + ) + for request_id in request_ids + ) + ) + real_write_new = product_cli_module._write_new + write_lock = threading.Lock() + active_writes = 0 + peak_active_writes = 0 + + def observe_parallel_writes( + path: Path, + content: bytes, + *, + durable: bool = True, + discard_on_error: bool = False, + ) -> None: + nonlocal active_writes, peak_active_writes + with write_lock: + active_writes += 1 + peak_active_writes = max(peak_active_writes, active_writes) + try: + time.sleep(0.01) + real_write_new( + path, + content, + durable=durable, + discard_on_error=discard_on_error, + ) + finally: + with write_lock: + active_writes -= 1 + + monkeypatch.setattr(product_cli_module, "_EVIDENCE_FLUSH_CONCURRENCY", 2) + monkeypatch.setattr(product_cli_module, "_write_new", observe_parallel_writes) + await access.flush_trial_evidence("trial-concurrent") + access.end_trial_evidence("trial-concurrent") + + metadata = [ + json.loads(path.read_text(encoding="utf-8")) + for path in (run_root / "cli-subprocesses").glob("*.json") + ] + assert sorted(item["request_id"] for item in metadata) == request_ids + assert len({item["invocation_id"] for item in metadata}) == len(request_ids) + assert len( + { + response.transport_evidence["metadata_path"] + for response in responses + if response.transport_evidence is not None + } + ) == len(request_ids) + assert peak_active_writes == 2 + + +@pytest.mark.asyncio +async def test_trial_evidence_lifecycle_rejects_nested_and_mismatched_calls( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + raise AssertionError("lifecycle test must not launch a process") + + access, _, _ = _access(tmp_path, monkeypatch, create_process) + with pytest.raises(ProductAccessError, match="not active"): + await access.flush_trial_evidence("trial-1") + with pytest.raises(ProductAccessError, match="not active"): + access.end_trial_evidence("trial-1") + + access.begin_trial_evidence("trial-1") + with pytest.raises(ProductAccessError, match="already active"): + access.begin_trial_evidence("trial-1") + with pytest.raises(ProductAccessError, match="does not match"): + await access.flush_trial_evidence("trial-2") + with pytest.raises(ProductAccessError, match="does not match"): + access.end_trial_evidence("trial-2") + + await access.flush_trial_evidence("trial-1") + access.end_trial_evidence("trial-1") + + +@pytest.mark.asyncio +async def test_failed_trial_flush_commits_other_markers_and_closes_fail_closed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + access.begin_trial_evidence("trial-failed-flush") + await asyncio.gather( + *( + access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id=f"trial-failed-flush.request.{index}", + ) + for index in range(3) + ) + ) + + real_write_new = product_cli_module._write_new + failed = False + failure_lock = threading.Lock() + + def fail_one_marker( + path: Path, + content: bytes, + *, + durable: bool = True, + discard_on_error: bool = False, + ) -> None: + nonlocal failed + with failure_lock: + inject_failure = path.suffix == ".json" and not failed + failed = failed or inject_failure + if inject_failure: + raise OSError("injected grouped marker failure") + real_write_new( + path, + content, + durable=durable, + discard_on_error=discard_on_error, + ) + + monkeypatch.setattr(product_cli_module, "_write_new", fail_one_marker) + with pytest.raises( + BaseExceptionGroup, match="CLI trial evidence commit failed" + ) as captured: + await access.flush_trial_evidence("trial-failed-flush") + assert any( + isinstance(error, OSError) and "injected grouped marker failure" in str(error) + for error in captured.value.exceptions + ) + assert len(list((run_root / "cli-subprocesses").glob("*.json"))) == 2 + with pytest.raises(ProductAccessError, match="1 pending commit marker"): + access.end_trial_evidence("trial-failed-flush") + + monkeypatch.setattr(product_cli_module, "_write_new", real_write_new) + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="outside-after-failed-flush", + ) + assert len(list((run_root / "cli-subprocesses").glob("*.json"))) == 3 + + +@pytest.mark.asyncio +async def test_end_trial_evidence_rejects_and_discards_unflushed_markers( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(b'{"sandboxes":[]}\n') + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + access.begin_trial_evidence("trial-unflushed") + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="trial-unflushed.request.0", + ) + + with pytest.raises(ProductAccessError, match="1 pending commit marker"): + access.end_trial_evidence("trial-unflushed") + assert not list((run_root / "cli-subprocesses").glob("*.json")) + + access.begin_trial_evidence("trial-next") + await access.flush_trial_evidence("trial-next") + access.end_trial_evidence("trial-next") + + +def test_failed_metadata_flush_leaves_redacted_uncommitted_payloads( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + raise AssertionError("direct persistence test must not launch a process") + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + real_fsync = product_cli_module.os.fsync + + def fail_fsync(_descriptor: int) -> None: + raise OSError("injected metadata flush failure") + + monkeypatch.setattr(product_cli_module.os, "fsync", fail_fsync) + with pytest.raises(OSError, match="injected metadata flush failure"): + access._persist_invocation( + "manager", + "list_sandboxes", + "run-1.flush-failure", + ["sandbox-manager-cli.exe", "--gateway-auth-token=[REDACTED]"], + 10, + 20, + 0, + b'{"token":"-secret-token"}\n', + b"-secret-token warning\n", + "transport_error:credential_echo", + ) + + evidence = run_root / "cli-subprocesses" + assert not list(evidence.glob("*.json")) + payloads = sorted(evidence.iterdir()) + assert [path.suffix for path in payloads] == [".stderr", ".stdout"] + assert all(b"-secret-token" not in path.read_bytes() for path in payloads) + before_retry = {path.name: path.read_bytes() for path in payloads} + + monkeypatch.setattr(product_cli_module.os, "fsync", real_fsync) + with pytest.raises(FileExistsError): + access._persist_invocation( + "manager", + "list_sandboxes", + "run-1.flush-failure", + ["sandbox-manager-cli.exe", "--gateway-auth-token=[REDACTED]"], + 10, + 20, + 0, + b"different stdout\n", + b"", + "passed", + ) + assert {path.name: path.read_bytes() for path in payloads} == before_retry + assert not list(evidence.glob("*.json")) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("stdout", "stderr", "returncode", "exception"), + [ + (b"not-json\n", b"", 0, GatewayTransportError), + (b'{"sandboxes":[]}\n', b"warning\n", 0, GatewayTransportError), + ( + b"", + b'{"error":{"kind":"invalid_request","message":"bad"}}\n', + 2, + GatewayProductError, + ), + ], +) +async def test_invalid_cli_results_fail_closed_and_are_retained( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + stdout: bytes, + stderr: bytes, + returncode: int, + exception: type[BaseException], +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return FakeProcess(stdout, stderr, returncode) + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + with pytest.raises(exception): + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=1, + request_id="run-1.invalid", + ) + metadata = json.loads( + next((run_root / "cli-subprocesses").glob("*.json")).read_text(encoding="utf-8") + ) + assert metadata["response_validation"] != "passed" + assert metadata["return_code"] == returncode + + +@pytest.mark.asyncio +async def test_timeout_kills_reaps_and_records_cli_process( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + process = FakeProcess(b"", block_until_killed=True) + + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return process + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + with pytest.raises(GatewayTransportError, match="cli_timeout"): + await access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=0.001, + request_id="run-1.timeout", + ) + assert process.killed + metadata = json.loads( + next((run_root / "cli-subprocesses").glob("*.json")).read_text(encoding="utf-8") + ) + assert metadata["response_validation"] == "timeout" + assert metadata["return_code"] == -9 + + +@pytest.mark.asyncio +async def test_cancellation_kills_reaps_and_records_cli_process( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + process = FakeProcess(b"", block_until_killed=True) + + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return process + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + task = asyncio.create_task( + access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=10, + request_id="run-1.cancelled", + ) + ) + await process.started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert process.killed + metadata = json.loads( + next((run_root / "cli-subprocesses").glob("*.json")).read_text(encoding="utf-8") + ) + assert metadata["response_validation"] == "cancelled" + assert metadata["return_code"] == -9 + + +@pytest.mark.asyncio +async def test_cancelled_trial_invocation_is_buffered_and_committed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + process = FakeProcess(b"", block_until_killed=True) + + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + return process + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + access.begin_trial_evidence("trial-cancelled") + task = asyncio.create_task( + access._invoke( + "manager", + "list_sandboxes", + [], + timeout_seconds=10, + request_id="trial-cancelled.request.0", + ) + ) + await process.started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + evidence = run_root / "cli-subprocesses" + assert process.killed + assert not list(evidence.glob("*.json")) + await access.flush_trial_evidence("trial-cancelled") + access.end_trial_evidence("trial-cancelled") + + metadata = json.loads(next(evidence.glob("*.json")).read_text(encoding="utf-8")) + assert metadata["response_validation"] == "cancelled" + assert metadata["return_code"] == -9 + + +@pytest.mark.asyncio +async def test_file_write_uses_exact_bounded_content_file_and_removes_it( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + observed_content: bytes | None = None + observed_path: Path | None = None + captured_argv: tuple[str, ...] = () + + async def create_process(*argv: str, **_kwargs: Any) -> FakeProcess: + nonlocal observed_content, observed_path, captured_argv + captured_argv = argv + content_index = argv.index("--content-file") + 1 + observed_path = Path(argv[content_index]) + observed_content = observed_path.read_bytes() + return FakeProcess(b'{"path":"payload.txt","bytes_written":262144}\n') + + access, _, _ = _access(tmp_path, monkeypatch, create_process) + access._sandboxes.add("sandbox-1") + content = ("x\n" * 131_072)[:262_144] + await access.stage_file_write_content(content, request_id="run-1.write.0") + staged_path = access._content_path("run-1.write.0") + assert staged_path.read_bytes() == content.encode() + response = await access.file_write( + "sandbox-1", + session_id="session-1", + path="payload.txt", + content=content, + timeout_ms=1000, + request_id="run-1.write.0", + ) + + assert observed_content == content.encode() + assert observed_path is not None and not observed_path.exists() + assert "--content-file" in captured_argv + assert "--content" not in captured_argv + assert response.value["bytes_written"] == 262_144 + + +@pytest.mark.asyncio +async def test_staged_file_write_rejects_content_mismatch_and_can_be_discarded( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + raise AssertionError("mismatched staged content must not launch a process") + + access, _, _ = _access(tmp_path, monkeypatch, create_process) + access._sandboxes.add("sandbox-1") + await access.stage_file_write_content("expected", request_id="run-1.write.mismatch") + + with pytest.raises(ProductAccessError, match="does not match"): + await access.file_write( + "sandbox-1", + session_id=None, + path="payload.txt", + content="different", + timeout_ms=1000, + request_id="run-1.write.mismatch", + ) + + access.discard_file_write_content("run-1.write.mismatch") + assert not access._content_path("run-1.write.mismatch").exists() + + +@pytest.mark.asyncio +@pytest.mark.skipif(os.name != "nt", reason="Windows extended-path identity regression") +async def test_create_accepts_equivalent_windows_extended_workspace_path( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + workspace_holder: dict[str, Path] = {} + + async def create_process(*_argv: str, **_kwargs: Any) -> FakeProcess: + workspace = workspace_holder["workspace"].resolve(strict=True) + extended = f"\\\\?\\{workspace}" + payload = { + "id": "sandbox-1", + "workspace_root": extended, + "state": "ready", + "activity_revision": 0, + "daemon": {"host": "127.0.0.1", "port": 32768}, + "daemon_http": {"host": "127.0.0.1", "port": 32769}, + "shared_base": { + "source": "C:\\cache\\base", + "target": "/eos/layer-stack/base", + "root_hash": "abc", + "readonly": True, + }, + "resource_profile": { + "name": "standard", + "nano_cpus": 1_000_000_000, + "memory_high_bytes": 402_653_184, + "memory_max_bytes": 536_870_912, + "pids_max": 256, + "workload_memory_high_bytes": 402_653_184, + "workload_memory_max_bytes": 402_653_184, + "workload_pids_max": 224, + "control_plane_pids_reserve": 32, + "daemon_runtime_profile": "standard", + "separate_workload_cgroup": True, + }, + } + return FakeProcess((json.dumps(payload) + "\n").encode()) + + access, run_root, _ = _access(tmp_path, monkeypatch, create_process) + workspace = run_root / "workspace" + workspace.mkdir() + workspace_holder["workspace"] = workspace + + record, response = await access.create_sandbox( + "image@sha256:digest", + workspace, + request_id="run-1.create", + ) + + assert record.id == "sandbox-1" + assert response.value["workspace_root"].startswith("\\\\?\\") + assert access.owned_sandboxes == {"sandbox-1"} diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_resource_sampling.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_resource_sampling.py new file mode 100644 index 0000000..be59f2f --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_resource_sampling.py @@ -0,0 +1,869 @@ +import asyncio +import os +import time +from pathlib import Path +from types import MethodType + +import pytest + +from benchmark_lab import resource_sampling +from benchmark_lab.observability import parse_cgroup, parse_snapshot +from benchmark_lab.resource_sampling import ( + ResourceSamplingError, + TrialResourceSampler, + WorkspaceMetricCache, + _runner_rss, +) + +METRIC_IDS = ( + "runner_rss_bytes", + "daemon_rss_bytes", + "daemon_cpu_time_ns", + "sandbox_memory_current_bytes", + "sandbox_memory_peak_bytes", + "sandbox_cpu_time_ns", + "sandbox_block_read_bytes", + "sandbox_block_write_bytes", + "workspace_logical_bytes", + "workspace_allocated_bytes", + "workspace_file_count", + "layerstack_bytes", + "upperdir_bytes", + "host_free_bytes", +) + + +def _test_records(index: int) -> list[dict]: + return [ + { + "cell_id": "cell-1", + "trial_id": "trial-1", + "request_id": None, + "reading": { + "metric_id": metric_id, + "monotonic_offset_ns": index, + "value": { + "availability": "unavailable", + "source": "test", + "reason": "test collector", + }, + }, + } + for metric_id in METRIC_IDS + ] + + +class FastProduct: + def __init__(self) -> None: + self.cgroup_timestamp = 0 + self.snapshot_timestamp = 0 + self.daemon_calls = 0 + self.timestamp_base_ms = time.time_ns() // 1_000_000 + 10_000 + + async def observe_cgroup(self, sandbox: str, *, request_id: str): + self.cgroup_timestamp += 1 + return parse_cgroup( + { + "view": "cgroup", + "scope": "sandbox", + "availability": "available", + "errors": [], + "topology": { + "daemon": { + "available": True, + "pid": 7, + "resident_memory_bytes": 1024, + "peak_resident_memory_bytes": 2048, + "cpu_time_us": 1, + } + }, + "series": [ + { + "ts": self.timestamp_base_ms + self.cgroup_timestamp, + "sample_delta_ms": None, + "metrics": { + "metrics_source": "docker_engine", + "cpu_usec": 1, + "mem_cur": 1024, + "mem_max": 2048, + "io_rbytes": 0, + "io_wbytes": 0, + }, + "deltas": {}, + } + ], + } + ) + + async def observe_snapshot(self, sandbox: str, *, request_id: str): + self.snapshot_timestamp += 1 + return parse_snapshot( + { + "sandbox_id": sandbox, + "lifecycle_state": "ready", + "availability": "available", + "sampled_at_unix_ms": ( + self.timestamp_base_ms + self.snapshot_timestamp + ), + "errors": [], + "daemon": { + "daemon_pid": 7, + "runtime_dir": "/run/fake", + "event_store": { + "dropped_storage": 0, + "dropped_oversized": 0, + "truncated_records": 0, + }, + }, + "resources": {"latest": None, "history": []}, + "workspaces": [ + { + "workspace_id": "workspace-1", + "lifecycle_state": "active", + "finalization_state": "active", + "network_profile": "shared", + "finalize_policy": "no_op", + "layers": {"base_root_hash": None, "layer_count": 1}, + "namespace_fd_count": 0, + "resources": { + "latest": { + "ts": ( + self.timestamp_base_ms + self.snapshot_timestamp + ), + "sample_delta_ms": None, + "metrics": { + "disk_allocated_bytes": 8192, + "disk_truncated": False, + }, + "deltas": {}, + }, + "history": [], + }, + "active_namespace_executions": [], + } + ], + "stack": None, + }, + sandbox, + ) + + async def observe_daemon(self, sandbox: str, *, request_id: str): + self.daemon_calls += 1 + raise AssertionError("resource sampling must reuse cgroup.topology.daemon") + + +class DelayedBoundaryProduct(FastProduct): + def __init__(self) -> None: + super().__init__() + self.cgroup_request_ids: list[str] = [] + self.snapshot_request_ids: list[str] = [] + + async def observe_cgroup(self, sandbox: str, *, request_id: str): + self.cgroup_request_ids.append(request_id) + view = await super().observe_cgroup(sandbox, request_id=request_id) + if len(self.cgroup_request_ids) == 1: + view.availability = "partial" + view.errors = ["resource ring is not available yet"] + view.series = [] + return view + + async def observe_snapshot(self, sandbox: str, *, request_id: str): + self.snapshot_request_ids.append(request_id) + view = await super().observe_snapshot(sandbox, request_id=request_id) + if len(self.snapshot_request_ids) == 1: + view.workspaces[0].resources.latest = None + return view + + +class StalePostBoundaryProduct(FastProduct): + async def observe_cgroup(self, sandbox: str, *, request_id: str): + view = await super().observe_cgroup(sandbox, request_id=request_id) + if self.cgroup_timestamp == 2: + view.series[-1].ts = self.timestamp_base_ms + 1 + return view + + async def observe_snapshot(self, sandbox: str, *, request_id: str): + view = await super().observe_snapshot(sandbox, request_id=request_id) + if self.snapshot_timestamp == 2: + view.workspaces[0].resources.latest.ts = self.timestamp_base_ms + 1 + return view + + +class PreThresholdPostBoundaryProduct(FastProduct): + def __init__(self) -> None: + super().__init__() + self.timestamp_base_ms = time.time_ns() // 1_000_000 - 10_000 + self.cgroup_emitted_ms: list[int] = [] + self.workspace_emitted_ms: list[int] = [] + + async def observe_cgroup(self, sandbox: str, *, request_id: str): + view = await super().observe_cgroup(sandbox, request_id=request_id) + if self.cgroup_timestamp >= 3: + view.series[-1].ts = time.time_ns() // 1_000_000 + 10_000 + self.cgroup_emitted_ms.append(view.series[-1].ts) + return view + + async def observe_snapshot(self, sandbox: str, *, request_id: str): + view = await super().observe_snapshot(sandbox, request_id=request_id) + if self.snapshot_timestamp >= 3: + timestamp = time.time_ns() // 1_000_000 + 10_000 + view.sampled_at_unix_ms = timestamp + view.workspaces[0].resources.latest.ts = timestamp + self.workspace_emitted_ms.append(view.workspaces[0].resources.latest.ts) + return view + + +class PendingBoundaryProduct(FastProduct): + async def observe_cgroup(self, sandbox: str, *, request_id: str): + view = await super().observe_cgroup(sandbox, request_id=request_id) + view.availability = "partial" + view.errors = ["resource ring is not available yet"] + view.series = [] + return view + + +class WorkspaceAppearsProduct(FastProduct): + async def observe_snapshot(self, sandbox: str, *, request_id: str): + view = await super().observe_snapshot(sandbox, request_id=request_id) + if self.snapshot_timestamp < 3: + view.workspaces = [] + return view + + +class MalformedEmbeddedDaemonProduct(FastProduct): + async def observe_cgroup(self, sandbox: str, *, request_id: str): + view = await super().observe_cgroup(sandbox, request_id=request_id) + view.topology["daemon"]["cpu_time_us"] = "invalid" + return view + + +class BlockingPeriodicProduct(FastProduct): + def __init__(self) -> None: + super().__init__() + self.periodic_started = asyncio.Event() + self.periodic_release = asyncio.Event() + self.final_boundary_started_ns: int | None = None + + async def observe_cgroup(self, sandbox: str, *, request_id: str): + if request_id.endswith(".observe.cgroup.1"): + self.periodic_started.set() + await self.periodic_release.wait() + elif request_id.endswith(".observe.cgroup.2.boundary.0"): + self.final_boundary_started_ns = time.monotonic_ns() + return await super().observe_cgroup(sandbox, request_id=request_id) + + async def observe_snapshot(self, sandbox: str, *, request_id: str): + if request_id.endswith(".observe.snapshot.1"): + self.periodic_started.set() + await self.periodic_release.wait() + return await super().observe_snapshot(sandbox, request_id=request_id) + + +@pytest.mark.asyncio +async def test_mandatory_and_periodic_samples_reuse_exact_cgroup_daemon( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + product = FastProduct() + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + ) + + await asyncio.wait_for(sampler.start(), timeout=1) + async with asyncio.timeout(1): + while product.cgroup_timestamp < 2: + await asyncio.sleep(0.01) + await asyncio.wait_for(sampler.stop(), timeout=1) + + assert product.daemon_calls == 0 + assert product.cgroup_timestamp >= 3 + daemon_readings = [ + item["reading"] + for batch in batches + for item in batch + if item["reading"]["metric_id"] == "daemon_rss_bytes" + ] + assert len(daemon_readings) >= 3 + assert all( + reading["source"] + == "product_observability.cgroup.topology.daemon.peak_resident_memory_bytes" + and reading["value"] == {"availability": "available", "value": 2048.0} + for reading in daemon_readings + ) + + +@pytest.mark.asyncio +async def test_malformed_embedded_daemon_is_never_used_as_metric_data( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + product = MalformedEmbeddedDaemonProduct() + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + ) + + await sampler.start() + await sampler.stop() + + assert product.daemon_calls == 0 + for batch in batches: + daemon_values = [ + item["reading"]["value"] + for item in batch + if item["reading"]["metric_id"] + in {"daemon_rss_bytes", "daemon_cpu_time_ns"} + ] + assert len(daemon_values) == 2 + assert all( + value["availability"] == "unavailable" + and "ObservabilityError" in value["reason"] + for value in daemon_values + ) + + +@pytest.mark.asyncio +async def test_sampler_discards_transient_boundary_polls( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + product = DelayedBoundaryProduct() + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + ) + + await sampler.start() + await sampler.stop() + + assert len(batches) == 2 + assert len(product.cgroup_request_ids) == 3 + assert len(product.snapshot_request_ids) == 3 + assert all(".boundary." in item for item in product.cgroup_request_ids) + for metric_id in ( + "sandbox_cpu_time_ns", + "sandbox_block_read_bytes", + "sandbox_block_write_bytes", + "upperdir_bytes", + ): + readings = [ + next( + item["reading"] + for item in batch + if item["reading"]["metric_id"] == metric_id + ) + for batch in batches + ] + assert all( + reading["value"]["availability"] == "available" for reading in readings + ) + + +@pytest.mark.asyncio +async def test_sampler_waits_for_post_boundary_newer_than_baseline( + tmp_path: Path, +) -> None: + product = StalePostBoundaryProduct() + batches: list[list[dict]] = [] + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + ) + + await sampler.start() + await sampler.stop() + + assert len(batches) == 2 + assert product.cgroup_timestamp == 3 + assert product.snapshot_timestamp == 3 + + +@pytest.mark.asyncio +async def test_sampler_delays_first_post_response_boundary_poll( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + product = FastProduct() + delays: list[tuple[float, int]] = [] + batches: list[list[dict]] = [] + original_sleep = asyncio.sleep + + async def tracked_sleep(delay: float) -> None: + delays.append((delay, product.cgroup_timestamp)) + await original_sleep(0) + + async def sink(batch: list[dict]) -> None: + batches.append(batch) + + monkeypatch.setattr(resource_sampling.asyncio, "sleep", tracked_sleep) + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=sink, + ) + + await sampler.start() + assert delays == [] + await sampler.stop() + + assert delays[0][0] == pytest.approx( + resource_sampling._POST_BOUNDARY_FIRST_POLL_DELAY_SECONDS, + abs=0.01, + ) + assert delays[0][1] == 1 + assert product.cgroup_timestamp == 2 + assert product.snapshot_timestamp == 2 + assert len(batches) == 2 + + +@pytest.mark.asyncio +async def test_sampler_rejects_post_sample_taken_before_response_threshold( + tmp_path: Path, +) -> None: + product = PreThresholdPostBoundaryProduct() + batches: list[list[dict]] = [] + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + ) + + await sampler.start() + await sampler.stop() + + assert len(batches) == 2 + assert product.cgroup_timestamp == 3 + assert product.snapshot_timestamp == 3 + assert sampler._post_response_threshold_ms is not None + assert ( + product.cgroup_emitted_ms[0] + < product.cgroup_emitted_ms[1] + <= sampler._post_response_threshold_ms + < product.cgroup_emitted_ms[2] + ) + assert ( + product.workspace_emitted_ms[0] + < product.workspace_emitted_ms[1] + <= sampler._post_response_threshold_ms + < product.workspace_emitted_ms[2] + ) + + +@pytest.mark.asyncio +async def test_workspace_create_post_boundary_waits_for_created_workspace( + tmp_path: Path, +) -> None: + product = WorkspaceAppearsProduct() + batches: list[list[dict]] = [] + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0, result=batches.append(batch)), + expected_post_workspace_count=1, + ) + + await sampler.start() + await sampler.stop() + + assert len(batches) == 2 + assert product.cgroup_timestamp == 3 + assert product.snapshot_timestamp == 3 + upperdir = [ + next( + item["reading"] + for item in batch + if item["reading"]["metric_id"] == "upperdir_bytes" + ) + for batch in batches + ] + assert [reading["value"]["value"] for reading in upperdir] == [0.0, 8192.0] + + +@pytest.mark.asyncio +async def test_sampler_fails_closed_when_boundary_never_becomes_ready( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(resource_sampling, "_BOUNDARY_READINESS_TIMEOUT_SECONDS", 0.0) + sampler = TrialResourceSampler( + product=PendingBoundaryProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + ) + + with pytest.raises(ResourceSamplingError, match="readiness timed out"): + await sampler.start() + + +@pytest.mark.asyncio +async def test_sampler_uses_fixed_deadlines_and_defers_ordered_persistence( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + + async def sink(batch: list[dict]) -> None: + batches.append(batch) + + started_ns = time.monotonic_ns() + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=started_ns, + sink=sink, + ) + await sampler.start() + await asyncio.sleep(0.24) + + # Collection runs independently, but journal persistence is deferred until + # the explicit trial-boundary stop. + assert batches == [] + await sampler.stop() + + assert len(batches) >= 4 + assert all(len(batch) == 14 for batch in batches) + periodic = [ + batch[0]["reading"]["scheduled_monotonic_offset_ns"] + for batch in batches + if batch[0]["reading"].get("sampled") is True + ] + assert len(periodic) >= 2 + assert 50_000_000 <= periodic[1] - periodic[0] <= 175_000_000 + offsets = [batch[0]["reading"]["monotonic_offset_ns"] for batch in batches] + assert offsets == sorted(offsets) + + +@pytest.mark.asyncio +async def test_sampler_records_saturation_as_explicit_unavailability( + tmp_path: Path, +) -> None: + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + ) + release = asyncio.Event() + + # Keep the collector implementation tiny while deterministically occupying + # the single expensive-sample admission slot. + async def blocked(self, *, index: int, sampled: bool, scheduled_ns: int): + await release.wait() + return _test_records(index) + + sampler._sample = MethodType(blocked, sampler) + for _ in range(2): + sampler._launch(sampled=True) + await asyncio.sleep(0) + + saturated = await sampler._samples[1] + assert len(saturated) == 14 + assert all( + item["reading"]["value"]["availability"] == "unavailable" + and "concurrency cap" in item["reading"]["value"]["reason"] + and item["reading"]["collection_started_monotonic_offset_ns"] is None + for item in saturated + ) + release.set() + await sampler._samples[0] + + +@pytest.mark.asyncio +async def test_sampler_awaits_baseline_before_start_returns(tmp_path: Path) -> None: + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + ) + entered = asyncio.Event() + release = asyncio.Event() + + async def controlled( + self, *, index: int, sampled: bool, scheduled_ns: int + ) -> list[dict]: + if index == 0: + entered.set() + await release.wait() + return _test_records(index) + + sampler._sample = MethodType(controlled, sampler) + starting = asyncio.create_task(sampler.start()) + await entered.wait() + assert not starting.done() + release.set() + await starting + await sampler.stop() + + +@pytest.mark.asyncio +async def test_stop_serializes_boundary_after_active_periodic_collector( + tmp_path: Path, +) -> None: + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=10, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + ) + periodic_started = asyncio.Event() + release_periodic = asyncio.Event() + active = 0 + maximum_active = 0 + actual_samples: list[tuple[int, bool]] = [] + + async def controlled( + self, *, index: int, sampled: bool, scheduled_ns: int + ) -> list[dict]: + nonlocal active, maximum_active + active += 1 + maximum_active = max(maximum_active, active) + actual_samples.append((index, sampled)) + try: + if sampled: + periodic_started.set() + await release_periodic.wait() + return _test_records(index) + finally: + active -= 1 + + sampler._sample = MethodType(controlled, sampler) + await sampler.start() + await periodic_started.wait() + stopping = asyncio.create_task(sampler.stop()) + await asyncio.sleep(0) + + assert actual_samples == [(0, False), (1, True)] + assert not stopping.done() + release_periodic.set() + await stopping + + assert maximum_active == 1 + assert len([sample for sample in actual_samples if sample[1] is False]) == 2 + + +@pytest.mark.asyncio +async def test_stop_overlaps_freshness_wait_with_active_periodic_drain( + tmp_path: Path, +) -> None: + product = BlockingPeriodicProduct() + sampler = TrialResourceSampler( + product=product, + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=10, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + ) + + await sampler.start() + await asyncio.wait_for(product.periodic_started.wait(), timeout=1) + stop_started_ns = time.monotonic_ns() + stopping = asyncio.create_task(sampler.stop()) + + # Keep the admitted periodic collector active beyond the complete + # post-response freshness wait. The mandatory product query must launch + # promptly after the collector drains, without starting before 100 ms. + await asyncio.sleep(0.15) + periodic_released_ns = time.monotonic_ns() + product.periodic_release.set() + await asyncio.wait_for(stopping, timeout=1) + + assert product.final_boundary_started_ns is not None + assert product.final_boundary_started_ns - stop_started_ns >= int( + resource_sampling._POST_BOUNDARY_FIRST_POLL_DELAY_SECONDS * 1_000_000_000 + ) + assert product.final_boundary_started_ns - periodic_released_ns < 75_000_000 + + +@pytest.mark.asyncio +async def test_workspace_tree_is_cached_but_dynamic_metrics_are_not( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + counts = {"workspace": 0, "dynamic": 0} + + def workspace_metrics(path: Path) -> dict: + counts["workspace"] += 1 + return { + "workspace_logical_bytes": (1, None), + "workspace_allocated_bytes": (1, None), + "workspace_file_count": (1, None), + } + + def dynamic_metrics(path: Path) -> dict: + counts["dynamic"] += 1 + return { + "runner_rss_bytes": (1, None), + "host_free_bytes": (1, None), + } + + monkeypatch.setattr(resource_sampling, "_workspace_metrics", workspace_metrics) + monkeypatch.setattr(resource_sampling, "_dynamic_local_metrics", dynamic_metrics) + cache = WorkspaceMetricCache() + for trial in ("trial-1", "trial-2"): + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id=trial, + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=lambda batch: asyncio.sleep(0), + workspace_cache=cache, + ) + await sampler.start() + await sampler.stop() + + assert counts == {"workspace": 1, "dynamic": 4} + + +@pytest.mark.asyncio +async def test_upperdir_delta_reading_uses_complete_product_workspace_scope( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + + async def sink(batch: list[dict]) -> None: + batches.append(batch) + + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=sink, + ) + await sampler.start() + await sampler.stop() + + readings = [ + next( + item["reading"] + for item in batch + if item["reading"]["metric_id"] == "upperdir_bytes" + ) + for batch in batches + ] + assert len(readings) == 2 + assert all( + reading["metric_semantic_revision"] == 2 + and reading["scope"] == "workspace" + and reading["kind"] == "gauge" + and reading["aggregation"] == "delta" + and reading["source"] + == "product_observability.snapshot.workspaces.disk_allocated_bytes.sum" + and reading["value"] == {"availability": "available", "value": 8192.0} + for reading in readings + ) + + +@pytest.mark.asyncio +async def test_create_sandbox_counter_deltas_are_explicitly_inapplicable( + tmp_path: Path, +) -> None: + batches: list[list[dict]] = [] + + async def sink(batch: list[dict]) -> None: + batches.append(batch) + + sampler = TrialResourceSampler( + product=FastProduct(), + sandbox="sandbox-1", + workspace=tmp_path, + cell_id="cell-1", + trial_id="trial-1", + interval_ms=100, + campaign_started_ns=time.monotonic_ns(), + sink=sink, + counter_delta_applicable=False, + ) + await sampler.start() + await sampler.stop() + + for batch in batches: + readings = {item["reading"]["metric_id"]: item["reading"] for item in batch} + for metric_id in ( + "daemon_cpu_time_ns", + "sandbox_cpu_time_ns", + "sandbox_block_read_bytes", + "sandbox_block_write_bytes", + ): + value = readings[metric_id]["value"] + assert value["availability"] == "unavailable" + assert "pre-create baseline cannot exist" in value["reason"] + assert ( + readings["sandbox_memory_current_bytes"]["value"]["availability"] + == "available" + ) + + +@pytest.mark.skipif(os.name != "nt", reason="Windows process API") +def test_windows_runner_rss_uses_full_width_process_handle() -> None: + value, reason = _runner_rss() + assert reason is None + assert value is not None and value > 0 diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_command.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_command.py index 9e081ee..97cfbae 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_command.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_command.py @@ -1,6 +1,10 @@ from pathlib import Path -from benchmark_lab.runner import TrialContext, _operation_evidence +from benchmark_lab.runner import ( + TrialContext, + _expected_command_output, + _operation_evidence, +) from benchmark_lab.transport import TimedGatewayResponse @@ -39,3 +43,7 @@ def test_command_evidence_preserves_stdout_and_empty_stderr_contract() -> None: "stdout": empty, "stderr": empty, } + + +def test_fixture_read_expected_output_matches_released_line_window() -> None: + assert _expected_command_output("fixture_read") == "4096" diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_squash.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_squash.py index 848c8fa..8de928a 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_squash.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_runner_squash.py @@ -22,6 +22,9 @@ ROOT = Path(__file__).resolve().parents[3] GOLDEN = ROOT / "tests/fixtures/golden/rust/quick-smoke-completed" +FROZEN_SQUASH_EVIDENCE = ( + ROOT / "tests/fixtures/golden/artifacts/operation-evidence-v1-squash.json" +) def _roots(tmp_path: Path) -> BenchmarkRoots: @@ -128,14 +131,13 @@ def test_squash_evidence_preserves_the_frozen_public_shape() -> None: } evidence = _operation_evidence(cell, [], context)["evidence"] - golden_file = next( - path - for path in (GOLDEN / "cells").rglob("operation-evidence-*.json") - if json.loads(path.read_text())["data"]["operation"] == "squash_layerstack" - ) - frozen = json.loads(golden_file.read_text())["data"]["evidence"] + frozen_envelope = json.loads(FROZEN_SQUASH_EVIDENCE.read_text()) + assert frozen_envelope["schema_name"] == "eos_benchmark_operation_evidence" + assert frozen_envelope["schema_version"] == 1 + assert frozen_envelope["data"]["operation"] == "squash_layerstack" + frozen = frozen_envelope["data"]["evidence"] - assert set(evidence) == set(frozen) + assert evidence == frozen assert evidence["observed_replaced_layer_count"] == 2 assert evidence["reclaimed_bytes"] == {"availability": "available", "value": 24} assert evidence["manifest_reduced"] is True diff --git a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_safety.py b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_safety.py index 259e7a6..b84a6b6 100644 --- a/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_safety.py +++ b/ephemeral-sandbox-v1/benchmark/backend/tests/unit/test_safety.py @@ -2,6 +2,7 @@ import pytest +from benchmark_lab.fixtures import native_filesystem_path from benchmark_lab.models import OwnedPathMarker from benchmark_lab.paths import BenchmarkRoots, MARKER_NAME from benchmark_lab.safety import OwnershipError, OwnershipLedger @@ -34,6 +35,26 @@ def test_deletion_requires_exact_marker_and_active_ledger(tmp_path: Path) -> Non assert not target.exists() +def test_deletion_supports_deep_owned_tree_on_windows(tmp_path: Path) -> None: + test, product, binaries = repositories(tmp_path) + roots = BenchmarkRoots.resolve(test, product, binaries, initialize=True) + target = roots.runs / "run-1" / "trial-1" + target.mkdir(parents=True) + ledger = OwnershipLedger(roots) + ledger.register(target, marker()) + + deepest = target + while len(str(deepest / "segment")) <= 280: + deepest /= "segment" + native_deepest = native_filesystem_path(deepest) + native_deepest.mkdir(parents=True) + (native_deepest / "sentinel").write_text("owned") + + ledger.remove(target, marker()) + + assert not target.exists() + + def test_outside_paths_roles_and_corrupt_markers_fail_closed(tmp_path: Path) -> None: test, product, binaries = repositories(tmp_path) roots = BenchmarkRoots.resolve(test, product, binaries, initialize=True) @@ -54,7 +75,9 @@ def test_outside_paths_roles_and_corrupt_markers_fail_closed(tmp_path: Path) -> assert target.exists() -def test_symlink_target_never_gains_cleanup_authority(tmp_path: Path) -> None: +def test_symlink_target_never_gains_cleanup_authority( + tmp_path: Path, symlink_or_skip +) -> None: test, product, binaries = repositories(tmp_path) roots = BenchmarkRoots.resolve(test, product, binaries, initialize=True) outside = roots.tmp / "outside" @@ -62,20 +85,22 @@ def test_symlink_target_never_gains_cleanup_authority(tmp_path: Path) -> None: sentinel = outside / "sentinel" sentinel.write_text("keep") link = roots.runs / "linked" - link.symlink_to(outside, target_is_directory=True) + symlink_or_skip(link, outside, target_is_directory=True) with pytest.raises(OwnershipError, match="symlink"): OwnershipLedger(roots).register(link, marker()) assert sentinel.read_text() == "keep" -def test_symlink_ancestor_is_rejected_even_when_it_points_inside_role(tmp_path: Path) -> None: +def test_symlink_ancestor_is_rejected_even_when_it_points_inside_role( + tmp_path: Path, symlink_or_skip +) -> None: test, product, binaries = repositories(tmp_path) roots = BenchmarkRoots.resolve(test, product, binaries, initialize=True) real = roots.runs / "real" target = real / "trial-1" target.mkdir(parents=True) link = roots.runs / "linked" - link.symlink_to(real, target_is_directory=True) + symlink_or_skip(link, real, target_is_directory=True) with pytest.raises(OwnershipError, match="crosses a symlink"): OwnershipLedger(roots).register(link / "trial-1", marker()) assert not (target / MARKER_NAME).exists() diff --git a/ephemeral-sandbox-v1/benchmark/defaults/definition-catalog.json b/ephemeral-sandbox-v1/benchmark/defaults/definition-catalog.json index 10d051e..9e2f8bc 100644 --- a/ephemeral-sandbox-v1/benchmark/defaults/definition-catalog.json +++ b/ephemeral-sandbox-v1/benchmark/defaults/definition-catalog.json @@ -1,6 +1,13 @@ { "schema_version": 2, "families": [ + { + "id": "sandbox_lifecycle", + "label": "Sandbox Lifecycle", + "help": "Fresh sandbox creation through the public manager operation over a deterministic benchmark-owned base mount.", + "research_question": "What is the native Windows CLI time to create one ready sandbox over the fixed paper workspace?", + "measured_boundary": "A fresh benchmark-owned fixture exists before the timer; the measured manager CLI subprocess creates one sandbox and returns its ready record." + }, { "id": "command", "label": "Command", @@ -18,7 +25,7 @@ { "id": "workspace_lifecycle", "label": "Workspace Lifecycle", - "help": "Concurrent explicit no_op session creation through the exact internal test adapter.", + "help": "Concurrent explicit no_op session creation through the public runtime operation.", "research_question": "How do workspace scale, network profile, and concurrent session count affect time to ready?", "measured_boundary": "A prepared sandbox and fixture precede the barrier; the measured operation creates C independent sessions and observes readiness." }, @@ -157,12 +164,12 @@ }, { "id": "upperdir_bytes", - "semantic_revision": 1, + "semantic_revision": 2, "unit": "bytes", - "scope": "layerstack", + "scope": "workspace", "kind": "gauge", "availability": "explicit_unavailable", - "aggregation": "maximum", + "aggregation": "delta", "direction": "lower_is_preferred" }, { @@ -207,6 +214,20 @@ "maximum_depth": 8 } }, + { + "schema_version": 1, + "id": "paper-100m", + "version": 1, + "label": "Paper 100 MiB, depth 100", + "help": "Paper stress profile with 4,000 files, exactly 100 MiB of deterministic logical content, and maximum depth 100.", + "generator_version": 1, + "standard": true, + "fixture": { + "file_count": 4000, + "logical_bytes": 104857600, + "maximum_depth": 100 + } + }, { "schema_version": 1, "id": "small", @@ -224,6 +245,83 @@ ] }, "operations": [ + { + "id": "create_sandbox", + "family": "sandbox_lifecycle", + "label": "Create sandbox", + "help": "Creates exactly one sandbox with the public manager CLI over a fresh benchmark-owned paper fixture.", + "measured_boundary": "The fixture is materialized before the trial; elapsed time spans native CLI process creation through validated JSON for a ready sandbox.", + "count_semantics_help": "Each trial issues exactly one create_sandbox product request.", + "semantic_revision": 1, + "factor_schema_revision": 1, + "count_semantics": { + "kind": "single_request" + }, + "execution_shape": "single_request", + "isolation": "fresh_sandbox_per_trial", + "cleanup": "destroy_sandbox_and_verify_registry", + "product_access": { + "kind": "public_gateway", + "action": "create_sandbox" + }, + "supported_cohorts": [ + "direct_client", + "product_cli" + ], + "security_class": "public_system_lifecycle", + "factors": [ + { + "id": "workspace_profile", + "label": "Workspace profile", + "help": "Deterministic materialized base mount presented to the sandbox manager.", + "value_kind": "choice", + "unit": null, + "constraint": { + "kind": "profile_catalog", + "catalog": "workspace_profiles" + }, + "comparison": "scientific_invariant" + }, + { + "id": "network_profile", + "label": "Network profile", + "help": "Controlled network profile for the sandbox lifecycle cell.", + "value_kind": "choice", + "unit": null, + "constraint": { + "kind": "choices", + "values": [ + "shared" + ] + }, + "comparison": "scientific_invariant" + } + ], + "checks": [ + { + "id": "sandbox_ready", + "label": "Sandbox ready", + "help": "The returned and inspected sandbox records agree and report ready state.", + "semantic_revision": 1, + "evidence_limit": 8 + }, + { + "id": "sandbox_workspace_identity", + "label": "Workspace identity", + "help": "The product record resolves to the benchmark-owned fixture selected for the cell.", + "semantic_revision": 1, + "evidence_limit": 8 + } + ], + "phases": [], + "comparison": { + "semantic_revision": 1, + "factors": [ + "workspace_profile", + "network_profile" + ] + } + }, { "id": "exec_command", "family": "command", @@ -245,7 +343,8 @@ "action": "exec_command" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "bounded_shell", "factors": [ @@ -360,7 +459,8 @@ "action": "file_read" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "public_read_only", "factors": [ @@ -465,7 +565,8 @@ "action": "file_write" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "public_mutation", "factors": [ @@ -570,7 +671,8 @@ "action": "file_edit" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "public_mutation", "factors": [ @@ -706,7 +808,8 @@ "action": "file_blame" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "public_read_only", "factors": [ @@ -786,7 +889,7 @@ "id": "create_workspace", "family": "workspace_lifecycle", "label": "Create workspace", - "help": "Benchmarks the exact allowlisted create_workspace_session test adapter for explicit no_op sessions; it is not a public operation.", + "help": "Benchmarks the public runtime CLI create_workspace_session operation for explicit no_op sessions.", "measured_boundary": "C independent session-creation requests start together after one sandbox and fixture are prepared; readiness is measured, while residual cleanup is not.", "count_semantics_help": "Workspace count C is the number of independent product requests in one measured trial.", "semantic_revision": 1, @@ -799,13 +902,14 @@ "isolation": "prepared_sandbox_per_cell", "cleanup": "destroy_sessions_and_verify_baseline", "product_access": { - "kind": "internal_workspace", - "action": "create_no_op_session" + "kind": "public_gateway", + "action": "create_workspace_session" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], - "security_class": "internal_workspace_lifecycle", + "security_class": "public_workspace_lifecycle", "factors": [ { "id": "workspace_count", @@ -900,7 +1004,8 @@ "action": "squash_layerstacks" }, "supported_cohorts": [ - "direct_client" + "direct_client", + "product_cli" ], "security_class": "destructive_manager_mutation", "factors": [ diff --git a/ephemeral-sandbox-v1/benchmark/defaults/gateway.yml b/ephemeral-sandbox-v1/benchmark/defaults/gateway.yml index 21c2265..f602fc2 100644 --- a/ephemeral-sandbox-v1/benchmark/defaults/gateway.yml +++ b/ephemeral-sandbox-v1/benchmark/defaults/gateway.yml @@ -2,7 +2,10 @@ daemon: server: socket_path: /eos/runtime/daemon/runtime.sock pid_path: /eos/runtime/daemon/runtime.pid - max_worker_threads: 32 + worker_threads: 2 + max_blocking_threads: 8 + blocking_thread_keep_alive_s: 5 + max_concurrent_connections: 64 runtime: workspace: diff --git a/ephemeral-sandbox-v1/benchmark/docs/phase-3-product-boundary.md b/ephemeral-sandbox-v1/benchmark/docs/phase-3-product-boundary.md index 136aadb..a72d055 100644 --- a/ephemeral-sandbox-v1/benchmark/docs/phase-3-product-boundary.md +++ b/ephemeral-sandbox-v1/benchmark/docs/phase-3-product-boundary.md @@ -18,7 +18,7 @@ The catalog exporter is selected only from the explicit canonical product binary Gateway requests use a new connection and request ID for each operation. Connection setup and response verification are outside the primary latency; the clock begins immediately before write/drain/read and ends as soon as the complete first response line arrives. The client then verifies EOF so a second response is rejected. Product error envelopes remain distinct from transport failures, and neither registered credentials nor sensitive diagnostics enter exceptions or logs. -An isolated campaign requires an exact marker-owned run directory, canonical gateway binary, fixed architecture-specific Linux ELF daemon package, and canonical Git toolchain archives. Startup creates a marker-owned `runtime/` with mode-0600 token, config, owner-process metadata, and sanitized JSONL logs. Readiness requires both the gateway-owned PID file and an authenticated `list_sandboxes` response. Shutdown attempts the final sandbox sweep, process-group termination, labeled container/volume cleanup, log persistence, token deletion, and runtime removal independently; any failure blocks cleanup and preserves marker-owned evidence. +An isolated campaign requires an exact marker-owned run directory, canonical gateway binary, and fixed architecture-specific Linux ELF daemon package. Startup creates a marker-owned `runtime/` with mode-0600 token, config, owner-process metadata, and sanitized JSONL logs. Readiness requires both the gateway-owned PID file and an authenticated `list_sandboxes` response. Shutdown attempts the final sandbox sweep, process-group termination, labeled container/volume cleanup, log persistence, token deletion, and runtime removal independently; any failure blocks cleanup and preserves marker-owned evidence. Stale recovery validates the exact runtime marker, metadata schema, binary path and digest, process-group identity, and OS process birth identity before sending a signal. A reused PID is never signaled. Once identity is proven, recovery aggregates gateway, process, labeled Docker, credential, and runtime cleanup. Failed recovery deletes credentials when safe but retains the owned runtime directory and blocks completion. diff --git a/ephemeral-sandbox-v1/benchmark/presets/paper-env-smoke.yml b/ephemeral-sandbox-v1/benchmark/presets/paper-env-smoke.yml index e24d2ac..b9e3f7d 100644 --- a/ephemeral-sandbox-v1/benchmark/presets/paper-env-smoke.yml +++ b/ephemeral-sandbox-v1/benchmark/presets/paper-env-smoke.yml @@ -1,6 +1,6 @@ schema_version: 1 id: paper-env-smoke -version: 1 +version: 2 plan: schema_version: 1 name: paper-env-smoke @@ -11,7 +11,7 @@ plan: seed: 20260712 environment: image: ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf - client_cohort: direct_client + client_cohort: product_cli protocol: order: randomized_blocks resource_interval_ms: 100 @@ -22,18 +22,50 @@ plan: default: 120000 squash_layerstack: 120000 operations: + - operation: create_sandbox + configuration: + enabled: true + factors: + workspace_profile: {role: controlled, values: [paper-100m]} + network_profile: {role: controlled, values: [shared]} - operation: exec_command configuration: enabled: true factors: - concurrent_requests: {role: controlled, values: [1]} - workspace_profile: {role: controlled, values: [small]} + concurrent_requests: {role: varied, values: [1, 5], control: 1} + workspace_profile: {role: controlled, values: [paper-100m]} session_mode: {role: controlled, values: [explicit]} - command_case: {role: controlled, values: [noop]} + command_case: {role: varied, values: [noop, fixture_read], control: noop} + - operation: file_read + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + returned_bytes: {role: varied, values: [4096, 262144], control: 4096} + source: {role: controlled, values: [snapshot]} + target_mode: {role: controlled, values: [independent]} + - operation: file_write + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + content_bytes: {role: varied, values: [4096, 262144], control: 4096} + destination: {role: controlled, values: [session]} + target_mode: {role: controlled, values: [independent]} + - operation: file_edit + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + file_bytes: {role: varied, values: [4096, 262144], control: 4096} + replacement_count: {role: controlled, values: [1]} + match_density: {role: controlled, values: [1.0]} + destination: {role: controlled, values: [session]} + target_mode: {role: controlled, values: [independent]} - operation: create_workspace configuration: enabled: true factors: - workspace_count: {role: controlled, values: [1]} - workspace_profile: {role: controlled, values: [small]} + workspace_count: {role: varied, values: [1, 5], control: 1} + workspace_profile: {role: controlled, values: [paper-100m]} network_profile: {role: controlled, values: [shared]} diff --git a/ephemeral-sandbox-v1/benchmark/presets/paper-good-pass.yml b/ephemeral-sandbox-v1/benchmark/presets/paper-good-pass.yml index 496f47c..0d5adc8 100644 --- a/ephemeral-sandbox-v1/benchmark/presets/paper-good-pass.yml +++ b/ephemeral-sandbox-v1/benchmark/presets/paper-good-pass.yml @@ -1,6 +1,6 @@ schema_version: 1 id: paper-good-pass -version: 1 +version: 2 plan: schema_version: 1 name: paper-good-pass @@ -11,7 +11,7 @@ plan: seed: 20260712 environment: image: ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf - client_cohort: direct_client + client_cohort: product_cli protocol: order: randomized_blocks resource_interval_ms: 100 @@ -22,6 +22,12 @@ plan: default: 120000 squash_layerstack: 120000 operations: + - operation: create_sandbox + configuration: + enabled: true + factors: + workspace_profile: {role: controlled, values: [paper-100m]} + network_profile: {role: controlled, values: [shared]} - operation: exec_command configuration: enabled: true diff --git a/ephemeral-sandbox-v1/benchmark/presets/paper-pilot.yml b/ephemeral-sandbox-v1/benchmark/presets/paper-pilot.yml new file mode 100644 index 0000000..3e65e1f --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/presets/paper-pilot.yml @@ -0,0 +1,71 @@ +schema_version: 1 +id: paper-pilot +version: 1 +plan: + schema_version: 1 + name: paper-pilot + configuration_base: + id: standard-local + version: 1 + scope: all + seed: 20260712 + environment: + image: ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf + client_cohort: product_cli + protocol: + order: randomized_blocks + resource_interval_ms: 100 + trial_defaults: + fast: {warmups: 2, measured_trials: 5} + destructive: {warmups: 2, measured_trials: 5} + timeout_ms: + default: 120000 + squash_layerstack: 120000 + operations: + - operation: create_sandbox + configuration: + enabled: true + factors: + workspace_profile: {role: controlled, values: [paper-100m]} + network_profile: {role: controlled, values: [shared]} + - operation: exec_command + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + workspace_profile: {role: controlled, values: [paper-100m]} + session_mode: {role: controlled, values: [explicit]} + command_case: {role: varied, values: [noop, fixture_read], control: noop} + - operation: file_read + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + returned_bytes: {role: varied, values: [4096, 262144], control: 4096} + source: {role: controlled, values: [snapshot]} + target_mode: {role: controlled, values: [independent]} + - operation: file_write + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + content_bytes: {role: varied, values: [4096, 262144], control: 4096} + destination: {role: controlled, values: [session]} + target_mode: {role: controlled, values: [independent]} + - operation: file_edit + configuration: + enabled: true + factors: + concurrent_requests: {role: varied, values: [1, 5], control: 1} + file_bytes: {role: varied, values: [4096, 262144], control: 4096} + replacement_count: {role: controlled, values: [1]} + match_density: {role: controlled, values: [1.0]} + destination: {role: controlled, values: [session]} + target_mode: {role: controlled, values: [independent]} + - operation: create_workspace + configuration: + enabled: true + factors: + workspace_count: {role: varied, values: [1, 5], control: 1} + workspace_profile: {role: controlled, values: [paper-100m]} + network_profile: {role: controlled, values: [shared]} diff --git a/ephemeral-sandbox-v1/benchmark/tests/fixtures/golden/artifacts/operation-evidence-v1-squash.json b/ephemeral-sandbox-v1/benchmark/tests/fixtures/golden/artifacts/operation-evidence-v1-squash.json new file mode 100644 index 0000000..edd9689 --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/tests/fixtures/golden/artifacts/operation-evidence-v1-squash.json @@ -0,0 +1,231 @@ +{ + "schema_name": "eos_benchmark_operation_evidence", + "schema_version": 1, + "data": { + "operation": "squash_layerstack", + "evidence": { + "requested_live_sessions": 1, + "observed_migrated_sessions": 1, + "observed_non_migrated_sessions": 0, + "dispositions": { + "migrated": 1, + "identity": 0, + "leased": 0, + "faulty": 0, + "session_gone": 0 + }, + "effective_remount_parallelism": 4, + "observed_squashed_block_count": 1, + "observed_replaced_layer_count": 2, + "source_layer_ids": [ + "L1", + "L2" + ], + "retained_source_layer_ids": [], + "source_layer_allocations": [ + { + "layer_id": "L1", + "logical_bytes": { + "availability": "available", + "value": 4 + }, + "allocated_bytes": { + "availability": "available", + "value": 8 + } + }, + { + "layer_id": "L2", + "logical_bytes": { + "availability": "available", + "value": 5 + }, + "allocated_bytes": { + "availability": "available", + "value": 16 + } + } + ], + "reclaimed_bytes": { + "availability": "available", + "value": 24 + }, + "s0_baseline": { + "monotonic_offset_ns": { + "availability": "available", + "value": 10 + }, + "sampled": false, + "manifest_version": { + "availability": "available", + "value": 2 + }, + "root_hash": { + "availability": "available", + "value": "before" + }, + "active_layer_count": { + "availability": "available", + "value": 2 + }, + "active_lease_count": { + "availability": "available", + "value": 1 + }, + "active_logical_bytes": { + "availability": "available", + "value": 9 + }, + "active_allocated_bytes": { + "availability": "available", + "value": 24 + }, + "storage_logical_bytes": { + "availability": "available", + "value": 109 + }, + "storage_allocated_bytes": { + "availability": "available", + "value": 224 + }, + "staging_entry_count": { + "availability": "available", + "value": 0 + } + }, + "s1_sampled_peak": { + "monotonic_offset_ns": { + "availability": "available", + "value": 10 + }, + "sampled": true, + "manifest_version": { + "availability": "available", + "value": 2 + }, + "root_hash": { + "availability": "available", + "value": "before" + }, + "active_layer_count": { + "availability": "available", + "value": 2 + }, + "active_lease_count": { + "availability": "available", + "value": 1 + }, + "active_logical_bytes": { + "availability": "available", + "value": 9 + }, + "active_allocated_bytes": { + "availability": "available", + "value": 24 + }, + "storage_logical_bytes": { + "availability": "available", + "value": 109 + }, + "storage_allocated_bytes": { + "availability": "available", + "value": 224 + }, + "staging_entry_count": { + "availability": "available", + "value": 0 + } + }, + "s2_post_commit": { + "monotonic_offset_ns": { + "availability": "available", + "value": 20 + }, + "sampled": false, + "manifest_version": { + "availability": "available", + "value": 3 + }, + "root_hash": { + "availability": "available", + "value": "after" + }, + "active_layer_count": { + "availability": "available", + "value": 1 + }, + "active_lease_count": { + "availability": "unavailable", + "source": "product_observability.trace", + "reason": "active lease count is not emitted at the commit boundary" + }, + "active_logical_bytes": { + "availability": "available", + "value": 9 + }, + "active_allocated_bytes": { + "availability": "available", + "value": 12 + }, + "storage_logical_bytes": { + "availability": "available", + "value": 109 + }, + "storage_allocated_bytes": { + "availability": "available", + "value": 212 + }, + "staging_entry_count": { + "availability": "available", + "value": 0 + } + }, + "s3_settled": { + "monotonic_offset_ns": { + "availability": "available", + "value": 30 + }, + "sampled": false, + "manifest_version": { + "availability": "available", + "value": 3 + }, + "root_hash": { + "availability": "available", + "value": "after" + }, + "active_layer_count": { + "availability": "available", + "value": 1 + }, + "active_lease_count": { + "availability": "available", + "value": 1 + }, + "active_logical_bytes": { + "availability": "available", + "value": 9 + }, + "active_allocated_bytes": { + "availability": "available", + "value": 12 + }, + "storage_logical_bytes": { + "availability": "available", + "value": 109 + }, + "storage_allocated_bytes": { + "availability": "available", + "value": 212 + }, + "staging_entry_count": { + "availability": "available", + "value": 0 + } + }, + "manifest_reduced": true, + "content_equivalent": true, + "usable_session_count": 1 + } + } +} diff --git a/ephemeral-sandbox-v1/benchmark/uv.lock b/ephemeral-sandbox-v1/benchmark/uv.lock new file mode 100644 index 0000000..4df6f19 --- /dev/null +++ b/ephemeral-sandbox-v1/benchmark/uv.lock @@ -0,0 +1,372 @@ +version = 1 +revision = 3 +requires-python = ">=3.13" + +[[package]] +name = "annotated-doc" +version = "0.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/8e/38aa427ed5402449e226975b649c5dc73ccadfefeb95e6aecb8f8ea4b6b6/annotated_doc-0.0.5.tar.gz", hash = "sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb", size = 10758, upload-time = "2026-07-28T13:50:58.129Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/30/e900b21425a860e195f32e37657aa1f7c7f2b1bfb26f03ca209b90933c06/annotated_doc-0.0.5-py3-none-any.whl", hash = "sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101", size = 5302, upload-time = "2026-07-28T13:50:57.239Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5f/56/a8120250d128bed162cd73c76d45f6ef9991f3e068f62a8ee060afa3104a/annotated_types-0.8.0.tar.gz", hash = "sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7", size = 15893, upload-time = "2026-07-23T20:16:13.995Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, +] + +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "click" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "ephemeralos-benchmark" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "fastapi" }, + { name = "pydantic" }, + { name = "pyyaml" }, + { name = "uvicorn" }, +] + +[package.optional-dependencies] +test = [ + { name = "httpx" }, + { name = "pytest" }, + { name = "pytest-asyncio" }, +] + +[package.metadata] +requires-dist = [ + { name = "fastapi", specifier = ">=0.116,<1" }, + { name = "httpx", marker = "extra == 'test'", specifier = ">=0.28,<1" }, + { name = "pydantic", specifier = ">=2.11,<3" }, + { name = "pytest", marker = "extra == 'test'", specifier = ">=8.4,<10" }, + { name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=1.1,<2" }, + { name = "pyyaml", specifier = ">=6,<7" }, + { name = "uvicorn", specifier = ">=0.35,<1" }, +] +provides-extras = ["test"] + +[[package]] +name = "fastapi" +version = "0.141.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8a/02/91e3416a8fdd715abb903a952a6bec7cdd8d14eed55d415fc8595524c319/fastapi-0.141.1.tar.gz", hash = "sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1", size = 425799, upload-time = "2026-07-29T17:18:05.568Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pydantic" +version = "2.13.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/18/a5/b60d21ac674192f8ab0ba4e9fd860690f9b4a6e51ca5df118733b487d8d6/pydantic-2.13.4.tar.gz", hash = "sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6", size = 844775, upload-time = "2026-05-06T13:43:05.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl", hash = "sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba", size = 472262, upload-time = "2026-05-06T13:43:02.641Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.46.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9d/56/921726b776ace8d8f5db44c4ef961006580d91dc52b803c489fafd1aa249/pydantic_core-2.46.4.tar.gz", hash = "sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1", size = 471464, upload-time = "2026-05-06T13:37:06.98Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/a2/5d30b469c5267a17b39dec53208222f76a8d351dfac4af661888c5aee77d/pydantic_core-2.46.4-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008", size = 2106306, upload-time = "2026-05-06T13:37:48.029Z" }, + { url = "https://files.pythonhosted.org/packages/c1/81/4fa520eaffa8bd7d1525e644cd6d39e7d60b1592bc5b516693c7340b50f1/pydantic_core-2.46.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4", size = 1951906, upload-time = "2026-05-06T13:37:17.012Z" }, + { url = "https://files.pythonhosted.org/packages/03/d5/fd02da45b659668b05923b17ba3a0100a0a3d5541e3bd8fcc4ecb711309e/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76", size = 1976802, upload-time = "2026-05-06T13:37:35.113Z" }, + { url = "https://files.pythonhosted.org/packages/21/f2/95727e1368be3d3ed485eaab7adbd7dda408f33f7a36e8b48e0144002b91/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3", size = 2052446, upload-time = "2026-05-06T13:37:12.313Z" }, + { url = "https://files.pythonhosted.org/packages/9c/86/5d99feea3f77c7234b8718075b23db11532773c1a0dbd9b9490215dc2eeb/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76", size = 2232757, upload-time = "2026-05-06T13:39:01.149Z" }, + { url = "https://files.pythonhosted.org/packages/d2/3a/508ac615935ef7588cf6d9e9b91309fdc2da751af865e02a9098de88258c/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4", size = 2309275, upload-time = "2026-05-06T13:37:41.406Z" }, + { url = "https://files.pythonhosted.org/packages/07/f8/41db9de19d7987d6b04715a02b3b40aea467000275d9d758ffaa31af7d50/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a", size = 2094467, upload-time = "2026-05-06T13:39:18.847Z" }, + { url = "https://files.pythonhosted.org/packages/2c/e2/f35033184cb11d0052daf4416e8e10a502ea2ac006fc4f459aee872727d1/pydantic_core-2.46.4-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262", size = 2134417, upload-time = "2026-05-06T13:40:17.944Z" }, + { url = "https://files.pythonhosted.org/packages/7e/7b/6ceeb1cc90e193862f444ebe373d8fdf613f0a82572dde03fb10734c6c71/pydantic_core-2.46.4-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e", size = 2179782, upload-time = "2026-05-06T13:40:32.618Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f2/c8d7773ede6af08036423a00ae0ceffce266c3c52a096c435d68c896083f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd", size = 2188782, upload-time = "2026-05-06T13:36:51.018Z" }, + { url = "https://files.pythonhosted.org/packages/59/31/0c864784e31f09f05cdd87606f08923b9c9e7f6e51dd27f20f62f975ce9f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be", size = 2328334, upload-time = "2026-05-06T13:40:37.764Z" }, + { url = "https://files.pythonhosted.org/packages/c2/eb/4f6c8a41efa30baa755590f4141abf3a8c370fab610915733e74134a7270/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d", size = 2372986, upload-time = "2026-05-06T13:39:34.152Z" }, + { url = "https://files.pythonhosted.org/packages/5b/24/b375a480d53113860c299764bfe9f349a3dc9108b3adc0d7f0d786492ebf/pydantic_core-2.46.4-cp313-cp313-win32.whl", hash = "sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb", size = 1973693, upload-time = "2026-05-06T13:37:55.072Z" }, + { url = "https://files.pythonhosted.org/packages/7e/e8/cff247591966f2d22ec8c003cd7587e27b7ba7b81ab2fb888e3ab75dc285/pydantic_core-2.46.4-cp313-cp313-win_amd64.whl", hash = "sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292", size = 2071819, upload-time = "2026-05-06T13:38:49.139Z" }, + { url = "https://files.pythonhosted.org/packages/c6/1a/f4aee670d5670e9e148e0c82c7db98d780be566c6e6a97ee8035528ca0b3/pydantic_core-2.46.4-cp313-cp313-win_arm64.whl", hash = "sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d", size = 2027411, upload-time = "2026-05-06T13:40:45.796Z" }, + { url = "https://files.pythonhosted.org/packages/8d/74/228a26ddad29c6672b805d9fd78e8d251cd04004fa7eed0e622096cd0250/pydantic_core-2.46.4-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb", size = 2102079, upload-time = "2026-05-06T13:38:41.019Z" }, + { url = "https://files.pythonhosted.org/packages/ad/1f/8970b150a4b4365623ae00fc88603491f763c627311ae8031e3111356d6e/pydantic_core-2.46.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462", size = 1952179, upload-time = "2026-05-06T13:36:59.812Z" }, + { url = "https://files.pythonhosted.org/packages/95/30/5211a831ae054928054b2f79731661087a2bc5c01e825c672b3a4a8f1b3e/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9", size = 1978926, upload-time = "2026-05-06T13:37:39.933Z" }, + { url = "https://files.pythonhosted.org/packages/57/e9/689668733b1eb67adeef047db3c2e8788fcf65a7fd9c9e2b46b7744fe245/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4", size = 2046785, upload-time = "2026-05-06T13:38:01.995Z" }, + { url = "https://files.pythonhosted.org/packages/60/d9/6715260422ff50a2109878fd24d948a6c3446bb2664f34ee78cd972b3acd/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914", size = 2228733, upload-time = "2026-05-06T13:40:50.371Z" }, + { url = "https://files.pythonhosted.org/packages/18/ae/fdb2f64316afca925640f8e70bb1a564b0ec2721c1389e25b8eb4bf9a299/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28", size = 2307534, upload-time = "2026-05-06T13:37:21.531Z" }, + { url = "https://files.pythonhosted.org/packages/89/1d/8eff589b45bb8190a9d12c49cfad0f176a5cbd1534908a6b5125e2886239/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b", size = 2099732, upload-time = "2026-05-06T13:39:31.942Z" }, + { url = "https://files.pythonhosted.org/packages/06/d5/ee5a3366637fee41dee51a1fc91562dcf12ddbc68fda34e6b253da2324bb/pydantic_core-2.46.4-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c", size = 2129627, upload-time = "2026-05-06T13:37:25.033Z" }, + { url = "https://files.pythonhosted.org/packages/94/33/2414be571d2c6a6c4d08be21f9292b6d3fdb08949a97b6dfe985017821db/pydantic_core-2.46.4-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb", size = 2179141, upload-time = "2026-05-06T13:37:14.046Z" }, + { url = "https://files.pythonhosted.org/packages/7b/79/7daa95be995be0eecc4cf75064cb33f9bbbfe3fe0158caf2f0d4a996a5c7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898", size = 2184325, upload-time = "2026-05-06T13:36:53.615Z" }, + { url = "https://files.pythonhosted.org/packages/9f/cb/d0a382f5c0de8a222dc61c65348e0ce831b1f68e0a018450d31c2cace3a5/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e", size = 2323990, upload-time = "2026-05-06T13:40:29.971Z" }, + { url = "https://files.pythonhosted.org/packages/05/db/d9ba624cc4a5aced1598e88c04fdbd8310c8a69b9d38b9a3d39ce3a61ed7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519", size = 2369978, upload-time = "2026-05-06T13:37:23.027Z" }, + { url = "https://files.pythonhosted.org/packages/f2/20/d15df15ba918c423461905802bfd2981c3af0bfa0e40d05e13edbfa48bc3/pydantic_core-2.46.4-cp314-cp314-win32.whl", hash = "sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4", size = 1966354, upload-time = "2026-05-06T13:38:03.499Z" }, + { url = "https://files.pythonhosted.org/packages/fc/b6/6b8de4c0a7d7ab3004c439c80c5c1e0a3e8d78bbae19379b01960383d9e5/pydantic_core-2.46.4-cp314-cp314-win_amd64.whl", hash = "sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac", size = 2072238, upload-time = "2026-05-06T13:39:40.807Z" }, + { url = "https://files.pythonhosted.org/packages/32/36/51eb763beec1f4cf59b1db243a7dcc39cbb41230f050a09b9d69faaf0a48/pydantic_core-2.46.4-cp314-cp314-win_arm64.whl", hash = "sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a", size = 2018251, upload-time = "2026-05-06T13:37:26.72Z" }, + { url = "https://files.pythonhosted.org/packages/e8/91/855af51d625b23aa987116a19e231d2aaef9c4a415273ddc189b79a45fee/pydantic_core-2.46.4-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0", size = 2099593, upload-time = "2026-05-06T13:39:47.682Z" }, + { url = "https://files.pythonhosted.org/packages/fb/1b/8784a54c65edb5f49f0a14d6977cf1b209bba85a4c77445b255c2de58ab3/pydantic_core-2.46.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d", size = 1935226, upload-time = "2026-05-06T13:40:40.428Z" }, + { url = "https://files.pythonhosted.org/packages/e8/e7/1955d28d1afc56dd4b3ad7cc0cf39df1b9852964cf16e5d13912756d6d6b/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b", size = 1974605, upload-time = "2026-05-06T13:37:32.029Z" }, + { url = "https://files.pythonhosted.org/packages/93/e2/3fedbf0ba7a22850e6e9fd78117f1c0f10f950182344d8a6c535d468fdd8/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000", size = 2030777, upload-time = "2026-05-06T13:38:55.239Z" }, + { url = "https://files.pythonhosted.org/packages/f8/61/46be275fcaaba0b4f5b9669dd852267ce1ff616592dccf7a7845588df091/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e", size = 2236641, upload-time = "2026-05-06T13:37:08.096Z" }, + { url = "https://files.pythonhosted.org/packages/60/db/12e93e46a8bac9988be3c016860f83293daea8c716c029c9ace279036f2f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd", size = 2286404, upload-time = "2026-05-06T13:40:20.221Z" }, + { url = "https://files.pythonhosted.org/packages/e2/4a/4d8b19008f38d31c53b8219cfedc2e3d5de5fe99d90076b7e767de29274f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3", size = 2109219, upload-time = "2026-05-06T13:38:12.153Z" }, + { url = "https://files.pythonhosted.org/packages/88/70/3cbc40978fefb7bb09c6708d40d4ad1a5d70fd7213c3d17f971de868ec1f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7", size = 2110594, upload-time = "2026-05-06T13:40:02.971Z" }, + { url = "https://files.pythonhosted.org/packages/9d/20/b8d36736216e29491125531685b2f9e61aa5b4b2599893f8268551da3338/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff", size = 2159542, upload-time = "2026-05-06T13:39:27.506Z" }, + { url = "https://files.pythonhosted.org/packages/1d/a2/367df868eb584dacf6bf82a389272406d7178e301c4ac82545ab98bc2dd9/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424", size = 2168146, upload-time = "2026-05-06T13:38:31.93Z" }, + { url = "https://files.pythonhosted.org/packages/c1/b8/4460f77f7e201893f649a29ab355dddd3beee8a97bcb1a320db414f9a06e/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6", size = 2306309, upload-time = "2026-05-06T13:37:44.717Z" }, + { url = "https://files.pythonhosted.org/packages/64/c4/be2639293acd87dc8ddbcec41a73cee9b2ebf996fe6d892a1a74e88ad3f7/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565", size = 2369736, upload-time = "2026-05-06T13:37:05.645Z" }, + { url = "https://files.pythonhosted.org/packages/30/a6/9f9f380dbb301f67023bf8f707aaa75daadf84f7152d95c410fd7e81d994/pydantic_core-2.46.4-cp314-cp314t-win32.whl", hash = "sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02", size = 1955575, upload-time = "2026-05-06T13:38:51.116Z" }, + { url = "https://files.pythonhosted.org/packages/40/1f/f1eb9eb350e795d1af8586289746f5c5677d16043040d63710e22abc43c9/pydantic_core-2.46.4-cp314-cp314t-win_amd64.whl", hash = "sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5", size = 2051624, upload-time = "2026-05-06T13:38:21.672Z" }, + { url = "https://files.pythonhosted.org/packages/f6/d2/42dd53d0a85c27606f316d3aa5d2869c4e8470a5ed6dec30e4a1abe19192/pydantic_core-2.46.4-cp314-cp314t-win_arm64.whl", hash = "sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596", size = 2017325, upload-time = "2026-05-06T13:40:52.723Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "pytest-asyncio" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/7c/d36d04db312ecf4298932ef77e6e4a9e8ad017906e24e34f0b0c361a2473/pytest_asyncio-1.4.0.tar.gz", hash = "sha256:c6c0d2259945122819f171a32ecea2c349ead889ee28176caaf492143424be42", size = 58514, upload-time = "2026-05-26T09:56:04.083Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/e2/08a497ef684b88559c9cc5f4ad53a37e7b99e727094a86d6ea32536d5d3c/pytest_asyncio-1.4.0-py3-none-any.whl", hash = "sha256:933ca923a23075a87fb7070c0ec272a6848489824d887c85c812670932835aa1", size = 16930, upload-time = "2026-05-26T09:56:02.576Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "starlette" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/eb/e3/7c1dc7381d9f8ab7d854328ebfa884e62cb3f3d8549ddfd37c7814f42afa/starlette-1.3.1.tar.gz", hash = "sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0", size = 2703240, upload-time = "2026-06-12T09:23:11.602Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/bb/2799cc2ede3ed41131f8975621e7213dfc7ef4acbbaadfa440f32500c370/starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6", size = 73632, upload-time = "2026-06-12T09:23:10.017Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] + +[[package]] +name = "uvicorn" +version = "0.52.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/05/c8/2d307868453a4bca6e64fa3581d122ae0748a0869c53f159339def179c7c/uvicorn-0.52.0.tar.gz", hash = "sha256:ca8876ad6c1983f394157c168b39d52f6dd56dabf5602fa0982751cffc2293ae", size = 97504, upload-time = "2026-07-29T08:45:34.065Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/39/e6/b5c0630ace9757232aec07112be8146b812787db52141ff9d50674aa7634/uvicorn-0.52.0-py3-none-any.whl", hash = "sha256:3d887809810b89ed33501bcf0a9aba469b06ecd608158efce04bd6b48d8c9b08", size = 79058, upload-time = "2026-07-29T08:45:32.492Z" }, +] diff --git a/ephemeral-sandbox-v1/benchmark/web/src/api/types.ts b/ephemeral-sandbox-v1/benchmark/web/src/api/types.ts index a4d129f..ded3049 100644 --- a/ephemeral-sandbox-v1/benchmark/web/src/api/types.ts +++ b/ephemeral-sandbox-v1/benchmark/web/src/api/types.ts @@ -18,7 +18,7 @@ export type OperationId = | "squash_layerstack"; export type ConfigurationScope = "all" | "command" | "files" | "workspace" | "layerstack"; -export type ClientCohort = "direct_client" | "cli_e2e"; +export type ClientCohort = "direct_client" | "product_cli"; export type FactorRole = "varied" | "controlled"; export type WorkspaceProfileId = string; export type AllowedNetworkProfile = "shared" | "isolated"; diff --git a/ephemeral-sandbox-v1/benchmark/web/src/components/DefaultPlanLauncher.tsx b/ephemeral-sandbox-v1/benchmark/web/src/components/DefaultPlanLauncher.tsx index ca89a4a..afd665a 100644 --- a/ephemeral-sandbox-v1/benchmark/web/src/components/DefaultPlanLauncher.tsx +++ b/ephemeral-sandbox-v1/benchmark/web/src/components/DefaultPlanLauncher.tsx @@ -460,7 +460,7 @@ function ProtocolAndEnvironmentControls({ onChange({ ...plan, seed: number(value, plan.seed) })} /> onChange({ ...plan, protocol: { ...plan.protocol, resource_interval_ms: number(value, plan.protocol.resource_interval_ms) } })} /> - ({ value, label: value === "direct_client" ? "Direct client" : "Product CLI" }))} value={plan.environment.client_cohort} onChange={(value) => { if (value === "direct_client" || value === "product_cli") onChange({ ...plan, environment: { ...plan.environment, client_cohort: value } }); }} allowDeselect={false} /> onChange({ ...plan, protocol: { ...plan.protocol, trial_defaults: { ...plan.protocol.trial_defaults, fast: { ...plan.protocol.trial_defaults.fast, warmups: number(value, plan.protocol.trial_defaults.fast.warmups) } } } })} /> onChange({ ...plan, protocol: { ...plan.protocol, trial_defaults: { ...plan.protocol.trial_defaults, fast: { ...plan.protocol.trial_defaults.fast, measured_trials: number(value, plan.protocol.trial_defaults.fast.measured_trials) } } } })} /> onChange({ ...plan, protocol: { ...plan.protocol, trial_defaults: { ...plan.protocol.trial_defaults, destructive: { ...plan.protocol.trial_defaults.destructive, warmups: number(value, plan.protocol.trial_defaults.destructive.warmups) } } } })} /> diff --git a/ephemeral-sandbox-v1/benchmark/web/tests/unit/plan-workflow.test.tsx b/ephemeral-sandbox-v1/benchmark/web/tests/unit/plan-workflow.test.tsx index 225f8de..e73d0e5 100644 --- a/ephemeral-sandbox-v1/benchmark/web/tests/unit/plan-workflow.test.tsx +++ b/ephemeral-sandbox-v1/benchmark/web/tests/unit/plan-workflow.test.tsx @@ -103,7 +103,7 @@ const operationDefinition: DefinitionsResponse["catalog"]["operations"][number] isolation: "session_mode_dependent", cleanup: "resolve_from_isolation", product_access: { kind: "public_gateway", action: "exec_command" }, - supported_cohorts: ["direct_client", "cli_e2e"], + supported_cohorts: ["direct_client", "product_cli"], security_class: "bounded_shell", factors, checks: [], @@ -355,6 +355,6 @@ describe("typed plan workflow", () => { const cohort = screen.getByRole("combobox", { name: "Client cohort" }) as HTMLInputElement; expect(cohort.value).toBe("Direct client"); - expect(screen.queryByText("CLI end to end")).toBeNull(); + expect(screen.queryByText("Product CLI")).toBeNull(); }); }); diff --git a/ephemeral-sandbox-v1/build_check.md b/ephemeral-sandbox-v1/build_check.md index 44df464..04a325f 100644 --- a/ephemeral-sandbox-v1/build_check.md +++ b/ephemeral-sandbox-v1/build_check.md @@ -1,33 +1,28 @@ -# PW0 build check +# PW4--PW7 build check -**Status:** Passed with executed attestation. +**Status:** passed, executed attestation. -The declared command is: +The declared build command completed with exit code zero: ```text latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex ``` -The skill build recorder executed the declared command successfully at `2026-07-30T00:23:10.503310+00:00` using the user-local TinyTeX installation at `C:\Users\yifan\AppData\Roaming\TinyTeX`. - -## Tool versions - -- Latexmk 4.88, dated 2026-03-09. -- pdfTeX 3.141592653-2.6-1.40.29, TeX Live 2026. -- BibTeX 0.99e, TeX Live 2026. - -## Attestation - -- Exit code: 0. -- PDF: `main.pdf`. -- PDF SHA-256: `dd63976a4714299aa99f564e2669c1e8a438dad6fe6eeaf1f4eeaadd50ef4406`. -- Input SHA-256: `e0c64365a445533c8efdb74068d5d5d28945871853d26c06692d0f791b48868e`. -- Build log: `plan/pw0-build-attempt.log`. -- Build-log SHA-256: `b0ad2eb77266611365f2af42a8b146fa25d8fdbd29d4cce599d7106b8b1d1e94`. -- Attestation: executed. - -The parsed LaTeX log contains zero errors, emergency stops, undefined citations, undefined references, missing files, or overfull boxes. It reports five underfull boxes in the provisional cost table and one expected empty-bibliography warning. - -Visual verification rendered all four US-letter pages. Text, headings, equations, table borders, page numbers, and draft markers are legible with no clipping, overlap, or broken glyphs. Because `references.bib` is intentionally comment-only, page 4 is an empty References page at this scaffold stage. - -PW0's build blocker is closed. Other scientific, author, source-freeze, and evaluation blockers remain recorded in `paper_state.json`. +| Field | Recorded value | +| --- | --- | +| Attestation time | 2026-08-02T05:28:00.336951+00:00 | +| PDF | `main.pdf` | +| PDF SHA-256 | `6ffb429c6acb0f27af6dc493cd3fcc43a1710a22cba17fb669a11129ad06e8b2` | +| Build-input SHA-256 | `6f1675a8021bf6acc982d9b7dc9f6c4d5755832ca937334c6212eb15b867b6a7` | +| Attestation log | `plan/pw4-pw7-build-attestation.log` | +| Attestation-log SHA-256 | `cffe435373442861725c7ca42e572c0a33ea029f02d1031fa38ba731745157c1` | + +The separate final log scan reports no overfull boxes and no unresolved +citations or references. Citation-key, citation-lock, numeric-evidence, and +full-paper quality gates pass. The PDF has 17 pages and was rendered in color +and grayscale for visual QA; the record and final figure waivers are in +`figures/concept-figure-review.md`. + +The local LaTeX wrapper emitted only non-fatal locale and console-code-page +messages. They do not affect the recorded zero exit code, source digest, or PDF +digest. diff --git a/ephemeral-sandbox-v1/citation_lock.json b/ephemeral-sandbox-v1/citation_lock.json new file mode 100644 index 0000000..d844409 --- /dev/null +++ b/ephemeral-sandbox-v1/citation_lock.json @@ -0,0 +1,489 @@ +{ + "schema_version": "ai-research-writing/citation-lock-v1", + "request_sha256": "5029a69bc9012311aaa042cfe197b5861c602f848b80a8dd84554935e7facb0f", + "records": [ + { + "key": "swe_minisandbox_2026", + "identifier": "arxiv:2602.11210", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2602.11210", + "checked_at": "2026-08-02T05:08:44.142706+00:00", + "metadata": { + "title": "SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents", + "authors": [ + "Danlong Yuan", + "Wei Wu", + "Enhan Zhao", + "Zhengren Wang", + "Xueliang Zhao", + "Huishuai Zhang", + "Dongyan Zhao" + ], + "year": 2026, + "arxiv_id": "2602.11210", + "type": "preprint" + }, + "metadata_sha256": "b0afd7985d642e29d2c1b9d76c972c304d155199a8bf9f170f81f56f342441fc", + "claim_support_sha256": "cd01bc6af0e22e12a90b29db47c974b8c3532b369b650f3972b717af17de5d21", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "agentbay_2025", + "identifier": "arxiv:2512.04367", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2512.04367", + "checked_at": "2026-08-02T05:08:45.877064+00:00", + "metadata": { + "title": "AgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systems", + "authors": [ + "Yun Piao", + "Hongbo Min", + "Hang Su", + "Leilei Zhang", + "Lei Wang", + "Yue Yin", + "Xiao Wu", + "Zhejing Xu", + "Liwei Qu", + "Hang Li", + "Xinxin Zeng", + "Wei Tian", + "Fei Yu", + "Xiaowei Li", + "Jiayi Jiang", + "Tongxu Liu", + "Hao Tian", + "Yufei Que", + "Xiaobing Tu", + "Bing Suo", + "Yuebing Li", + "Xiangting Chen", + "Zeen Zhao", + "Jiaming Tang", + "Wei Huang", + "Xuguang Li", + "Jing Zhao", + "Jin Li", + "Jie Shen", + "Jinkui Ren", + "Xiantao Zhang" + ], + "year": 2025, + "arxiv_id": "2512.04367", + "type": "preprint" + }, + "metadata_sha256": "90a25f817006a62691aa2747110fd25764f2508d87b63579b7ab2ef89561800a", + "claim_support_sha256": "0dfaa81ba8d187a1a5638f6fed7659e5e78fb1e56d351a2c84766e19933ff9ed", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "deltabox_2026", + "identifier": "arxiv:2605.22781", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2605.22781", + "checked_at": "2026-08-02T05:08:47.051113+00:00", + "metadata": { + "title": "DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback", + "authors": [ + "Yunpeng Dong", + "Jingkai He", + "Shiqi Liu", + "Yuze Hou", + "Dong Du", + "Zhonghu Xu", + "Si Yu", + "Baochuan Yang", + "Yubin Xia", + "Haibo Chen" + ], + "year": 2026, + "arxiv_id": "2605.22781", + "type": "preprint" + }, + "metadata_sha256": "756eb36421b9eb3fd7a0ae1f47b1c0fcfcffc09e6e83ec67a87055fba33a30c5", + "claim_support_sha256": "05dc256113e2b7c5726e1314b1c6f31924f755767f428a47157c6989d0b33c36", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "shepherd_2026", + "identifier": "arxiv:2605.10913", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2605.10913", + "checked_at": "2026-08-02T05:08:48.240303+00:00", + "metadata": { + "title": "Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Traces", + "authors": [ + "Simon Yu", + "Derek Chong", + "Ananjan Nandi", + "Dilara Soylu", + "Jiuding Sun", + "Christopher D Manning", + "Weiyan Shi" + ], + "year": 2026, + "arxiv_id": "2605.10913", + "type": "preprint" + }, + "metadata_sha256": "750bddd6526be452f540fa7456249fbfecc1f6a6a36b9f5c1457ed1c09488d3a", + "claim_support_sha256": "3b6f67ca2479f91614b26fc4cb0827b6c689ed25c28eb4af4c72032cd74f0e60", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "caid_2026", + "identifier": "arxiv:2603.21489", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2603.21489", + "checked_at": "2026-08-02T05:08:49.225891+00:00", + "metadata": { + "title": "Effective Strategies for Asynchronous Software Engineering Agents", + "authors": [ + "Jiayi Geng", + "Graham Neubig" + ], + "year": 2026, + "arxiv_id": "2603.21489", + "type": "preprint" + }, + "metadata_sha256": "779ed2ce28d937276e523c8d4b79df1237417465afbba365c9526398c9063901", + "claim_support_sha256": "d64d926345c793643178870ccde09d910ee6ec13f704f26f9400e3aca05804b6", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "coagent_2026", + "identifier": "arxiv:2606.15376", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2606.15376", + "checked_at": "2026-08-02T05:08:50.392870+00:00", + "metadata": { + "title": "CoAgent: Concurrency Control for Multi-Agent Systems", + "authors": [ + "Hongtao Lyu", + "Dingyan Zhang", + "Mingyu Wu", + "Xingda Wei", + "Haibo Chen" + ], + "year": 2026, + "arxiv_id": "2606.15376", + "type": "preprint" + }, + "metadata_sha256": "0b90714028413d3e7a05dd50361c88011741403943dbc4d92a6d9072f9155e37", + "claim_support_sha256": "7b74920a4de2436d488af500dd41f352eb4690a1f07cb0c1bfd8d0e91892812a", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "claimplane_2026", + "identifier": "arxiv:2607.21909", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2607.21909", + "checked_at": "2026-08-02T05:08:51.520148+00:00", + "metadata": { + "title": "Claim Plane: Enforceable Change Intents and Dynamic Scope for Parallel Coding Agents", + "authors": [ + "Maxim Nikolaev" + ], + "year": 2026, + "arxiv_id": "2607.21909", + "type": "preprint" + }, + "metadata_sha256": "509ad70806c21cca3e87dfe4fc1565d260f116eea851adbfaec515e4f443b52b", + "claim_support_sha256": "b4a4614815d051b10f5c8cee6f0bd7a596ec2b1d6efe3239699bd52f469f8db6", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "palantir_2012", + "identifier": "doi:10.1109/TSE.2011.64", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1109%2FTSE.2011.64", + "checked_at": "2026-08-02T05:08:52.666125+00:00", + "metadata": { + "title": "Palantir: Early Detection of Development Conflicts Arising from Parallel Code Changes", + "authors": [ + "A. Sarma", + "D. F. Redmiles", + "A. van der Hoek" + ], + "year": 2012, + "doi": "10.1109/tse.2011.64", + "type": "journal-article" + }, + "metadata_sha256": "7cf44d76fed3c80abc38116f4277e833353a5d021a55129fd3c25ef99176d334", + "claim_support_sha256": "08c49c0ce3ef99e7259abf3554a822a6bc345f1978a290d0b9ccf61a2add5355", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "crystal_2013", + "identifier": "doi:10.1109/TSE.2013.28", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1109%2FTSE.2013.28", + "checked_at": "2026-08-02T05:08:53.800139+00:00", + "metadata": { + "title": "Early Detection of Collaboration Conflicts and Risks", + "authors": [ + "Yuriy Brun", + "Reid Holmes", + "Michael D. Ernst", + "David Notkin" + ], + "year": 2013, + "doi": "10.1109/tse.2013.28", + "type": "journal-article" + }, + "metadata_sha256": "7dc1e1d072373eefb76c8c13e279d5b2f797c0f512ca3c360c1c2341ba1640e8", + "claim_support_sha256": "d48823a32bc32a4f7b02f756c4dcc9155ce06f98c0eaee6bc5ed6709bcee1c06", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "threewaymerge_2018", + "identifier": "doi:10.1145/3276535", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1145%2F3276535", + "checked_at": "2026-08-02T05:08:54.963170+00:00", + "metadata": { + "title": "Verified three-way program merge", + "authors": [ + "Marcelo Sousa", + "Isil Dillig", + "Shuvendu K. Lahiri" + ], + "year": 2018, + "doi": "10.1145/3276535", + "type": "journal-article" + }, + "metadata_sha256": "0d3ed2e208c29cac492e1e26e8ac39ac404525a4bdac10c952ef8defcf709b9b", + "claim_support_sha256": "036316e17f07492fe04051987f5b60770d722c7502d50e6b60dae81f5f6ab8ae", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "occ_1981", + "identifier": "doi:10.1145/319566.319567", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1145%2F319566.319567", + "checked_at": "2026-08-02T05:08:56.107483+00:00", + "metadata": { + "title": "On optimistic methods for concurrency control", + "authors": [ + "H. T. Kung", + "John T. Robinson" + ], + "year": 1981, + "doi": "10.1145/319566.319567", + "type": "journal-article" + }, + "metadata_sha256": "2068d14d308175a88e222b38f6e0cf2c20b46b48c8f637b3ce076425ed480bbf", + "claim_support_sha256": "705ecb260f1167746c36a2d8477c4e1b428eec6bddee7bcb9b3b0915c6cfad1c", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "snapshot_isolation_1995", + "identifier": "doi:10.1145/223784.223785", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1145%2F223784.223785", + "checked_at": "2026-08-02T05:08:57.251944+00:00", + "metadata": { + "title": "A critique of ANSI SQL isolation levels", + "authors": [ + "Hal Berenson", + "Phil Bernstein", + "Jim Gray", + "Jim Melton", + "Elizabeth O'Neil", + "Patrick O'Neil" + ], + "year": 1995, + "doi": "10.1145/223784.223785", + "type": "proceedings-article" + }, + "metadata_sha256": "e8ba388f67baf7343a6dacad33f419d4352fe0522522353791261c773ce7b3fc", + "claim_support_sha256": "f371f87496b09d16804c5455a10ee8e83e86ed4e6a047d9430777af38665d085", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "cooperbench_2026", + "identifier": "arxiv:2601.13295", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2601.13295", + "checked_at": "2026-08-02T05:08:58.426179+00:00", + "metadata": { + "title": "CooperBench: Why Coding Agents Cannot be Your Teammates Yet", + "authors": [ + "Arpandeep Khatua", + "Hao Zhu", + "Peter Tran", + "Arya Prabhudesai", + "Frederic Sadrieh", + "Johann K. Lieberwirth", + "Xinkai Yu", + "Yicheng Fu", + "Michael J. Ryan", + "Jiaxin Pei", + "Diyi Yang" + ], + "year": 2026, + "arxiv_id": "2601.13295", + "type": "preprint" + }, + "metadata_sha256": "aa935a9150889c3a639b2000a80dcc489761fd6d189448832238317cb3d861ab", + "claim_support_sha256": "b70d8589c7fc8240b58457f285a395cd5d4c60e1c36315f4e8c1ecf136e97b8e", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "agenticflict_2026", + "identifier": "doi:10.1145/3805760.3814923", + "status": "verified", + "provider": "crossref", + "request_url": "https://api.crossref.org/works/10.1145%2F3805760.3814923", + "checked_at": "2026-08-02T05:09:00.326711+00:00", + "metadata": { + "title": "AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub", + "authors": [ + "Daniel Ogenrwot", + "John Businge" + ], + "year": 2026, + "doi": "10.1145/3805760.3814923", + "type": "proceedings-article" + }, + "metadata_sha256": "07b395ac90ed34fea3e0b0a69bff48abe02b5a3efc2bec92b0d3d0f94479c72e", + "claim_support_sha256": "c838113a574bd5470056da53fc00a3bb9eeb4d540573a9aa399e9993a3649d8f", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "teambench_2026", + "identifier": "arxiv:2605.07073", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2605.07073", + "checked_at": "2026-08-02T05:09:01.459991+00:00", + "metadata": { + "title": "TeamBench: Evaluating Agent Coordination under Enforced Role Separation", + "authors": [ + "Yubin Kim", + "Chanwoo Park", + "Taehan Kim", + "Eugene Park", + "Samuel Schmidgall", + "Salman Rahman", + "Chunjong Park", + "Cynthia Breazeal", + "Xin Liu", + "Hamid Palangi", + "Hae Won Park", + "Daniel McDuff" + ], + "year": 2026, + "arxiv_id": "2605.07073", + "type": "preprint" + }, + "metadata_sha256": "e898a4d5400fe6bc4cf769626376c3df4aaf6d21b16f9783650730118117dd41", + "claim_support_sha256": "763d2827ba4ac2b7ce15bed95fdf72aa0cda3f271dff77d35b371fa101aed986", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "swebench_2024", + "identifier": "arxiv:2310.06770", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2310.06770", + "checked_at": "2026-08-02T05:09:02.490988+00:00", + "metadata": { + "title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", + "authors": [ + "Carlos E. Jimenez", + "John Yang", + "Alexander Wettig", + "Shunyu Yao", + "Kexin Pei", + "Ofir Press", + "Karthik Narasimhan" + ], + "year": 2023, + "arxiv_id": "2310.06770", + "type": "preprint" + }, + "metadata_sha256": "6e5af16c31da21b3108f7a19ded4b85b70b49607e5516029882b862614a91b79", + "claim_support_sha256": "6c9a13e4f79fe8cc6c83ce667fee469353e28891c66e3fc212d508ca548b36e3", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + }, + { + "key": "paperbench_2025", + "identifier": "arxiv:2504.01848", + "status": "verified", + "provider": "arxiv", + "request_url": "https://export.arxiv.org/api/query?id_list=2504.01848", + "checked_at": "2026-08-02T05:09:03.869088+00:00", + "metadata": { + "title": "PaperBench: Evaluating AI's Ability to Replicate AI Research", + "authors": [ + "Giulio Starace", + "Oliver Jaffe", + "Dane Sherburn", + "James Aung", + "Jun Shern Chan", + "Leon Maksin", + "Rachel Dias", + "Evan Mays", + "Benjamin Kinsella", + "Wyatt Thompson", + "Johannes Heidecke", + "Amelia Glaese", + "Tejal Patwardhan" + ], + "year": 2025, + "arxiv_id": "2504.01848", + "type": "preprint" + }, + "metadata_sha256": "1671122a5a16e1f1d44046462e292dc73ecac9245dcfe59f350744465169acb3", + "claim_support_sha256": "5a6d52b8e6a5c14a2a69b5f982d89e6156a29c4ff18ad4a5a2c350cd58d986b1", + "cross_check": null, + "cross_check_sha256": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", + "error": "" + } + ] +} diff --git a/ephemeral-sandbox-v1/citation_requests.json b/ephemeral-sandbox-v1/citation_requests.json new file mode 100644 index 0000000..72a2b9d --- /dev/null +++ b/ephemeral-sandbox-v1/citation_requests.json @@ -0,0 +1,90 @@ +{ + "schema_version": "ai-research-writing/citation-requests-v1", + "entries": [ + { + "key": "swe_minisandbox_2026", + "identifier": "arxiv:2602.11210", + "claim_support": [{"claim": "SWE-MiniSandbox presents container-free isolated workspaces for scaling software-engineering agent training.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes kernel-level isolation and a container-free approach.", "source": "references/related_work.md section 1"}] + }, + { + "key": "agentbay_2025", + "identifier": "arxiv:2512.04367", + "claim_support": [{"claim": "AgentBay provides sandbox sessions with hybrid human and agent control across execution environments.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes isolated execution environments and a hybrid control interface.", "source": "references/related_work.md section 1"}] + }, + { + "key": "deltabox_2026", + "identifier": "arxiv:2605.22781", + "claim_support": [{"claim": "DeltaBox studies change-based sandbox checkpoint and rollback, including process-state support.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract distinguishes filesystem and process-state checkpoint/rollback mechanisms.", "source": "references/related_work.md section 2"}] + }, + { + "key": "shepherd_2026", + "identifier": "arxiv:2605.10913", + "claim_support": [{"claim": "Shepherd records agent-environment interactions in a typed execution trace and supports forking and replay.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes a Git-like execution trace and replayable past state.", "source": "references/related_work.md section 2"}] + }, + { + "key": "caid_2026", + "identifier": "arxiv:2603.21489", + "claim_support": [{"claim": "CAID centrally delegates dependency-aware work, executes subtasks in isolated workspaces, and integrates with executable verification.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes centralized asynchronous isolated delegation and structured integration.", "source": "references/related_work.md section 3"}] + }, + { + "key": "coagent_2026", + "identifier": "arxiv:2606.15376", + "claim_support": [{"claim": "CoAgent coordinates speculative in-place shared-state effects through a predetermined order and agent repair.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes MTPO, speculative writes, notifications, and repair.", "source": "references/related_work.md section 3"}] + }, + { + "key": "claimplane_2026", + "identifier": "arxiv:2607.21909", + "claim_support": [{"claim": "Claim Plane treats concurrent software change as an admission and dynamic-scope control problem.", "relation": "contrast", "status": "verified", "evidence": "Primary abstract describes versioned change intents, atomic admission, and scope promotion.", "source": "references/related_work.md section 3"}] + }, + { + "key": "palantir_2012", + "identifier": "doi:10.1109/TSE.2011.64", + "claim_support": [{"claim": "Palantir studies early detection of development conflicts arising from parallel code changes.", "relation": "background", "status": "verified", "evidence": "The DOI record and source inventory identify early conflict detection for parallel code changes.", "source": "references/related_work.md section 3"}] + }, + { + "key": "crystal_2013", + "identifier": "doi:10.1109/TSE.2013.28", + "claim_support": [{"claim": "Crystal detects collaboration conflicts and risks before integration.", "relation": "background", "status": "verified", "evidence": "The DOI record and source inventory identify early detection of collaboration conflicts and risks.", "source": "references/related_work.md section 3"}] + }, + { + "key": "threewaymerge_2018", + "identifier": "doi:10.1145/3276535", + "claim_support": [{"claim": "Verified Three-Way Program Merge addresses semantic conflict-freedom beyond line-oriented text merging.", "relation": "contrast", "status": "verified", "evidence": "The DOI record and source inventory identify verified semantic program merge.", "source": "references/related_work.md section 3"}] + }, + { + "key": "occ_1981", + "identifier": "doi:10.1145/319566.319567", + "claim_support": [{"claim": "Optimistic concurrency control uses tentative work followed by validation rather than long-held locks.", "relation": "background", "status": "verified", "evidence": "The DOI record identifies the foundational optimistic concurrency-control work.", "source": "references/related_work.md section 4"}] + }, + { + "key": "snapshot_isolation_1995", + "identifier": "doi:10.1145/223784.223785", + "claim_support": [{"claim": "Snapshot-isolation terminology must not be imported as a claim of serializable filesystem semantics.", "relation": "background", "status": "verified", "evidence": "The cited critique is used only as a guardrail on terminology and does not establish system behavior here.", "source": "references/related_work.md section 4"}] + }, + { + "key": "cooperbench_2026", + "identifier": "arxiv:2601.13295", + "claim_support": [{"claim": "CooperBench evaluates coordination among coding agents on collaborative coding tasks.", "relation": "background", "status": "verified", "evidence": "Primary abstract identifies a benchmark for collaborative coding and coordination failures.", "source": "references/related_work.md section 5"}] + }, + { + "key": "agenticflict_2026", + "identifier": "doi:10.1145/3805760.3814923", + "claim_support": [{"claim": "AgenticFlict is used only as narrow motivation about textual Git-conflict outcomes, not as a universal interference rate.", "relation": "background", "status": "verified", "evidence": "The DOI record is scoped in the inventory to a selected simulated-pull-request dataset.", "source": "references/related_work.md section 5"}] + }, + { + "key": "teambench_2026", + "identifier": "arxiv:2605.07073", + "claim_support": [{"claim": "TeamBench evaluates agent coordination under enforced role separation.", "relation": "background", "status": "verified", "evidence": "The primary record is used only for role-separated coordination evaluation context.", "source": "references/related_work.md section 5"}] + }, + { + "key": "swebench_2024", + "identifier": "arxiv:2310.06770", + "claim_support": [{"claim": "SWE-bench provides executable repository-level issue-resolution tasks for software agents.", "relation": "background", "status": "verified", "evidence": "The primary preprint is the source record for the benchmark.", "source": "references/related_work.md section 5"}] + }, + { + "key": "paperbench_2025", + "identifier": "arxiv:2504.01848", + "claim_support": [{"claim": "PaperBench supplies long-horizon research-replication tasks but is not an integration-effect benchmark.", "relation": "background", "status": "verified", "evidence": "The primary record is used as workload background with that limitation stated.", "source": "references/related_work.md section 5"}] + } + ] +} diff --git a/ephemeral-sandbox-v1/citation_verification.md b/ephemeral-sandbox-v1/citation_verification.md index fd75ee9..86ef7cb 100644 --- a/ephemeral-sandbox-v1/citation_verification.md +++ b/ephemeral-sandbox-v1/citation_verification.md @@ -1,7 +1,32 @@ # Citation verification -PW0 introduces no manuscript citation and no BibTeX entry. `references.bib` is comment-only. Citation metadata and sentence-level support remain a later evidence-gated task. +All scholarly entries used by `main.tex` have a terminal record in +`citation_lock.json`. The lock was regenerated from `citation_requests.json` +on 2026-08-02 with the arXiv and Crossref primary providers. Sentence-level +claim support is intentionally limited to the relationship stated in the +manuscript. -| Key | Status | Provenance | Sentence-level relation | -|---|---|---|---| -| None | not-applicable | PW0 scaffold | No cited sentence exists. | +| Key | Status | Primary provenance | Manuscript relation | +| --- | --- | --- | --- | +| `swe_minisandbox_2026` | verified | arXiv | Isolated-execution contrast. | +| `agentbay_2025` | verified | arXiv | Hybrid sandbox-session contrast. | +| `deltabox_2026` | verified | arXiv | Checkpoint/rollback contrast. | +| `shepherd_2026` | verified | arXiv | Reversible-trace contrast. | +| `caid_2026` | verified | arXiv | Asynchronous coding-orchestration contrast. | +| `coagent_2026` | verified | arXiv | Shared-state concurrency-control contrast. | +| `claimplane_2026` | verified | arXiv | Intent-admission/control-plane contrast. | +| `palantir_2012` | verified | Crossref | Early conflict-awareness background. | +| `crystal_2013` | verified | Crossref | Collaboration-risk diagnosis background. | +| `threewaymerge_2018` | verified | Crossref | Semantic-merge limitation boundary. | +| `occ_1981` | verified | Crossref | Optimistic-validation background. | +| `snapshot_isolation_1995` | verified | Crossref | Terminology guardrail. | +| `cooperbench_2026` | verified | arXiv | Collaborative-coding evaluation context. | +| `agenticflict_2026` | verified | Crossref | Narrow textual-conflict motivation. | +| `teambench_2026` | verified | arXiv | Role-separation evaluation context. | +| `swebench_2024` | verified | arXiv | Repository-task evaluation context. | +| `paperbench_2025` | verified | arXiv | Long-horizon workload context. | + +The union-mount primary proceedings record is cited as an official URL in a +footnote because its bibliographic identity is not a DOI or arXiv identifier, +the two identifier forms supported by the lock contract. Its official USENIX +record is listed in `literature/paper_inventory.md`. diff --git a/ephemeral-sandbox-v1/claim_evidence_map.md b/ephemeral-sandbox-v1/claim_evidence_map.md index fb5d3f2..2527297 100644 --- a/ephemeral-sandbox-v1/claim_evidence_map.md +++ b/ephemeral-sandbox-v1/claim_evidence_map.md @@ -1,109 +1,54 @@ -# Ephemeral Sandbox v1 claim–evidence map +# Ephemeral Sandbox v1 claim-evidence map -Status date: 2026-07-30. Source rows point to baseline commit [`b22862550e0a7cb4fe61ce581831e9244cc492b5`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5), not yet to the required annotated `paper-v1-freeze` tag. No performance, resource, productivity, or multi-agent workflow result is paper-ready. +**Scope date:** 2026-08-02. Source claims are revalidated against measured +product commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` and tag +`paper-v1.1-freeze`. Quantitative claims are bound to final EXP1 v1.1 numeric +evidence only. -## Evidence levels +## Evidence classes -- **Implemented:** directly visible in source. -- **Contract-tested:** tested at the CLI/catalog boundary; does not establish runtime correctness. -- **Correctness-tested:** tested at the workspace/layer/runtime boundary. -- **Documented only:** public framing that still requires frozen source or experiment support. -- **Hypothesis:** motivates an experiment; must not be written as a result. -- **Out of scope:** explicitly disclaimed. -- **Blocked:** missing artifact, freeze, audit, or measurement. +- **Source:** visible in the measured source snapshot. +- **Contract-tested:** interface test existence; not empirical correctness. +- **Final RQ3:** one numeric-evidence v2 selector from the immutable archive. +- **Limitation:** intentionally not claimed as a measured result. -## Proposed paper contributions +## Retained contributions -| ID | Proposed contribution | Exact current evidence | Status and allowed wording | Required before submission | -|---|---|---|---|---| -| C1 | A LayerStack-backed workspace-session protocol: each sessionless command tool call creates an implicit isolated session, while an explicit session can group multiple command/file calls before publication or discard. | Per-command implicit create and `PublishThenDestroy`: [`exec_command.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/command/service/exec_command.rs), [`service/model.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs); explicit session contract: [`workspace_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime/workspace_session.rs); session snapshot/overlay: [`create_workspace.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/service/impls/create_workspace.rs), [`lifecycle/create.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/lifecycle/create.rs), [`kernel_mount.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs); lease value: [`stack/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/mod.rs). Tests: [`workspace_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session.rs), [`layerstack_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs), capture and squash/lease tests. | Implemented and partially correctness-tested. “Per tool call” is exact for sessionless `exec_command`, not every runtime operation. Claim the Linux mechanism, not measured isolation strength, security, or cheapness. | Relink final tag; test concurrent per-call sessions, explicit multi-call sessions, cross-session visibility/process/network isolation, and lease-held compaction/restart. | -| C2 | A capture and publication protocol that validates a whole data changeset against the active head, performs bounded eligible text merges, rejects unresolved conflicts, and exposes the accepted state through an atomic layer/manifest transition. | Capture: [`capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs). Plan/resolve/merge: [`plan.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/plan.rs), [`resolve.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs), [`merge.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/merge.rs). Durable transition: [`ops/publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs), [`storage/fs.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/storage/fs.rs). Tests: [`layerstack/tests/unit/publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/publish.rs), [`workspace_session_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs). | Implemented and correctness-tested. “Atomic” applies to one resolved **data changeset at manifest visibility**, not audit, accounting, cleanup, semantic correctness, or every platform's crash model. | Relink final tag; run concurrent-publisher and injected-fault durability tests on the paper platform; define the abstract state transition. | -| C3 | A role-separated, catalog-derived operational contract for management, sandbox-scoped runtime work, and read-only observability, with request correlation and machine-readable shell behavior. | Exact projections: [`manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/manager.rs), [`runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/runtime.rs), [`observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/observability.rs). Scope/routes: [`routed.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/routed.rs). Request/config/output/help: [`request.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/request.rs), [`config.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/config.rs), [`output.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/output.rs), [`help.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/help.rs). Tests: [`projection_integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/projection_integrity.rs) and three client test files. | Implemented and contract-tested. At baseline: 8/10/8 operations. Claim role-specific exposure and orchestration utility, not an authorization/security boundary. | Generate/export contract from final tag; archive help/JSON fixtures and binary digests; reconcile live-site drift. | -| C4 | Explicit lifecycle semantics distinguish persistent sessions, publish/retry, discard, implicit publish-then-destroy, and post-commit cleanup failure. | Finalization policy: [`service/model.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs). Explicit publish state machine: [`publish_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs). Implicit finalization: [`finalize_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/finalize_session.rs). Retry/partial-success tests: [`workspace_session_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs), [`layerstack_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs). | Implemented and correctness-tested. This contribution is about explicit failure semantics, not automatic recovery from every crash. | Relink final tag; resolve explicit/implicit protected-drop asymmetry; test daemon restart/reconciliation and abandoned-session cleanup. | -| C5 | An evaluation framework for determining whether the mechanism raises a workload-specific useful-work concurrency ceiling. | External lab at [`ephemeral-sandbox-test@d456187`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/tree/d45618733c8bfe75466947fdb9c47bea67f74b78), including [`concurrency-scaling.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/blob/d45618733c8bfe75466947fdb9c47bea67f74b78/benchmark/presets/concurrency-scaling.yml) and [`publication.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/blob/d45618733c8bfe75466947fdb9c47bea67f74b78/benchmark/presets/publication.yml). Existing demo runs are pre-freeze/exploratory. | Protocol-ready, **unmeasured**. In an evaluation-ready paper this can become a methodological contribution; it is not currently a result contribution. | Freeze code and benchmark commits; preregister comparisons/metrics; preserve raw data/provenance; run enough seeds/repeats; publish analysis and uncertainty. | - -## Design and interface claim matrix - -| Claim ID | Candidate paper sentence | Evidence/status | Boundary or missing evidence | -|---|---|---|---| -| D1 | A sessionless `exec_command` tool call creates one implicit workspace session over a leased LayerStack snapshot and uses `publish_then_destroy` after its command ledger drains. | Implemented/correctness-tested: [`exec_command.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/command/service/exec_command.rs), [`service/model.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs), [`layerstack_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs). | The automatically created session currently uses the shared network profile. Publication occurs when the command ledger reaches terminal/drained state, not necessarily when the initial CLI response returns. | -| D2 | Multiple command and file tool calls targeting the same explicit session share its private live overlay until explicit publication or discard; distinct sessions have distinct writable upperdirs. | Implemented: [`session/state.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/session/state.rs), [`workspace_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime/workspace_session.rs), C1 creation sources. | Add direct two-session visibility tests for the final artifact. | -| D2a | Sessionless `file_read` projects the active LayerStack, while sessionless `file_write` and `file_edit` amend the current head directly rather than creating workspace sessions. | Implemented: [`read.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/read.rs), [`write.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/write.rs), [`edit.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/edit.rs). | Required qualifier: do not generalize the per-call workspace-session model to every runtime tool operation. | -| D3 | A lease captures one manifest and ordered layer-path set and prevents GC of those referenced layers until safe release/substitution. | Implemented/correctness-tested: [`stack/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/mod.rs), [`lease/cleanup.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/cleanup.rs), [`squash.rs` tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/squash.rs). | Do not call this serializable snapshot isolation. Restart ordering remains to test. | -| D4 | Capture derives changes from kernel overlay metadata in the private upperdir. | Implemented/correctness-tested: [`capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs), [`overlay_capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/tests/unit/overlay_capture.rs). | Unsupported entries become protected drops; capture is filesystem delta, not process state. | -| D5 | Publication uses path fingerprints against the current head and can three-way merge eligible exact-file text divergence. | Implemented/correctness-tested: C2 plan/resolve/merge sources and tests. | Merge is line-oriented, bounded to 8 MiB, and rejects binary/invalid UTF-8/conflicting edits. No semantic guarantee. | -| D6 | An unresolved change rejects the whole resolved data changeset; it does not publish an unrelated subset. | Implemented/correctness-tested: [`resolve.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs), `source_occ_conflict_rejects_without_publishing_ignored_changes` in [`publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/publish.rs). | This excludes audit/accounting/cleanup from the transaction. | -| D7 | Successful publication durably promotes a new layer/digest and makes it visible by atomic active-manifest replacement. | Implemented/correctness-tested: C2 publish/storage sources and injected-failure tests. | Paper platform only after fault tests; avoid generic Windows durability language. | -| D8 | Publication attribution is best-effort and appended after the data commit. | Implemented: [`publish_changes.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/layerstack/service/impls/publish_changes.rs), [`file/audit.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/audit.rs). | Required limitation. Do not say data and attribution commit atomically; blame can be `unknown`. | -| D9 | A committed publication can be reported as published-but-not-closed if later session cleanup fails. | Implemented/correctness-tested: [`publish_session.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs), `committed_destroy_failure_is_partial_success...` in [`workspace_session_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs). | Required lifecycle boundary; cleanup is not transactionally rolled back. | -| D10 | Source-derived client surfaces contain 8 management, 10 runtime, and 8 observability operations at the baseline. | Implemented/contract-tested: three projections and [`projection_integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/projection_integrity.rs). | Must be regenerated from final tag; site is currently 8/7/5. | -| D11 | Runtime requests are sandbox-scoped; only runtime accepts an explicit correlated request ID, while other clients generate UUIDv4 IDs. | Implemented/contract-tested: [`client/request.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/request.rs), [`cli/runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/runtime.rs), [`request_builder.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/request_builder.rs). | Correlation supports orchestration/debugging; it does not prove causal attribution of every filesystem line. | -| D12 | Successful operations write JSON to stdout; remote/transport failures and local usage failures use JSON stderr with distinct exits. | Implemented/contract-tested: [`output.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/output.rs), client CLI tests. | Help is text on stdout. Manager progress may share stderr with a successful final stdout result. | -| D13 | Help is derived by joining the catalog and CLI projection. | Implemented/contract-tested: [`help.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/help.rs), [`help` tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/help.rs). | Does not make the live website auto-synchronized. | - -## Operational cost and evolution claims - -The detailed variable definitions, derivations, and measurement matrix are in [`complexity_and_evolution.md`](./complexity_and_evolution.md). These rows authorize source-derived cost-driver language, not measured performance adjectives. - -| Claim ID | Candidate paper sentence | Evidence/status | Boundary or missing evidence | +| ID | Claim | Evidence | Allowed wording and boundary | |---|---|---|---| -| K1 | Session acquisition and userspace OverlayFS setup process every manifest layer, while each live lease retains a cloned manifest; source-level setup metadata is \(O(L)\) per session and live lease-manifest metadata is \(O(R)\), commonly \(O(SL)\). | Implemented/source-derived: [`stack/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/mod.rs), [`lease/registry.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/registry.rs), [`kernel_mount.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs). | This is userspace algorithm/representation analysis, not a startup-latency or total-RSS result. Measure \(S \times L\); kernel mount cost is not claimed asymptotically. | -| K2 | Capture is metadata-only for regular-file payloads, visits the private upperdir, and sorts entries per directory; its retained change metadata is \(O(U)\), with sorting work \(O(\sum_j d_j\log d_j)\). | Implemented/source-derived: [`capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs). | Metadata-only capture does not mean the session upper or later publication is byte-free. Measure entry/fan-out/path/xattr cases. | -| K3 | v1 publication serializes active-head resolution, eligible merge, payload hashing/copying/syncing, and manifest replacement; its byte path is at least proportional to published regular-file bytes and can accumulate writer queueing under concurrent publishers. | Source-proven mechanism/hypothesis: [`ops/publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs), [`layer/write.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/layer/write.rs), [`model/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/model/mod.rs). | The critical section and copy path are implemented; their observed latency, queue saturation, and physical allocation are unmeasured. Record writer wait and hold separately. | -| K4 | The eligible text merge is byte-limited to 8 MiB per input, but its retained Myers trace remains edit-distance dependent; for \(K=\min(n,200{,}000)\) and reached distance \(D\), the implementation retains \(O(KD)\) trace state and can approach an \(O(K^2)\) envelope. | Implemented/source-derived: [`merge.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/merge.rs), [`resolve.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs). | Do not call merge linear or robustly resource-bounded solely from the byte cap. Add adversarial line-count/edit-distance CPU/RSS tests and preferably a stricter implementation bound. | -| K5 | Squash builds outside the writer lock and hardlinks regular-file winners, while live lease boundaries constrain compaction and lease release retains any layer referenced by active history or another lease. | Implemented/correctness-tested: [`squash.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/squash.rs), [`flatten.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/squash/flatten.rs), [`lease/cleanup.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/cleanup.rs). | Do not infer physical storage savings or compaction latency. Measure layer/entry count, lease age, peak storage, and residual storage after release. | -| K6 | LayerStack 2.0 is a candidate evolution path for reducing byte-copy and allocated-storage amplification through a qualified single storage domain and reflink-backed transfer. | Future-work protocol only at [`ephemeral-sandbox-layerstack-2-experiment@6e486fc`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/tree/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e), especially [`README.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/README.md) and [`EXPERIMENT.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/EXPERIMENT.md). | Not a v1 capability, result, or cross-platform guarantee. Reflink is extent/filesystem dependent and must not be called \(O(1)\). | -| K7 | The pinned stock Windows Docker Desktop/WSL 2 LayerStack 2.0 feasibility cell failed direct `FICLONE` with `errno=95`; the integrated Windows candidate remained blocked, and a copy fallback is required for correctness. | Negative experiment result: [`CONCLUSION.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/CONCLUSION.md) and [sealed report](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/reports/20260719T100800Z-x86_64-6af011f4-s0001.md). | Scope only to the recorded Windows/Docker/WSL/storage cell. It does not prove reflink unavailable on every Windows configuration or say anything positive about Linux/macOS. | +| C1 | Leased LayerStack workspaces provide private overlay views; sessionless `exec_command` uses an implicit session and explicit sessions retain selected multi-call private state. | Source: workspace/layer/session paths at `5c48dae`; prior tests are contract/correctness-test context only. | Describe implementation, not measured isolation, security, or every-operation behavior. Sessionless file paths are different. | +| C2 | Capture and current-head reconciliation implement source-defined candidate handling, bounded eligible text merge, whole-candidate rejection, and active-manifest data visibility. | Source: capture, publish plan/resolve/merge, layer writer at `5c48dae`. | `Atomic` applies only to resolved data visibility, not semantic validity, audit, accounting, cleanup, or every crash model. | +| C3 | The public interface contains 8 management, 10 runtime, and 8 observability operations. | Regenerated [`cli_contract_matrix.md`](cli_contract_matrix.md), measured projection sources, and contract tests. | All three clients support optional validated request IDs; Windows default is local named pipe, with TCP compatibility. This is not a security boundary. | +| C4 | Lifecycle distinguishes retryable precommit failure, discard, no-op, committed publication, and published-but-not-closed cleanup failure. | Source: session finalization/publish paths at `5c48dae`. | Do not claim automatic crash recovery, consistent post-commit attribution, or tested restart semantics. | +| C5 | EXP1 v1.1 describes startup, public-CLI operations, and seven resource rows for one treatment. | Final archive plus 153 selector-bound entries in `experiments/analysis/final-v11-019fb86c-tables-a/numeric-evidence.json`. | Descriptive only; not comparative, causal, reliable-in-general, or useful-work evidence. | -## Motivation and concurrency claims +## Eligible quantitative claims -| Claim ID | Claim | Classification | Evidence and safe treatment | +| ID | Sentence family | Numeric evidence binding | Boundary | |---|---|---|---| -| M0 | Conventional operating-system abstractions do not provide a first-class coding-agent role, delegated task, workspace-session base, resource-ownership record, or publish/reject event. | Systems framing and interface observation. | Primary OS documentation establishes process/file/path/lock/event semantics; Git documents worktree boundaries. State this as an abstraction mismatch, not that operating systems are broken, exclusively designed for humans, or incapable of hosting agents. See [`project_inventory.md`](./project_inventory.md). | -| M0a | Current agent-team products make parallel research, review, planning, and separated modules useful or legible, while fine-grained interdependent writers leave runtime questions open. | Engineering motivation only. | Product documentation and the Agent Infra Foundation's [agent-team comparison](https://agent-infra-foundation.org/blog/2026/07/the-concurrency-ceiling-of-coding-agents/) support the interface comparison. They do not establish a general empirical concurrency limit or show that Ephemeral raises it. | -| M0b | Structured agent teams and exploratory agent swarms can encounter different concurrency limits. | Evaluation model/hypothesis. | Teams motivate dependency, handoff, shared-service, and integration-lane workloads; swarms motivate duplication, hot-path collision, staleness, resource saturation, and selection workloads. Treat these as workload families to instantiate and measure, not universal product categories. | -| M1 | Concurrent coding-agent tool calls can interfere through shared mutable workspace/execution state and can return stale or incompatible session deltas. | Motivation with mechanism examples. | Git worktrees and private-workspace literature establish the design problem; CooperBench v2 and AgenticFlict provide bounded integration evidence. Do not say runtime interference is always dominant. See [`references/related_work.md`](./references/related_work.md). | -| M2 | The practical concurrency ceiling is workload- and runtime-dependent. | Definition/hypothesis, not a universal result. | Define useful-work rate \(U(W,R,A,n)\) as verification-passing, durably accepted contribution units per wall time or cost. A ceiling region begins where marginal workers no longer improve it because conflict, retry, integration, verification, or resource costs dominate accepted progress. | -| M3 | Coding swarms stall at a single-digit worker count. | Unsupported generalization. | The [Agent Infra Foundation article](https://agent-infra-foundation.org/blog/2026/07/the-concurrency-ceiling-of-coding-agents/) is framing only. CooperBench's scaling probe is 2–4 agents on 46 tasks, not a universal threshold. Avoid. | -| M4 | Ephemeral Sandbox raises the useful-work concurrency ceiling. | Hypothesis requiring new experiment. | No frozen comparison or paper-ready number exists. Requires controlled shared-directory/worktree/Ephemeral runs, identical agents/budgets/tasks, multiple scales/seeds, accepted-work and resource metrics. | -| M5 | More agents increase throughput. | Unsupported/untrue in general. | CAID and CooperBench show configuration- and workload-dependent, sometimes non-monotonic outcomes. Never assert monotonic scaling. | -| M6 | Workspace isolation solves coding-agent coordination. | Out of scope. | Isolated CooperBench agents still exhibit coordination/integration failures. v1 does not solve model quality, decomposition, semantic agreement, or general communication/scheduling. | +| E1 | Sandbox create plus base mount had the Table 2 p50/p95/p99 values in the tested environment. | `table2.create_sandbox.none.c1.{p50_ms,p95_ms,p99_ms}` | Native Windows/Docker Desktop, pinned image, `paper-100m`, `product_cli`, concurrency 1. | +| E2 | A public CLI row had the Table 3 p50/p95/p99 and mean throughput values for its fixed payload and concurrency. | `table3.*` selectors used by deterministic table source | Throughput change is not individual-latency improvement, linear scaling, or superiority. | +| E3 | Table 4 reports available sampled resource metrics for seven preregistered rows. | `table4.*` selectors | `upperdir_bytes` is not unavailable host workspace allocated blocks; unavailable values remain unavailable. | +| E4 | The sole final completed 1,900/1,900 reportable measured trials without a classified campaign failure. | campaign/report provenance recorded in final handoff; numerical table claims remain selector-bound | Campaign completeness only, not product-wide reliability. | -## Evaluation claims and required artifacts +## Explicit limitations and removed claims -| Evaluation question | Minimum protocol | Measures | Status | -|---|---|---|---| -| Isolation correctness | Two or more leased sessions from controlled bases; cross-session file/process/network probes; cleanup and compaction during a lease; isolated and shared network profiles. | Visibility matrix, namespace identity, leakage failures, cleanup state, exact manifest/lease IDs. | Not yet run with v1-freeze provenance. | -| Publication correctness | Disjoint stale-base writes, same-path clean text merge, binary/structural conflict, protected/drop cases, concurrent publishers, injected storage faults, cleanup faults. | Accepted/rejected paths and reasons, exact pre/post manifest root/version, layer count/digest, unrelated-path leakage, retry behavior. | Unit/integration coverage exists; paper-grade matrix/fault run still required. | -| Latency/resource scaling | 1/2/4/8/... concurrent workers on fixed no-op, command, capture, publish, and conflict workloads; independently vary \(L,S,U,C,F,B_p\), merge line/edit-distance shape, lease age, and storage backend; use warmups and repeated measured runs. | Start/exec/capture/publish/squash latency distributions, writer wait/hold, CPU, RSS/PSS, I/O, logical/allocated/shared/exclusive storage, retained history, queue/retry/conflict rates, failures. | Source-derived cost model exists; no paper-ready runs. | -| Multi-agent workflow effect | Shared mutable directory vs Git worktrees vs Ephemeral on overlap-controlled tasks, same models/prompts/budgets/tooling/integration tests and matched seeds/repeats. Instantiate both structured-team workloads with explicit dependencies/handoffs and exploratory-swarm workloads with redundant or competing proposals. | Verification-passing durably accepted units per time/cost, conflict/retry/integration time, stale or duplicated work, selection cost, and clean textual publishes that fail tests. | Not run. | -| Operational attribution/recovery | Correlated requests through accept, reject, retry, cleanup failure, daemon restart. | Availability/correctness of base, request, paths, result and audit; unknown/missing attribution rate; recovery time/state. | Source exposes pieces; best-effort audit and restart behavior need measurement. | - -Every reported number must archive: source/tag and tag object, benchmark-repository commit, dirty status, OS/kernel/filesystem, hardware, runtime/cgroup settings, toolchain, binary/image digests, workload commit, exact commands/environment, seeds/repeats, raw NDJSON/logs, exclusions, and analysis-code commit. - -## Claims to avoid or state explicitly as limitations - -| Avoided claim | Reason | +| Area | Status | |---|---| -| Formal sandbox security or complete noninterference | No formal model/proof; namespace/overlay boundaries are narrower. | -| Universal egress denial | Networking can be shared; isolated network is a profile, not a universal policy proof. | -| Process-state checkpoint or rollback | v1 captures filesystem changes, not process state; DeltaBox and Shepherd cover broader state. | -| Windows overlay/reflink/durability support | Baseline overlay is Linux-only, no reflink implementation is evidenced, and Windows directory sync is a no-op. | -| Filesystem-enforced immutability of every layer | History is append-oriented and content-digested, but the audit found no enforcement proof that every promoted directory cannot be mutated externally. | -| Serializable snapshot isolation | A leased stable view and path-level validation do not establish database serializability. | -| Semantic merge correctness | Bounded line merge and fingerprints do not prove behavioral compatibility. | -| Atomic data + attribution + cleanup | Only resolved data publication is committed at the manifest boundary; audit/accounting/cleanup are separate. | -| Cheap, fast, scalable, or productivity-improving | No paper-ready measurements. | -| Linear-time or safely memory-bounded merge | The current byte limit does not eliminate edit-distance/line-count-dependent trace growth. | -| \(O(1)\) reflink publication or storage | Clone work/allocation is filesystem- and extent-dependent, LayerStack 2.0 is not a v1 capability, and the Windows feasibility cell failed. | -| Universal agent-count threshold | The useful limit depends on workload, runtime, models, orchestration, verification, and resources. | - -## Current blockers - -1. `paper-v1-freeze` does not yet exist. -2. Attribution is best-effort rather than atomically coupled to publication; the paper must decide whether to weaken “attributable” or change/test the implementation before freeze. -3. Explicit versus implicit protected-drop policy needs maintainer confirmation. -4. In-memory lease/substitution behavior across daemon restart needs fault testing. -5. Claim Plane's full manuscript needs a complete novelty audit. -6. The merge implementation needs adversarial CPU/RSS characterization or a stricter independent resource bound. -7. No final isolation, fault, scaling, resource, or multi-agent workflow dataset exists. +| RQ1 private-view/isolation correctness | No eligible final empirical matrix; source and test existence are not results. | +| RQ2 publication, fault, and semantic correctness | No eligible paper-platform fault matrix; source protocol only. | +| RQ4 competitive baseline, deeper scaling, useful work, productivity | Not measured; removed from Results and framing. | +| RQ5 attribution and restart recovery | Best-effort post-commit attribution and in-memory lease/restart gap remain limitations. | +| Security, egress, rollback, serializability | Out of scope. | +| Reflink/LayerStack 2.0 | Future work; recorded Windows `FICLONE errno=95` feasibility result is narrow and non-performance evidence. | +| Merge resource bound | The 8 MiB input gate does not independently bound line/edit-distance trace memory. | + +## Mandatory wording guards + +- Never pool v1.0, qualifier, smoke, pilot, projection, setup, verification, + teardown, partial, simulated, or unavailable values with final results. +- Never imply that a source link or test implies observed correctness. +- Never claim broad speed, scalability, superiority, production readiness, + security, or multi-agent productivity. +- Retain the local named-pipe change as preregistered treatment remediation and + disclose the Table-1 analysis compatibility erratum as numeric-neutral. diff --git a/ephemeral-sandbox-v1/cli_contract_matrix.md b/ephemeral-sandbox-v1/cli_contract_matrix.md index 78828ff..73cd8bf 100644 --- a/ephemeral-sandbox-v1/cli_contract_matrix.md +++ b/ephemeral-sandbox-v1/cli_contract_matrix.md @@ -1,133 +1,87 @@ # Ephemeral Sandbox v1 CLI contract matrix -Audited 2026-07-30 against baseline commit [`b22862550e0a7cb4fe61ce581831e9244cc492b5`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5). This is a provisional source snapshot: regenerate it from the eventual annotated `paper-v1-freeze` tag. - -## Contract authority and known documentation drift - -The primary interface evidence is the requested source projection directory: - -- Management: [`crates/sandbox-cli/src/projection/manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/manager.rs) — 8 operations. -- Runtime: [`crates/sandbox-cli/src/projection/runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/runtime.rs) — 10 operations. -- Observability: [`crates/sandbox-cli/src/projection/observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/observability.rs) — 8 operations. - -The live [CLI documentation](https://ephemeral-sandbox.com/docs/cli) lists 8/7/5. Relative to the baseline source, its runtime page omits `create_workspace_session`, `publish_workspace_session`, and `destroy_workspace_session`; its observability page omits `resources`, `daemon`, and `topology`. The source projection and semantic catalog, not the live site, define the paper interface. - -The semantic catalog supplies operation names, descriptions, argument types/defaults, visibility, and routing. The CLI projection supplies the shell path, flags/positionals, usage, and examples. [`projection_integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/projection_integrity.rs) tests a bidirectional match between public catalog routes and projections. - -## Why three clients - -The baseline builds three feature-gated executables and no combined CLI ([`sandbox-cli/Cargo.toml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/Cargo.toml), [`sandbox-cli/src/lib.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/lib.rs)): - -- **Management** owns host/fleet lifecycle, selection, compaction, and export. Requests use system envelope scope even when a target sandbox ID is an operation argument. -- **Runtime** owns stateful work inside one selected sandbox: execution, file operations, and explicit workspace-session lifecycle. Every request requires sandbox envelope scope. -- **Observability** contains read-only inspection operations. `snapshot` and `resources` can aggregate at system scope or target one sandbox; the other operations target a sandbox. - -This separation gives an orchestrator smaller role-specific operation sets and makes accidental cross-authority invocation testable ([manager rejection tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/manager.rs), [observability rejection tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/observability.rs)). It is an interface/least-exposure property, not by itself an authorization or sandbox-security guarantee. - -## Boundary: operational contract versus coordination plane - -The three-client division exposes lifecycle, runtime work, request correlation, and read-only inspection in role-specific surfaces. It does **not** make the baseline a complete agent-team or swarm control plane. The audited source contract does not establish: - -- declared coding intent or path/interface claims before execution; -- agent roles, task dependencies, ownership transfers, or handoff acceptance; -- port leases, discoverable service endpoints, or general resource-budget admission; -- semantic test compatibility or automated integration-lane scheduling; -- transactionally complete attribution across data publication, audit, accounting, and cleanup. - -These are relevant public-coordination facts in high-concurrency agent systems, but they remain external orchestration concerns or future runtime work unless they are implemented and source-proven before `paper-v1-freeze`. The paper can claim that the current interfaces provide typed building blocks for orchestration; it cannot claim that v1 implements the full coordination plane proposed by the concurrency-ceiling article. +**Measured-source audit:** 2026-08-02 against annotated +[`paper-v1.1-freeze`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8), commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`. This replaces the provisional +baseline matrix. It is source/contract evidence, not a runtime-correctness or +security result. + +## Final snapshot differences that affect the paper + +Relative to historical baseline `b22862550e0a7cb4fe61ce581831e9244cc492b5`, +the measured snapshot preserves the 8/10/8 operation sets but changes the +gateway interface. All three CLIs accept an optional validated `--request-id`; +`--gateway-endpoint URI` is the canonical global flag and +`--gateway-socket` remains a visible compatibility alias. Endpoint parsing +accepts TCP, local Windows named-pipe, and Unix-domain-socket forms. On Windows, +the default endpoint is `npipe://./pipe/ephemeral-sandbox-gateway`; TCP remains +an explicit compatibility endpoint. The EXP1 treatment used per-execution-block +local named pipes and must not be generalized to every deployment. + +Primary source: [`manager`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/manager.rs), +[`runtime`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/runtime.rs), +[`observability`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/observability.rs), +[`endpoint parser`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-operations/client/src/endpoint.rs), +and [`gateway listener`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-gateway/src/gateway/listener.rs). ## Management client: 8 system-scoped operations -Primary evidence: [`manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/manager.rs); semantics: [`catalog/src/manager`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/manager); tests: [`cli/tests/manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/manager.rs). - -| Operation | Source-projected usage | Effect | Envelope scope | -|---|---|---|---| -| `create_sandbox` | `sandbox-manager-cli create_sandbox --image IMAGE --workspace-bind-root PATH [--count N]` | Creates manager/runtime sandbox records and daemon(s); mutating. `--workspace-root` remains an accepted alias. | System | -| `list_docker_images` | `sandbox-manager-cli list_docker_images` | Reads available images. | System | -| `list_workspace_directories` | `sandbox-manager-cli list_workspace_directories [--path PATH]` | Reads host directory choices. | System | -| `destroy_sandbox` | `sandbox-manager-cli destroy_sandbox --sandbox-id ID` | Stops and removes the selected sandbox; mutating/destructive. | System; ID is an argument | -| `list_sandboxes` | `sandbox-manager-cli list_sandboxes` | Reads manager records. | System | -| `inspect_sandbox` | `sandbox-manager-cli inspect_sandbox --sandbox-id ID` | Reads one manager record. | System; ID is an argument | -| `squash_layerstacks` | `sandbox-manager-cli squash_layerstacks --sandbox-id ID` | Requests compaction and live-session migration; mutating representation while preserving the intended logical history. | System; ID is an argument | -| `export_changes` | `sandbox-manager-cli export_changes --sandbox-id ID --dest PATH [--format dir\|tar\|tar-zst]` | Reads a published delta and writes the requested destination artifact. It does not publish a workspace session. | System; ID is an argument | - -Manager `--progress` writes progress records to stderr while retaining the final JSON result on stdout; this separation is tested in [`manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/manager.rs). - -## Runtime client: 10 sandbox-scoped operations - -Primary evidence: [`runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/runtime.rs); semantics: [`catalog/src/runtime`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime); tests: [`cli/tests/runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/runtime.rs). - -All runtime invocations require the global `--sandbox-id ID`; the selector becomes the request-envelope scope and is removed from operation arguments ([request builder tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/request_builder.rs)). - -The workspace-session model has two source-distinct paths: - -- A sessionless `exec_command` creates one implicit private workspace session with `publish_then_destroy`; after its command ledger drains, the runtime captures the session delta, publishes or rejects it against LayerStack, and destroys the session ([`exec_command.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/command/service/exec_command.rs), [`service/model.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs)). -- An explicit `create_workspace_session` returns an ID that multiple command and file calls can share before deliberate `publish_workspace_session` merge-back or `destroy_workspace_session` discard ([session catalog](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime/workspace_session.rs)). - -This should not be generalized to every sessionless file operation: `file_read` projects the current LayerStack, while `file_write` and `file_edit` amend the current head directly ([read](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/read.rs), [write](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/write.rs), [edit](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/edit.rs)). - -| Operation | Source-projected usage | Effect | +| Operation | Measured-source usage | Effect | |---|---|---| -| `exec_command` | `sandbox-runtime-cli --sandbox-id ID exec_command [--workspace-session-id ID] [--timeout-ms N] [--yield-time-ms N] COMMAND` | Runs a command in a workspace view. It can mutate private state; without an explicit session, implicit finalization can publish then destroy. | -| `write_command_stdin` | `... write_command_stdin --command-session-id ID [--yield-time-ms N] TEXT` | Mutates a live command session's input/process state. | -| `read_command_lines` | `... read_command_lines --command-session-id ID [--start-offset N] [--limit N]` | Reads buffered command output. | -| `file_read` | `... file_read --path FILE [--offset N] [--limit N] [--workspace-session-id ID]` | Reads file contents from an implicit or explicit workspace view. | -| `file_write` | `... file_write --path FILE --content TEXT [--workspace-session-id ID]` | Writes the private view; an implicit operation can finalize through publication. | -| `file_edit` | `... file_edit --path FILE --edits JSON [--workspace-session-id ID]` | Applies validated string edits to the private view; an implicit operation can finalize through publication. | -| `file_blame` | `... file_blame --path FILE` | Reads best-effort publication attribution. It must not be presented as transactionally complete provenance. | -| `create_workspace_session` | `... create_workspace_session [--network-profile PROFILE]` | Creates a persistent private workspace session. `isolated` requests a network namespace; shared networking is otherwise possible. | -| `publish_workspace_session` | `... publish_workspace_session --workspace-session-id ID [--grace-s SECONDS]` | Captures and performs all-or-none data publication, then closes. Rejection/precommit failure preserves the session for retry; committed cleanup failure is reported as partial success. | -| `destroy_workspace_session` | `... destroy_workspace_session --workspace-session-id ID [--grace-s SECONDS]` | Tears down the session and discards unpublished private state; destructive to that session. | - -An operation-level command exit status is response data. The CLI exit status reports whether the requested runtime operation/protocol succeeded; a gateway-returned runtime failure envelope exits 1 even when its details contain a command exit code ([runtime CLI test](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/runtime.rs)). - -## Observability client: 8 read-only operations - -Primary evidence: [`observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/observability.rs); semantics: [`catalog/src/observability`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/observability); routing tests: [`catalog/tests/observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/tests/observability.rs). - -| Operation | Source-projected usage | View | Envelope scope | -|---|---|---|---| -| `snapshot` | `sandbox-observability-cli snapshot [--sandbox-id ID]` | Aggregate manager/gateway snapshot or one sandbox snapshot. | System if omitted; sandbox if supplied | -| `trace` | `... trace --sandbox-id ID [--trace-id TRACE\|last]` | One request trace, defaulting to the latest. | Sandbox required | -| `events` | `... events --sandbox-id ID [--name NAME] [--since-ms MS] [--last-n N]` | Bounded/filterable event records. | Sandbox required | -| `resources` | `... resources [--sandbox-id ID] [--window-ms MS]` | Fleet current-usage map at system scope or daemon-sampled sandbox history. | System if omitted; sandbox if supplied | -| `daemon` | `... daemon --sandbox-id ID` | Bounded daemon self/process/ownership diagnostics. | Sandbox required | -| `topology` | `... topology --sandbox-id ID` | Explicit bounded workspace process-topology collection. | Sandbox required | -| `cgroup` | `... cgroup --sandbox-id ID [--scope SCOPE] [--window-ms MS]` | Host/daemon resource counters for the sandbox or workspace scope. It is exposed read-only even though catalog routing can be manager-owned. | Sandbox required | -| `layerstack` | `... layerstack --sandbox-id ID [--workspace-id WS] [--window-ms MS]` | LayerStack/session history and current view. | Sandbox required | - -Only `snapshot` and `resources` are aggregate-capable in the baseline; this is asserted in [`catalog/tests/observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/tests/observability.rs). Aggregate/scoped request construction and one-request behavior are tested in [`cli/tests/observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/observability.rs). - -## Shared request, connection, and authentication discovery +| `create_sandbox` | `sandbox-manager-cli create_sandbox --image IMAGE --workspace-bind-root PATH [--count N]` | Creates sandbox records and daemons. | +| `list_docker_images` | `sandbox-manager-cli list_docker_images` | Reads image choices. | +| `list_workspace_directories` | `sandbox-manager-cli list_workspace_directories [--path PATH]` | Reads workspace choices. | +| `destroy_sandbox` | `sandbox-manager-cli destroy_sandbox --sandbox-id ID` | Removes the selected sandbox. | +| `list_sandboxes` | `sandbox-manager-cli list_sandboxes` | Reads manager records. | +| `inspect_sandbox` | `sandbox-manager-cli inspect_sandbox --sandbox-id ID` | Reads one manager record. | +| `squash_layerstacks` | `sandbox-manager-cli squash_layerstacks --sandbox-id ID` | Requests compaction/remount work. | +| `export_changes` | `sandbox-manager-cli export_changes --sandbox-id ID --dest PATH [--format dir\|tar\|tar-zst]` | Exports a published delta; it does not publish a session. | -The request envelope is `{op, request_id, scope, args}` ([request contract](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/contract/src/request.rs)). +Projection source: [`projection/manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/projection/manager.rs). -- Default request IDs are UUIDv4. Only the runtime CLI exposes `--request-id`; it accepts 1–128 ASCII alphanumeric characters plus `.`, `_`, `:`, and `-` ([runtime CLI input](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/runtime.rs), [request builder](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/request.rs), [tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/request_builder.rs)). -- Gateway connection resolution is CLI override (`--gateway-socket`, `--gateway-auth-token`) → `SANDBOX_GATEWAY_SOCKET` / `SANDBOX_GATEWAY_AUTH_TOKEN` → default socket `127.0.0.1:7878`; a blank configured value is rejected ([client configuration](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/config.rs), [config tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/tests/config.rs)). -- The direct client opens a TCP connection per request, inserts the token field, and exchanges newline-framed JSON ([client](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/src/client.rs), [protocol auth field](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-protocol/src/auth.rs)). -- The CLI client does **not** automatically discover a token file. Setup documentation shows an operator reading a token file into the environment, for example [Linux setup](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/docs/linux-setup.md). The token check is an implemented gateway authentication mechanism, not a general sandbox-security guarantee. - -## JSON streams and process exits - -[`sandbox-cli/src/output.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/output.rs) and the client-specific tests establish: - -| Condition | stdout | stderr | CLI exit | -|---|---|---|---:| -| Help or successful operation | Help text, or one JSON result line | Empty, except manager progress if requested | 0 | -| Gateway operation failure | Empty | Gateway JSON error envelope | 1 | -| Transport/protocol failure | Empty | Local JSON error envelope | 1 | -| Local syntax, unknown operation, invalid argument, or configuration/build error | Empty | Local JSON error envelope | 2 | - -The response envelope is defined in [`sandbox-operations/contract/src/response.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/contract/src/response.rs). Compatibility fixtures cover unknown operations and exit 2 ([`compatibility.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/compatibility.rs)). - -## Catalog-derived help contract - -Help joins semantic catalog entries with the CLI projection to render usage, arguments, defaults, examples, and related operations ([`help.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/help.rs)). Exact help snapshots and operation sets are tested in [`manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/manager.rs), [`runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/runtime.rs), and [`observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/observability.rs). These are CLI contract tests; they do not establish workspace isolation, publication correctness, or performance. - -## Freeze actions - -1. At `paper-v1-freeze`, export the semantic catalog and all three help snapshots from the tagged binary. -2. Record the tag object, exact commit, build toolchain, binary digests, command line, and raw exported JSON/text. -3. Diff tagged counts/usage/defaults against this matrix and the live site. -4. Treat any site discrepancy as documentation drift, not as permission to edit the paper contract. +## Runtime client: 10 sandbox-scoped operations +Every runtime request requires `--sandbox-id`; it becomes the envelope scope. +The operation set is: + +`exec_command`, `write_command_stdin`, `read_command_lines`, `file_read`, +`file_write`, `file_edit`, `file_blame`, `create_workspace_session`, +`publish_workspace_session`, and `destroy_workspace_session`. + +`exec_command` may use `--workspace-session-id`; a sessionless command has the +documented implicit-session path. Sessionless `file_read` projects the active +LayerStack, while sessionless `file_write` and `file_edit` amend the current +head directly. Do not generalize the implicit-session rule to every runtime +operation. Projection source: [`projection/runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/projection/runtime.rs). + +## Read-only observability client: 8 operations + +| Operation | Scope | +|---|---| +| `snapshot` | System when `--sandbox-id` is omitted; sandbox otherwise. | +| `trace` | Sandbox required. | +| `events` | Sandbox required. | +| `resources` | System when `--sandbox-id` is omitted; sandbox otherwise. | +| `daemon` | Sandbox required. | +| `topology` | Sandbox required. | +| `cgroup` | Sandbox required. | +| `layerstack` | Sandbox required. | + +Projection source: [`projection/observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8/crates/sandbox-cli/src/projection/observability.rs). + +## Shared contract and evidence boundary + +The envelope contains operation, request ID, scope, and arguments. Successful +operations produce a JSON result on stdout with exit 0. Remote or transport +failures use a JSON stderr envelope and exit 1; local usage/configuration +errors use JSON stderr and exit 2. Manager progress can share stderr with a +successful stdout result. Contract tests exercise catalog/projection integrity, +CLI input validation, output conventions, and endpoint parsing. They do not +measure workspace isolation, publication correctness, security, recovery, or +performance. + +The client division exposes narrower role-specific operation sets but does not +implement task intent, ownership, handoffs, service/port leases, admission, +semantic verification, or a general coordination plane. The public website is +not the paper authority because its operation counts may drift from this +measured source snapshot. diff --git a/ephemeral-sandbox-v1/experiment_inventory.md b/ephemeral-sandbox-v1/experiment_inventory.md index 1924fac..46afe97 100644 --- a/ephemeral-sandbox-v1/experiment_inventory.md +++ b/ephemeral-sandbox-v1/experiment_inventory.md @@ -1,15 +1,39 @@ # Focused sandbox performance experiment specification -**Protocol ID:** `ephemeral-sandbox-v1-practical-performance-v0.1` -**Status:** Draft; not protocol-locked and not approved for final measurement -**Last updated:** 2026-07-30 +**Active protocol ID:** `ephemeral-sandbox-v1-practical-performance-v1.1` + +**Status:** v1.1 Gates 0--7 complete; final evidence ready for author review + +**Last updated:** 2026-07-31 **Primary tracker:** the phase checklists in this document **Run log:** [`experiments/experiment_log.md`](experiments/experiment_log.md) +The fixed study design below originated in v1.0, whose sole final attempt +failed and remains permanently ineligible. The completed treatment and active +Gate decisions are governed by the +[`EXP1 v1.1 local-IPC protocol amendment`](experiments/exp1-v1.1-protocol-amendment.md), +which changes only the scientific treatment's CLI-to-gateway transport to an +isolated Windows named pipe per gateway execution block and preserves all +other scientific choices. Its sole schema clarification adds a non-numeric +`Gateway transport` provenance row; measured table definitions are unchanged. +The v1.0 and v1.1 corpora must never be pooled or numerically compared. + +The preregistered v1.1 qualification, fresh smoke, five-sample pilot, +conservative runtime projection, product/paper freeze, strict final preflight, +sole final, immutable archival, deterministic tables, numeric-evidence v2, and +claim handoff all passed on 2026-07-31. The final run is +`019fb86c-096e-7589-a0a4-a6d6ef5d7f8b`; its archive tree is +`sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. +The retained evidence, hashes, post-freeze analysis compatibility erratum, and +Gate decisions are recorded in +[`experiments/experiment_log.md`](experiments/experiment_log.md) and the +[`final Gate report`](experiments/analysis/exp1-gate0-7-final-report.md). + ## Purpose and claim boundary This experiment characterizes whether Ephemeral Sandbox has practically useful -startup and public-operation performance in one disclosed Linux environment. +startup and public-operation performance on one disclosed native-Windows host +using Docker Desktop's pinned Linux sandbox environment. It does not attempt to show that Ephemeral beats another sandbox, shared directory, Git worktree, or agent framework. @@ -36,18 +60,24 @@ charter in [`lanes/experiments.md`](lanes/experiments.md). | Item | Locked choice for this protocol | |---|---| | Product branch | `main` | -| Product source baseline | `b22862550e0a7cb4fe61ce581831e9244cc492b5`; final clean commit/tag pending | +| Product source baseline | clean `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`, annotated tag `v0.1.4` | | Upstream benchmark snapshot | `d45618733c8bfe75466947fdb9c47bea67f74b78` | | Paper-local benchmark | [`benchmark/`](benchmark/) plus documented paper-local changes | -| Host | Ubuntu Server 24.04 LTS, Linux x86-64 | -| Host minimum | 8 vCPU, 16 GiB RAM, 100 GiB local NVMe-backed ext4 | -| Runtime | Docker Engine, cgroup v2 | +| Computer name | `DESKTOP-OLP1ADS` | +| Host | Native 64-bit Windows build 26200 | +| Qualified host capacity | 48 logical CPUs, 137,438,953,472 bytes memory, NTFS | +| Runtime | Docker Desktop 29.0.1; Linux AMD64 engine, `overlayfs`, cgroup v2 | +| Product path | `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox` | +| Paper path | `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1` | +| Environment qualifier | Native PowerShell; no Python dependency | | Sandbox image | `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` | | Sandbox resource profile | `standard`: 1 vCPU, 512 MiB memory maximum, 256 PIDs | | Network profile | `shared` | -| Client cohort | `direct_client` | -| Base workspace profile | `paper-100m` | -| Base workspace | 4,000 deterministic files, 100 MiB logical content, maximum depth 100 | +| Client cohort | `product_cli`; manager, runtime, and observability subprocesses | +| Environment qualification workspace | Two tiny isolated fixtures inside the qualification artifact directory | +| Canonical repo-backed workspace | `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\workspace-base\ephemeral-sandbox-v0.1.4` | +| Performance-only workspace profile | `paper-100m` (outside the environment task) | +| Performance-only base workspace | 4,000 deterministic files, 100 MiB logical content, maximum depth 100 | | Generator ceiling | Maximum depth 499 | | Seed | `20260712` | | Operation concurrency | 1 and 5 | @@ -59,8 +89,8 @@ charter in [`lanes/experiments.md`](lanes/experiments.md). | Retries | None | The image digest above was resolved from the Docker registry for -`linux/amd64` on 2026-07-30. Phase 1 must independently pull and inspect the -same digest on the final host before it is accepted. +`linux/amd64` on 2026-07-30 and inspected through Docker Desktop during the +accepted Windows qualification. ## Workspace construction @@ -81,32 +111,49 @@ maximum depth: 100 seed: 20260712 ``` -The current benchmark schema applies the selected base-workspace profile -directly to command and workspace-readiness cells. File-operation cells create -their deterministic target files during untimed setup. Before protocol lock, -we must either: - -1. extend file-operation cells to mount the same `paper-100m` base; or -2. state in the paper that the 100 MiB base applies only to startup/command - cells and that file-operation rows use operation-specific clean fixtures. - -No final run may proceed while this boundary is ambiguous. +The selected `paper-100m` base applies to every final measured cell, including +file read, write, and edit. Operation-specific targets are created during +untimed setup inside a fresh per-cell copy of that base. The current benchmark +schema applies the selected profile to command, workspace-readiness, read, +write, and edit cells, as required by the completed Phase 3 instrumentation. +Neither the product checkout, paper checkout, canonical repository-backed +qualification workspace, nor a clean empty fixture is a substitute for the +performance base. ## Measured operations and boundaries | Operation | Timed boundary | Untimed work | |---|---|---| -| Workspace/session create | Concurrent internal no-op session requests until ready, against a prepared sandbox and base fixture | Base generation, sandbox creation, verification, teardown | -| `exec_command` | One public command request against a prepared explicit session | Sandbox/session setup, result verification, teardown | -| File read | One public `file_read` request over a prepared snapshot target | Target generation and content verification | -| File write | One public `file_write` request in a fresh prepared session | Target/session setup, read-back verification, teardown | -| File edit | One public `file_edit` request in a fresh prepared session | Target/session setup, read-back verification, teardown | - -Initial sandbox creation and mounting are not currently a first-class measured -operation: the runner invokes `create_sandbox` during setup and discards its -timed response. If the paper reports "sandbox create + mount" latency, Phase 3 -must add an explicit metric with request-to-ready timing. Session-create -latency must not be relabeled as sandbox-create latency. +| Sandbox create + base mount | One native manager-CLI subprocess from process launch through successful validated ready response | Base generation/copy, gateway start, verification, teardown | +| Workspace/session create | Concurrent native runtime-CLI `create_workspace_session` subprocesses against a prepared sandbox and `paper-100m` base | Base generation/copy, sandbox creation, verification, teardown | +| `exec_command` | One native runtime-CLI subprocess against a prepared explicit session | Sandbox/session setup, result verification, teardown | +| File read | One native runtime-CLI `file_read` subprocess over a prepared target in the `paper-100m` base | Target generation and content verification | +| File write | One native runtime-CLI `file_write` subprocess in a fresh prepared session over the `paper-100m` base | Target/session setup, read-back verification, teardown | +| File edit | One native runtime-CLI `file_edit` subprocess in a fresh prepared session over the `paper-100m` base | Target/session setup, read-back verification, teardown | + +The focused campaign includes explicit manager-CLI sandbox-create-to-ready +timing. The runner records the dedicated `create_sandbox` process-launch through +validated-ready boundary separately from untimed setup. Session-create latency +remains a distinct runtime-CLI operation and must never be relabeled as +sandbox-create latency. + +### CLI timing contract + +- A measured operation begins immediately before the native Windows CLI + subprocess is created and ends only after it exits and its stdout and stderr + have been captured, parsed, and validated. +- Primary latency therefore includes executable launch, argument parsing, + CLI-to-gateway transport, gateway execution, response serialization, and + subprocess exit. This is the user-visible cost of the required public CLI + boundary. +- Product-reported internal command or gateway durations may be archived as + separate secondary observations when available. They must not replace, + subtract from, or be mixed with the primary end-to-end CLI latency. +- Concurrent batches release their CLI subprocesses from one benchmark barrier + and measure makespan until every subprocess has produced a valid result. +- Setup, correctness verification, observability sampling, and teardown also + use released product CLIs where they are sandbox operations, but remain + outside the measured operation interval. ## Draft good-pass matrix @@ -115,12 +162,13 @@ The executable draft is | Operation | Cases | Concurrency | Measured cells | |---|---|---:|---:| +| Sandbox create + base mount | `paper-100m`, shared network | 1 | 1 | | Workspace/session create | `paper-100m`, shared network | 1, 5 | 2 | | `exec_command` | no-op, 4 KiB fixture read | 1, 5 | 4 | | File read | 4 KiB, 256 KiB | 1, 5 | 4 | | File write | 4 KiB, 256 KiB, session-local | 1, 5 | 4 | | File edit | 4 KiB, 256 KiB, one exact replacement | 1, 5 | 4 | -| **Total** | | | **18** | +| **Total** | | | **19** | Each cell has 2 warmups and 100 measured trials. This sample count permits a reported empirical p99 without silently treating a smaller sample's maximum as @@ -130,7 +178,8 @@ a stable tail estimate. ### Primary -- client-observed batch makespan latency in milliseconds; +- end-to-end native product-CLI subprocess latency and concurrent-batch + makespan in milliseconds; - p50, p95, and p99 over the 100 measured trials; - throughput in completed operation requests per second. @@ -146,8 +195,9 @@ a stable tail estimate. ### Metric definitions -- A trial's latency is its timed operation boundary, excluding setup, - verification, and teardown. +- A trial's latency follows the CLI timing contract above. It includes native + CLI process launch and exit while excluding separate setup, verification, and + teardown operations. - For a concurrent batch, throughput is `completed_requests / batch_makespan_seconds`. - Warmups never contribute to aggregates. @@ -184,15 +234,34 @@ The targets below are acceptance budgets, not measured claims: | Warm, network-free environment preflight | <= 60 seconds | | Minimal live environment smoke | <= 3 minutes | | Five-sample exploratory pilot | <= 5 minutes | -| Complete good pass | <= 20 minutes | +| Complete good pass | <= 1,400 seconds (23 minutes 20 seconds) | | Deterministic analysis and table generation | <= 2 minutes | -If the good pass exceeds 20 minutes during the pilot, reduce the matrix before -protocol lock. Do not reduce trials or remove slow cells after seeing final -results. +The original v1.0 pre-lock option to reduce the matrix no longer applies. V1.1 +retains the locked 19-cell matrix unchanged. If the conservative pilot +projection exceeds 1,400 seconds, the final remains blocked; do not reduce +trials or remove slow cells after seeing v1.1 pilot or final evidence without a +new, explicitly authorized protocol amendment. ## Phases and acceptance tracker +### Environment decision + +The native Windows workstation with Docker Desktop is the selected and +qualified environment. Windows launches the released gateway and CLIs; Docker +Desktop supplies the Linux engine and runs the pinned Ubuntu sandbox image. +The exact qualification command and evidence are fixed in +[`experiments/environment_setup.md`](experiments/environment_setup.md). + +Every sandbox lifecycle, command, file, and observation operation used for +qualification or a future experiment must execute through +`sandbox-manager-cli`, `sandbox-runtime-cli`, or +`sandbox-observability-cli`. The imported benchmark currently implements only +`direct_client`; its performance presets are therefore prohibited until a +product-CLI subprocess cohort is implemented and reviewed. The complete +execution contract is +[`plan/task-packets/exp1-cli-performance-campaign.md`](plan/task-packets/exp1-cli-performance-campaign.md). + ### Phase 0 - Reproducibility package - [x] Create the paper-local benchmark snapshot. @@ -201,91 +270,130 @@ results. - [x] Define the environment, table schemas, and append-only log. - [x] Create a network-free fast preflight script. - [x] Create minimal-smoke and good-pass presets. -- [ ] Review and approve this draft protocol. +- [x] Review and approve this protocol. **Gate 0:** all files exist and cross-links/configuration validate. ### Phase 1 - Verify the final environment first -- [ ] Provision the single Ubuntu 24.04 x86-64 host. -- [ ] Place prebuilt product binaries; do not build on the measurement host. -- [ ] Pull the pinned image before the measurement window. -- [ ] Run `experiments/scripts/verify_environment.sh`. -- [ ] Confirm ext4, cgroup v2, Docker server, CPU/RAM/free-space thresholds. -- [ ] Confirm clean product `main` and record the exact commit. -- [ ] Confirm binary, daemon, toolchain-archive, and image digests. -- [ ] Archive the preflight output in the run directory. -- [ ] Complete the preflight in 60 seconds or less with a warm local image. +- [x] Confirm native Windows x64 host identity and build. +- [x] Stage the official `v0.1.4` Windows AMD64 package under ignored `target/`. +- [x] Confirm NTFS, CPU/RAM/free-space thresholds. +- [x] Confirm Docker Desktop 29.0.1, Linux AMD64, `overlayfs`, and cgroup v2. +- [x] Confirm the pinned Ubuntu image is present locally. +- [x] Confirm clean product `main` and the exact commit. +- [x] Confirm archive, gateway, CLI, config, daemon, and image digests. +- [x] Archive the complete native Windows preflight output. **Gate 1:** every environment acceptance item passes. Any failure stops work; do not compensate by building, installing, or changing the host mid-run. ### Phase 2 - Minimal live smoke -- [ ] Run the `paper-env-smoke` preset. -- [ ] Confirm one sandbox/session lifecycle completes. -- [ ] Confirm one `exec_command` completes and verifies. -- [ ] Confirm cleanup leaves no owned sandbox, process, or runtime residue. -- [ ] Record elapsed time and keep the smoke below 3 minutes. -- [ ] Mark all smoke samples exploratory and ineligible for paper tables. +- [x] Run the native Windows CLI-only environment smoke. +- [x] Confirm two independent sandbox lifecycles complete. +- [x] Confirm command and file-operation correctness in both batches. +- [x] Confirm cleanup leaves no qualifier-owned container, volume, or process. +- [x] Record elapsed time and keep the smoke below 3 minutes. +- [x] Mark the elapsed time as environment evidence, ineligible for paper tables. + +**Gate 2:** both product-CLI-controlled lifecycles complete, all 20 expected +CLI calls pass strict response checks, and the unique gateway instance leaves +no containers, volumes, processes, or runtime state. + +The earlier 2026-07-30 WSL smoke used the now-disallowed `direct_client` +cohort. It remains diagnostic implementation evidence but is not +Gate 2-equivalent under the product-CLI-only contract. -**Gate 2:** live create, execute, observe, and cleanup succeed once. +A later WSL/Docker Desktop diagnostic executed the replacement product-CLI +smoke successfully for two lifecycles. Because WSL is ineligible, this validates +the automation only and still does not satisfy Gate 1 or Gate 2. + +The accepted native Windows run is +`qualification-windows-docker-20260730-final-6`. It satisfies Gate 1 and Gate 2 +with the official Windows release gateway and CLIs controlling Docker Desktop. ### Phase 3 - Instrumentation and five-sample pilot -- [ ] Decide and implement the file-operation base-workspace boundary. -- [ ] Add explicit sandbox-create + mount timing if Table 2 will report it. -- [ ] Confirm p99, throughput, and resource fields are emitted by analysis. -- [ ] Run a five-sample pilot over every final cell. -- [ ] Confirm operation setup is excluded from operation latency. -- [ ] Confirm all correctness checks and cleanup gates pass. -- [ ] Confirm raw observations deterministically regenerate draft tables. -- [ ] Confirm the projected good-pass duration is <= 20 minutes. +- [x] Implement and review the `product_cli` subprocess cohort for manager, + runtime, and observability operations; make paper presets reject + `direct_client`. +- [x] Apply `paper-100m` to command, lifecycle, read, write, and edit cells. +- [x] Add explicit manager-CLI sandbox-create + base-mount timing for Table 2. +- [x] Enforce the end-to-end CLI timing contract and retain any internal + product timing only as separate secondary evidence. +- [x] Confirm p99, throughput, and resource fields are emitted by analysis. +- [x] Run a five-sample pilot over all 19 final cells with the final two + warmups retained. +- [x] Confirm operation setup is excluded from operation latency. +- [x] Confirm all correctness checks and cleanup gates pass. +- [x] Confirm raw observations deterministically regenerate draft tables. +- [x] Confirm the projected good-pass duration is <= 1,400 seconds. **Gate 3:** measurement boundaries, outputs, and runtime are demonstrated. ### Phase 4 - Protocol lock and freeze -- [ ] Resolve every Gate 0-3 item. -- [ ] Freeze the product commit/tag and record a clean tree. -- [ ] Freeze the paper-local benchmark commit and plan hash. -- [ ] Freeze image and binary digests. -- [ ] Freeze table columns, metric definitions, seed, trials, and exclusions. -- [ ] Record protocol version `v1.0` in the log. - -**Gate 4:** no scientific decision remains conditional or ambiguous. +- [x] Resolve every Gate 0-3 item. +- [x] Freeze the product commit/tag and record a clean tree. +- [x] Freeze the paper-local benchmark commit and plan hash. +- [x] Freeze image and binary digests. +- [x] Freeze table columns, metric definitions, seed, trials, and exclusions. +- [x] Record active protocol version `v1.1` and preserve historical v1.0. + +**Gate 4: PASS.** Measurement source was frozen at paper commit +`1680b599129532f72e706b6acb12ef62c63759e2`, product commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, and annotated tag object +`834c84534359f37653fb25ac45304091e82c37a6`. No scientific decision remained +conditional. The later canonical-host reader correction is a separately +identified post-freeze analysis erratum and did not alter the measurement +freeze. ### Phase 5 - Good pass -- [ ] Re-run the fast environment preflight. -- [ ] Confirm no build, image pull, package install, or source mutation occurs. -- [ ] Run `paper-good-pass` once with the frozen plan. -- [ ] Preserve run manifest, raw observations, traces, resources, and logs. -- [ ] Preserve failed or partial evidence if the run does not complete. -- [ ] Confirm all 18 cells have 100 reportable measured trials. - -**Gate 5:** one complete, provenance-rich measured corpus exists. +- [x] Re-run the fast environment preflight. +- [x] Confirm no build, image pull, package install, or source mutation occurs. +- [x] Run `paper-good-pass` once with the frozen plan. +- [x] Preserve run manifest, raw observations, traces, resources, and logs. +- [x] Preserve failed or partial evidence if the run does not complete. +- [x] Confirm all 19 cells have 100 reportable measured trials. + +**Gate 5: PASS.** The sole v1.1 final +`019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` completed exactly 19 cells, 1,938 +batches, 38 warmups, 1,900 successful/reportable measured trials, and 5,610 +issued requests with zero classified failure or warning. The 3,139,214,747-byte +archive was independently verified at content tree +`sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. +The earlier v1.0 failed run remains immutable and ineligible. ### Phase 6 - Deterministic analysis and tables -- [ ] Validate the raw corpus before aggregation. -- [ ] Generate all four expected tables from archived data. -- [ ] Confirm every displayed value traces to a run/cell/metric selector. -- [ ] Confirm no verification/pass column appears in the CLI table. -- [ ] Confirm negative or unavailable resource fields are disclosed. -- [ ] Record analysis command, commit, and output hashes. +- [x] Validate the immutable v1.1 raw corpus before aggregation. +- [x] Generate all four expected tables from archived data twice. +- [x] Confirm every displayed value traces to a run/cell/metric selector. +- [x] Confirm no verification/pass column appears in the CLI table. +- [x] Confirm negative or unavailable resource fields are disclosed. +- [x] Record analysis command, frozen/corrected identities, and output hashes. -**Gate 6:** tables regenerate exactly and contain no manual numeric edits. +**Gate 6: PASS.** Two nine-file generations are byte-identical at output tree +`sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. +Numeric-evidence v2 contains 153 unique selector-bound values. A frozen-reader +schema mismatch was corrected only for canonical host metadata; the original +and corrected generator identities are both disclosed, Tables 2--4 and numeric +artifacts were proven unaffected, and the archive remained unchanged. ### Phase 7 - Paper handoff -- [ ] Map each table to the exact supported descriptive claim. -- [ ] Record wording that remains unsafe. -- [ ] Update the claim-evidence map with run and table identifiers. -- [ ] Add numeric-evidence selectors before inserting numbers into LaTeX. +- [x] Map every final result family to exact supported descriptive wording. +- [x] Record wording that remains unsafe. +- [x] Update the claim-evidence map with the final run and eligibility boundary. +- [x] Add numeric-evidence selectors before inserting numbers into LaTeX. - [ ] Obtain author review of environment, boundaries, and interpretation. -**Gate 7:** the paper uses only reproducible, scoped evidence. +**Gate 7: PASS.** The claim-mapped final handoff, supported/unsafe wording, +unavailable fields, exclusions, table hashes, and 153-row numeric provenance +are complete. Author review remains the next manuscript action, not an +experiment execution blocker; no number was manually inserted into LaTeX. ## Artifact layout diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-final-failure-diagnostic.json b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-failure-diagnostic.json new file mode 100644 index 0000000..3784a33 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-failure-diagnostic.json @@ -0,0 +1,156 @@ +{ + "schema_version": 1, + "diagnostic_id": "exp1-final-019fb6e5-tcp-endpoint-reuse", + "run": { + "run_id": "019fb6e5-c00b-7b02-8a3c-d76bd1346eb4", + "name": "paper-good-pass", + "disposition": "final", + "run_status": "failed", + "eligibility": "failed_ineligible", + "plan_hash": "sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b", + "started_at": "2026-07-31T06:39:01.112787Z", + "ended_at": "2026-07-31T06:50:13.449078Z", + "elapsed_seconds": "672.336291", + "completed_trial_batches": 853, + "total_trial_batches": 1938, + "issued_operation_requests": 2077, + "expected_operation_requests": 5610, + "failure_count": 1, + "warning_count": 0 + }, + "failed_trial": { + "cell_id": "sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16", + "family": "files", + "operation": "file_read", + "trial_id": "trial-242749a59d59cc16-measured-000034", + "trial_kind": "measured", + "sequence_in_cell": 34, + "status": "infrastructure_failed", + "product_succeeded": true, + "cleanup_baseline_restored": true, + "checks_passed": false, + "reportable": false, + "operation_latency_ns_ineligible": 13526000, + "resource_records": 14, + "resource_batches": 1, + "expected_boundary_batches": 2, + "failure_boundary": "mandatory post-response resource boundary" + }, + "root_cause": { + "classification": "windows_tcp_dynamic_endpoint_reuse_pressure", + "confidence": "high", + "summary": "A fresh observability CLI process could not connect to the isolated loopback gateway during the mandatory final resource sample. Windows returned WSAEADDRINUSE (10048), and the System TCP/IP provider emitted Event 4227 at the same instant, attributing the failure to high-rate outbound connection churn and reuse of a recently used local endpoint.", + "failed_cli_invocation": { + "relative_path": "raw/cli-subprocesses/de511434eb741316c53eba28b2e8586c402560f3d346b593c3f37c36e4547311.json", + "sha256": "sha256:3e188c6e284c0f65e2a909df854ac18a5f550ea1daffb86f2eb3d02ebc26ab0f", + "request_id": "trial-242749a59d59cc16-measured-000034.observe.snapshot.1.boundary.0", + "executable_role": "observability", + "operation": "snapshot", + "return_code": 1, + "response_validation": "transport_error:gateway transport failed (connection_error)", + "stderr_bytes": 187, + "stderr_sha256": "sha256:b61e0c2f370d165f3aee73407c216ca82db0bed41112c5462f1257f9888bab5c", + "winsock_error": { + "symbol": "WSAEADDRINUSE", + "code": 10048 + } + }, + "system_event": { + "log": "System", + "provider": "Tcpip", + "event_id": 4227, + "record_id": 88385, + "level": "Warning", + "created_at_utc": "2026-07-31T06:50:09.9652511Z", + "message_sha256": "sha256:535f882dfc5e90d98a9ffb429ef7a2c3d5eeb45d71892b54eb0af2bb1bc548dc", + "primary_capture_path": "experiments/analysis/exp1-final-system-event-4227.json", + "primary_capture_sha256": "sha256:b6eac476b6ecf8c20de529be5c5ca8de297874ae9273c65a4baaf6ffc34ac89d", + "capture_timing": "post-run read-only retrieval from the Windows System event log; the immutable run archive was not modified", + "meaning": "TCP/IP could not establish an outbound connection because the selected local endpoint had recently been used for the same remote endpoint; the event identifies high-rate open/close churn as the typical cause." + }, + "connection_pressure": { + "ipv4_dynamic_tcp_start_port": 49152, + "ipv4_dynamic_tcp_port_count": 16384, + "ipv6_dynamic_tcp_start_port": 49152, + "ipv6_dynamic_tcp_port_count": 16384, + "persisted_cli_invocations_before_stop": 7992, + "persisted_cli_invocations_in_preceding_240_seconds": 6760, + "roles": { + "manager": 328, + "runtime": 3829, + "observability": 3835 + }, + "return_code_zero": 7991, + "return_code_nonzero": 1 + }, + "boundary_evidence": { + "preceding_measured_trials_27_through_33": "Each retained two complete 14-metric resource batches and passed.", + "failed_trial": "Only the first 14-metric boundary batch was durably recorded. The product file-read request succeeded, then the final snapshot CLI failed to connect before verification.", + "event_alignment": "The TCP/IP Event 4227 timestamp aligns with the failed trial event at approximately 668.853 seconds after run start." + }, + "uncertainty": "The evidence establishes endpoint-reuse pressure and the exact failed connection. It does not prove that all 16,384 dynamic ports were simultaneously occupied, nor does it attribute the condition to a single internal socket beyond the recorded observability CLI connection.", + "official_references": [ + "https://learn.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2", + "https://learn.microsoft.com/en-us/troubleshoot/windows-client/networking/tcp-ip-port-exhaustion-troubleshooting" + ] + }, + "evidence_integrity": { + "committed_cli_invocation_records": 7992, + "committed_cli_invocation_records_passing": 7991, + "committed_cli_invocation_records_failing": 1, + "raw_cli_projection_basenames": 7993, + "incomplete_projection_group": { + "basename": "766ed434998f2dc7c002bac6dd08c3d642cf1ec416eb2d2cc7df65a6906471a0", + "stdout_path": "raw/cli-subprocesses/766ed434998f2dc7c002bac6dd08c3d642cf1ec416eb2d2cc7df65a6906471a0.stdout", + "stdout_bytes": 20165, + "stdout_sha256": "sha256:0c53f4064938e550a540f0b7004f3ee0f131e097bfabb39a3874e34ff03e4c5b", + "stdout_is_valid_cgroup_json": true, + "stderr_path": "raw/cli-subprocesses/766ed434998f2dc7c002bac6dd08c3d642cf1ec416eb2d2cc7df65a6906471a0.stderr", + "stderr_bytes": 0, + "stderr_sha256": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "metadata_commit_marker_present": false + }, + "interpretation": "The orphan stdout/stderr pair is consistent with the concurrent cgroup sibling finishing while the snapshot exception unwound the resource-boundary gather, but the exact persistence race is not logged and is not claimed as proven. The defect is secondary to the directly recorded snapshot connection failure." + }, + "protocol_decision": { + "gate_5": "fail", + "gate_6": "fail_not_authorized_for_aggregation", + "gate_7": "fail_no_numeric_claim_handoff", + "rerun": "prohibited_under_frozen_v1.0", + "reason": "The protocol allows exactly one eligible frozen final attempt and requires a stop for a missing resource boundary or measured cell with fewer than 100 reportable trials.", + "required_new_decision": "Any new attempt requires a documented protocol amendment, a new source/environment freeze, and explicit author authorization. It cannot replace or overwrite this failed archive." + }, + "cleanup": { + "initial_archive_attempt": "failed because the owned run workspace still existed", + "explicit_cleanup": { + "exit_code": 0, + "cleaned": true, + "terminalized": false, + "run_workspace_exists_after": false + }, + "final_proof": { + "runtime_exists": false, + "matching_product_processes": [], + "run_labeled_containers": [], + "gateway_labeled_containers": [], + "run_labeled_volumes": [], + "gateway_labeled_volumes": [], + "product_branch": "main", + "product_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "product_status_porcelain": "" + } + }, + "archive": { + "path": "experiments/runs/019fb6e5-c00b-7b02-8a3c-d76bd1346eb4", + "verified": true, + "file_count": 24867, + "bytes": 620311242, + "content_tree_sha256": "sha256:7efa643b12aba09f0ba5ecfbed5b5692a166a5c12931490402d3992d92f3ae6a", + "archive_manifest_sha256": "sha256:5e0a3c4f7c864df8070a668d2f373b75bece3c2a57cc4340cd89ece292cc7927", + "campaign_manifest_sha256": "sha256:8eefbec9772406943bb1baa2476b181c7436a6fafd7a6d7984874e8889f96982", + "raw_file_count": 24841, + "raw_bytes": 547911783, + "raw_content_tree_sha256": "sha256:1cc85e7883136ede15e342aa6f2ac50d72bdf6d4eace340dc0e6dba9e992f5b5", + "independent_verify_output_sha256": "sha256:13b4b44f9a6f2e8835fd85b81d73de0bba7a65ac0eb7c110a57ad21168f43630" + } +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-final-handoff.md b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-handoff.md new file mode 100644 index 0000000..6d2f435 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-handoff.md @@ -0,0 +1,137 @@ +# EXP1 v1.1 final evidence handoff + +Status date: 2026-07-31. + +## Outcome + +EXP1 v1.1 completed Gates 0--7. The sole eligible final run +`019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` contains exactly 19 cells, 1,938 +batches, 38 warmups, 1,900 successful/reportable measured trials, and 5,610 +issued product requests. Correctness passed, report warnings are empty, and +all product, correctness, infrastructure, cleanup, and missing-latency failure +counts are zero. + +The immutable archive is: + +`C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` + +It contains 82,051 files and 3,139,214,747 bytes with content tree +`sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. +Its raw subtree is bound by +`sha256:561dd3bd8ac1a7106fcf970acdcd6972a76da24fa07da147e4f19d49c83f3981`. + +The earlier v1.0 final remains `failed_ineligible`; no v1.0, qualifier, smoke, +pilot, projection, or partial-final value is included in this handoff. + +## Analysis identity and erratum + +The measurement freeze records table generator +`sha256:7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7`. +Its first post-final invocation failed before output because it did not accept +the archive's canonical `os_caption` and `os_build_number` keys. + +Post-freeze analysis-only correction commit +`538f6c98233863957082620329203348ddaa781c` has generator hash +`sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b`. +The correction changes only Table 1 host-field compatibility. Against the +frozen pilot archive, numeric evidence, numeric provenance, and Tables 2--4 +remain byte-identical. The final archive was not changed and the final was not +rerun. + +## Deterministic outputs + +Primary output: + +`C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\analysis\final-v11-019fb86c-tables-a` + +Independent repeat: + +`C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\analysis\final-v11-019fb86c-tables-b` + +Both directories contain the same nine files and 231,047 bytes. Every path, +byte count, and SHA-256 matches. Output tree: +`sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. + +Use these artifacts: + +- `table-1-environment.md` for the tested environment and workload; +- `table-2-startup.md` for sandbox/workspace startup and first no-op; +- `table-3-cli-operations.md` for the 16 public CLI rows; +- `table-4-resources.md` for the seven preregistered resource rows; +- `numeric-provenance.csv` for the 153 raw source/hash/selector/aggregate + bindings; +- `numeric-evidence.json` as the + `ai-research-writing/numeric-evidence-v2` import registry; +- `output-manifest.json` for generator, archive, protocol, and output hashes. + +## RQ3 claim mapping + +| Result family | Supported wording | Direct evidence | Boundary | +|---|---|---|---| +| Environment and treatment | "The result uses native Windows build 26200, Docker Desktop 29.0.1, a pinned Ubuntu 24.04 image, `paper-100m`, and local Windows named pipes rotated per execution block." | Table 1, campaign/run/environment manifests. | One disclosed host and treatment only. | +| Startup | "In the tested environment, sandbox create plus base mount had p50/p95/p99 1,659.811/1,749.739/1,794.902 ms." | Table 2 and `table2.create_sandbox.none.c1.*` numeric selectors. | Not a cold-machine, remote, or cross-platform result. | +| Workspace/first command | "Session create and first no-op values are those in Table 2 for concurrency 1 or 5." | Table 2 and matching numeric selectors. | Session create is distinct from sandbox create. | +| CLI latency | "For each fixed operation/payload/concurrency row, end-to-end native CLI p50/p95/p99 is the value in Table 3." | Table 3; `batch_makespan_ns` selectors over 100 reportable trials. | Includes native process launch, CLI/gateway transport, validation, and process exit. | +| CLI throughput | "Mean completed-operation throughput changed between concurrency 1 and 5 as shown in Table 3." | Table 3; `throughput_ops_s` selectors. | Describe change; do not claim linear scaling or improved per-request latency. | +| Resources | "Peak/mean resource observations for the seven preregistered rows are those in Table 4." | Table 4 and matching daemon/cgroup/upperdir selectors. | Sampled/available metrics only; unavailable values are not zero. | +| Completeness | "The sole v1.1 final completed all 1,900 measured trials with zero classified failures." | Report counts, campaign manifest, archive verifier. | Does not establish broader reliability. | + +Every displayed table value has a unique numeric-provenance row. No table +contains a verification/pass column, and no number was manually copied into +LaTeX during this campaign. + +## Supported representative sentences + +- "Sandbox create plus base mount had p50/p95/p99 latency of + 1,659.811/1,749.739/1,794.902 ms in the tested environment." +- "No-op native CLI throughput changed from 37.28 operations/s at concurrency + 1 to 107.61 operations/s at concurrency 5; p50 batch makespan changed from + 26.719 to 45.998 ms." +- "At concurrency 5, the 256-KiB file-edit row had p50/p95/p99 + 187.357/208.008/294.392 ms and mean throughput 26.36 operations/s." +- "For the preregistered 256-KiB/concurrency-5 resource rows, peak measured + sandbox memory was 14.977 MiB for read, 54.270 MiB for write, and 66.715 MiB + for edit." + +## Unsafe wording + +- "Ephemeral Sandbox is fast/scalable/cheap" without the tested-environment + qualifier and exact workload. +- Any superiority or competitive-baseline claim. +- Any claim beyond concurrency 1 and 5, the fixed payloads, `paper-100m`, the + pinned image, this host, or this source identity. +- Any causal interpretation that concurrency improves individual latency. +- Any performance use of v1.0, qualifier, smoke, pilot, projection, setup, + verification, teardown, partial, or unavailable evidence. +- Any security, isolation, publication correctness, fault tolerance, + multi-agent quality, or productivity conclusion from EXP1. + +## Unavailable and excluded evidence + +Final failures, report warnings, and excluded final trials: none. + +Explicitly unavailable at the reportable-trial level: + +- the frozen OS caption's localized edition suffix is represented literally + as `???`; Table 1 preserves that archived string. A later read-only registry + query found `EditionID=Core`, but it is external context and is not inserted + into the archive-derived table; +- LayerStack allocated storage: 1,900/1,900; +- host workspace allocated blocks: 1,900/1,900; +- create-sandbox daemon/sandbox CPU and block-I/O deltas: 100/1,900 each, + because no sandbox-scoped pre-create baseline can exist; +- sandbox current and peak memory before the resource ring exists: 50/1,900 + each. + +These values remain unavailable. Table 4's workspace column is the separately +defined, available `upperdir_bytes` delta. + +## Cleanup and next action + +The final workspace/runtime are absent, no owned process/container/volume +remains, product `main` is clean at `5c48dae1`, and protected gateway PID 62980 +is alive and untouched. Nothing was pushed. + +There is no remaining EXP1 blocker. The next external action is author review +of the generated tables and bounded sentences before evidence-backed values +are imported into `main.tex`. diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-final-system-event-4227.json b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-system-event-4227.json new file mode 100644 index 0000000..fcb94d9 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-final-system-event-4227.json @@ -0,0 +1,33 @@ +{ + "schema_version": 1, + "artifact_kind": "post_run_primary_system_event_capture", + "capture": { + "captured_at_utc": "2026-07-31T07:16:34.8242109Z", + "source": "Windows System event log", + "retrieval": "Get-WinEvent filtered to System/Tcpip event 4227 from 2026-07-31T06:50:00Z through 2026-07-31T06:50:20Z, then exact RecordId 88385", + "read_only": true, + "immutable_run_archive_modified": false + }, + "event": { + "id": 4227, + "record_id": 88385, + "provider_name": "Tcpip", + "log_name": "System", + "level": 3, + "level_display_name": "Warning", + "time_created_utc": "2026-07-31T06:50:09.9652511Z", + "machine_name": "DESKTOP-OLP1ADS", + "message": "TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.", + "message_sha256": "sha256:535f882dfc5e90d98a9ffb429ef7a2c3d5eeb45d71892b54eb0af2bb1bc548dc", + "xml": "422703000x8000000000000088385SystemDESKTOP-OLP1ADS00000000010000000000000083100080000000000000000000000000000000000000000000000000", + "xml_sha256": "sha256:864d02daffa7ea1c6030f520ccbe7983279229ea8f6ff0e2a5d468bc0a134c34" + }, + "interpretation_boundary": { + "directly_supported": "The host TCP/IP provider recorded an outgoing connection failure at the final-run failure instant because the selected local endpoint had recently been used for the same remote endpoint; the message identifies high-rate connection open/close churn as the typical cause.", + "not_proven": [ + "All 16384 dynamic ports were simultaneously occupied.", + "A universal Windows connection ceiling exists at the observed invocation count.", + "The event alone identifies every socket or process contributing to endpoint pressure." + ] + } +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-freeze-record-eb10c26-0392b299.json b/ephemeral-sandbox-v1/experiments/analysis/exp1-freeze-record-eb10c26-0392b299.json new file mode 100644 index 0000000..eb0e655 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-freeze-record-eb10c26-0392b299.json @@ -0,0 +1,100 @@ +{ + "schema_version": 1, + "created_at": "2026-07-31T06:34:57.974Z", + "protocol": { + "name": "EXP1 CLI-only focused performance campaign", + "version": "v1.0", + "campaign_file": { + "path": "plan/task-packets/exp1-cli-performance-campaign.md", + "sha256": "sha256:c021ecdd47cc80b1601171c7c175ed4f42d12dfe51229aa4c36b603b8d0a18ce" + }, + "expected_tables": { + "path": "experiments/expected_tables.md", + "sha256": "sha256:7c1ae3d78a9ca7a5b6cbb4a9cc80bac555d676c5833a40d2846597884f8f50da" + }, + "metrics_exclusions_and_stopping_rules": "Frozen by the two files above; no retries, no silent outlier removal, and only one eligible final attempt.", + "seed": 20260712 + }, + "paper_source": { + "repository_commit": "eb10c26d1bfd632772baf1bc331c985d0231f52d", + "branch": "agent/complete-pw3-and-final-host-prep", + "frozen_scope_clean": true, + "benchmark_source": { + "file_count": 207, + "bytes": 16290860, + "content_tree_sha256": "sha256:1efeff548dd664580dcb452829d86e1ae114477828a1af65589b7e34cc311b67" + } + }, + "product": { + "repository_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "branch": "main", + "clean": true, + "freeze_tag": { + "name": "paper-v1-freeze", + "object_type": "tag", + "tag_object": "0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d", + "peeled_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3" + }, + "baseline_release": { + "name": "v0.1.4", + "tag_object": "981b99fec4df576bb4326b3c1a145be4471766ce", + "peeled_commit": "b22862550e0a7cb4fe61ce581831e9244cc492b5" + }, + "package": { + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299.zip", + "bytes": 5685130, + "sha256": "sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e" + }, + "binaries": { + "sandbox-gateway.exe": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285", + "sandbox-manager-cli.exe": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed", + "sandbox-runtime-cli.exe": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8", + "sandbox-observability-cli.exe": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26", + "ephemeral-sandbox-daemon": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a" + } + }, + "image": { + "reference": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + }, + "fixture": { + "fixture_hash": "sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7", + "tree_hash": "sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f", + "pilot_manifest_sha256": "sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1" + }, + "final_plan": { + "preset": "paper-good-pass", + "preset_sha256": "sha256:4d7ea2903e7c6e6a2a0d28acec44d5427b3c42008227b2273958b0ad25870ce9", + "plan_hash": "sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b", + "reviewed_expansion_sha256": "sha256:4d2389e1782a5075bf4130b3a158b55cc39cc697a1f53593f8c93ff2388e7c92", + "client_cohort": "product_cli", + "workspace_profile": "paper-100m", + "cell_count": 19, + "warmups_per_cell": 2, + "measured_trials_per_cell": 100, + "trial_batch_count": 1938, + "issued_operation_request_count": 5610 + }, + "analysis": { + "archive_script": { + "path": "experiments/scripts/archive_exp1_run.py", + "sha256": "sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e" + }, + "table_generator": { + "path": "experiments/analysis/scripts/generate_exp1_tables.py", + "sha256": "sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3" + }, + "runtime_projection": { + "path": "experiments/scripts/project_exp1_final_runtime.py", + "sha256": "sha256:444eafef866c9d8be5a00f63a80eeb3727b6dfddd0faa0128d2f54deb161319f" + }, + "gate3_projection": { + "path": "experiments/analysis/pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json", + "sha256": "sha256:8b02470b98a8ff2374d140398e6a7bb00abb469fa96810db4a680ab00079a4a8", + "central_seconds": "1170.818322650", + "envelope_seconds": "1307.100411100", + "limit_seconds": "1400.000000000", + "decision": "pass_runtime_projection" + } + }, + "gate4_decision": "pass_protocol_and_source_freeze" +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-gate0-7-final-report.md b/ephemeral-sandbox-v1/experiments/analysis/exp1-gate0-7-final-report.md new file mode 100644 index 0000000..b887f62 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-gate0-7-final-report.md @@ -0,0 +1,382 @@ +# EXP1 CLI-only performance campaign: v1.1 final Gate 0--7 report + +Report date: 2026-07-31. + +Protocol: `ephemeral-sandbox-v1-practical-performance-v1.1`. + +Outcome: **EXP1 is complete. Gates 0--7 pass.** The sole eligible v1.1 final +completed, its immutable archive was independently verified, and all four +tables plus numeric-evidence v2 provenance regenerate byte-identically. This +closes the focused RQ3 experiment; it does not by itself make the full paper +submission-ready. + +The earlier v1.0 final remains permanently failed and ineligible. Its corpus +was neither replaced, pooled, nor numerically compared with v1.1. + +## Gate decisions + +| Gate | Phase | Decision | Evidence | +|---|---|---:|---| +| Gate 0 | Audit and protocol review | PASS | The fixed environment, CLI-only ownership, timing boundaries, workload, correctness, resource policy, table schemas, exclusions, and stop conditions were cross-checked before live work. | +| Gate 1 | Native-Windows `product_cli` and permanent local IPC | PASS | Product `main` at `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` implements no-fallback local named-pipe transport for the paper treatment. The preregistered qualifier completed 25,000/25,000 native CLI calls with no owned TCP endpoint, new TCP/IP 4227/4231 event, transport failure, or resource-bound violation. | +| Gate 2 | Fresh v1.1 CLI integration smoke | PASS | Run `019fb83a-54bc-79db-b6ac-6189fb28f5f2` completed all 19 cells and 55 requests through four local named-pipe execution blocks, with correctness, stderr, warning, cleanup, archive, and protected-process gates passing. | +| Gate 3 | Five-sample pilot and runtime projection | PASS | Run `019fb84e-aef1-7fdc-9a56-1adbe712f30d` completed 19 cells, 133 batches, 95 measured trials, and 385 requests with zero failures. Repeated projection outputs were byte-identical; the 1,303.732241600-second envelope was below the fixed 1,400-second limit. Pilot values remain exploratory/ineligible. | +| Gate 4 | Measurement protocol/source/treatment freeze | PASS | Paper measurement commit `1680b599129532f72e706b6acb12ef62c63759e2`, product commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, annotated tag object `834c84534359f37653fb25ac45304091e82c37a6`, plan, package, image, fixture, metrics, exclusions, seed, trials, and analysis identities were frozen before the final. The later Table-1 reader compatibility erratum is separately identified and does not claim that the frozen generator passed unchanged. | +| Gate 5 | Sole eligible v1.1 final | PASS | Run `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` completed exactly once: 19 cells, 1,938 batches, 38 warmups, 1,900 successful/reportable measured trials, 5,610 product requests, 4,800 correctness checks, zero failures, zero warnings, and clean teardown. | +| Gate 6 | Deterministic final aggregation | PASS | The immutable 3,139,214,747-byte archive reverified at its original tree hash. Two corrected-generator builds produced nine files/231,047 bytes each and matched byte-for-byte at output tree `sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. Numeric-evidence v2 contains 153 unique selector-bound values. | +| Gate 7 | Numeric evidence and RQ3 handoff | PASS | All four archive-derived tables, 153-row numeric provenance, supported wording, unsafe wording, unavailable metrics, exclusions, and claim boundaries are recorded. No result number was manually entered into LaTeX. | + +## Frozen measurement identities + +- Paper measurement commit: + `1680b599129532f72e706b6acb12ef62c63759e2`. +- Benchmark-source inventory: 213 files, 16,451,598 bytes, + `sha256:c060e397ce3511a7839c71e13506dd4db99c9ad774464d0a0555f6949319dabd`. +- Product commit: `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, + clean direct local `main`. +- Annotated product tag: `paper-v1.1-freeze`; tag object + `834c84534359f37653fb25ac45304091e82c37a6`; peeled commit + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`. +- Preserved v1.0 tag object: `paper-v1-freeze`, + `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d`. +- Windows package: 5,739,735 bytes, + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`. +- Final plan: + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`; + 19 cells, two warmups and 100 measured trials per cell, 1,938 batches, + 5,610 product requests, `product_cli`, `paper-100m`, seed 20260712. +- Freeze record: + `experiments/analysis/exp1-v11-freeze-record-1680b59-5c48dae1.json`; + 6,888 bytes; + `sha256:5b8ca3962f479f1776be0298acbbe7620b683a334c1964889b031122a0ffdc32`. +- Frozen table generator: + `sha256:7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7`. +- No commit or tag was pushed. + +## Sole final result and archive + +Run `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` started at +`2026-07-31T13:45:18.307991Z` and ended at +`2026-07-31T14:05:29.554460Z`; exact corpus elapsed time was 1,211.246469 +seconds. The benchmark and supervisor exited 0, and both stderr captures are +empty. + +The terminal report is non-provisional, correctness `pass`, and warning-free: + +- 19/19 cells; +- 1,938/1,938 attempted batches; +- 38 warmups and 1,900/1,900 successful reportable measured trials; +- 5,610 issued product requests; +- 4,800/4,800 correctness checks passed; +- zero product, correctness, infrastructure, cleanup, or primary-latency + failures; +- 26,692/26,692 native CLI invocations returned 0, passed response validation + and authentication redaction, used unique request IDs, and emitted zero + stderr bytes; +- four execution blocks, exclusively `windows_named_pipe`, `local_only`, and + `per_execution_block`; +- 76,276 retained observations: 1,938 trials, 1,938 operations, 5,610 + requests, 4,896 checks, and 61,894 resource observations. + +Absolute archive path: + +`C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` + +Archive identities: + +- 82,051 files; 3,139,214,747 bytes; +- content tree + `sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`; +- raw subtree: 82,025 files; 2,964,792,876 bytes; tree + `sha256:561dd3bd8ac1a7106fcf970acdcd6972a76da24fa07da147e4f19d49c83f3981`; +- archive manifest + `sha256:239dbedb781f2e427fb61b316629ea57393d1a92a3be56a45bd107e998d9131c`; +- campaign manifest + `sha256:93dd241e38c48b2a3f337d66065492f101d8945f7189c2f6402a32fa0fd7e7cf`; +- final report input + `sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982`. + +Archive creation and two verify-only replays exited 0. The post-analysis replay +reproduced the same file count, byte count, and content tree. + +## Post-freeze analysis compatibility erratum + +The first final table-generation attempt failed closed before creating an +output directory: + +`ERROR: final archive lacks required environment fields: host OS edition, host OS build` + +The archive and archiver use the canonical verified keys `os_caption` and +`os_build_number`; the frozen reader accepted only synthetic legacy aliases +`os_edition` and `os_build`. Both canonical values were already present in the +immutable archive. No raw file, selector, exclusion, metric, aggregate, +eligibility decision, or measured result was missing or changed. + +Local post-freeze correction commit +`538f6c98233863957082620329203348ddaa781c` accepts the canonical keys, retains +legacy aliases, avoids duplicating the generic OS family before a full caption, +and still fails closed when caption/edition or build evidence is absent. The +corrected generator is +`sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b`. + +This correction is disclosed as an analysis erratum, not as an unchanged +frozen generator. Its numeric neutrality was tested three ways: + +1. Targeted generator tests, including explicit v1.1 canonical-only host + metadata and missing-field negatives, pass 22/22. +2. The complete backend plus analysis suite passes 380 tests with five + expected Windows symlink-privilege skips. +3. Replaying the corrected generator on the frozen pilot archive produces + byte-identical `numeric-evidence.json`, `numeric-provenance.csv`, and Tables + 2--4 relative to the frozen generator. Only the intended non-numeric Table + 1 host presentation changes. + +A rerun would have violated the exactly-one-final rule, and archive mutation +would have violated corpus immutability; neither occurred. + +## Final tables and numeric provenance + +The two fresh output directories are: + +- `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\analysis\final-v11-019fb86c-tables-a` +- `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\analysis\final-v11-019fb86c-tables-b` + +Each contains nine files and 231,047 bytes. Path, byte-count, and SHA-256 +inventories are identical. The output content tree is +`sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. + +| Artifact | SHA-256 | +|---|---| +| `generation-log.txt` | `481f89a1900a051649102b986ac3ad6a7083b53959cfa4a9bcf419e383d3d0b3` | +| `numeric-evidence.json` | `8a247f492b5bbf5bb35f1a89e0ed7dc3effcaaab0f935b746e1dcd89d62a367c` | +| `numeric-provenance.csv` | `e967539c2b62af1ab6c5c369c5825604ea545a2090b481544e74d9f174bff3d1` | +| `output-manifest.json` | `a77b30469c3d46c26ed045f748f09313a646ef13b571f40c45a7a31f7a72505e` | +| `table-1-environment.md` | `ebce68ac85292844cd1f397494211e3fe44250d9b5bf7dae019b62d10993b575` | +| `table-2-startup.md` | `9887ffedc5dd1bd98c5385bd2470cc2aa5e11bb07843a3bc4c8058a9601bdea6` | +| `table-3-cli-operations.md` | `6c194fb477a8279c31cf45f498954fdb80d7751328d1b4bef0f222f643410b80` | +| `table-4-resources.md` | `ed3fd61377ab075fc870194769919b138432053e0906b05a4d1c96cd5c6c1d78` | +| `tables.json` | `c8aaa13c58d0dad900f3d08d6a926d7736c281bacdb665e3976da269cbdab3dd` | + +Numeric provenance is schema `ai-research-writing/numeric-evidence-v2`: 153 +entries, 153 unique evidence IDs, no duplicates or value mismatches. Every +archive source hash, output-manifest entry, and `reportable_measured` selector +was independently rechecked. + +## Result summary and evidence boundary + +The complete reportable values are the generated tables, not this prose +summary. Representative bounded results are: + +- sandbox create plus base mount: p50/p95/p99 + 1,659.811/1,749.739/1,794.902 ms, mean throughput 0.62 ready/s; +- no-op CLI command at concurrency 1 versus 5: p50 26.719 versus 45.998 ms, + and mean throughput 37.28 versus 107.61 operations/s; +- 256-KiB file edit at concurrency 5: p50/p95/p99 + 187.357/208.008/294.392 ms, mean throughput 26.36 operations/s; +- for that same preregistered resource row, peak daemon RSS was 62.602 MiB, + peak sandbox memory 66.715 MiB, mean sandbox CPU 148.732 ms/trial, and mean + workspace allocated delta 1.258 MiB. + +These are descriptive values for one disclosed native-Windows/Docker Desktop +host, one pinned image, one fixed 100-MiB fixture, one product/benchmark +revision, and concurrency 1 or 5. They are not competitive, universal, or +multi-agent productivity claims. + +## Failures, exclusions, warnings, and unavailable metrics + +The v1.1 final has no product, correctness, infrastructure, cleanup, or +primary-latency failure; no report warning; no excluded final trial; no retry; +and no silent outlier removal. Smoke, pilot, projection values, the failed +v1.0 partial final, setup/verification/teardown timings, and unavailable +observations remain excluded from manuscript result rows. + +Explicit final-report unavailability is retained rather than encoded as zero: + +- the archived canonical OS caption is the literal + `Microsoft Windows 11 ???`; its localized edition suffix is not legible in + the frozen capture. Build 26200, version, architecture, computer name, and + all other host fields remain present. A post-run read-only registry query + reported `EditionID=Core`, but that external check is not substituted into + the archive-derived Table 1; +- `layerstack_bytes`: 1,900/1,900 reportable trials unavailable because the + product does not report LayerStack allocated storage; +- `workspace_allocated_bytes`: 1,900/1,900 unavailable because host metadata + does not expose allocated block counts; +- `daemon_cpu_time_ns`, `sandbox_cpu_time_ns`, and sandbox block read/write: + 100/1,900 unavailable for create-sandbox trials where a sandbox-scoped + pre-create counter baseline cannot exist; +- sandbox current/peak memory: 50/1,900 unavailable before the resource ring + exists. + +Table 4 uses the available, preregistered `upperdir_bytes` workspace delta for +the rows it reports; it does not silently substitute unavailable +`workspace_allocated_bytes`. + +## Code and documentation changed + +The v1.1 product delta from the frozen v1.0 treatment +`0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3` through `5c48dae1` changed exactly: + +```text +README.md +bin/package-windows-amd64-release.ps1 +bin/start-sandbox-windows-docker-gateway.ps1 +crates/sandbox-cli/src/manager.rs +crates/sandbox-cli/src/observability.rs +crates/sandbox-cli/src/runtime.rs +crates/sandbox-cli/tests/manager.rs +crates/sandbox-config/src/configs/gateway.rs +crates/sandbox-config/tests/unit/configs/gateway.rs +crates/sandbox-gateway/src/gateway/config.rs +crates/sandbox-gateway/src/gateway/lifecycle.rs +crates/sandbox-gateway/src/gateway/listener.rs +crates/sandbox-gateway/src/gateway/main.rs +crates/sandbox-gateway/src/gateway/mod.rs +crates/sandbox-gateway/src/lib.rs +crates/sandbox-gateway/tests/gateway_server.rs +crates/sandbox-gateway/tests/local_daemon_installer.rs +crates/sandbox-mcp/src/config.rs +crates/sandbox-mcp/src/lib.rs +crates/sandbox-operations/client/src/client.rs +crates/sandbox-operations/client/src/config.rs +crates/sandbox-operations/client/src/endpoint.rs +crates/sandbox-operations/client/src/lib.rs +crates/sandbox-operations/client/tests/config.rs +crates/sandbox-operations/client/tests/endpoint.rs +crates/sandbox-operations/client/tests/transport.rs +docs/windows-setup.md +``` + +The v1.1 paper/benchmark measurement delta through frozen commit `1680b59` +changed these exact experiment-scoped paths: + +```text +benchmark/PAPER_ARTIFACT.md +benchmark/backend/benchmark_lab/cli.py +benchmark/backend/benchmark_lab/gateway.py +benchmark/backend/benchmark_lab/ipc_qualification.py +benchmark/backend/benchmark_lab/metadata.py +benchmark/backend/benchmark_lab/product_cli.py +benchmark/backend/benchmark_lab/runner.py +benchmark/backend/benchmark_lab/transport.py +benchmark/backend/tests/integration/test_gateway_lifecycle.py +benchmark/backend/tests/integration/test_gateway_transport.py +benchmark/backend/tests/integration/test_runner.py +benchmark/backend/tests/unit/test_exp1_archive.py +benchmark/backend/tests/unit/test_exp1_runtime_projection.py +benchmark/backend/tests/unit/test_ipc_qualification.py +benchmark/backend/tests/unit/test_metadata.py +benchmark/backend/tests/unit/test_product_cli.py +claim_evidence_map.md +experiment_inventory.md +experiments/analysis/exp1-final-failure-diagnostic.json +experiments/analysis/exp1-final-handoff.md +experiments/analysis/exp1-final-system-event-4227.json +experiments/analysis/exp1-freeze-record-eb10c26-0392b299.json +experiments/analysis/exp1-gate0-7-final-report.md +experiments/analysis/exp1-v1.1-remediation-decision.json +experiments/analysis/exp1-v1.1-remediation-decision.md +experiments/analysis/pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json +experiments/analysis/scripts/generate_exp1_tables.py +experiments/analysis/tests/test_generate_exp1_tables.py +experiments/environment_setup.md +experiments/exp1-v1.1-protocol-amendment.md +experiments/expected_tables.md +experiments/experiment_log.md +experiments/scripts/archive_exp1_run.py +experiments/scripts/project_exp1_final_runtime.py +paper_state.json +plan/progress.md +plan/task-packets/exp1-cli-performance-campaign.md +progress.md +``` + +The post-freeze analysis erratum changes only +`experiments/analysis/scripts/generate_exp1_tables.py` and its test file at +commit `538f6c9`. Post-final evidence updates modify this report, +`exp1-final-handoff.md`, the claim map, experiment inventory, two progress +trackers, paper state, benchmark artifact note, and the append-only experiment +log. The freeze record and two byte-identical final-v1.1 table trees are +explicitly versioned with `-text` Git attributes so checkout preserves their +recorded SHA-256 identities. Large run archives, exploratory tables, and +temporary diagnostics remain outside source commits. + +## Tests and material commands + +| Command or command class | Exit/status | +|---|---| +| Changed product-crate tests, real named-pipe request/response and concurrency-5 tests | Exit 0. | +| Changed product-crate all-target Clippy with warnings denied; formatting/diff checks | Exit 0. | +| Native release packaging and packaged concurrency-5 manager-CLI round trip | Exit 0 after the first policy-blocked `.ps1` launch was corrected to the documented bypass invocation. | +| Preregistered v1.1 IPC qualifier | PASS: 25,000/25,000 calls, 5,000 batches, zero transport/TCP/event/resource/cleanup gate failures. | +| Full prequalification benchmark/analysis suite | Exit 0: 354 passed, five expected skips. | +| Fresh v1.1 smoke, archive creation, and independent verification | Exit 0; terminal `completed`. | +| Fresh v1.1 pilot, archive creation, two table builds, projection, and independent verification | Exit 0; terminal `completed`; repeated outputs byte-identical. | +| Final freeze record, strict final preflight, package/tag/plan verification | Exit 0; final validation 19 cells/1,938 batches/5,610 requests and zero findings/warnings. | +| Sole v1.1 final benchmark and supervisor | Exit 0; terminal `completed`; stderr empty. | +| Final archive creation and two verify-only replays | Exit 0; all reproduce 82,051 files, 3,139,214,747 bytes, and the same content tree. | +| Initial frozen table-generator attempt | Exit 1 before output: canonical host schema was not recognized. No archive/output mutation. | +| Corrected generator targeted tests | Exit 0: 22 passed. | +| Corrected complete backend plus analysis suite | Exit 0: 380 passed, five expected Windows symlink skips. | +| Corrected final table generation into A and B | Exit 0 twice; 9/9 paths byte-identical. | +| Numeric-provenance/hash verification and frozen-pilot neutrality replay | Exit 0: 153/153 selectors verified; numeric artifacts and Tables 2--4 byte-identical to frozen-generator pilot output. | + +The experiment log also retains non-result helper failures: read-only `rg` +exit propagation, one malformed PowerShell quote, one Docker template parse, +one pre-file-existence monitor read, a 10-second aggregate timeout followed by +a successful 60-second retry, unavailable venv Ruff, unsupported PowerShell +hex helpers, and two PowerShell numeric-audit false alarms corrected by exact +Python verification. None mutated a corpus, treatment, archive, Docker object, +host setting, or protected process, and none consumed another final attempt. + +## Cleanup outcome + +The final run workspace and runtime are absent. No matching benchmark process, +run- or gateway-labeled container, or volume remains. The product checkout is +clean direct `main` at `5c48dae1`. Protected pre-existing gateway PID 62980 is +alive and was never touched. The pre-existing exited container and five +pre-existing volumes remain unchanged. + +## Supported wording + +- "Under the disclosed native-Windows/Docker Desktop EXP1 v1.1 environment, + the archive-derived latency, throughput, and resource values are those in + Tables 2--4 for the fixed `paper-100m` workload and concurrency 1 or 5." +- "Sandbox create plus base mount had p50/p95/p99 latency of + 1,659.811/1,749.739/1,794.902 ms in this environment." +- "For the no-op native CLI command, mean throughput changed from 37.28 + operations/s at concurrency 1 to 107.61 operations/s at concurrency 5, + while p50 batch makespan changed from 26.719 to 45.998 ms." +- "For the preregistered 256-KiB/concurrency-5 resource rows, peak measured + sandbox memory ranged from 14.977 MiB for read to 66.715 MiB for edit." +- "The sole v1.1 final completed all 1,900 measured trials without a product, + correctness, infrastructure, cleanup, or primary-latency failure." +- "The v1.0 failed corpus remains ineligible and was not pooled or compared + numerically with v1.1." + +## Wording that remains unsafe + +- Any claim that Ephemeral Sandbox is universally fast, cheap, scalable, + superior, or production-ready. +- Any competitive claim: no independent sandbox/worktree/container baseline + was measured by this focused campaign. +- Any extrapolation beyond this host, image, product revision, fixture, + workload, concurrency 1/5, or end-to-end native CLI boundary. +- Any causal claim that concurrency improves individual latency; the measured + batch makespan and throughput must be reported separately. +- Any use of qualifier, smoke, pilot, projection, failed-v1.0 partial, setup, + verification, teardown, unavailable, or silently rounded values as final + manuscript results. +- Any security, isolation-correctness, multi-agent task-quality, publication, + fault-tolerance, or useful-work claim from EXP1. + +## Remaining blockers and next external action + +There is **no remaining EXP1 execution or analysis blocker** and no rerun is +authorized or needed. The single next external action is author review of the +generated tables and bounded wording before selected numeric-evidence-backed +values are imported into `main.tex`. + +Broader paper blockers—other RQs, competitive baselines, author metadata, +figure normalization, novelty review, and submission checks—remain outside +this focused experiment. diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.json b/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.json new file mode 100644 index 0000000..fb9ff74 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.json @@ -0,0 +1,298 @@ +{ + "schema_version": 1, + "decision_id": "exp1-v1.1-windows-endpoint-pressure-remediation", + "created_at_utc": "2026-07-31T07:37:02.435Z", + "status": "superseded_unexecuted_by_authorized_local_ipc_treatment", + "supersession": { + "authorized_at_utc": "2026-07-31T07:59:50.209Z", + "selected_treatment": "windows_named_pipe_local_ipc", + "active_protocol": "ephemeral-sandbox-v1-practical-performance-v1.1", + "active_protocol_amendment": "experiments/exp1-v1.1-protocol-amendment.md", + "host_network_mutation_performed": false, + "historical_record_retained": true + }, + "scope": { + "failed_protocol": "ephemeral-sandbox-v1-practical-performance-v1.0", + "proposed_protocol": "ephemeral-sandbox-v1-practical-performance-v1.1", + "failed_run_id": "019fb6e5-c00b-7b02-8a3c-d76bd1346eb4", + "failed_archive_content_tree_sha256": "sha256:7efa643b12aba09f0ba5ecfbed5b5692a166a5c12931490402d3992d92f3ae6a", + "proposal_is_performance_evidence": false, + "v1_0_rerun_authorized": false, + "host_or_source_mutation_performed": false, + "live_probe_performed": false + }, + "direct_failure_evidence": { + "failed_request_id": "trial-242749a59d59cc16-measured-000034.observe.snapshot.1.boundary.0", + "failed_invocation_sha256": "sha256:3e188c6e284c0f65e2a909df854ac18a5f550ea1daffb86f2eb3d02ebc26ab0f", + "winsock_error": { + "symbol": "WSAEADDRINUSE", + "code": 10048 + }, + "system_event": { + "event_id": 4227, + "record_id": 88385, + "created_at_utc": "2026-07-31T06:50:09.9652511Z", + "primary_capture_path": "experiments/analysis/exp1-final-system-event-4227.json", + "primary_capture_sha256": "sha256:b6eac476b6ecf8c20de529be5c5ca8de297874ae9273c65a4baaf6ffc34ac89d" + }, + "claim_boundary": "The evidence proves one local endpoint-reuse failure under high-rate connection churn. It does not prove simultaneous occupancy of every dynamic port or an exact universal failure threshold." + }, + "connection_forensics": { + "tcp_attempts_identified": 7993, + "committed_cli_metadata_records": 7992, + "orphan_completed_cgroup_projection": 1, + "trace_span_seconds": "589.946", + "average_attempts_per_second": "13.549", + "roles": { + "manager": 328, + "runtime": 3829, + "observability": 3836 + }, + "operations": { + "create_sandbox": 110, + "destroy_sandbox": 110, + "inspect_sandbox": 102, + "list_sandboxes": 6, + "create_workspace_session": 208, + "destroy_workspace_session": 208, + "exec_command": 1224, + "file_read": 1567, + "file_write": 622, + "cgroup": 1918, + "snapshot": 1918 + }, + "failure_centered_windows": [ + { + "seconds": 10, + "attempts": 186, + "attempts_per_second": "18.600" + }, + { + "seconds": 30, + "attempts": 657, + "attempts_per_second": "21.900" + }, + { + "seconds": 60, + "attempts": 1349, + "attempts_per_second": "22.483" + }, + { + "seconds": 120, + "attempts": 3145, + "attempts_per_second": "26.208" + }, + { + "seconds": 240, + "attempts": 6760, + "attempts_per_second": "28.167" + } + ], + "maximum_rolling_windows": [ + { + "seconds": 10, + "attempts_per_second": "43.700" + }, + { + "seconds": 30, + "attempts_per_second": "42.767" + }, + { + "seconds": 60, + "attempts_per_second": "41.667" + }, + { + "seconds": 120, + "attempts_per_second": "33.100" + }, + { + "seconds": 240, + "attempts_per_second": "28.404" + } + ], + "resource_sampling": { + "mandatory_connections": 3626, + "periodic_connections": 210, + "mandatory_fraction_of_observability_percent": "94.53", + "periodic_fraction_of_all_attempts_percent": "2.63", + "persisted_metric_batches": 1810, + "mandatory_metric_batches": 1705, + "periodic_metric_batches": 105, + "missing_mandatory_batch": "the failed post-response snapshot boundary", + "baseline_repoll_connections": 214, + "complete_overlapping_cgroup_snapshot_pairs": 1917, + "incomplete_pair": "the failed snapshot plus one valid orphan cgroup response" + }, + "observed_gateway_endpoints": [ + { + "endpoint": "127.0.0.1:53160", + "committed_connections": 2878, + "duration_seconds": "113.651400", + "family": "command" + }, + { + "endpoint": "127.0.0.1:60545", + "committed_connections": 4398, + "duration_seconds": "153.901352", + "family": "files", + "failed_endpoint": true + }, + { + "endpoint": "127.0.0.1:54620", + "committed_connections": 716, + "duration_seconds": "321.535579", + "family": "sandbox_lifecycle" + } + ], + "accepted_pilot_structural_projection": { + "method": "Scale each pilot cell's trial-scoped CLI count from seven trials to 102 trials. Periodic and freshness-poll counts make this a diagnostic structural estimate, not an exact final count.", + "projected_trial_scoped_connections": 26392, + "projected_by_family": { + "files": 20578, + "command": 2856, + "workspace_lifecycle": 2244, + "sandbox_lifecycle": 714 + }, + "largest_projected_single_cell_connections": 4532, + "largest_projected_single_cell": "file_edit, concurrency 5, 262144-byte file", + "interpretation": "One gateway per cell would reduce family-level concentration, but its worst unexecuted cell still projects slightly above the 4398 committed connections observed at the failed endpoint. Per-cell rotation alone is therefore not accepted as a sufficient remedy without a stronger same-endpoint qualification." + } + }, + "frozen_source_mechanism": { + "product_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "paper_commit": "eb10c26d1bfd632772baf1bc331c985d0231f52d", + "client_source": { + "path": "crates/sandbox-operations/client/src/client.rs", + "sha256": "sha256:61ed900ae92a6911649390966b7dc47aec9eff98898f1942305c77ef4740edcc", + "behavior": "GatewayClient::send_with_logs creates one fresh tokio TcpStream::connect for every request and shuts down its write side after the request line." + }, + "gateway_source": { + "path": "crates/sandbox-gateway/src/gateway/connection.rs", + "sha256": "sha256:0cd590e9e3bc3db0c3c4eb0cd878bda44b7085487f3a91c5a514604b49be8a8b", + "behavior": "The gateway reads one request line, emits one response stream/final response, and shuts down the connection." + }, + "sampler_source": { + "path": "benchmark/backend/benchmark_lab/resource_sampling.py", + "sha256": "sha256:c051089fa2730e00962aebcf26d2be0777e6c5e00df25867fb95a933222a07b3", + "behavior": "Every product resource collection starts cgroup and snapshot CLI calls concurrently; mandatory baseline/final boundaries cannot be removed." + } + }, + "current_windows_tcp_state": { + "captured_read_only": true, + "host": "DESKTOP-OLP1ADS", + "windows_build": 26200, + "ipv4_dynamic_tcp_start_port": 49152, + "ipv4_dynamic_tcp_port_count": 16384, + "ipv6_dynamic_tcp_start_port": 49152, + "ipv6_dynamic_tcp_port_count": 16384, + "tcp_profiles_auto_reuse_start_port": 0, + "tcp_profiles_auto_reuse_port_count": 0, + "tcp_timed_wait_delay_registry": "not_configured_default", + "max_user_port_registry": "not_configured_default", + "transport_filter_query": "access_denied_in_current_non_elevated_session" + }, + "candidate_assessment": [ + { + "candidate": "temporary_active_store_ipv4_dynamic_range_expansion", + "rank_for_completing_exp1": 1, + "decision": "recommended_pending_authorization_and_qualification", + "proposed_active_range": { + "start_port": 1025, + "port_count": 64511, + "end_port": 65535, + "persistence": "active_store_only_until_reboot" + }, + "advantages": [ + "Preserves one fresh native CLI subprocess and one TCP connection inside every primary latency measurement.", + "Requires no product, operation, metric, trial-count, exclusion, or table-schema change.", + "Expands the disclosed pool from 16384 to 64511 ports for this boot." + ], + "risks": [ + "Requires an elevated host-wide network setting mutation.", + "Changes the qualified environment and therefore requires protocol v1.1, fresh qualification, a new freeze, and explicit disclosure.", + "Is a bounded experiment workaround rather than the product's long-term connection-pooling fix.", + "Success is not guaranteed without a same-rate, same-endpoint qualification." + ], + "exact_elevated_command_not_run": "netsh interface ipv4 set dynamicportrange protocol=tcp startport=1025 numberofports=64511 store=active", + "exact_elevated_rollback_not_run": "netsh interface ipv4 set dynamicportrange protocol=tcp startport=49152 numberofports=16384 store=active" + }, + { + "candidate": "one_gateway_execution_block_per_cell", + "rank_for_completing_exp1": 2, + "decision": "insufficient_alone_without_stronger_qualification", + "advantages": [ + "Paper-local planning change; no product or host setting mutation.", + "Preserves one-shot CLI timing and all primary operation definitions." + ], + "risks": [ + "Changes untimed gateway lifecycle and warm-state disclosure.", + "The accepted-pilot structural estimate has a 4532-connection worst cell, slightly above the failed endpoint's 4398 committed connections.", + "The five-sample pilot itself cannot exercise final-scale endpoint pressure." + ] + }, + { + "candidate": "combine_cgroup_and_snapshot_in_one_observability_request", + "rank_for_completing_exp1": 3, + "decision": "engineering_option_not_sufficiently_validated", + "observed_reduction": { + "connections": 1918, + "percent_of_all_attempts": "24.00", + "failure_centered_240_second_attempts_before": 6760, + "failure_centered_240_second_attempts_after_structural_estimate": 5250 + }, + "risks": [ + "Requires product operation/schema changes and a new package/treatment.", + "Does not establish a recurrence-safe connection level.", + "Changes the resource-evidence implementation." + ] + }, + { + "candidate": "persistent_or_multiplexed_cli_gateway_transport", + "rank_for_completing_exp1": 4, + "decision": "preferred_long_term_product_fix_but_separate_treatment", + "risks": [ + "Both the gateway and CLI protocol need redesign because the frozen sides are one-request-per-connection.", + "Using persistence for measured calls would remove fresh process/transport work from the frozen primary-latency definition.", + "Requires separate product qualification and scientific treatment." + ] + }, + { + "candidate": "remove_periodic_sampling_or_retry_failed_connections", + "decision": "rejected", + "reason": "Only 210 attempts, 2.63 percent, were periodic; the failed call was mandatory. Retries and reduced sampling violate frozen correctness/resource decisions and can bias results." + }, + { + "candidate": "shorten_tcp_time_wait_or_enable_so_reuseaddr", + "decision": "rejected", + "reason": "TIME_WAIT registry mutation is host-wide and changes safety behavior; Microsoft warns ordinary applications away from SO_REUSEADDR because behavior can be nondeterministic and vulnerable to port hijacking." + } + ], + "proposed_v1_1_qualification_and_execution_sequence": [ + "Obtain explicit author authorization for protocol v1.1, the temporary elevated active-store IPv4 range mutation, the qualification soak, a fresh smoke/pilot, a new freeze, and exactly one new final.", + "Capture active/persistent IPv4 and IPv6 dynamic ranges, excluded port ranges, TCP profiles, transport filters, relevant registry values, event-log cursor, product/paper Git state, and protected baseline before mutation.", + "Apply only the proposed IPv4 active-store range; verify 1025 plus 64511 without changing IPv6, TIME_WAIT, auto-reuse, firewall, routing, Docker, or protected resources.", + "After prior endpoint states quiesce, run an ineligible same-endpoint strict native-CLI churn qualifier for at least 700 seconds and at least 20000 successful connections at no less than the failed run's 28.167 attempts-per-second 240-second sustained rate. Capture BOUND/TIME_WAIT counts and Event 4227/4231 cursors. Any transport failure or new event fails qualification.", + "Clean up the qualifier, allow endpoint state to quiesce, verify the protected baseline, and repeat the complete 19-cell smoke and five-sample pilot with no automatic retries.", + "Require smoke/pilot correctness and cleanup, a fresh no-more-than-1400-second projection, deterministic exploratory regeneration, and an anomaly review before protocol/source/environment v1.1 freeze.", + "Freeze all prior identities plus exact active IPv4 range and qualifier provenance in a new paper commit/environment record. Preserve the failed v1.0 archive and do not compare or pool its partial values.", + "After a final read-only preflight, run exactly one newly authorized v1.1 final. Restore the active IPv4 range after immutable archival and cleanup even if the run fails.", + "Only a complete eligible v1.1 corpus may generate deterministic tables, numeric-evidence v2 records, and RQ3 numeric wording." + ], + "authorization_blocker": { + "status": "superseded_not_applicable", + "reason": "The proposed host-wide TCP change was not selected and was never applied.", + "single_external_decision": "none; active local-IPC work proceeds under experiments/exp1-v1.1-protocol-amendment.md", + "current_session_can_apply_setting": false, + "observed_privilege_evidence": "Get-NetTransportFilter returned Windows access denied; no setting mutation was attempted." + }, + "primary_references": [ + "https://learn.microsoft.com/en-us/troubleshoot/windows-client/networking/tcp-ip-port-exhaustion-troubleshooting", + "https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/connect/intermittent-periodic-network-issue", + "https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/default-dynamic-port-range-tcpip-chang", + "https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/netsh-interface", + "https://learn.microsoft.com/en-us/powershell/module/nettcpip/set-nettcpsetting", + "https://learn.microsoft.com/en-us/windows/win32/fwp/wmi/nettcpipprov/msft-nettcpsetting", + "https://learn.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2", + "https://learn.microsoft.com/en-us/windows/win32/winsock/using-so-reuseaddr-and-so-exclusiveaddruse" + ] +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.md b/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.md new file mode 100644 index 0000000..7ddf970 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-v1.1-remediation-decision.md @@ -0,0 +1,166 @@ +# EXP1 v1.1 endpoint-pressure remediation decision + +Status: superseded without execution by the authorized local-IPC treatment. + +Date: 2026-07-31. + +This document is failure analysis and protocol planning, not performance +evidence. It does not authorize a v1.0 rerun, alter the immutable failed +archive, change the host, or launch a live probe. + +The author subsequently selected a permanent CLI-focused product fix. No +dynamic-port command in this document was run and no host network setting was +changed. Active work is governed by +[`../exp1-v1.1-protocol-amendment.md`](../exp1-v1.1-protocol-amendment.md), +which uses Windows named pipes, retains explicit TCP compatibility outside the +paper treatment, and preregisters the replacement qualification and freeze +sequence. The material below remains an immutable historical decision record. + +## Decision + +For the shortest path to a scientifically usable EXP1 result, prefer a +temporary **active-store IPv4 dynamic-port range expansion** from +49,152–65,535 (16,384 ports) to 1,025–65,535 (64,511 ports), followed by a +same-rate qualification, fresh smoke/pilot, protocol/environment v1.1 freeze, +and exactly one newly authorized final. + +This is not the ideal product architecture. Persistent connection reuse is the +long-term engineering fix. It is nevertheless the least disruptive EXP1 +remediation because it preserves every measured request as a fresh native CLI +subprocess with its TCP transport inside the primary timing window. No product +operation, payload, cell, trial count, metric, exclusion, or table schema needs +to change. + +The host mutation is global, elevated, and scientifically material. It must +not be applied without explicit author authorization. The current session is +not elevated, and no network setting was changed. + +## What the failed corpus proves + +- The immutable final contains 7,992 committed CLI metadata records plus one + identifiable completed cgroup sibling without its metadata commit marker: + 7,993 client-to-gateway TCP attempts in total. +- Every frozen `GatewayClient` request opens a fresh `TcpStream`; the gateway + reads one request and closes the connection. +- The failure-centered 240-second window contains 6,760 attempts, or 28.167 + attempts per second. The maximum rolling 240-second rate is 28.404 per + second. +- Exactly one committed invocation failed: the mandatory post-response + snapshot returned WSAEADDRINUSE 10048. Its concurrent cgroup sibling + completed successfully, the measured file-read had already succeeded, and + the gateway remained live. +- TCP/IP Event 4227 at the same instant records local-endpoint reuse under + high-rate connection open/close churn. + +This proves endpoint-reuse pressure and the proximate failed connection. It +does not prove that all 16,384 dynamic ports were simultaneously occupied, an +exact TIME_WAIT count, or a universal numerical failure threshold. + +## Why reducing resource cadence is not a remedy + +Of 3,836 observability connections, 3,626 (94.53%) are mandatory and only 210 +are periodic. Periodic sampling is 2.63% of all identified attempts. The +failed snapshot belongs to the mandatory post-response boundary. + +Removing periodic samples would therefore weaken the resource protocol while +removing only 210 attempts. Automatic retries are also prohibited because they +would hide a failed sample and can bias the retained distribution. + +## Why per-cell gateway rotation is not accepted alone + +One gateway per cell is an attractive paper-local change and would preserve +one-shot CLI timing. However, the accepted pilot’s trial-scoped CLI counts, +scaled diagnostically from seven to 102 trials, project: + +| Scope | Projected trial-scoped CLI connections | +|---|---:| +| Complete final | 26,392 | +| Files family | 20,578 | +| Largest single cell | 4,532 | + +The largest cell is the 256-KiB, concurrency-5 file-edit cell. Its projected +4,532 connections are slightly above the 4,398 committed connections already +observed at the failed files endpoint. The sole final never reached that cell, +so reasoning only from the largest completed cell would understate the risk. +Per-cell rotation remains a fallback to qualify, not a sufficient remedy on +current evidence. + +## Other candidates + +| Candidate | Decision | Reason | +|---|---|---| +| Merge cgroup and snapshot into one product request | Engineering option, not selected | Removes 1,918 observed attempts (24.00%) and the incomplete-pair failure mode, but requires a new product operation/schema and still leaves no validated safe churn level. | +| Persistent/multiplexed CLI transport | Long-term product fix | Both client and gateway are one-request-per-connection. Redesign is required, and using persistence for measured calls would change the defining process/transport timing construct. | +| Auto-reuse port range | Qualify only as fallback | The host reports zero/disabled auto-reuse ranges. Configuration requires elevation and stronger same-destination validation; it is not the first experiment remedy. | +| Shorter TIME_WAIT | Rejected initially | Host-wide safety/timing change with restart implications; no need to combine it with the first remedy. | +| `SO_REUSEADDR` | Rejected | Microsoft warns that ordinary use can be nondeterministic and vulnerable to port hijacking. | +| Reboot, pacing, sleeps, or retries | Rejected | Reboot is temporary; pacing changes scheduling/runtime; retries violate the protocol. | + +## Proposed v1.1 authorization scope + +If approved, the host change would be active-store only: + +```powershell +netsh interface ipv4 set dynamicportrange protocol=tcp startport=1025 numberofports=64511 store=active +``` + +The exact active-store rollback would be: + +```powershell +netsh interface ipv4 set dynamicportrange protocol=tcp startport=49152 numberofports=16384 store=active +``` + +Neither command has been run. IPv6, TIME_WAIT, auto-reuse, routing, firewall, +Docker, and protected resources would remain unchanged. + +Approval must cover this complete sequence: + +1. Capture active/persistent TCP ranges, excluded ranges, TCP profiles, + transport filters, registry values, event-log cursor, Git identities, and + protected baseline. +2. Apply and verify only the temporary IPv4 active-store range. +3. After prior endpoint states quiesce, run an ineligible strict native-CLI + same-endpoint qualifier for at least 700 seconds and at least 20,000 + successful connections, sustaining at least the failed run’s 28.167 + attempts/second 240-second rate. Capture BOUND/TIME_WAIT counts and Event + 4227/4231 cursors. Any transport error or new event fails qualification. +4. Clean up and quiesce, then repeat the complete 19-cell smoke and + five-sample pilot with no retries. +5. Require correctness, cleanup, deterministic exploratory regeneration, and + a fresh projection no greater than the approved 1,400 seconds. +6. Freeze protocol/source/environment v1.1 in a new paper commit and record + the exact active IPv4 range and qualification evidence. +7. Run exactly one newly authorized v1.1 final after a read-only preflight. +8. Archive and clean up even on failure, then restore the active IPv4 range. + Only a complete eligible corpus may generate tables and numeric evidence. + +The failed v1.0 archive remains immutable and cannot be replaced, pooled, or +numerically compared with v1.1. + +## Evidence and references + +- Machine-readable decision: + `experiments/analysis/exp1-v1.1-remediation-decision.json` +- Failed archive: + `experiments/runs/019fb6e5-c00b-7b02-8a3c-d76bd1346eb4` +- Failed archive content tree: + `sha256:7efa643b12aba09f0ba5ecfbed5b5692a166a5c12931490402d3992d92f3ae6a` +- Microsoft port-exhaustion and Event 4227 guidance: + +- Microsoft dynamic-port-range support and syntax: + +- Microsoft active versus persistent `netsh interface` store: + +- Microsoft guidance to address connection creation/pooling before relying on + range/TIME_WAIT workarounds: + +- Microsoft Winsock 10048 definition: + +- Microsoft `SO_REUSEADDR` safety guidance: + + +## Exact blocker + +This proposal has no remaining blocker because it was superseded and will not +be executed. The active local-IPC campaign is blocked only by its in-tree +qualification, smoke, pilot, projection, and freeze gates. diff --git a/ephemeral-sandbox-v1/experiments/analysis/exp1-v11-freeze-record-1680b59-5c48dae1.json b/ephemeral-sandbox-v1/experiments/analysis/exp1-v11-freeze-record-1680b59-5c48dae1.json new file mode 100644 index 0000000..c91db59 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/exp1-v11-freeze-record-1680b59-5c48dae1.json @@ -0,0 +1,169 @@ +{ + "schema_version": 1, + "created_at": "2026-07-31T13:40:38.895Z", + "protocol": { + "name": "EXP1 CLI-only focused performance campaign", + "id": "ephemeral-sandbox-v1-practical-performance-v1.1", + "version": "v1.1", + "freeze_state": "frozen", + "protocol_files": [ + { + "path": "progress.md", + "bytes": 15817, + "sha256": "sha256:d7b49d15a2bff5f59b73aefeda6b5bc32423737de34cdd8b78e162e178988b37" + }, + { + "path": "plan/task-packets/exp1-cli-performance-campaign.md", + "bytes": 25212, + "sha256": "sha256:a090d2733e821d6d3990d16155169d3bcd4d1a0f01b85c8f84cccee274bd13d9" + }, + { + "path": "experiment_inventory.md", + "bytes": 19447, + "sha256": "sha256:6ece88800fb7bc20b692d4b2a041434c41b1aa8c0c727c1a90e8e09b42e7dd88" + }, + { + "path": "experiments/exp1-v1.1-protocol-amendment.md", + "bytes": 9972, + "sha256": "sha256:e2f96acede3f0f8ac40e84b0e1945f307df938c58060ad94cde92a2c3f854d01" + }, + { + "path": "experiments/environment_setup.md", + "bytes": 11551, + "sha256": "sha256:9d404fa1623e546b8123deb561ffb94501b3234edf15f8ab72b54020a8b8f213" + }, + { + "path": "experiments/expected_tables.md", + "bytes": 12021, + "sha256": "sha256:6802bb831d5b9ef05d2685f5005a115faecdd06024872d07abff0bc75e6c7eae" + }, + { + "path": "experiments/experiment_log.md", + "bytes": 407071, + "sha256": "sha256:e932fd7cfc2ce11d3545edae39b19c5d7bc75f5761b18ee4fae2314205bca080" + }, + { + "path": "benchmark/PAPER_ARTIFACT.md", + "bytes": 15092, + "sha256": "sha256:c55b85d464c9e1a8fa486f72009481720abe7b4b603f2f0baa4a751a058c9966" + }, + { + "path": "paper_state.json", + "bytes": 3565, + "sha256": "sha256:582f5b2c1e87c5858ea3428ca652211f00c6c85998783c2461a6d3c4decd495b" + }, + { + "path": "plan/progress.md", + "bytes": 30535, + "sha256": "sha256:8a0f34397a238228a0e86c37356283e720408f0f054e6ec1f2aadf8fabfe2d58" + } + ], + "metrics_exclusions_and_stopping_rules": "Frozen by the task packet, v1.1 amendment, inventory, and expected-table contract; no retries, no silent outlier removal, and exactly one eligible v1.1 final attempt.", + "seed": 20260712 + }, + "paper_source": { + "repository_commit": "1680b599129532f72e706b6acb12ef62c63759e2", + "branch": "agent/complete-pw3-and-final-host-prep", + "frozen_scope_clean": true, + "freeze_state": "clean_frozen_commit", + "benchmark_source": { + "file_count": 213, + "bytes": 16451598, + "content_tree_sha256": "sha256:c060e397ce3511a7839c71e13506dd4db99c9ad774464d0a0555f6949319dabd" + } + }, + "product": { + "repository_commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8", + "branch": "main", + "clean": true, + "transport": { + "kind": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", + "fallback": "none" + }, + "freeze_tag": { + "name": "paper-v1.1-freeze", + "object_type": "tag", + "tag_object": "834c84534359f37653fb25ac45304091e82c37a6", + "peeled_commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8" + }, + "preserved_v1_tag": { + "name": "paper-v1-freeze", + "tag_object": "0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d" + }, + "baseline_release": { + "name": "v0.1.4", + "tag_object": "981b99fec4df576bb4326b3c1a145be4471766ce", + "peeled_commit": "b22862550e0a7cb4fe61ce581831e9244cc492b5" + }, + "package": { + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-5c48dae1.zip", + "bytes": 5739735, + "sha256": "sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506" + }, + "binaries": { + "sandbox-gateway.exe": "sha256:42e7642dd025487811abbcd78dcc5513760f2aaa1e6057cfdfa3e74c03748358", + "sandbox-manager-cli.exe": "sha256:e1faa2fe0e9f4909fa2d694166784ac65dde40ba82795b7e0c503eb5fea86513", + "sandbox-runtime-cli.exe": "sha256:e18827cf765945c958e169748575b89645c730b310ee5ffc1b42c382b44a0e26", + "sandbox-observability-cli.exe": "sha256:2b1c13bba36c9486f768824178d1e2ea8d2b1da019bd21cd1f9ea250d5da34c5", + "ephemeral-sandbox-daemon": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a" + } + }, + "environment": { + "host": "native Windows x64 DESKTOP-OLP1ADS build 26200", + "docker": "Docker Desktop 29.0.1 Linux amd64 engine, overlayfs, cgroup v2", + "image": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + "sandbox_limits": { + "profile": "standard", + "nano_cpus": 1000000000, + "vcpus": 1, + "memory_bytes": 536870912, + "pids_limit": 256 + } + }, + "fixture": { + "profile": "paper-100m", + "fixture_hash": "sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7", + "tree_hash": "sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f", + "pilot_manifest_sha256": "sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1" + }, + "final_plan": { + "preset": "paper-good-pass", + "preset_sha256": "sha256:4d7ea2903e7c6e6a2a0d28acec44d5427b3c42008227b2273958b0ad25870ce9", + "plan_hash": "sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b", + "reviewed_expansion_path": "tmp/validate-paper-good-pass-v11-20260731T1327Z.json", + "reviewed_expansion_sha256": "sha256:7c4d2fc4f086e06971146dcdf38028ecd4dff9865745b16b238305594e51138c", + "client_cohort": "product_cli", + "workspace_profile": "paper-100m", + "cell_count": 19, + "warmups_per_cell": 2, + "measured_trials_per_cell": 100, + "trial_batch_count": 1938, + "issued_operation_request_count": 5610 + }, + "analysis": { + "archive_script": { + "path": "experiments/scripts/archive_exp1_run.py", + "sha256": "sha256:353a45bf8780146b19d61a07ce258fb10b9f01c6ba695de7c264e85efdbb714c" + }, + "table_generator": { + "path": "experiments/analysis/scripts/generate_exp1_tables.py", + "sha256": "sha256:7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7" + }, + "runtime_projection": { + "path": "experiments/scripts/project_exp1_final_runtime.py", + "sha256": "sha256:d6d73f75f3a2186eef3f92ae8ef8f176f60133e366d89f6ae69a78ad5aa47781" + }, + "gate3_projection": { + "path": "experiments/analysis/pilot-v11-019fb84e-final-runtime-prefreeze.json", + "sha256": "sha256:a7b4eda8cd1f15e59bca2e6495cb8b8c36914619e4c812143ac361bdb1803822", + "pilot_seconds": "276.094047000", + "central_seconds": "1179.784426150", + "envelope_seconds": "1303.732241600", + "limit_seconds": "1400.000000000", + "decision": "pass_runtime_projection" + } + }, + "gate4_decision": "pass_protocol_and_source_freeze" +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/generation-log.txt b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/generation-log.txt new file mode 100644 index 0000000..5cc008d --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/generation-log.txt @@ -0,0 +1,14 @@ +EXP1 deterministic table generation +generator_schema_version=2 +generator_sha256=sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b +archive_run_id=019fb86c-096e-7589-a0a4-a6d6ef5d7f8b +archive_disposition=final +archive_eligibility=frozen_final_candidate +protocol_version=v1.1 +archive_content_tree_sha256=sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986 +archive_inventory_verified=true +semantic_identity_verified=true +correctness_verified=true +command=python experiments/analysis/scripts/generate_exp1_tables.py --protocol-version --archive --output +output_path_embedded=false +wall_clock_embedded=false diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-evidence.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-evidence.json new file mode 100644 index 0000000..574d0c4 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-evidence.json @@ -0,0 +1,2606 @@ +{ + "entries": [ + { + "aggregate": "identity", + "note": "table1.cpu.logical_processors; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/logical_processors\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.cpu.logical_processors" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 48.0 + }, + { + "aggregate": "identity", + "note": "table1.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/total_memory_bytes\"}; units=bytes->bytes; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.memory_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 137438953472.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/memory_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.memory_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 512.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.pids_limit; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/pids_limit\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.pids_limit" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 256.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.vcpus; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/vcpus\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.vcpus" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.0 + }, + { + "aggregate": "identity", + "note": "table1.seed; upstream=fixture-manifest.json#{\"pointer\":\"/identity/seed\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.seed" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20260712.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.measured; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/measured_trials\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.measured" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.warmups; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/warmups\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.warmups" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 2.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.file_count; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/file_count\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.file_count" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 4000.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.logical_mib; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/logical_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.logical_mib" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.maximum_depth; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/maximum_depth\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.maximum_depth" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1659.8107 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1749.7388 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1794.9019450000008 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.6185790825907366 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 32.89655 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 35.6847 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 36.096376 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.359842997206 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 109.11905 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 120.107005 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 131.58766000000003 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.3204753420739 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 27.23455 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.937635 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 32.66597900000007 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 36.44471318802348 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.44085 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 50.973254999999995 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 56.30326700000011 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 108.88291336149418 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.9976 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.281615 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 60.09194300000015 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 107.61432697082469 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 48.93455 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.328845 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 63.09657600000165 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.171524365907672 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 187.35685 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 208.008075 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 294.3917890000002 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.357148435723182 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.6474 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 42.61549 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 46.87779200000013 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.608367032181523 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 134.61035 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 141.75804 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 146.72477300000017 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.216098524026926 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.7811 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 24.366515 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.48393800000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 47.68014601293585 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.78825 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 57.79372999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 60.15640100000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 125.54290904967944 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 11.4248 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 15.772725 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 19.96439500000001 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 84.9999254477021 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.575 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.591984999999998 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 39.73821200000005 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 185.30259988470985 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.06585 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 49.94722 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.543718 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 22.02875636155663 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 125.9685 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 137.255055 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 147.37442700000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 39.354743449708884 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.8126 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 41.38427999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 51.56190400000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.551107882015895 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 108.75565 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 111.979635 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 116.06709000000002 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.9991049224615 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.5 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0108203125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.4712 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.171875 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.00390625 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 34.125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0270703125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 79.50166 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 43.52734375 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.01953125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 13.875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.001171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 35.00335 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.03125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 13.125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0026171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 63.07729 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 21.5546875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 62.6015625 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2505859375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 148.73234 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 66.71484375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2578125 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 14.47265625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.43103 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 14.9765625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 51.04296875 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.250625 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 95.11521 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 54.26953125 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2578125 + } + ], + "schema_version": "ai-research-writing/numeric-evidence-v2" +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv new file mode 100644 index 0000000..3bbf824 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv @@ -0,0 +1,154 @@ +evidence_id,value,source_file,source_sha256,selector_kind,selector_json,input_unit,output_unit,aggregation,aggregation_parameters_json,eligibility +table1.cpu.logical_processors,48,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/recorded_run_environment/host/logical_processors""}",count,count,identity,{},frozen_final_candidate +table1.memory_bytes,137438953472,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/recorded_run_environment/host/total_memory_bytes""}",bytes,bytes,identity,{},frozen_final_candidate +table1.sandbox_limits.memory_bytes,512,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/memory_bytes""}",bytes,MiB,identity,"{""divisor"":1048576}",frozen_final_candidate +table1.sandbox_limits.pids_limit,256,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/pids_limit""}",count,count,identity,{},frozen_final_candidate +table1.sandbox_limits.vcpus,1,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/vcpus""}",count,count,identity,{},frozen_final_candidate +table1.seed,20260712,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/seed""}",count,count,identity,{},frozen_final_candidate +table1.trials.measured,100,expanded-plan.json,sha256:fa04fe9254e071ea6c65f906b9e62854a60330d0fdca8c41cb41297f3c923b2e,archive-json,"{""invariant"":""all equal"",""pointer"":""/data/cells/*/protocol/measured_trials""}",count,count,identity,{},frozen_final_candidate +table1.trials.warmups,2,expanded-plan.json,sha256:fa04fe9254e071ea6c65f906b9e62854a60330d0fdca8c41cb41297f3c923b2e,archive-json,"{""invariant"":""all equal"",""pointer"":""/data/cells/*/protocol/warmups""}",count,count,identity,{},frozen_final_candidate +table1.workspace_fixture.file_count,4000,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/file_count""}",count,count,identity,{},frozen_final_candidate +table1.workspace_fixture.logical_mib,100,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/logical_bytes""}",bytes,MiB,identity,"{""divisor"":1048576}",frozen_final_candidate +table1.workspace_fixture.maximum_depth,100,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/maximum_depth""}",count,count,identity,{},frozen_final_candidate +table2.create_sandbox.none.c1.p50_ms,1659.8107,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.p95_ms,1749.7388000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.p99_ms,1794.9019450000008,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_sandbox.none.c1.throughput_ops_s,0.61857908259073657,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.create_workspace.none.c1.p50_ms,32.896549999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.p95_ms,35.684699999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.p99_ms,36.096375999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_workspace.none.c1.throughput_ops_s,30.359842997206002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.create_workspace.none.c5.p50_ms,109.11905,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.p95_ms,120.107005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.p99_ms,131.58766000000003,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_workspace.none.c5.throughput_ops_s,45.320475342073898,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.exec_command.noop.c1.p50_ms,26.71865,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.p95_ms,28.291429999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.p99_ms,28.996465000000022,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.exec_command.noop.c1.throughput_ops_s,37.28339526491812,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p50_ms,27.234549999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p95_ms,28.937635,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p99_ms,32.665979000000071,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.fixture_read.c1.throughput_ops_s,36.444713188023478,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p50_ms,45.440849999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p95_ms,50.973254999999995,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p99_ms,56.303267000000112,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.fixture_read.c5.throughput_ops_s,108.88291336149418,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.noop.c1.p50_ms,26.71865,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.p95_ms,28.291429999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.p99_ms,28.996465000000022,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.noop.c1.throughput_ops_s,37.28339526491812,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.noop.c5.p50_ms,45.997599999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.p95_ms,53.281615000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.p99_ms,60.09194300000015,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.noop.c5.throughput_ops_s,107.61432697082469,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.262144.c1.p50_ms,48.934550000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.p95_ms,53.328845000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.p99_ms,63.096576000001647,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.262144.c1.throughput_ops_s,20.171524365907672,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.262144.c5.p50_ms,187.35685000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.p95_ms,208.00807499999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.p99_ms,294.39178900000019,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.262144.c5.throughput_ops_s,26.357148435723182,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.4096.c1.p50_ms,38.647399999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.p95_ms,42.615490000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.p99_ms,46.877792000000127,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.4096.c1.throughput_ops_s,25.608367032181523,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.4096.c5.p50_ms,134.61035000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.p95_ms,141.75803999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.p99_ms,146.72477300000017,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.4096.c5.throughput_ops_s,37.216098524026926,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.262144.c1.p50_ms,20.781099999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.p95_ms,24.366515,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.p99_ms,25.48393800000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.262144.c1.throughput_ops_s,47.68014601293585,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.262144.c5.p50_ms,37.788249999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.p95_ms,57.793729999999989,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.p99_ms,60.156401000000017,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.262144.c5.throughput_ops_s,125.54290904967944,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.4096.c1.p50_ms,11.424799999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.p95_ms,15.772724999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.p99_ms,19.96439500000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.4096.c1.throughput_ops_s,84.999925447702097,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.4096.c5.p50_ms,26.574999999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.p95_ms,30.591984999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.p99_ms,39.738212000000047,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.4096.c5.throughput_ops_s,185.30259988470985,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.262144.c1.p50_ms,45.065849999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.p95_ms,49.947220000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.p99_ms,53.543717999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.262144.c1.throughput_ops_s,22.028756361556631,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.262144.c5.p50_ms,125.96850000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.p95_ms,137.255055,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.p99_ms,147.37442700000005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.262144.c5.throughput_ops_s,39.354743449708884,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.4096.c1.p50_ms,38.812600000000003,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.p95_ms,41.38427999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.p99_ms,51.561904000000048,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.4096.c1.throughput_ops_s,25.551107882015895,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.4096.c5.p50_ms,108.75565,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.p95_ms,111.979635,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.p99_ms,116.06709000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.4096.c5.throughput_ops_s,45.999104922461498,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table4.create_workspace.none.c1.daemon_rss_bytes,25.5,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_block_write_bytes,0.0108203125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_cpu_time_ns,30.4712,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_memory_peak_bytes,28.171875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.upperdir_bytes,0.00390625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.daemon_rss_bytes,34.125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_block_write_bytes,0.027070312499999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_cpu_time_ns,79.501660000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_memory_peak_bytes,43.52734375,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.upperdir_bytes,0.01953125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.daemon_rss_bytes,13.875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_block_write_bytes,0.001171875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_cpu_time_ns,35.003349999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_memory_peak_bytes,20.03125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.daemon_rss_bytes,13.125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_block_write_bytes,0.0026171875000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_cpu_time_ns,63.077289999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_memory_peak_bytes,21.5546875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.daemon_rss_bytes,62.6015625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_block_write_bytes,1.2505859374999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_cpu_time_ns,148.73233999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_memory_peak_bytes,66.71484375,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.upperdir_bytes,1.2578125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.daemon_rss_bytes,14.47265625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_block_write_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_cpu_time_ns,38.43103,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_memory_peak_bytes,14.9765625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.daemon_rss_bytes,51.04296875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_block_write_bytes,1.2506250000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_cpu_time_ns,95.115210000000005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_memory_peak_bytes,54.26953125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.upperdir_bytes,1.2578125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/output-manifest.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/output-manifest.json new file mode 100644 index 0000000..0ee38c7 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/output-manifest.json @@ -0,0 +1,52 @@ +{ + "archive_content_tree_sha256": "sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986", + "archive_disposition": "final", + "archive_run_id": "019fb86c-096e-7589-a0a4-a6d6ef5d7f8b", + "eligibility": "frozen_final_candidate", + "files": [ + { + "bytes": 673, + "path": "generation-log.txt", + "sha256": "sha256:481f89a1900a051649102b986ac3ad6a7083b53959cfa4a9bcf419e383d3d0b3" + }, + { + "bytes": 123328, + "path": "numeric-evidence.json", + "sha256": "sha256:8a247f492b5bbf5bb35f1a89e0ed7dc3effcaaab0f935b746e1dcd89d62a367c" + }, + { + "bytes": 74608, + "path": "numeric-provenance.csv", + "sha256": "sha256:e967539c2b62af1ab6c5c369c5825604ea545a2090b481544e74d9f174bff3d1" + }, + { + "bytes": 2155, + "path": "table-1-environment.md", + "sha256": "sha256:ebce68ac85292844cd1f397494211e3fe44250d9b5bf7dae019b62d10993b575" + }, + { + "bytes": 612, + "path": "table-2-startup.md", + "sha256": "sha256:9887ffedc5dd1bd98c5385bd2470cc2aa5e11bb07843a3bc4c8058a9601bdea6" + }, + { + "bytes": 1612, + "path": "table-3-cli-operations.md", + "sha256": "sha256:6c194fb477a8279c31cf45f498954fdb80d7751328d1b4bef0f222f643410b80" + }, + { + "bytes": 932, + "path": "table-4-resources.md", + "sha256": "sha256:ed3fd61377ab075fc870194769919b138432053e0906b05a4d1c96cd5c6c1d78" + }, + { + "bytes": 25385, + "path": "tables.json", + "sha256": "sha256:c8aaa13c58d0dad900f3d08d6a926d7736c281bacdb665e3976da269cbdab3dd" + } + ], + "generator_schema_version": 2, + "generator_sha256": "sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b", + "protocol_version": "v1.1", + "schema_version": 2 +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-1-environment.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-1-environment.md new file mode 100644 index 0000000..9ee6b23 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-1-environment.md @@ -0,0 +1,23 @@ +# EXP1 environment and workload + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Field | Archived value | Evidence source | +| --- | --- | --- | +| Host OS | Microsoft Windows 11 ??? build 26200 | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Container engine OS | linux | environment-preflight.txt#{"pointer":"/docker/os_type"} | +| Architecture | x64 | environment-preflight.txt#{"pointer":"/recorded_run_environment/host/architecture"} | +| CPU | AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Memory | 137,438,953,472 bytes | environment-preflight.txt#{"pointer":"/recorded_run_environment/host/total_memory_bytes"} | +| Storage | NTFS | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Docker Engine | 29.0.1 | environment-preflight.txt#{"pointer":"/docker/server_version"} | +| Cgroup | 2 | environment-preflight.txt#{"pointer":"/docker/cgroup_version"} | +| Product commit/tag | 5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8; annotated tag paper-v1.1-freeze | campaign-manifest.json#{"pointer":"/product"} | +| Benchmark commit | 1680b599129532f72e706b6acb12ef62c63759e2 | campaign-manifest.json#{"pointer":"/paper_git/commit"} | +| Sandbox image | ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf | campaign-manifest.json#{"pointer":"/image/requested"} | +| Sandbox limits | 1 vCPU / 512 MiB / 256 PIDs | environment-preflight.txt#{"pointer":"/sandbox_limits"} | +| Workspace | 100 MiB / 4000 files / depth 100 | fixture-manifest.json#{"pointer":"/identity/fixture"} | +| Client | product_cli | campaign-manifest.json#{"pointer":"/plan/client_cohort"} | +| Gateway transport | windows_named_pipe; local_only; per_execution_block | run-manifest.json#{"pointer":"/data/environment/gateway_transport","protocol_version":"v1.1"} | +| Seed | 20260712 | fixture-manifest.json#{"pointer":"/identity/seed"} | +| Trials | 2 warm-up + 100 measured | expanded-plan.json#{"pointer":"/data/cells/*/protocol"} | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-2-startup.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-2-startup.md new file mode 100644 index 0000000..6c3efcb --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-2-startup.md @@ -0,0 +1,10 @@ +# EXP1 startup and workspace creation + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Stage | Concurrent creates | Samples | p50 (ms) | p95 (ms) | p99 (ms) | Throughput (ready/s) | +| --- | --- | --- | --- | --- | --- | --- | +| Sandbox create + base mount | 1 | 100 | 1659.811 | 1749.739 | 1794.902 | 0.62 | +| Session create to ready | 1 | 100 | 32.897 | 35.685 | 36.096 | 30.36 | +| Session create to ready | 5 | 100 | 109.119 | 120.107 | 131.588 | 45.32 | +| First no-op command | 1 | 100 | 26.719 | 28.291 | 28.996 | 37.28 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-3-cli-operations.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-3-cli-operations.md new file mode 100644 index 0000000..4bdbc29 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-3-cli-operations.md @@ -0,0 +1,22 @@ +# EXP1 public CLI operations + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Operation | Case | Payload/file size | Concurrency | Samples | p50 (ms) | p95 (ms) | p99 (ms) | Throughput (ops/s) | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `exec_command` | no-op | -- | 1 | 100 | 26.719 | 28.291 | 28.996 | 37.28 | +| `exec_command` | no-op | -- | 5 | 100 | 45.998 | 53.282 | 60.092 | 107.61 | +| `exec_command` | fixture read | 4 KiB | 1 | 100 | 27.235 | 28.938 | 32.666 | 36.44 | +| `exec_command` | fixture read | 4 KiB | 5 | 100 | 45.441 | 50.973 | 56.303 | 108.88 | +| Read | snapshot | 4 KiB | 1 | 100 | 11.425 | 15.773 | 19.964 | 85 | +| Read | snapshot | 4 KiB | 5 | 100 | 26.575 | 30.592 | 39.738 | 185.3 | +| Read | snapshot | 256 KiB | 1 | 100 | 20.781 | 24.367 | 25.484 | 47.68 | +| Read | snapshot | 256 KiB | 5 | 100 | 37.788 | 57.794 | 60.156 | 125.54 | +| Write | session-local | 4 KiB | 1 | 100 | 38.813 | 41.384 | 51.562 | 25.55 | +| Write | session-local | 4 KiB | 5 | 100 | 108.756 | 111.98 | 116.067 | 46 | +| Write | session-local | 256 KiB | 1 | 100 | 45.066 | 49.947 | 53.544 | 22.03 | +| Write | session-local | 256 KiB | 5 | 100 | 125.969 | 137.255 | 147.374 | 39.35 | +| Edit | one replacement | 4 KiB | 1 | 100 | 38.647 | 42.615 | 46.878 | 25.61 | +| Edit | one replacement | 4 KiB | 5 | 100 | 134.61 | 141.758 | 146.725 | 37.22 | +| Edit | one replacement | 256 KiB | 1 | 100 | 48.935 | 53.329 | 63.097 | 20.17 | +| Edit | one replacement | 256 KiB | 5 | 100 | 187.357 | 208.008 | 294.392 | 26.36 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-4-resources.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-4-resources.md new file mode 100644 index 0000000..f4ea7e4 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/table-4-resources.md @@ -0,0 +1,13 @@ +# EXP1 resource observations + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Operation/case | Concurrency | Peak daemon RSS (MiB) | Peak sandbox memory (MiB) | Sandbox CPU (ms/trial) | Block read (MiB/trial) | Block write (MiB/trial) | Workspace allocated delta (MiB) | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Workspace create, 100 MiB/depth 100 | 1 | 25.5 | 28.172 | 30.471 | 0 | 0.011 | 0.004 | +| Workspace create, 100 MiB/depth 100 | 5 | 34.125 | 43.527 | 79.502 | 0 | 0.027 | 0.02 | +| `exec_command`, no-op | 1 | 13.875 | 20.031 | 35.003 | 0 | 0.001 | 0 | +| `exec_command`, no-op | 5 | 13.125 | 21.555 | 63.077 | 0 | 0.003 | 0 | +| Read, 256 KiB | 5 | 14.473 | 14.977 | 38.431 | 0 | 0 | 0 | +| Write, 256 KiB | 5 | 51.043 | 54.27 | 95.115 | 0 | 1.251 | 1.258 | +| Edit, 256 KiB | 5 | 62.602 | 66.715 | 148.732 | 0 | 1.251 | 1.258 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/tables.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/tables.json new file mode 100644 index 0000000..3f0be9c --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-a/tables.json @@ -0,0 +1,841 @@ +{ + "archive": { + "content_tree_sha256": "sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986", + "disposition": "final", + "run_id": "019fb86c-096e-7589-a0a4-a6d6ef5d7f8b" + }, + "eligibility": "frozen_final_candidate", + "generator_schema_version": 2, + "protocol_version": "v1.1", + "schema_version": 2, + "tables": { + "environment": { + "fields": [ + { + "display": "Microsoft Windows 11 ??? build 26200", + "field": "Host OS", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "Microsoft Windows 11 ??? build 26200" + }, + { + "display": "linux", + "field": "Container engine OS", + "selector": { + "pointer": "/docker/os_type" + }, + "source": "environment-preflight.txt", + "value": "linux" + }, + { + "display": "x64", + "field": "Architecture", + "selector": { + "pointer": "/recorded_run_environment/host/architecture" + }, + "source": "environment-preflight.txt", + "value": "x64" + }, + { + "display": "AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors", + "field": "CPU", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors" + }, + { + "display": "137,438,953,472 bytes", + "field": "Memory", + "selector": { + "pointer": "/recorded_run_environment/host/total_memory_bytes" + }, + "source": "environment-preflight.txt", + "value": 137438953472 + }, + { + "display": "NTFS", + "field": "Storage", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "NTFS" + }, + { + "display": "29.0.1", + "field": "Docker Engine", + "selector": { + "pointer": "/docker/server_version" + }, + "source": "environment-preflight.txt", + "value": "29.0.1" + }, + { + "display": "2", + "field": "Cgroup", + "selector": { + "pointer": "/docker/cgroup_version" + }, + "source": "environment-preflight.txt", + "value": "2" + }, + { + "display": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8; annotated tag paper-v1.1-freeze", + "field": "Product commit/tag", + "selector": { + "pointer": "/product" + }, + "source": "campaign-manifest.json", + "value": { + "commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8", + "freeze_tag": { + "availability": "available", + "name": "paper-v1.1-freeze", + "object_type": "tag", + "peeled_commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8", + "reference": "refs/tags/paper-v1.1-freeze", + "tag_object": "834c84534359f37653fb25ac45304091e82c37a6" + } + } + }, + { + "display": "1680b599129532f72e706b6acb12ef62c63759e2", + "field": "Benchmark commit", + "selector": { + "pointer": "/paper_git/commit" + }, + "source": "campaign-manifest.json", + "value": "1680b599129532f72e706b6acb12ef62c63759e2" + }, + { + "display": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + "field": "Sandbox image", + "selector": { + "pointer": "/image/requested" + }, + "source": "campaign-manifest.json", + "value": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + }, + { + "display": "1 vCPU / 512 MiB / 256 PIDs", + "field": "Sandbox limits", + "selector": { + "pointer": "/sandbox_limits" + }, + "source": "environment-preflight.txt", + "value": "1 vCPU / 512 MiB / 256 PIDs" + }, + { + "display": "100 MiB / 4000 files / depth 100", + "field": "Workspace", + "selector": { + "pointer": "/identity/fixture" + }, + "source": "fixture-manifest.json", + "value": "100 MiB / 4000 files / depth 100" + }, + { + "display": "product_cli", + "field": "Client", + "selector": { + "pointer": "/plan/client_cohort" + }, + "source": "campaign-manifest.json", + "value": "product_cli" + }, + { + "display": "windows_named_pipe; local_only; per_execution_block", + "field": "Gateway transport", + "selector": { + "pointer": "/data/environment/gateway_transport", + "protocol_version": "v1.1" + }, + "source": "run-manifest.json", + "value": { + "rotation": "per_execution_block", + "scope": "local_only", + "transport": "windows_named_pipe" + } + }, + { + "display": "20260712", + "field": "Seed", + "selector": { + "pointer": "/identity/seed" + }, + "source": "fixture-manifest.json", + "value": 20260712 + }, + { + "display": "2 warm-up + 100 measured", + "field": "Trials", + "selector": { + "pointer": "/data/cells/*/protocol" + }, + "source": "expanded-plan.json", + "value": "2 warm-up + 100 measured" + } + ] + }, + "public_cli_operations": { + "rows": [ + { + "case": "no-op", + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "Exec no-op", + "operation_label": "`exec_command`", + "payload_or_file_size": "--", + "values": { + "p50_ms": 26.71865, + "p95_ms": 28.29143, + "p99_ms": 28.996465000000022, + "samples": 100, + "throughput_ops_s": 37.28339526491812 + } + }, + { + "case": "no-op", + "cell_id": "sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d", + "concurrency": 5, + "key": [ + "exec_command", + "noop", + 5 + ], + "operation": "Exec no-op", + "operation_label": "`exec_command`", + "payload_or_file_size": "--", + "values": { + "p50_ms": 45.9976, + "p95_ms": 53.281615, + "p99_ms": 60.09194300000015, + "samples": 100, + "throughput_ops_s": 107.61432697082469 + } + }, + { + "case": "fixture read", + "cell_id": "sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf", + "concurrency": 1, + "key": [ + "exec_command", + "fixture_read", + 1 + ], + "operation": "Exec fixture read (4 KiB)", + "operation_label": "`exec_command`", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 27.23455, + "p95_ms": 28.937635, + "p99_ms": 32.66597900000007, + "samples": 100, + "throughput_ops_s": 36.44471318802348 + } + }, + { + "case": "fixture read", + "cell_id": "sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb", + "concurrency": 5, + "key": [ + "exec_command", + "fixture_read", + 5 + ], + "operation": "Exec fixture read (4 KiB)", + "operation_label": "`exec_command`", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 45.44085, + "p95_ms": 50.973254999999995, + "p99_ms": 56.30326700000011, + "samples": 100, + "throughput_ops_s": 108.88291336149418 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16", + "concurrency": 1, + "key": [ + "file_read", + 4096, + 1 + ], + "operation": "Read snapshot (4 KiB)", + "operation_label": "Read", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 11.4248, + "p95_ms": 15.772725, + "p99_ms": 19.96439500000001, + "samples": 100, + "throughput_ops_s": 84.9999254477021 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185", + "concurrency": 5, + "key": [ + "file_read", + 4096, + 5 + ], + "operation": "Read snapshot (4 KiB)", + "operation_label": "Read", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 26.575, + "p95_ms": 30.591984999999998, + "p99_ms": 39.73821200000005, + "samples": 100, + "throughput_ops_s": 185.30259988470985 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0", + "concurrency": 1, + "key": [ + "file_read", + 262144, + 1 + ], + "operation": "Read snapshot (256 KiB)", + "operation_label": "Read", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 20.7811, + "p95_ms": 24.366515, + "p99_ms": 25.48393800000002, + "samples": 100, + "throughput_ops_s": 47.68014601293585 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603", + "concurrency": 5, + "key": [ + "file_read", + 262144, + 5 + ], + "operation": "Read snapshot (256 KiB)", + "operation_label": "Read", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 37.78825, + "p95_ms": 57.79372999999999, + "p99_ms": 60.15640100000002, + "samples": 100, + "throughput_ops_s": 125.54290904967944 + } + }, + { + "case": "session-local", + "cell_id": "sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6", + "concurrency": 1, + "key": [ + "file_write", + 4096, + 1 + ], + "operation": "Write session (4 KiB)", + "operation_label": "Write", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 38.8126, + "p95_ms": 41.38427999999999, + "p99_ms": 51.56190400000005, + "samples": 100, + "throughput_ops_s": 25.551107882015895 + } + }, + { + "case": "session-local", + "cell_id": "sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce", + "concurrency": 5, + "key": [ + "file_write", + 4096, + 5 + ], + "operation": "Write session (4 KiB)", + "operation_label": "Write", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 108.75565, + "p95_ms": 111.979635, + "p99_ms": 116.06709000000002, + "samples": 100, + "throughput_ops_s": 45.9991049224615 + } + }, + { + "case": "session-local", + "cell_id": "sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75", + "concurrency": 1, + "key": [ + "file_write", + 262144, + 1 + ], + "operation": "Write session (256 KiB)", + "operation_label": "Write", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 45.06585, + "p95_ms": 49.94722, + "p99_ms": 53.543718, + "samples": 100, + "throughput_ops_s": 22.02875636155663 + } + }, + { + "case": "session-local", + "cell_id": "sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa", + "concurrency": 5, + "key": [ + "file_write", + 262144, + 5 + ], + "operation": "Write session (256 KiB)", + "operation_label": "Write", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 125.9685, + "p95_ms": 137.255055, + "p99_ms": 147.37442700000005, + "samples": 100, + "throughput_ops_s": 39.354743449708884 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e", + "concurrency": 1, + "key": [ + "file_edit", + 4096, + 1 + ], + "operation": "Edit replacement (4 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 38.6474, + "p95_ms": 42.61549, + "p99_ms": 46.87779200000013, + "samples": 100, + "throughput_ops_s": 25.608367032181523 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e", + "concurrency": 5, + "key": [ + "file_edit", + 4096, + 5 + ], + "operation": "Edit replacement (4 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 134.61035, + "p95_ms": 141.75804, + "p99_ms": 146.72477300000017, + "samples": 100, + "throughput_ops_s": 37.216098524026926 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55", + "concurrency": 1, + "key": [ + "file_edit", + 262144, + 1 + ], + "operation": "Edit replacement (256 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 48.93455, + "p95_ms": 53.328845, + "p99_ms": 63.09657600000165, + "samples": 100, + "throughput_ops_s": 20.171524365907672 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab", + "concurrency": 5, + "key": [ + "file_edit", + 262144, + 5 + ], + "operation": "Edit replacement (256 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 187.35685, + "p95_ms": 208.008075, + "p99_ms": 294.3917890000002, + "samples": 100, + "throughput_ops_s": 26.357148435723182 + } + } + ] + }, + "resources": { + "rows": [ + { + "cell_id": "sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd", + "concurrency": 1, + "key": [ + "create_workspace", + null, + 1 + ], + "operation": "Workspace create, 100 MiB/depth 100", + "values": { + "daemon_rss_bytes": { + "display": "25.5", + "value": 25.5 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.011", + "value": 0.0108203125 + }, + "sandbox_cpu_time_ns": { + "display": "30.471", + "value": 30.4712 + }, + "sandbox_memory_peak_bytes": { + "display": "28.172", + "value": 28.171875 + }, + "upperdir_bytes": { + "display": "0.004", + "value": 0.00390625 + } + } + }, + { + "cell_id": "sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21", + "concurrency": 5, + "key": [ + "create_workspace", + null, + 5 + ], + "operation": "Workspace create, 100 MiB/depth 100", + "values": { + "daemon_rss_bytes": { + "display": "34.125", + "value": 34.125 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.027", + "value": 0.0270703125 + }, + "sandbox_cpu_time_ns": { + "display": "79.502", + "value": 79.50166 + }, + "sandbox_memory_peak_bytes": { + "display": "43.527", + "value": 43.52734375 + }, + "upperdir_bytes": { + "display": "0.02", + "value": 0.01953125 + } + } + }, + { + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "`exec_command`, no-op", + "values": { + "daemon_rss_bytes": { + "display": "13.875", + "value": 13.875 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.001", + "value": 0.001171875 + }, + "sandbox_cpu_time_ns": { + "display": "35.003", + "value": 35.00335 + }, + "sandbox_memory_peak_bytes": { + "display": "20.031", + "value": 20.03125 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d", + "concurrency": 5, + "key": [ + "exec_command", + "noop", + 5 + ], + "operation": "`exec_command`, no-op", + "values": { + "daemon_rss_bytes": { + "display": "13.125", + "value": 13.125 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.003", + "value": 0.0026171875 + }, + "sandbox_cpu_time_ns": { + "display": "63.077", + "value": 63.07729 + }, + "sandbox_memory_peak_bytes": { + "display": "21.555", + "value": 21.5546875 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603", + "concurrency": 5, + "key": [ + "file_read", + 262144, + 5 + ], + "operation": "Read, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "14.473", + "value": 14.47265625 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_cpu_time_ns": { + "display": "38.431", + "value": 38.43103 + }, + "sandbox_memory_peak_bytes": { + "display": "14.977", + "value": 14.9765625 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa", + "concurrency": 5, + "key": [ + "file_write", + 262144, + 5 + ], + "operation": "Write, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "51.043", + "value": 51.04296875 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "1.251", + "value": 1.250625 + }, + "sandbox_cpu_time_ns": { + "display": "95.115", + "value": 95.11521 + }, + "sandbox_memory_peak_bytes": { + "display": "54.27", + "value": 54.26953125 + }, + "upperdir_bytes": { + "display": "1.258", + "value": 1.2578125 + } + } + }, + { + "cell_id": "sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab", + "concurrency": 5, + "key": [ + "file_edit", + 262144, + 5 + ], + "operation": "Edit, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "62.602", + "value": 62.6015625 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "1.251", + "value": 1.2505859375 + }, + "sandbox_cpu_time_ns": { + "display": "148.732", + "value": 148.73234 + }, + "sandbox_memory_peak_bytes": { + "display": "66.715", + "value": 66.71484375 + }, + "upperdir_bytes": { + "display": "1.258", + "value": 1.2578125 + } + } + } + ] + }, + "startup": { + "rows": [ + { + "cell_id": "sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2", + "concurrency": 1, + "key": [ + "create_sandbox", + null, + 1 + ], + "operation": "Create sandbox", + "stage": "Sandbox create + base mount", + "values": { + "p50_ms": 1659.8107, + "p95_ms": 1749.7388, + "p99_ms": 1794.9019450000008, + "samples": 100, + "throughput_ops_s": 0.6185790825907366 + } + }, + { + "cell_id": "sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd", + "concurrency": 1, + "key": [ + "create_workspace", + null, + 1 + ], + "operation": "Create workspace (session)", + "stage": "Session create to ready", + "values": { + "p50_ms": 32.89655, + "p95_ms": 35.6847, + "p99_ms": 36.096376, + "samples": 100, + "throughput_ops_s": 30.359842997206 + } + }, + { + "cell_id": "sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21", + "concurrency": 5, + "key": [ + "create_workspace", + null, + 5 + ], + "operation": "Create workspace (session)", + "stage": "Session create to ready", + "values": { + "p50_ms": 109.11905, + "p95_ms": 120.107005, + "p99_ms": 131.58766000000003, + "samples": 100, + "throughput_ops_s": 45.3204753420739 + } + }, + { + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "Exec no-op", + "stage": "First no-op command", + "values": { + "p50_ms": 26.71865, + "p95_ms": 28.29143, + "p99_ms": 28.996465000000022, + "samples": 100, + "throughput_ops_s": 37.28339526491812 + } + } + ] + } + } +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/generation-log.txt b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/generation-log.txt new file mode 100644 index 0000000..5cc008d --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/generation-log.txt @@ -0,0 +1,14 @@ +EXP1 deterministic table generation +generator_schema_version=2 +generator_sha256=sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b +archive_run_id=019fb86c-096e-7589-a0a4-a6d6ef5d7f8b +archive_disposition=final +archive_eligibility=frozen_final_candidate +protocol_version=v1.1 +archive_content_tree_sha256=sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986 +archive_inventory_verified=true +semantic_identity_verified=true +correctness_verified=true +command=python experiments/analysis/scripts/generate_exp1_tables.py --protocol-version --archive --output +output_path_embedded=false +wall_clock_embedded=false diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-evidence.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-evidence.json new file mode 100644 index 0000000..574d0c4 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-evidence.json @@ -0,0 +1,2606 @@ +{ + "entries": [ + { + "aggregate": "identity", + "note": "table1.cpu.logical_processors; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/logical_processors\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.cpu.logical_processors" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 48.0 + }, + { + "aggregate": "identity", + "note": "table1.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/total_memory_bytes\"}; units=bytes->bytes; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.memory_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 137438953472.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/memory_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.memory_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 512.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.pids_limit; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/pids_limit\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.pids_limit" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 256.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.vcpus; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/vcpus\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.vcpus" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.0 + }, + { + "aggregate": "identity", + "note": "table1.seed; upstream=fixture-manifest.json#{\"pointer\":\"/identity/seed\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.seed" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20260712.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.measured; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/measured_trials\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.measured" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.warmups; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/warmups\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.warmups" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 2.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.file_count; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/file_count\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.file_count" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 4000.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.logical_mib; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/logical_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.logical_mib" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.maximum_depth; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/maximum_depth\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.maximum_depth" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1659.8107 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1749.7388 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1794.9019450000008 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.6185790825907366 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 32.89655 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 35.6847 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 36.096376 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.359842997206 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 109.11905 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 120.107005 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 131.58766000000003 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.3204753420739 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 27.23455 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.937635 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 32.66597900000007 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 36.44471318802348 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.44085 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 50.973254999999995 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 56.30326700000011 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 108.88291336149418 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.9976 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.281615 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 60.09194300000015 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 107.61432697082469 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 48.93455 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.328845 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 63.09657600000165 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.171524365907672 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 187.35685 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 208.008075 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 294.3917890000002 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.357148435723182 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.6474 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 42.61549 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 46.87779200000013 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.608367032181523 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 134.61035 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 141.75804 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 146.72477300000017 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.216098524026926 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.7811 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 24.366515 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.48393800000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 47.68014601293585 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 37.78825 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 57.79372999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 60.15640100000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 125.54290904967944 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 11.4248 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 15.772725 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 19.96439500000001 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 84.9999254477021 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 26.575 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.591984999999998 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 39.73821200000005 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 185.30259988470985 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.06585 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 49.94722 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 53.543718 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 22.02875636155663 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 125.9685 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 137.255055 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 147.37442700000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 39.354743449708884 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.8126 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 41.38427999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 51.56190400000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.551107882015895 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p50_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 108.75565 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p95_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 111.979635 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p99_ms" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 116.06709000000002 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.samples" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.throughput_ops_s" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 45.9991049224615 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 25.5 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0108203125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 30.4712 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 28.171875 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.00390625 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 34.125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0270703125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 79.50166 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 43.52734375 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.01953125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 13.875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.001171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 35.00335 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 20.03125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 13.125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0026171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 63.07729 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 21.5546875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 62.6015625 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2505859375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 148.73234 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 66.71484375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2578125 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 14.47265625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 38.43103 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 14.9765625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.daemon_rss_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 51.04296875 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.250625 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 95.11521 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 54.26953125 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.upperdir_bytes" + } + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": 1.2578125 + } + ], + "schema_version": "ai-research-writing/numeric-evidence-v2" +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-provenance.csv b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-provenance.csv new file mode 100644 index 0000000..3bbf824 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/numeric-provenance.csv @@ -0,0 +1,154 @@ +evidence_id,value,source_file,source_sha256,selector_kind,selector_json,input_unit,output_unit,aggregation,aggregation_parameters_json,eligibility +table1.cpu.logical_processors,48,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/recorded_run_environment/host/logical_processors""}",count,count,identity,{},frozen_final_candidate +table1.memory_bytes,137438953472,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/recorded_run_environment/host/total_memory_bytes""}",bytes,bytes,identity,{},frozen_final_candidate +table1.sandbox_limits.memory_bytes,512,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/memory_bytes""}",bytes,MiB,identity,"{""divisor"":1048576}",frozen_final_candidate +table1.sandbox_limits.pids_limit,256,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/pids_limit""}",count,count,identity,{},frozen_final_candidate +table1.sandbox_limits.vcpus,1,environment-preflight.txt,sha256:064fa1a6d5a372d25b2a1aa1d886369719bd4da3139bb11c5db932ecf3504281,archive-json,"{""pointer"":""/sandbox_limits/vcpus""}",count,count,identity,{},frozen_final_candidate +table1.seed,20260712,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/seed""}",count,count,identity,{},frozen_final_candidate +table1.trials.measured,100,expanded-plan.json,sha256:fa04fe9254e071ea6c65f906b9e62854a60330d0fdca8c41cb41297f3c923b2e,archive-json,"{""invariant"":""all equal"",""pointer"":""/data/cells/*/protocol/measured_trials""}",count,count,identity,{},frozen_final_candidate +table1.trials.warmups,2,expanded-plan.json,sha256:fa04fe9254e071ea6c65f906b9e62854a60330d0fdca8c41cb41297f3c923b2e,archive-json,"{""invariant"":""all equal"",""pointer"":""/data/cells/*/protocol/warmups""}",count,count,identity,{},frozen_final_candidate +table1.workspace_fixture.file_count,4000,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/file_count""}",count,count,identity,{},frozen_final_candidate +table1.workspace_fixture.logical_mib,100,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/logical_bytes""}",bytes,MiB,identity,"{""divisor"":1048576}",frozen_final_candidate +table1.workspace_fixture.maximum_depth,100,fixture-manifest.json,sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1,archive-json,"{""pointer"":""/identity/fixture/maximum_depth""}",count,count,identity,{},frozen_final_candidate +table2.create_sandbox.none.c1.p50_ms,1659.8107,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.p95_ms,1749.7388000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.p99_ms,1794.9019450000008,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_sandbox.none.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_sandbox.none.c1.throughput_ops_s,0.61857908259073657,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.create_workspace.none.c1.p50_ms,32.896549999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.p95_ms,35.684699999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.p99_ms,36.096375999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_workspace.none.c1.throughput_ops_s,30.359842997206002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.create_workspace.none.c5.p50_ms,109.11905,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.p95_ms,120.107005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.p99_ms,131.58766000000003,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.create_workspace.none.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.create_workspace.none.c5.throughput_ops_s,45.320475342073898,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table2.exec_command.noop.c1.p50_ms,26.71865,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.p95_ms,28.291429999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.p99_ms,28.996465000000022,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table2.exec_command.noop.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table2.exec_command.noop.c1.throughput_ops_s,37.28339526491812,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p50_ms,27.234549999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p95_ms,28.937635,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.p99_ms,32.665979000000071,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.fixture_read.c1.throughput_ops_s,36.444713188023478,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p50_ms,45.440849999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p95_ms,50.973254999999995,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.p99_ms,56.303267000000112,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.fixture_read.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.fixture_read.c5.throughput_ops_s,108.88291336149418,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.noop.c1.p50_ms,26.71865,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.p95_ms,28.291429999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.p99_ms,28.996465000000022,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.noop.c1.throughput_ops_s,37.28339526491812,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.exec_command.noop.c5.p50_ms,45.997599999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.p95_ms,53.281615000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.p99_ms,60.09194300000015,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.exec_command.noop.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.exec_command.noop.c5.throughput_ops_s,107.61432697082469,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.262144.c1.p50_ms,48.934550000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.p95_ms,53.328845000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.p99_ms,63.096576000001647,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.262144.c1.throughput_ops_s,20.171524365907672,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.262144.c5.p50_ms,187.35685000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.p95_ms,208.00807499999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.p99_ms,294.39178900000019,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.262144.c5.throughput_ops_s,26.357148435723182,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.4096.c1.p50_ms,38.647399999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.p95_ms,42.615490000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.p99_ms,46.877792000000127,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.4096.c1.throughput_ops_s,25.608367032181523,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_edit.4096.c5.p50_ms,134.61035000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.p95_ms,141.75803999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.p99_ms,146.72477300000017,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_edit.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_edit.4096.c5.throughput_ops_s,37.216098524026926,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.262144.c1.p50_ms,20.781099999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.p95_ms,24.366515,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.p99_ms,25.48393800000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.262144.c1.throughput_ops_s,47.68014601293585,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.262144.c5.p50_ms,37.788249999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.p95_ms,57.793729999999989,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.p99_ms,60.156401000000017,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.262144.c5.throughput_ops_s,125.54290904967944,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.4096.c1.p50_ms,11.424799999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.p95_ms,15.772724999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.p99_ms,19.96439500000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.4096.c1.throughput_ops_s,84.999925447702097,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_read.4096.c5.p50_ms,26.574999999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.p95_ms,30.591984999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.p99_ms,39.738212000000047,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_read.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_read.4096.c5.throughput_ops_s,185.30259988470985,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.262144.c1.p50_ms,45.065849999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.p95_ms,49.947220000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.p99_ms,53.543717999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.262144.c1.throughput_ops_s,22.028756361556631,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.262144.c5.p50_ms,125.96850000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.p95_ms,137.255055,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.p99_ms,147.37442700000005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.262144.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.262144.c5.throughput_ops_s,39.354743449708884,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.4096.c1.p50_ms,38.812600000000003,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.p95_ms,41.38427999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.p99_ms,51.561904000000048,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c1.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.4096.c1.throughput_ops_s,25.551107882015895,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table3.file_write.4096.c5.p50_ms,108.75565,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p50,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.5,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.p95_ms,111.979635,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p95,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.95,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.p99_ms,116.06709000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,linear_quantile_p99,"{""interpolation"":""linear"",""position"":""(n-1)*q"",""probability"":0.99,""unit_divisor"":1000000}",frozen_final_candidate +table3.file_write.4096.c5.samples,100,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""available_n"",""metric_id"":""batch_makespan_ns"",""trial_scope"":""reportable_measured""}",count,count,identity,{},frozen_final_candidate +table3.file_write.4096.c5.throughput_ops_s,45.999104922461498,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce"",""field"":""raw_points[].value"",""metric_id"":""throughput_ops_s"",""trial_scope"":""reportable_measured""}",operations_per_second,operations_per_second,arithmetic_mean,"{""denominator"":100}",frozen_final_candidate +table4.create_workspace.none.c1.daemon_rss_bytes,25.5,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_block_write_bytes,0.0108203125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_cpu_time_ns,30.4712,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.create_workspace.none.c1.sandbox_memory_peak_bytes,28.171875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c1.upperdir_bytes,0.00390625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.daemon_rss_bytes,34.125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_block_write_bytes,0.027070312499999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_cpu_time_ns,79.501660000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.create_workspace.none.c5.sandbox_memory_peak_bytes,43.52734375,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.create_workspace.none.c5.upperdir_bytes,0.01953125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.daemon_rss_bytes,13.875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_block_write_bytes,0.001171875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_cpu_time_ns,35.003349999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.exec_command.noop.c1.sandbox_memory_peak_bytes,20.03125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c1.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.daemon_rss_bytes,13.125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_block_write_bytes,0.0026171875000000002,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_cpu_time_ns,63.077289999999998,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.exec_command.noop.c5.sandbox_memory_peak_bytes,21.5546875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.exec_command.noop.c5.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.daemon_rss_bytes,62.6015625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_block_write_bytes,1.2505859374999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_cpu_time_ns,148.73233999999999,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_edit.262144.c5.sandbox_memory_peak_bytes,66.71484375,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_edit.262144.c5.upperdir_bytes,1.2578125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.daemon_rss_bytes,14.47265625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_block_write_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_cpu_time_ns,38.43103,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_read.262144.c5.sandbox_memory_peak_bytes,14.9765625,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_read.262144.c5.upperdir_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.daemon_rss_bytes,51.04296875,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""daemon_rss_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_block_read_bytes,0,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_read_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_block_write_bytes,1.2506250000000001,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_block_write_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_cpu_time_ns,95.115210000000005,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_cpu_time_ns"",""trial_scope"":""reportable_measured""}",nanoseconds,milliseconds,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1000000}",frozen_final_candidate +table4.file_write.262144.c5.sandbox_memory_peak_bytes,54.26953125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""sandbox_memory_peak_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,maximum,"{""denominator"":null,""unit_divisor"":1048576}",frozen_final_candidate +table4.file_write.262144.c5.upperdir_bytes,1.2578125,report.json,sha256:ebea3dc4119919969370897b64ea004cbc9555e3403463bc44cba534461d2982,archive-json,"{""artifact"":""report.json"",""cell_id"":""sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa"",""field"":""raw_points[].value"",""metric_id"":""upperdir_bytes"",""trial_scope"":""reportable_measured""}",bytes,MiB,arithmetic_mean,"{""denominator"":100,""unit_divisor"":1048576}",frozen_final_candidate diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/output-manifest.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/output-manifest.json new file mode 100644 index 0000000..0ee38c7 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/output-manifest.json @@ -0,0 +1,52 @@ +{ + "archive_content_tree_sha256": "sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986", + "archive_disposition": "final", + "archive_run_id": "019fb86c-096e-7589-a0a4-a6d6ef5d7f8b", + "eligibility": "frozen_final_candidate", + "files": [ + { + "bytes": 673, + "path": "generation-log.txt", + "sha256": "sha256:481f89a1900a051649102b986ac3ad6a7083b53959cfa4a9bcf419e383d3d0b3" + }, + { + "bytes": 123328, + "path": "numeric-evidence.json", + "sha256": "sha256:8a247f492b5bbf5bb35f1a89e0ed7dc3effcaaab0f935b746e1dcd89d62a367c" + }, + { + "bytes": 74608, + "path": "numeric-provenance.csv", + "sha256": "sha256:e967539c2b62af1ab6c5c369c5825604ea545a2090b481544e74d9f174bff3d1" + }, + { + "bytes": 2155, + "path": "table-1-environment.md", + "sha256": "sha256:ebce68ac85292844cd1f397494211e3fe44250d9b5bf7dae019b62d10993b575" + }, + { + "bytes": 612, + "path": "table-2-startup.md", + "sha256": "sha256:9887ffedc5dd1bd98c5385bd2470cc2aa5e11bb07843a3bc4c8058a9601bdea6" + }, + { + "bytes": 1612, + "path": "table-3-cli-operations.md", + "sha256": "sha256:6c194fb477a8279c31cf45f498954fdb80d7751328d1b4bef0f222f643410b80" + }, + { + "bytes": 932, + "path": "table-4-resources.md", + "sha256": "sha256:ed3fd61377ab075fc870194769919b138432053e0906b05a4d1c96cd5c6c1d78" + }, + { + "bytes": 25385, + "path": "tables.json", + "sha256": "sha256:c8aaa13c58d0dad900f3d08d6a926d7736c281bacdb665e3976da269cbdab3dd" + } + ], + "generator_schema_version": 2, + "generator_sha256": "sha256:ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b", + "protocol_version": "v1.1", + "schema_version": 2 +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-1-environment.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-1-environment.md new file mode 100644 index 0000000..9ee6b23 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-1-environment.md @@ -0,0 +1,23 @@ +# EXP1 environment and workload + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Field | Archived value | Evidence source | +| --- | --- | --- | +| Host OS | Microsoft Windows 11 ??? build 26200 | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Container engine OS | linux | environment-preflight.txt#{"pointer":"/docker/os_type"} | +| Architecture | x64 | environment-preflight.txt#{"pointer":"/recorded_run_environment/host/architecture"} | +| CPU | AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Memory | 137,438,953,472 bytes | environment-preflight.txt#{"pointer":"/recorded_run_environment/host/total_memory_bytes"} | +| Storage | NTFS | environment-preflight.txt#{"pointer":"/recorded_run_environment/host"} | +| Docker Engine | 29.0.1 | environment-preflight.txt#{"pointer":"/docker/server_version"} | +| Cgroup | 2 | environment-preflight.txt#{"pointer":"/docker/cgroup_version"} | +| Product commit/tag | 5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8; annotated tag paper-v1.1-freeze | campaign-manifest.json#{"pointer":"/product"} | +| Benchmark commit | 1680b599129532f72e706b6acb12ef62c63759e2 | campaign-manifest.json#{"pointer":"/paper_git/commit"} | +| Sandbox image | ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf | campaign-manifest.json#{"pointer":"/image/requested"} | +| Sandbox limits | 1 vCPU / 512 MiB / 256 PIDs | environment-preflight.txt#{"pointer":"/sandbox_limits"} | +| Workspace | 100 MiB / 4000 files / depth 100 | fixture-manifest.json#{"pointer":"/identity/fixture"} | +| Client | product_cli | campaign-manifest.json#{"pointer":"/plan/client_cohort"} | +| Gateway transport | windows_named_pipe; local_only; per_execution_block | run-manifest.json#{"pointer":"/data/environment/gateway_transport","protocol_version":"v1.1"} | +| Seed | 20260712 | fixture-manifest.json#{"pointer":"/identity/seed"} | +| Trials | 2 warm-up + 100 measured | expanded-plan.json#{"pointer":"/data/cells/*/protocol"} | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-2-startup.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-2-startup.md new file mode 100644 index 0000000..6c3efcb --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-2-startup.md @@ -0,0 +1,10 @@ +# EXP1 startup and workspace creation + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Stage | Concurrent creates | Samples | p50 (ms) | p95 (ms) | p99 (ms) | Throughput (ready/s) | +| --- | --- | --- | --- | --- | --- | --- | +| Sandbox create + base mount | 1 | 100 | 1659.811 | 1749.739 | 1794.902 | 0.62 | +| Session create to ready | 1 | 100 | 32.897 | 35.685 | 36.096 | 30.36 | +| Session create to ready | 5 | 100 | 109.119 | 120.107 | 131.588 | 45.32 | +| First no-op command | 1 | 100 | 26.719 | 28.291 | 28.996 | 37.28 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-3-cli-operations.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-3-cli-operations.md new file mode 100644 index 0000000..4bdbc29 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-3-cli-operations.md @@ -0,0 +1,22 @@ +# EXP1 public CLI operations + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Operation | Case | Payload/file size | Concurrency | Samples | p50 (ms) | p95 (ms) | p99 (ms) | Throughput (ops/s) | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `exec_command` | no-op | -- | 1 | 100 | 26.719 | 28.291 | 28.996 | 37.28 | +| `exec_command` | no-op | -- | 5 | 100 | 45.998 | 53.282 | 60.092 | 107.61 | +| `exec_command` | fixture read | 4 KiB | 1 | 100 | 27.235 | 28.938 | 32.666 | 36.44 | +| `exec_command` | fixture read | 4 KiB | 5 | 100 | 45.441 | 50.973 | 56.303 | 108.88 | +| Read | snapshot | 4 KiB | 1 | 100 | 11.425 | 15.773 | 19.964 | 85 | +| Read | snapshot | 4 KiB | 5 | 100 | 26.575 | 30.592 | 39.738 | 185.3 | +| Read | snapshot | 256 KiB | 1 | 100 | 20.781 | 24.367 | 25.484 | 47.68 | +| Read | snapshot | 256 KiB | 5 | 100 | 37.788 | 57.794 | 60.156 | 125.54 | +| Write | session-local | 4 KiB | 1 | 100 | 38.813 | 41.384 | 51.562 | 25.55 | +| Write | session-local | 4 KiB | 5 | 100 | 108.756 | 111.98 | 116.067 | 46 | +| Write | session-local | 256 KiB | 1 | 100 | 45.066 | 49.947 | 53.544 | 22.03 | +| Write | session-local | 256 KiB | 5 | 100 | 125.969 | 137.255 | 147.374 | 39.35 | +| Edit | one replacement | 4 KiB | 1 | 100 | 38.647 | 42.615 | 46.878 | 25.61 | +| Edit | one replacement | 4 KiB | 5 | 100 | 134.61 | 141.758 | 146.725 | 37.22 | +| Edit | one replacement | 256 KiB | 1 | 100 | 48.935 | 53.329 | 63.097 | 20.17 | +| Edit | one replacement | 256 KiB | 5 | 100 | 187.357 | 208.008 | 294.392 | 26.36 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-4-resources.md b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-4-resources.md new file mode 100644 index 0000000..f4ea7e4 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/table-4-resources.md @@ -0,0 +1,13 @@ +# EXP1 resource observations + +> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only after the remaining paper evidence/build gates pass. + +| Operation/case | Concurrency | Peak daemon RSS (MiB) | Peak sandbox memory (MiB) | Sandbox CPU (ms/trial) | Block read (MiB/trial) | Block write (MiB/trial) | Workspace allocated delta (MiB) | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Workspace create, 100 MiB/depth 100 | 1 | 25.5 | 28.172 | 30.471 | 0 | 0.011 | 0.004 | +| Workspace create, 100 MiB/depth 100 | 5 | 34.125 | 43.527 | 79.502 | 0 | 0.027 | 0.02 | +| `exec_command`, no-op | 1 | 13.875 | 20.031 | 35.003 | 0 | 0.001 | 0 | +| `exec_command`, no-op | 5 | 13.125 | 21.555 | 63.077 | 0 | 0.003 | 0 | +| Read, 256 KiB | 5 | 14.473 | 14.977 | 38.431 | 0 | 0 | 0 | +| Write, 256 KiB | 5 | 51.043 | 54.27 | 95.115 | 0 | 1.251 | 1.258 | +| Edit, 256 KiB | 5 | 62.602 | 66.715 | 148.732 | 0 | 1.251 | 1.258 | diff --git a/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/tables.json b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/tables.json new file mode 100644 index 0000000..3f0be9c --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/final-v11-019fb86c-tables-b/tables.json @@ -0,0 +1,841 @@ +{ + "archive": { + "content_tree_sha256": "sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986", + "disposition": "final", + "run_id": "019fb86c-096e-7589-a0a4-a6d6ef5d7f8b" + }, + "eligibility": "frozen_final_candidate", + "generator_schema_version": 2, + "protocol_version": "v1.1", + "schema_version": 2, + "tables": { + "environment": { + "fields": [ + { + "display": "Microsoft Windows 11 ??? build 26200", + "field": "Host OS", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "Microsoft Windows 11 ??? build 26200" + }, + { + "display": "linux", + "field": "Container engine OS", + "selector": { + "pointer": "/docker/os_type" + }, + "source": "environment-preflight.txt", + "value": "linux" + }, + { + "display": "x64", + "field": "Architecture", + "selector": { + "pointer": "/recorded_run_environment/host/architecture" + }, + "source": "environment-preflight.txt", + "value": "x64" + }, + { + "display": "AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors", + "field": "CPU", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "AMD Ryzen Threadripper 7960X 24-Cores / 48 logical processors" + }, + { + "display": "137,438,953,472 bytes", + "field": "Memory", + "selector": { + "pointer": "/recorded_run_environment/host/total_memory_bytes" + }, + "source": "environment-preflight.txt", + "value": 137438953472 + }, + { + "display": "NTFS", + "field": "Storage", + "selector": { + "pointer": "/recorded_run_environment/host" + }, + "source": "environment-preflight.txt", + "value": "NTFS" + }, + { + "display": "29.0.1", + "field": "Docker Engine", + "selector": { + "pointer": "/docker/server_version" + }, + "source": "environment-preflight.txt", + "value": "29.0.1" + }, + { + "display": "2", + "field": "Cgroup", + "selector": { + "pointer": "/docker/cgroup_version" + }, + "source": "environment-preflight.txt", + "value": "2" + }, + { + "display": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8; annotated tag paper-v1.1-freeze", + "field": "Product commit/tag", + "selector": { + "pointer": "/product" + }, + "source": "campaign-manifest.json", + "value": { + "commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8", + "freeze_tag": { + "availability": "available", + "name": "paper-v1.1-freeze", + "object_type": "tag", + "peeled_commit": "5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8", + "reference": "refs/tags/paper-v1.1-freeze", + "tag_object": "834c84534359f37653fb25ac45304091e82c37a6" + } + } + }, + { + "display": "1680b599129532f72e706b6acb12ef62c63759e2", + "field": "Benchmark commit", + "selector": { + "pointer": "/paper_git/commit" + }, + "source": "campaign-manifest.json", + "value": "1680b599129532f72e706b6acb12ef62c63759e2" + }, + { + "display": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + "field": "Sandbox image", + "selector": { + "pointer": "/image/requested" + }, + "source": "campaign-manifest.json", + "value": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + }, + { + "display": "1 vCPU / 512 MiB / 256 PIDs", + "field": "Sandbox limits", + "selector": { + "pointer": "/sandbox_limits" + }, + "source": "environment-preflight.txt", + "value": "1 vCPU / 512 MiB / 256 PIDs" + }, + { + "display": "100 MiB / 4000 files / depth 100", + "field": "Workspace", + "selector": { + "pointer": "/identity/fixture" + }, + "source": "fixture-manifest.json", + "value": "100 MiB / 4000 files / depth 100" + }, + { + "display": "product_cli", + "field": "Client", + "selector": { + "pointer": "/plan/client_cohort" + }, + "source": "campaign-manifest.json", + "value": "product_cli" + }, + { + "display": "windows_named_pipe; local_only; per_execution_block", + "field": "Gateway transport", + "selector": { + "pointer": "/data/environment/gateway_transport", + "protocol_version": "v1.1" + }, + "source": "run-manifest.json", + "value": { + "rotation": "per_execution_block", + "scope": "local_only", + "transport": "windows_named_pipe" + } + }, + { + "display": "20260712", + "field": "Seed", + "selector": { + "pointer": "/identity/seed" + }, + "source": "fixture-manifest.json", + "value": 20260712 + }, + { + "display": "2 warm-up + 100 measured", + "field": "Trials", + "selector": { + "pointer": "/data/cells/*/protocol" + }, + "source": "expanded-plan.json", + "value": "2 warm-up + 100 measured" + } + ] + }, + "public_cli_operations": { + "rows": [ + { + "case": "no-op", + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "Exec no-op", + "operation_label": "`exec_command`", + "payload_or_file_size": "--", + "values": { + "p50_ms": 26.71865, + "p95_ms": 28.29143, + "p99_ms": 28.996465000000022, + "samples": 100, + "throughput_ops_s": 37.28339526491812 + } + }, + { + "case": "no-op", + "cell_id": "sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d", + "concurrency": 5, + "key": [ + "exec_command", + "noop", + 5 + ], + "operation": "Exec no-op", + "operation_label": "`exec_command`", + "payload_or_file_size": "--", + "values": { + "p50_ms": 45.9976, + "p95_ms": 53.281615, + "p99_ms": 60.09194300000015, + "samples": 100, + "throughput_ops_s": 107.61432697082469 + } + }, + { + "case": "fixture read", + "cell_id": "sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf", + "concurrency": 1, + "key": [ + "exec_command", + "fixture_read", + 1 + ], + "operation": "Exec fixture read (4 KiB)", + "operation_label": "`exec_command`", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 27.23455, + "p95_ms": 28.937635, + "p99_ms": 32.66597900000007, + "samples": 100, + "throughput_ops_s": 36.44471318802348 + } + }, + { + "case": "fixture read", + "cell_id": "sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb", + "concurrency": 5, + "key": [ + "exec_command", + "fixture_read", + 5 + ], + "operation": "Exec fixture read (4 KiB)", + "operation_label": "`exec_command`", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 45.44085, + "p95_ms": 50.973254999999995, + "p99_ms": 56.30326700000011, + "samples": 100, + "throughput_ops_s": 108.88291336149418 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16", + "concurrency": 1, + "key": [ + "file_read", + 4096, + 1 + ], + "operation": "Read snapshot (4 KiB)", + "operation_label": "Read", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 11.4248, + "p95_ms": 15.772725, + "p99_ms": 19.96439500000001, + "samples": 100, + "throughput_ops_s": 84.9999254477021 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185", + "concurrency": 5, + "key": [ + "file_read", + 4096, + 5 + ], + "operation": "Read snapshot (4 KiB)", + "operation_label": "Read", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 26.575, + "p95_ms": 30.591984999999998, + "p99_ms": 39.73821200000005, + "samples": 100, + "throughput_ops_s": 185.30259988470985 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0", + "concurrency": 1, + "key": [ + "file_read", + 262144, + 1 + ], + "operation": "Read snapshot (256 KiB)", + "operation_label": "Read", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 20.7811, + "p95_ms": 24.366515, + "p99_ms": 25.48393800000002, + "samples": 100, + "throughput_ops_s": 47.68014601293585 + } + }, + { + "case": "snapshot", + "cell_id": "sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603", + "concurrency": 5, + "key": [ + "file_read", + 262144, + 5 + ], + "operation": "Read snapshot (256 KiB)", + "operation_label": "Read", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 37.78825, + "p95_ms": 57.79372999999999, + "p99_ms": 60.15640100000002, + "samples": 100, + "throughput_ops_s": 125.54290904967944 + } + }, + { + "case": "session-local", + "cell_id": "sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6", + "concurrency": 1, + "key": [ + "file_write", + 4096, + 1 + ], + "operation": "Write session (4 KiB)", + "operation_label": "Write", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 38.8126, + "p95_ms": 41.38427999999999, + "p99_ms": 51.56190400000005, + "samples": 100, + "throughput_ops_s": 25.551107882015895 + } + }, + { + "case": "session-local", + "cell_id": "sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce", + "concurrency": 5, + "key": [ + "file_write", + 4096, + 5 + ], + "operation": "Write session (4 KiB)", + "operation_label": "Write", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 108.75565, + "p95_ms": 111.979635, + "p99_ms": 116.06709000000002, + "samples": 100, + "throughput_ops_s": 45.9991049224615 + } + }, + { + "case": "session-local", + "cell_id": "sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75", + "concurrency": 1, + "key": [ + "file_write", + 262144, + 1 + ], + "operation": "Write session (256 KiB)", + "operation_label": "Write", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 45.06585, + "p95_ms": 49.94722, + "p99_ms": 53.543718, + "samples": 100, + "throughput_ops_s": 22.02875636155663 + } + }, + { + "case": "session-local", + "cell_id": "sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa", + "concurrency": 5, + "key": [ + "file_write", + 262144, + 5 + ], + "operation": "Write session (256 KiB)", + "operation_label": "Write", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 125.9685, + "p95_ms": 137.255055, + "p99_ms": 147.37442700000005, + "samples": 100, + "throughput_ops_s": 39.354743449708884 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e", + "concurrency": 1, + "key": [ + "file_edit", + 4096, + 1 + ], + "operation": "Edit replacement (4 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 38.6474, + "p95_ms": 42.61549, + "p99_ms": 46.87779200000013, + "samples": 100, + "throughput_ops_s": 25.608367032181523 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e", + "concurrency": 5, + "key": [ + "file_edit", + 4096, + 5 + ], + "operation": "Edit replacement (4 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "4 KiB", + "values": { + "p50_ms": 134.61035, + "p95_ms": 141.75804, + "p99_ms": 146.72477300000017, + "samples": 100, + "throughput_ops_s": 37.216098524026926 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55", + "concurrency": 1, + "key": [ + "file_edit", + 262144, + 1 + ], + "operation": "Edit replacement (256 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 48.93455, + "p95_ms": 53.328845, + "p99_ms": 63.09657600000165, + "samples": 100, + "throughput_ops_s": 20.171524365907672 + } + }, + { + "case": "one replacement", + "cell_id": "sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab", + "concurrency": 5, + "key": [ + "file_edit", + 262144, + 5 + ], + "operation": "Edit replacement (256 KiB)", + "operation_label": "Edit", + "payload_or_file_size": "256 KiB", + "values": { + "p50_ms": 187.35685, + "p95_ms": 208.008075, + "p99_ms": 294.3917890000002, + "samples": 100, + "throughput_ops_s": 26.357148435723182 + } + } + ] + }, + "resources": { + "rows": [ + { + "cell_id": "sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd", + "concurrency": 1, + "key": [ + "create_workspace", + null, + 1 + ], + "operation": "Workspace create, 100 MiB/depth 100", + "values": { + "daemon_rss_bytes": { + "display": "25.5", + "value": 25.5 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.011", + "value": 0.0108203125 + }, + "sandbox_cpu_time_ns": { + "display": "30.471", + "value": 30.4712 + }, + "sandbox_memory_peak_bytes": { + "display": "28.172", + "value": 28.171875 + }, + "upperdir_bytes": { + "display": "0.004", + "value": 0.00390625 + } + } + }, + { + "cell_id": "sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21", + "concurrency": 5, + "key": [ + "create_workspace", + null, + 5 + ], + "operation": "Workspace create, 100 MiB/depth 100", + "values": { + "daemon_rss_bytes": { + "display": "34.125", + "value": 34.125 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.027", + "value": 0.0270703125 + }, + "sandbox_cpu_time_ns": { + "display": "79.502", + "value": 79.50166 + }, + "sandbox_memory_peak_bytes": { + "display": "43.527", + "value": 43.52734375 + }, + "upperdir_bytes": { + "display": "0.02", + "value": 0.01953125 + } + } + }, + { + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "`exec_command`, no-op", + "values": { + "daemon_rss_bytes": { + "display": "13.875", + "value": 13.875 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.001", + "value": 0.001171875 + }, + "sandbox_cpu_time_ns": { + "display": "35.003", + "value": 35.00335 + }, + "sandbox_memory_peak_bytes": { + "display": "20.031", + "value": 20.03125 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d", + "concurrency": 5, + "key": [ + "exec_command", + "noop", + 5 + ], + "operation": "`exec_command`, no-op", + "values": { + "daemon_rss_bytes": { + "display": "13.125", + "value": 13.125 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0.003", + "value": 0.0026171875 + }, + "sandbox_cpu_time_ns": { + "display": "63.077", + "value": 63.07729 + }, + "sandbox_memory_peak_bytes": { + "display": "21.555", + "value": 21.5546875 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603", + "concurrency": 5, + "key": [ + "file_read", + 262144, + 5 + ], + "operation": "Read, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "14.473", + "value": 14.47265625 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_cpu_time_ns": { + "display": "38.431", + "value": 38.43103 + }, + "sandbox_memory_peak_bytes": { + "display": "14.977", + "value": 14.9765625 + }, + "upperdir_bytes": { + "display": "0", + "value": 0.0 + } + } + }, + { + "cell_id": "sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa", + "concurrency": 5, + "key": [ + "file_write", + 262144, + 5 + ], + "operation": "Write, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "51.043", + "value": 51.04296875 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "1.251", + "value": 1.250625 + }, + "sandbox_cpu_time_ns": { + "display": "95.115", + "value": 95.11521 + }, + "sandbox_memory_peak_bytes": { + "display": "54.27", + "value": 54.26953125 + }, + "upperdir_bytes": { + "display": "1.258", + "value": 1.2578125 + } + } + }, + { + "cell_id": "sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab", + "concurrency": 5, + "key": [ + "file_edit", + 262144, + 5 + ], + "operation": "Edit, 256 KiB", + "values": { + "daemon_rss_bytes": { + "display": "62.602", + "value": 62.6015625 + }, + "sandbox_block_read_bytes": { + "display": "0", + "value": 0.0 + }, + "sandbox_block_write_bytes": { + "display": "1.251", + "value": 1.2505859375 + }, + "sandbox_cpu_time_ns": { + "display": "148.732", + "value": 148.73234 + }, + "sandbox_memory_peak_bytes": { + "display": "66.715", + "value": 66.71484375 + }, + "upperdir_bytes": { + "display": "1.258", + "value": 1.2578125 + } + } + } + ] + }, + "startup": { + "rows": [ + { + "cell_id": "sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2", + "concurrency": 1, + "key": [ + "create_sandbox", + null, + 1 + ], + "operation": "Create sandbox", + "stage": "Sandbox create + base mount", + "values": { + "p50_ms": 1659.8107, + "p95_ms": 1749.7388, + "p99_ms": 1794.9019450000008, + "samples": 100, + "throughput_ops_s": 0.6185790825907366 + } + }, + { + "cell_id": "sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd", + "concurrency": 1, + "key": [ + "create_workspace", + null, + 1 + ], + "operation": "Create workspace (session)", + "stage": "Session create to ready", + "values": { + "p50_ms": 32.89655, + "p95_ms": 35.6847, + "p99_ms": 36.096376, + "samples": 100, + "throughput_ops_s": 30.359842997206 + } + }, + { + "cell_id": "sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21", + "concurrency": 5, + "key": [ + "create_workspace", + null, + 5 + ], + "operation": "Create workspace (session)", + "stage": "Session create to ready", + "values": { + "p50_ms": 109.11905, + "p95_ms": 120.107005, + "p99_ms": 131.58766000000003, + "samples": 100, + "throughput_ops_s": 45.3204753420739 + } + }, + { + "cell_id": "sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5", + "concurrency": 1, + "key": [ + "exec_command", + "noop", + 1 + ], + "operation": "Exec no-op", + "stage": "First no-op command", + "values": { + "p50_ms": 26.71865, + "p95_ms": 28.29143, + "p99_ms": 28.996465000000022, + "samples": 100, + "throughput_ops_s": 37.28339526491812 + } + } + ] + } + } +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json b/ephemeral-sandbox-v1/experiments/analysis/pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json new file mode 100644 index 0000000..39896b2 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json @@ -0,0 +1,1324 @@ +{ + "analysis_script_sha256": "sha256:444eafef866c9d8be5a00f63a80eeb3727b6dfddd0faa0128d2f54deb161319f", + "decision": "pass_runtime_projection", + "decomposition": { + "cells": [ + { + "central_projected": { + "ceil_ns": 27240701475, + "denominator": 1, + "numerator_ns": 27240701475 + }, + "envelope_projected_ns": 28509628500, + "family_id": "command", + "fixed": { + "leading_ns": 2880014200, + "total_ns": 3855060500, + "trailing_ns": 975046300 + }, + "operation_id": "exec_command", + "semantic_sha256": "sha256:5c723b84eeb279b1fbe8c7602bfbdac4a7e79cacfb07ef3fc0ae3a698ae47f1f", + "source_samples": { + "pilot_measured_active_ns": [ + 213835100, + 217016900, + 223771000, + 227926200, + 225105200 + ], + "pilot_transition_gap_ns": [ + 2436900, + 2225500, + 2182400, + 2372600, + 2783400, + 2431700 + ], + "pilot_warmup_active_ns": [ + 772693800, + 213386800 + ], + "smoke_cold_active_ns": [ + 790412300 + ] + } + }, + { + "central_projected": { + "ceil_ns": 28974063750, + "denominator": 1, + "numerator_ns": 28974063750 + }, + "envelope_projected_ns": 30733645000, + "family_id": "command", + "fixed": { + "leading_ns": 2893232600, + "total_ns": 3883555800, + "trailing_ns": 990323200 + }, + "operation_id": "exec_command", + "semantic_sha256": "sha256:2229844a3f007ed6127c916f6453a85a011f3eb4cc63d750d10d46227d64f5f3", + "source_samples": { + "pilot_measured_active_ns": [ + 238244100, + 233326800, + 233324800, + 236859600, + 249632300 + ], + "pilot_transition_gap_ns": [ + 2293100, + 2420000, + 2342000, + 2340000, + 2253300, + 2214100 + ], + "pilot_warmup_active_ns": [ + 792564400, + 239030800 + ], + "smoke_cold_active_ns": [ + 821219600 + ] + } + }, + { + "central_projected": { + "ceil_ns": 26400966400, + "denominator": 1, + "numerator_ns": 26400966400 + }, + "envelope_projected_ns": 27062106000, + "family_id": "command", + "fixed": { + "leading_ns": 2879629000, + "total_ns": 3839546800, + "trailing_ns": 959917800 + }, + "operation_id": "exec_command", + "semantic_sha256": "sha256:ebf781b1e09fa308dba03d1deec25abe6c1f0af15fa0cc2ed546b1aae06e877f", + "source_samples": { + "pilot_measured_active_ns": [ + 213656300, + 214391500, + 213343100, + 213338900, + 212904200 + ], + "pilot_transition_gap_ns": [ + 2184000, + 2446000, + 2228100, + 2158200, + 2391000, + 2177500 + ], + "pilot_warmup_active_ns": [ + 768181600, + 214296700 + ], + "smoke_cold_active_ns": [ + 762436300 + ] + } + }, + { + "central_projected": { + "ceil_ns": 29473955800, + "denominator": 1, + "numerator_ns": 29473955800 + }, + "envelope_projected_ns": 30645122600, + "family_id": "command", + "fixed": { + "leading_ns": 2889041100, + "total_ns": 3921869200, + "trailing_ns": 1032828100 + }, + "operation_id": "exec_command", + "semantic_sha256": "sha256:5d6bc7cfd95c72d367f460e87f636134830c2ae134e26eee580ae0a00c515984", + "source_samples": { + "pilot_measured_active_ns": [ + 238046200, + 240297000, + 241055800, + 247685900, + 246115000 + ], + "pilot_transition_gap_ns": [ + 2229300, + 2287600, + 2246200, + 2277400, + 3017200, + 2278400 + ], + "pilot_warmup_active_ns": [ + 806628500, + 233918000 + ], + "smoke_cold_active_ns": [ + 824963100 + ] + } + }, + { + "central_projected": { + "ceil_ns": 56735904200, + "denominator": 1, + "numerator_ns": 56735904200 + }, + "envelope_projected_ns": 58081470900, + "family_id": "files", + "fixed": { + "leading_ns": 2904862600, + "total_ns": 3844108800, + "trailing_ns": 939246200 + }, + "operation_id": "file_write", + "semantic_sha256": "sha256:55478faab34c09c9714f7e612066213d3a896ab17ad43cfe9720d71dd960bc1f", + "source_samples": { + "pilot_measured_active_ns": [ + 520990600, + 504573200, + 511347400, + 512633900, + 510671400 + ], + "pilot_transition_gap_ns": [ + 2208600, + 2242400, + 2301000, + 2228800, + 2334300, + 2311500 + ], + "pilot_warmup_active_ns": [ + 951268900, + 504649400 + ], + "smoke_cold_active_ns": [ + 935241100 + ] + } + }, + { + "central_projected": { + "ceil_ns": 46671882600, + "denominator": 1, + "numerator_ns": 46671882600 + }, + "envelope_projected_ns": 53050942400, + "family_id": "files", + "fixed": { + "leading_ns": 2876666800, + "total_ns": 3817323600, + "trailing_ns": 940656800 + }, + "operation_id": "file_write", + "semantic_sha256": "sha256:d3b121f4e956174dc2e7e2fad099e327e33ab439bfb55ed23c4b7606bfdf9a03", + "source_samples": { + "pilot_measured_active_ns": [ + 460500900, + 333745700, + 467088700, + 329081700, + 471913400 + ], + "pilot_transition_gap_ns": [ + 2346100, + 2297100, + 2244400, + 2294400, + 2389200, + 2307200 + ], + "pilot_warmup_active_ns": [ + 900484800, + 474251800 + ], + "smoke_cold_active_ns": [ + 885725700 + ] + } + }, + { + "central_projected": { + "ceil_ns": 48389845575, + "denominator": 1, + "numerator_ns": 48389845575 + }, + "envelope_projected_ns": 59123374900, + "family_id": "files", + "fixed": { + "leading_ns": 2862161000, + "total_ns": 3824621800, + "trailing_ns": 962460800 + }, + "operation_id": "file_edit", + "semantic_sha256": "sha256:0535a942cf61b426e2caaf1e2e246c43ec87f91111f2d50102a3ba433c03b9b1", + "source_samples": { + "pilot_measured_active_ns": [ + 531096300, + 392800700, + 396842300, + 415546100, + 387200000 + ], + "pilot_transition_gap_ns": [ + 2329100, + 2241400, + 2220300, + 2286800, + 2240600, + 2282400 + ], + "pilot_warmup_active_ns": [ + 976942000, + 890508300 + ], + "smoke_cold_active_ns": [ + 942441800 + ] + } + }, + { + "central_projected": { + "ceil_ns": 43720503700, + "denominator": 1, + "numerator_ns": 43720503700 + }, + "envelope_projected_ns": 52187598600, + "family_id": "files", + "fixed": { + "leading_ns": 2918839000, + "total_ns": 3876183400, + "trailing_ns": 957344400 + }, + "operation_id": "file_edit", + "semantic_sha256": "sha256:05d7fb6f43125ef65f10d47af62645ee2778e791b127208f93920ccbf024394b", + "source_samples": { + "pilot_measured_active_ns": [ + 332904200, + 462986600, + 326733300, + 462916800, + 328109600 + ], + "pilot_transition_gap_ns": [ + 2214500, + 2191600, + 2430000, + 2167600, + 2224000, + 2257200 + ], + "pilot_warmup_active_ns": [ + 883662600, + 458541300 + ], + "smoke_cold_active_ns": [ + 869299100 + ] + } + }, + { + "central_projected": { + "ceil_ns": 26440725025, + "denominator": 1, + "numerator_ns": 26440725025 + }, + "envelope_projected_ns": 27881877500, + "family_id": "files", + "fixed": { + "leading_ns": 2814743900, + "total_ns": 3816858800, + "trailing_ns": 1002114900 + }, + "operation_id": "file_read", + "semantic_sha256": "sha256:04974b33b8fabefec26067ac273a142d7b513c3665a5dd7af23b0c1541f6c75a", + "source_samples": { + "pilot_measured_active_ns": [ + 211915300, + 220075600, + 208453100, + 210013900, + 213397000 + ], + "pilot_transition_gap_ns": [ + 2222800, + 2219100, + 2377500, + 2170000, + 2286900, + 2495500 + ], + "pilot_warmup_active_ns": [ + 902706600, + 208704700 + ], + "smoke_cold_active_ns": [ + 795913700 + ] + } + }, + { + "central_projected": { + "ceil_ns": 25320836350, + "denominator": 1, + "numerator_ns": 25320836350 + }, + "envelope_projected_ns": 26575413800, + "family_id": "files", + "fixed": { + "leading_ns": 2981995200, + "total_ns": 3916849400, + "trailing_ns": 934854200 + }, + "operation_id": "file_read", + "semantic_sha256": "sha256:f03cc8f32fafd88c0f85911ec3f27a822840f67fc856af60f898c5c403e2d87e", + "source_samples": { + "pilot_measured_active_ns": [ + 195689600, + 205239900, + 200713000, + 200736700, + 209399600 + ], + "pilot_transition_gap_ns": [ + 2243100, + 2250100, + 2223700, + 2252500, + 2252800, + 2209600 + ], + "pilot_warmup_active_ns": [ + 745535800, + 197151600 + ], + "smoke_cold_active_ns": [ + 740777000 + ] + } + }, + { + "central_projected": { + "ceil_ns": 74633235475, + "denominator": 1, + "numerator_ns": 74633235475 + }, + "envelope_projected_ns": 78937026600, + "family_id": "files", + "fixed": { + "leading_ns": 3264789000, + "total_ns": 4244460300, + "trailing_ns": 979671300 + }, + "operation_id": "file_write", + "semantic_sha256": "sha256:21fa1b1b6c18dfc47b84c3e476bb1c5a06d8b9636bf08368e9c4317cf286be62", + "source_samples": { + "pilot_measured_active_ns": [ + 564126400, + 718553500, + 716199300, + 705555700, + 711638500 + ], + "pilot_transition_gap_ns": [ + 2776100, + 2243500, + 2259500, + 2235600, + 2251700, + 2194900 + ], + "pilot_warmup_active_ns": [ + 1118364200, + 722616300 + ], + "smoke_cold_active_ns": [ + 1278415100 + ] + } + }, + { + "central_projected": { + "ceil_ns": 59017596875, + "denominator": 1, + "numerator_ns": 59017596875 + }, + "envelope_projected_ns": 65895211100, + "family_id": "files", + "fixed": { + "leading_ns": 3221161000, + "total_ns": 4153777700, + "trailing_ns": 932616700 + }, + "operation_id": "file_write", + "semantic_sha256": "sha256:3f8f5e708ffcb3aa506ac6f91099696a370c965d9a81f4c3b6867088f1205e8e", + "source_samples": { + "pilot_measured_active_ns": [ + 564445700, + 454756900, + 586814100, + 454259000, + 590848100 + ], + "pilot_transition_gap_ns": [ + 2226300, + 2298900, + 6277200, + 2341800, + 3202300, + 2312000 + ], + "pilot_warmup_active_ns": [ + 1011313100, + 475705700 + ], + "smoke_cold_active_ns": [ + 979248700 + ] + } + }, + { + "central_projected": { + "ceil_ns": 83886901650, + "denominator": 1, + "numerator_ns": 83886901650 + }, + "envelope_projected_ns": 87847049200, + "family_id": "files", + "fixed": { + "leading_ns": 3068511400, + "total_ns": 4026957400, + "trailing_ns": 958446000 + }, + "operation_id": "file_edit", + "semantic_sha256": "sha256:b2767eff29ef4da9984c36f18dc2c6a60980a0bfe33b93ec331c6655b2246c85", + "source_samples": { + "pilot_measured_active_ns": [ + 805341000, + 797306500, + 811257700, + 794169300, + 676939400 + ], + "pilot_transition_gap_ns": [ + 2309300, + 2251500, + 2249500, + 2343200, + 2187700, + 2263400 + ], + "pilot_warmup_active_ns": [ + 1228829300, + 702442100 + ], + "smoke_cold_active_ns": [ + 1199277300 + ] + } + }, + { + "central_projected": { + "ceil_ns": 63130366675, + "denominator": 1, + "numerator_ns": 63130366675 + }, + "envelope_projected_ns": 67045610600, + "family_id": "files", + "fixed": { + "leading_ns": 3101061700, + "total_ns": 4068561800, + "trailing_ns": 967500100 + }, + "operation_id": "file_edit", + "semantic_sha256": "sha256:71cf3ce6f552d4b65a5caaa401a0320b9ae9a2867ec9eb1fa6adb8d491bba16a", + "source_samples": { + "pilot_measured_active_ns": [ + 595739700, + 601501500, + 591008200, + 606916300, + 465967600 + ], + "pilot_transition_gap_ns": [ + 2351500, + 2332800, + 2300300, + 2736200, + 2215600, + 2298800 + ], + "pilot_warmup_active_ns": [ + 1004531300, + 593538500 + ], + "smoke_cold_active_ns": [ + 987554200 + ] + } + }, + { + "central_projected": { + "ceil_ns": 28743823850, + "denominator": 1, + "numerator_ns": 28743823850 + }, + "envelope_projected_ns": 29744328800, + "family_id": "files", + "fixed": { + "leading_ns": 3031575500, + "total_ns": 3980480600, + "trailing_ns": 948905100 + }, + "operation_id": "file_read", + "semantic_sha256": "sha256:9743a6076566c33ad2b68e32670a020d3bf0b33ddcfa9a10060abaae201c9595", + "source_samples": { + "pilot_measured_active_ns": [ + 230204300, + 236285200, + 234319300, + 235133100, + 237648000 + ], + "pilot_transition_gap_ns": [ + 2558900, + 2236700, + 4300000, + 2376800, + 2273200, + 2168600 + ], + "pilot_warmup_active_ns": [ + 782374100, + 229190200 + ], + "smoke_cold_active_ns": [ + 781943000 + ] + } + }, + { + "central_projected": { + "ceil_ns": 27306415125, + "denominator": 1, + "numerator_ns": 27306415125 + }, + "envelope_projected_ns": 28463560500, + "family_id": "files", + "fixed": { + "leading_ns": 3084974700, + "total_ns": 4033274300, + "trailing_ns": 948299600 + }, + "operation_id": "file_read", + "semantic_sha256": "sha256:714d6eb49ffc10e6c26dd8c8f304b74dd82b214f9b5b445eaba4506e4a7d4dd9", + "source_samples": { + "pilot_measured_active_ns": [ + 222406400, + 218461000, + 225891100, + 216296800, + 218982900 + ], + "pilot_transition_gap_ns": [ + 2280400, + 2337900, + 2248300, + 2402600, + 2196300, + 2223900 + ], + "pilot_warmup_active_ns": [ + 778074700, + 225174100 + ], + "smoke_cold_active_ns": [ + 799256800 + ] + } + }, + { + "central_projected": { + "ceil_ns": 31402415375, + "denominator": 1, + "numerator_ns": 31402415375 + }, + "envelope_projected_ns": 32524406100, + "family_id": "workspace_lifecycle", + "fixed": { + "leading_ns": 2858143600, + "total_ns": 3784635900, + "trailing_ns": 926492300 + }, + "operation_id": "create_workspace", + "semantic_sha256": "sha256:0e64b768b0d8bec9b56baa54ff6cac9d969c835c0b5e0c11cf3828172d973518", + "source_samples": { + "pilot_measured_active_ns": [ + 262241400, + 268866000, + 260911300, + 259849800, + 264117600 + ], + "pilot_transition_gap_ns": [ + 2259300, + 2187100, + 2165600, + 2266200, + 2268800, + 2184300 + ], + "pilot_warmup_active_ns": [ + 812010700, + 261699100 + ], + "smoke_cold_active_ns": [ + 799204300 + ] + } + }, + { + "central_projected": { + "ceil_ns": 48193569775, + "denominator": 1, + "numerator_ns": 48193569775 + }, + "envelope_projected_ns": 50176164700, + "family_id": "workspace_lifecycle", + "fixed": { + "leading_ns": 2880119000, + "total_ns": 3827457100, + "trailing_ns": 947338100 + }, + "operation_id": "create_workspace", + "semantic_sha256": "sha256:c69bdeadd954d34c8a837840be76b7a59d462a02ffab4143b534b9d227d65884", + "source_samples": { + "pilot_measured_active_ns": [ + 422513600, + 419672500, + 441255800, + 422855900, + 430342400 + ], + "pilot_transition_gap_ns": [ + 2431800, + 2352100, + 2276900, + 2394600, + 2394300, + 2322700 + ], + "pilot_warmup_active_ns": [ + 980769300, + 415390100 + ], + "smoke_cold_active_ns": [ + 988757900 + ] + } + }, + { + "central_projected": { + "ceil_ns": 313057341175, + "denominator": 1, + "numerator_ns": 313057341175 + }, + "envelope_projected_ns": 390538601500, + "family_id": "sandbox_lifecycle", + "fixed": { + "leading_ns": 4504300, + "total_ns": 6966000, + "trailing_ns": 2461700 + }, + "operation_id": "create_sandbox", + "semantic_sha256": "sha256:8b77b6a6a688d77ddfac7af0e796f0487ebc002fa44237d909b98c47078c5a91", + "source_samples": { + "pilot_measured_active_ns": [ + 2489825800, + 2826637600, + 2416021000, + 2422854400, + 2730702100 + ], + "pilot_transition_gap_ns": [ + 5434900, + 2289400, + 2240500, + 2422400, + 2492300, + 2335300 + ], + "pilot_warmup_active_ns": [ + 52429634100, + 2657308900 + ], + "smoke_cold_active_ns": [ + 53659475300 + ] + } + } + ], + "family_fixed_ns": { + "command": 278750600, + "files": 302454000, + "sandbox_lifecycle": 298739500, + "workspace_lifecycle": 275019400 + }, + "run_fixed_ns": 80922308300 + }, + "display_seconds": { + "central_structural": "1170.818322650", + "limit": "1400.000000000", + "observed_envelope": "1307.100411100", + "pilot_elapsed": "276.504428000" + }, + "final_plan": { + "cells": 19, + "issued_operation_requests": 5610, + "measured_trials_per_cell": 100, + "plan_hash": "sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b", + "sha256": "sha256:4d2389e1782a5075bf4130b3a158b55cc39cc697a1f53593f8c93ff2388e7c92", + "trial_batches": 1938, + "warmups_per_cell": 2 + }, + "gate_3_runtime_pass": true, + "inputs": { + "pilot": { + "archive_content_tree_sha256": "sha256:f81cf711bbb734f04201c7fdc09652e0d59c7cbdecd50fe3676370b1df77b93c", + "archive_manifest_sha256": "sha256:c2810ac9014f835c676dce294ba4a164c7cc979c4e9cd49be4c3239bdf9bdbc6", + "campaign_manifest_sha256": "sha256:de60036c6fa2bb5d715315e61c36909a50701462a030d9f2af649a079b1033cb", + "plan_hash": "sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f", + "provenance": { + "analysis_and_archiving_code": { + "files": [ + { + "bytes": 37142, + "path": "benchmark/backend/benchmark_lab/derivation.py", + "sha256": "sha256:14b82867bfca82ab252db5f28567c83ab890b892759c3309f8e75391b8d3d7f1" + }, + { + "bytes": 14044, + "path": "benchmark/backend/benchmark_lab/reports.py", + "sha256": "sha256:17731aa45d2ce86147a1b28fb5b75fde83957bc6d22ca2a1fd1f0d7b98250d50" + }, + { + "bytes": 62503, + "path": "experiments/scripts/archive_exp1_run.py", + "sha256": "sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e" + }, + { + "bytes": 63181, + "path": "experiments/analysis/scripts/generate_exp1_tables.py", + "sha256": "sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3" + } + ] + }, + "artifact_schemas": { + "bounded_evidence": { + "read_versions": [ + 1, + 2 + ], + "schema_name": "eos_benchmark_operation_evidence", + "write_version": 2 + }, + "definition_snapshot": { + "read_versions": [ + 2 + ], + "schema_name": "eos_benchmark_definition_snapshot", + "write_version": 2 + }, + "environment_metadata": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_environment_metadata", + "write_version": 1 + }, + "events": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_event", + "write_version": 1 + }, + "expanded_plan": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_expanded_plan", + "write_version": 1 + }, + "intent_plan": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_intent_plan", + "write_version": 1 + }, + "observations": { + "read_versions": [ + 1, + 2, + 3, + 4, + 5 + ], + "schema_name": "eos_benchmark_observation", + "write_version": 5 + }, + "run_manifest": { + "read_versions": [ + 1, + 2 + ], + "schema_name": "eos_benchmark_run_manifest", + "write_version": 2 + } + }, + "benchmark_source": { + "content_tree_sha256": "sha256:1efeff548dd664580dcb452829d86e1ae114477828a1af65589b7e34cc311b67", + "manifest_sha256": "sha256:d96cff9ace92f167004a71da2ec3055342dedde2debdbbdd30978ae36a423ce2" + }, + "definition_snapshot": { + "schema_name": "eos_benchmark_definition_snapshot", + "schema_version": 2, + "sha256": "sha256:cedb67107a75795ce38ee238341896fa41c75852c77d37996c632a969b35da55" + }, + "docker": { + "architecture": "x86_64", + "cgroup_version": "2", + "driver": "overlayfs", + "os_type": "linux", + "server_version": "29.0.1", + "warnings": null + }, + "fixture": { + "fixture_hash": "sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7", + "manifest_sha256": "sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1", + "tree_hash": "sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f" + }, + "gateway": { + "isolated_runtime_per_execution_block": true, + "loopback_only": true, + "maximum_connections": 256, + "mode": "isolated", + "readiness_poll_interval_ms": 50, + "readiness_probe_timeout_ms": 2000, + "readiness_timeout_ms": 60000, + "remount_sweep_widths": [ + 1 + ], + "semantic_revision": 1 + }, + "host": { + "architecture": "x64", + "computer_name": "DESKTOP-OLP1ADS", + "cpu_model": "AMD Ryzen Threadripper 7960X 24-Cores", + "docker_engine_version": "29.0.1", + "filesystem": "NTFS", + "logical_processors": 48, + "monotonic_clock": "time.monotonic_ns", + "operating_system": "windows", + "os_build_number": 26200, + "os_version": "10.0.26200", + "processor_logical_processors": 48, + "total_memory_bytes": 137438953472, + "volume_root": "C:\\" + }, + "image": { + "architecture": "amd64", + "id": "sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + "os": "linux", + "repo_digests": [ + "ubuntu@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + ], + "requested": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + }, + "lifecycle": { + "automatic_retries": 0, + "failure_revision": 1, + "lifecycle_revision": 1, + "one_active_campaign": true, + "sequential_families": true, + "stabilization_revision": 1 + }, + "product": { + "archive": { + "bytes": 5685130, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299.zip", + "sha256": "sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e" + }, + "binaries": { + "daemon": { + "bytes": 7171600, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\dist\\sandbox-daemon-linux-amd64", + "sha256": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a" + }, + "gateway": { + "bytes": 3452928, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-gateway.exe", + "sha256": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285" + }, + "manager": { + "bytes": 800768, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-manager-cli.exe", + "sha256": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed" + }, + "observability": { + "bytes": 800256, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-observability-cli.exe", + "sha256": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26" + }, + "runtime": { + "bytes": 805888, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-runtime-cli.exe", + "sha256": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8" + } + }, + "branch": "main", + "commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "dirty": false, + "recorded_treatment": { + "daemon_binary_hash": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a", + "gateway_binary_hash": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285", + "manager_cli_binary_hash": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed", + "observability_cli_binary_hash": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26", + "runtime_cli_binary_hash": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8", + "source_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "source_diff_hash": null, + "source_dirty": false + } + }, + "protocol_files": [ + { + "bytes": 24478, + "path": "plan/task-packets/exp1-cli-performance-campaign.md", + "sha256": "sha256:c021ecdd47cc80b1601171c7c175ed4f42d12dfe51229aa4c36b603b8d0a18ce" + }, + { + "bytes": 11529, + "path": "experiments/expected_tables.md", + "sha256": "sha256:7c1ae3d78a9ca7a5b6cbb4a9cc80bac555d676c5833a40d2846597884f8f50da" + }, + { + "bytes": 12623, + "path": "benchmark/PAPER_ARTIFACT.md", + "sha256": "sha256:90385fda5fc65e7cbe47bd3cea1ad2f97307b248f510ea021aa9b20c2bed9ecb" + } + ], + "sandbox_limits": { + "authority": { + "capture_boundary": "run_start_before_gateway_and_measurement", + "create_request_override": "none", + "effective_config_builder": "benchmark_lab.gateway._effective_config preserves the selected profile and resource override fields", + "kind": "released_gateway_configuration", + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\config\\windows-amd64.yml", + "selector": "manager.docker.resource_profiles.standard with manager.docker nano_cpus/memory_bytes overrides", + "sha256": "sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a" + }, + "memory_bytes": 536870912, + "nano_cpus": 1000000000, + "pids_limit": 256, + "profile": "standard", + "vcpus": 1 + }, + "treatment": { + "daemon_binary_hash": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a", + "gateway_binary_hash": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285", + "manager_cli_binary_hash": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed", + "observability_cli_binary_hash": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26", + "runtime_cli_binary_hash": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8", + "source_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "source_diff_hash": null, + "source_dirty": false + } + }, + "raw_events_sha256": "sha256:a6994cfc15076e997ad06798a61084adfc2ede2b49b181f85f0be9db7e59676b", + "raw_observations_sha256": "sha256:ddadfb11edab4eb3ce6e65512f4876a2a751651ed56c782f340ff6bf3e36a99c", + "run_id": "019fb6cf-6021-76d5-ab4f-c6ed53e1d293" + }, + "smoke": { + "archive_content_tree_sha256": "sha256:3ad6e3aba681cfdf257939df79a0561a5b73b710a73d46686146bc5780fe8a6b", + "archive_manifest_sha256": "sha256:6fc6924f5bf775b4572093d6f5d619c249e2362273b69dc5d1626ec2bb8e21d1", + "campaign_manifest_sha256": "sha256:bf301771c5c2fff3ee82687d8ee8982a6c6d8451635664b411a2140afd99a325", + "plan_hash": "sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937", + "provenance": { + "analysis_and_archiving_code": { + "files": [ + { + "bytes": 37142, + "path": "benchmark/backend/benchmark_lab/derivation.py", + "sha256": "sha256:14b82867bfca82ab252db5f28567c83ab890b892759c3309f8e75391b8d3d7f1" + }, + { + "bytes": 14044, + "path": "benchmark/backend/benchmark_lab/reports.py", + "sha256": "sha256:17731aa45d2ce86147a1b28fb5b75fde83957bc6d22ca2a1fd1f0d7b98250d50" + }, + { + "bytes": 62503, + "path": "experiments/scripts/archive_exp1_run.py", + "sha256": "sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e" + }, + { + "bytes": 63181, + "path": "experiments/analysis/scripts/generate_exp1_tables.py", + "sha256": "sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3" + } + ] + }, + "artifact_schemas": { + "bounded_evidence": { + "read_versions": [ + 1, + 2 + ], + "schema_name": "eos_benchmark_operation_evidence", + "write_version": 2 + }, + "definition_snapshot": { + "read_versions": [ + 2 + ], + "schema_name": "eos_benchmark_definition_snapshot", + "write_version": 2 + }, + "environment_metadata": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_environment_metadata", + "write_version": 1 + }, + "events": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_event", + "write_version": 1 + }, + "expanded_plan": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_expanded_plan", + "write_version": 1 + }, + "intent_plan": { + "read_versions": [ + 1 + ], + "schema_name": "eos_benchmark_intent_plan", + "write_version": 1 + }, + "observations": { + "read_versions": [ + 1, + 2, + 3, + 4, + 5 + ], + "schema_name": "eos_benchmark_observation", + "write_version": 5 + }, + "run_manifest": { + "read_versions": [ + 1, + 2 + ], + "schema_name": "eos_benchmark_run_manifest", + "write_version": 2 + } + }, + "benchmark_source": { + "content_tree_sha256": "sha256:1efeff548dd664580dcb452829d86e1ae114477828a1af65589b7e34cc311b67", + "manifest_sha256": "sha256:d96cff9ace92f167004a71da2ec3055342dedde2debdbbdd30978ae36a423ce2" + }, + "definition_snapshot": { + "schema_name": "eos_benchmark_definition_snapshot", + "schema_version": 2, + "sha256": "sha256:cedb67107a75795ce38ee238341896fa41c75852c77d37996c632a969b35da55" + }, + "docker": { + "architecture": "x86_64", + "cgroup_version": "2", + "driver": "overlayfs", + "os_type": "linux", + "server_version": "29.0.1", + "warnings": null + }, + "fixture": { + "fixture_hash": "sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7", + "manifest_sha256": "sha256:f35b5e9fd04986b4370fd17f09311ff870d7f1abaf02cdd02411cd0d8c4e7be1", + "tree_hash": "sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f" + }, + "gateway": { + "isolated_runtime_per_execution_block": true, + "loopback_only": true, + "maximum_connections": 256, + "mode": "isolated", + "readiness_poll_interval_ms": 50, + "readiness_probe_timeout_ms": 2000, + "readiness_timeout_ms": 60000, + "remount_sweep_widths": [ + 1 + ], + "semantic_revision": 1 + }, + "host": { + "architecture": "x64", + "computer_name": "DESKTOP-OLP1ADS", + "cpu_model": "AMD Ryzen Threadripper 7960X 24-Cores", + "docker_engine_version": "29.0.1", + "filesystem": "NTFS", + "logical_processors": 48, + "monotonic_clock": "time.monotonic_ns", + "operating_system": "windows", + "os_build_number": 26200, + "os_version": "10.0.26200", + "processor_logical_processors": 48, + "total_memory_bytes": 137438953472, + "volume_root": "C:\\" + }, + "image": { + "architecture": "amd64", + "id": "sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + "os": "linux", + "repo_digests": [ + "ubuntu@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + ], + "requested": "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" + }, + "lifecycle": { + "automatic_retries": 0, + "failure_revision": 1, + "lifecycle_revision": 1, + "one_active_campaign": true, + "sequential_families": true, + "stabilization_revision": 1 + }, + "product": { + "archive": { + "bytes": 5685130, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299.zip", + "sha256": "sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e" + }, + "binaries": { + "daemon": { + "bytes": 7171600, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\dist\\sandbox-daemon-linux-amd64", + "sha256": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a" + }, + "gateway": { + "bytes": 3452928, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-gateway.exe", + "sha256": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285" + }, + "manager": { + "bytes": 800768, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-manager-cli.exe", + "sha256": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed" + }, + "observability": { + "bytes": 800256, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-observability-cli.exe", + "sha256": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26" + }, + "runtime": { + "bytes": 805888, + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\bin\\sandbox-runtime-cli.exe", + "sha256": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8" + } + }, + "branch": "main", + "commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "dirty": false, + "recorded_treatment": { + "daemon_binary_hash": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a", + "gateway_binary_hash": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285", + "manager_cli_binary_hash": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed", + "observability_cli_binary_hash": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26", + "runtime_cli_binary_hash": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8", + "source_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "source_diff_hash": null, + "source_dirty": false + } + }, + "protocol_files": [ + { + "bytes": 24478, + "path": "plan/task-packets/exp1-cli-performance-campaign.md", + "sha256": "sha256:c021ecdd47cc80b1601171c7c175ed4f42d12dfe51229aa4c36b603b8d0a18ce" + }, + { + "bytes": 11529, + "path": "experiments/expected_tables.md", + "sha256": "sha256:7c1ae3d78a9ca7a5b6cbb4a9cc80bac555d676c5833a40d2846597884f8f50da" + }, + { + "bytes": 12623, + "path": "benchmark/PAPER_ARTIFACT.md", + "sha256": "sha256:90385fda5fc65e7cbe47bd3cea1ad2f97307b248f510ea021aa9b20c2bed9ecb" + } + ], + "sandbox_limits": { + "authority": { + "capture_boundary": "run_start_before_gateway_and_measurement", + "create_request_override": "none", + "effective_config_builder": "benchmark_lab.gateway._effective_config preserves the selected profile and resource override fields", + "kind": "released_gateway_configuration", + "path": "C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-0392b299\\config\\windows-amd64.yml", + "selector": "manager.docker.resource_profiles.standard with manager.docker nano_cpus/memory_bytes overrides", + "sha256": "sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a" + }, + "memory_bytes": 536870912, + "nano_cpus": 1000000000, + "pids_limit": 256, + "profile": "standard", + "vcpus": 1 + }, + "treatment": { + "daemon_binary_hash": "sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a", + "gateway_binary_hash": "sha256:9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285", + "manager_cli_binary_hash": "sha256:564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed", + "observability_cli_binary_hash": "sha256:25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26", + "runtime_cli_binary_hash": "sha256:c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8", + "source_commit": "0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3", + "source_diff_hash": null, + "source_dirty": false + } + }, + "raw_events_sha256": "sha256:28f4ea8d5eb8fe44dc66005b1864e508096f47f6fe561bfd1c0bd182f2e30af2", + "raw_observations_sha256": "sha256:f6b0c4617efca5a91a8211435cca7c0e3e7dffb94cb47f2e6833381e6847b405", + "run_id": "019fb6c5-dab0-7958-b7ba-94f2a9eda944" + } + }, + "limit_ns": 1400000000000, + "model_revision": 1, + "models": { + "central_structural": { + "ceil_ns": 1170818322650, + "denominator": 1, + "numerator_ns": 1170818322650 + }, + "observed_envelope_ns": 1307100411100, + "pilot_elapsed_lower_bound_ns": 276504428000 + }, + "pass_conditions": { + "central_structural_within_limit": true, + "observed_envelope_within_limit": true, + "pilot_elapsed_within_limit": true + }, + "purpose": "EXP1 Gate 3 runtime projection; not manuscript evidence", + "schema_version": 2, + "scientific_use": "Exploratory capacity decision only. No smoke, pilot, or projected value is eligible for manuscript tables." +} diff --git a/ephemeral-sandbox-v1/experiments/analysis/scripts/generate_exp1_tables.py b/ephemeral-sandbox-v1/experiments/analysis/scripts/generate_exp1_tables.py new file mode 100644 index 0000000..9325a6d --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/scripts/generate_exp1_tables.py @@ -0,0 +1,1944 @@ +#!/usr/bin/env python3 +"""Generate the four deterministic EXP1 tables from one immutable archive. + +The generator deliberately has no live-system fallback. Every displayed value +is selected from an archive member covered by archive-manifest.json. Numeric +outputs are accompanied by both a strict ai-research-writing v2 registry and a +machine-readable provenance CSV containing the upstream selector, unit, and +aggregation recipe. +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import io +import json +import math +import re +import sys +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +SCHEMA_VERSION = 2 +EVIDENCE_SCHEMA_VERSION = "ai-research-writing/numeric-evidence-v2" +SHA256_PREFIX = "sha256:" +MIB = 1024.0 * 1024.0 +EXPECTED_COUNTS = { + "exploratory": { + "preset": "paper-pilot", + "cells": 19, + "trial_batches": 133, + "requests": 385, + "warmups": 2, + "measured": 5, + "campaign_eligibility": "exploratory_ineligible", + "output_eligibility": "exploratory_ineligible", + }, + "final": { + "preset": "paper-good-pass", + "cells": 19, + "trial_batches": 1938, + "requests": 5610, + "warmups": 2, + "measured": 100, + "campaign_eligibility": "frozen_final_candidate", + "output_eligibility": "frozen_final_candidate", + }, +} +PROTOCOLS = { + "v1.0": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.0", + "final_tag": "paper-v1-freeze", + "environment_identity": "isolated_loopback_per_execution_block", + }, + "v1.1": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.1", + "final_tag": "paper-v1.1-freeze", + "environment_identity": ( + "isolated_windows_named_pipe_per_execution_block" + ), + }, +} +V11_GATEWAY_TRANSPORT = { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", +} +V10_GATEWAY_TRANSPORT = { + "transport": "tcp_loopback", + "scope": "local_only", + "rotation": "per_execution_block", +} +SAFE_NPIPE_ENDPOINT = re.compile( + r"npipe://\./pipe/[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z" +) + +PERFORMANCE_ROWS = [ + ("create_sandbox", "create_sandbox", None, 1, "Create sandbox"), + ("create_workspace", "create_workspace", None, 1, "Create workspace (session)"), + ("exec_command", "exec_command", "noop", 1, "Exec no-op"), + ("exec_command", "exec_command", "noop", 5, "Exec no-op"), + ("exec_command", "exec_command", "fixture_read", 1, "Exec fixture read (4 KiB)"), + ("exec_command", "exec_command", "fixture_read", 5, "Exec fixture read (4 KiB)"), + ("file_read", "file_read", 4096, 1, "Read snapshot (4 KiB)"), + ("file_read", "file_read", 4096, 5, "Read snapshot (4 KiB)"), + ("file_read", "file_read", 262144, 1, "Read snapshot (256 KiB)"), + ("file_read", "file_read", 262144, 5, "Read snapshot (256 KiB)"), + ("file_write", "file_write", 4096, 1, "Write session (4 KiB)"), + ("file_write", "file_write", 4096, 5, "Write session (4 KiB)"), + ("file_write", "file_write", 262144, 1, "Write session (256 KiB)"), + ("file_write", "file_write", 262144, 5, "Write session (256 KiB)"), + ("file_edit", "file_edit", 4096, 1, "Edit replacement (4 KiB)"), + ("file_edit", "file_edit", 4096, 5, "Edit replacement (4 KiB)"), + ("file_edit", "file_edit", 262144, 1, "Edit replacement (256 KiB)"), + ("file_edit", "file_edit", 262144, 5, "Edit replacement (256 KiB)"), + ("create_workspace", "create_workspace", None, 5, "Create workspace (session)"), +] + +STARTUP_KEYS = [ + ("create_sandbox", None, 1), + ("create_workspace", None, 1), + ("create_workspace", None, 5), + ("exec_command", "noop", 1), +] + +PUBLIC_KEYS = [ + ("exec_command", "noop", 1), + ("exec_command", "noop", 5), + ("exec_command", "fixture_read", 1), + ("exec_command", "fixture_read", 5), + ("file_read", 4096, 1), + ("file_read", 4096, 5), + ("file_read", 262144, 1), + ("file_read", 262144, 5), + ("file_write", 4096, 1), + ("file_write", 4096, 5), + ("file_write", 262144, 1), + ("file_write", 262144, 5), + ("file_edit", 4096, 1), + ("file_edit", 4096, 5), + ("file_edit", 262144, 1), + ("file_edit", 262144, 5), +] + +RESOURCE_KEYS = [ + ("create_workspace", None, 1), + ("create_workspace", None, 5), + ("exec_command", "noop", 1), + ("exec_command", "noop", 5), + ("file_read", 262144, 5), + ("file_write", 262144, 5), + ("file_edit", 262144, 5), +] + +RESOURCE_COLUMNS = [ + ("daemon_rss_bytes", "Peak daemon RSS (MiB)", "bytes", "MiB", "max"), + ("sandbox_memory_peak_bytes", "Peak sandbox memory (MiB)", "bytes", "MiB", "max"), + ( + "sandbox_cpu_time_ns", + "Sandbox CPU (ms/trial)", + "nanoseconds", + "milliseconds", + "mean", + ), + ("sandbox_block_read_bytes", "Block read (MiB/trial)", "bytes", "MiB", "mean"), + ("sandbox_block_write_bytes", "Block write (MiB/trial)", "bytes", "MiB", "mean"), + ( + "upperdir_bytes", + "Workspace allocated delta (MiB)", + "bytes", + "MiB", + "mean", + ), +] + + +class GenerationError(RuntimeError): + """A fail-closed archive or generation-contract violation.""" + + +def _sha256_bytes(data: bytes) -> str: + return SHA256_PREFIX + hashlib.sha256(data).hexdigest() + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return SHA256_PREFIX + digest.hexdigest() + + +def _json_bytes(value: Any) -> bytes: + return ( + json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n" + ).encode("utf-8") + + +def _load_json(path: Path, *, envelope: bool = False) -> Any: + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise GenerationError( + f"cannot read JSON archive member {path.name}: {exc}" + ) from exc + if envelope: + if not isinstance(parsed, dict) or not isinstance(parsed.get("data"), dict): + raise GenerationError(f"{path.name} is not a valid data envelope") + return parsed["data"] + return parsed + + +def _require(condition: bool, message: str) -> None: + if not condition: + raise GenerationError(message) + + +def _nested(value: Any, *path: str) -> Any: + for part in path: + if not isinstance(value, Mapping) or part not in value: + return None + value = value[part] + return value + + +def _first(value: Any, paths: Sequence[Sequence[str]]) -> Any: + for path in paths: + found = _nested(value, *path) + if found is not None: + return found + return None + + +def _normalize_sha(value: Any) -> str | None: + if not isinstance(value, str): + return None + if value.startswith(SHA256_PREFIX) and len(value) == 71: + return value + if len(value) == 64: + return SHA256_PREFIX + value + return value + + +def _is_git_sha1(value: Any) -> bool: + return ( + isinstance(value, str) + and len(value) == 40 + and all(character in "0123456789abcdef" for character in value) + ) + + +def _verify_archive_inventory( + archive: Path, manifest: Mapping[str, Any] +) -> dict[str, Mapping[str, Any]]: + entries = manifest.get("files") + _require(isinstance(entries, list), "archive manifest has no file inventory") + expected: dict[str, Mapping[str, Any]] = {} + for entry in entries: + _require(isinstance(entry, dict), "archive inventory entry is not an object") + relative = entry.get("path") + _require( + isinstance(relative, str) + and relative + and "\\" not in relative + and not relative.startswith("/") + and ".." not in Path(relative).parts, + f"unsafe archive inventory path: {relative!r}", + ) + _require( + relative not in expected, f"duplicate archive inventory path: {relative}" + ) + expected[relative] = entry + + actual_paths = sorted( + path.relative_to(archive).as_posix() + for path in archive.rglob("*") + if path.is_file() and path.name != "archive-manifest.json" + ) + _require( + actual_paths == sorted(expected), + "archive file set drifted from archive-manifest.json", + ) + + tree = hashlib.sha256() + total_bytes = 0 + for relative in actual_paths: + path = archive / Path(relative) + data = path.read_bytes() + digest = hashlib.sha256(data).digest() + entry = expected[relative] + _require( + entry.get("bytes") == len(data), f"archive byte-count drift: {relative}" + ) + _require( + _normalize_sha(entry.get("sha256")) == SHA256_PREFIX + digest.hex(), + f"archive content drift: {relative}", + ) + total_bytes += len(data) + tree.update(relative.encode("utf-8")) + tree.update(b"\0") + tree.update(digest) + tree.update(b"\n") + + _require( + manifest.get("archive_file_count") == len(actual_paths), + "archive file count drift", + ) + _require( + manifest.get("archive_bytes") == total_bytes, + "archive aggregate byte count drift", + ) + _require( + _normalize_sha(manifest.get("content_tree_sha256")) + == SHA256_PREFIX + tree.hexdigest(), + "archive content-tree hash drift", + ) + return expected + + +def _metric(cell: Mapping[str, Any], metric_id: str) -> Mapping[str, Any]: + matches = [ + metric + for metric in cell.get("metrics", []) + if isinstance(metric, Mapping) + and _nested(metric, "identity", "id") == metric_id + ] + _require( + len(matches) == 1, + f"cell {cell.get('cell_id')} must contain one {metric_id} metric", + ) + return matches[0] + + +def _cell_key(plan_cell: Mapping[str, Any]) -> tuple[str, Any, int]: + operation = plan_cell.get("operation_id") + factors = _nested(plan_cell, "operation", "cell") + _require( + isinstance(factors, Mapping), + f"cell {plan_cell.get('cell_id')} has no operation factors", + ) + if operation == "create_sandbox": + return operation, None, 1 + if operation == "create_workspace": + return operation, None, int(factors.get("workspace_count", -1)) + concurrency = int(factors.get("concurrent_requests", -1)) + if operation == "exec_command": + return operation, factors.get("command_case"), concurrency + if operation == "file_read": + return operation, int(factors.get("returned_bytes", -1)), concurrency + if operation == "file_write": + return operation, int(factors.get("content_bytes", -1)), concurrency + if operation == "file_edit": + return operation, int(factors.get("file_bytes", -1)), concurrency + raise GenerationError(f"unexpected EXP1 operation {operation!r}") + + +def _expected_keys() -> set[tuple[str, Any, int]]: + return { + (operation, variant, concurrency) + for _, operation, variant, concurrency, _ in PERFORMANCE_ROWS + } + + +def _validate_factor_contract( + key: tuple[str, Any, int], factors: Mapping[str, Any] +) -> None: + operation, variant, concurrency = key + _require( + factors.get("workspace_profile") == "paper-100m", + f"{key}: workspace profile drift", + ) + if operation == "create_sandbox": + _require( + factors.get("network_profile") == "shared", f"{key}: network policy drift" + ) + elif operation == "create_workspace": + _require( + factors.get("workspace_count") == concurrency, + f"{key}: workspace count drift", + ) + else: + _require( + factors.get("concurrent_requests") == concurrency, + f"{key}: concurrency drift", + ) + if operation == "exec_command": + _require(factors.get("command_case") == variant, f"{key}: command case drift") + _require( + factors.get("session_mode") == "explicit", f"{key}: session mode drift" + ) + elif operation == "file_read": + _require(factors.get("returned_bytes") == variant, f"{key}: read size drift") + _require(factors.get("source") == "snapshot", f"{key}: read source drift") + elif operation == "file_write": + _require(factors.get("content_bytes") == variant, f"{key}: write size drift") + _require( + factors.get("destination") == "session", f"{key}: write destination drift" + ) + elif operation == "file_edit": + _require(factors.get("file_bytes") == variant, f"{key}: edit size drift") + _require( + factors.get("destination") == "session", f"{key}: edit destination drift" + ) + _require( + factors.get("replacement_count") == 1, f"{key}: replacement count drift" + ) + + +def _raw_values( + metric: Mapping[str, Any], measured: int, *, allow_unavailable: bool +) -> list[float] | None: + attempted = metric.get("attempted_n") + available = metric.get("available_n") + failed = metric.get("failed_n") + unavailable = _nested(metric, "unavailable", "count") + _require( + attempted == measured, + f"metric {_nested(metric, 'identity', 'id')}: attempted_n drift", + ) + _require(failed == 0, f"metric {_nested(metric, 'identity', 'id')}: failed samples") + if available != measured or unavailable != 0: + if allow_unavailable: + return None + raise GenerationError( + f"metric {_nested(metric, 'identity', 'id')}: incomplete measured samples " + f"(available={available}, unavailable={unavailable}, expected={measured})" + ) + points = metric.get("raw_points") + _require( + isinstance(points, list) and len(points) == measured, + "raw metric point count drift", + ) + trial_ids: set[str] = set() + values: list[float] = [] + for point in points: + _require(isinstance(point, Mapping), "raw metric point is not an object") + trial_id = point.get("trial_id") + _require( + isinstance(trial_id, str) + and "-measured-" in trial_id + and trial_id not in trial_ids, + "raw metric trial identity drift", + ) + trial_ids.add(trial_id) + value = point.get("value") + _require( + isinstance(value, (int, float)) and math.isfinite(float(value)), + "non-finite raw metric value", + ) + values.append(float(value)) + return values + + +def _quantile(values: Sequence[float], probability: float) -> float: + _require(bool(values), "cannot calculate quantile of empty samples") + ordered = sorted(values) + position = (len(ordered) - 1) * probability + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + weight = position - lower + return ordered[lower] * (1.0 - weight) + ordered[upper] * weight + + +def _mean(values: Sequence[float]) -> float: + _require(bool(values), "cannot calculate mean of empty samples") + return math.fsum(values) / len(values) + + +def _convert(value: float, input_unit: str, output_unit: str) -> float: + if (input_unit, output_unit) == ("nanoseconds", "milliseconds"): + return value / 1_000_000.0 + if (input_unit, output_unit) == ("bytes", "MiB"): + return value / MIB + if input_unit == output_unit: + return value + raise GenerationError(f"unsupported unit conversion: {input_unit} -> {output_unit}") + + +def _display(value: float, *, digits: int = 3) -> str: + rendered = f"{value:.{digits}f}" + if "." in rendered: + rendered = rendered.rstrip("0").rstrip(".") + return rendered + + +def _archive_selector( + path: str, *, cell_id: str | None = None, metric_id: str | None = None +) -> dict[str, Any]: + selector: dict[str, Any] = {"artifact": path} + if cell_id is not None: + selector["cell_id"] = cell_id + if metric_id is not None: + selector["metric_id"] = metric_id + selector["field"] = "raw_points[].value" + selector["trial_scope"] = "reportable_measured" + return selector + + +class Evidence: + def __init__(self, source_hashes: Mapping[str, str], eligibility: str) -> None: + self.source_hashes = source_hashes + self.eligibility = eligibility + self.rows: list[dict[str, Any]] = [] + + def add( + self, + evidence_id: str, + value: float, + *, + source_file: str, + selector: Mapping[str, Any], + input_unit: str, + output_unit: str, + aggregation: str, + parameters: Mapping[str, Any] | None = None, + ) -> float: + _require( + source_file in self.source_hashes, + f"unmanifested evidence source: {source_file}", + ) + number = float(value) + _require(math.isfinite(number), f"non-finite evidence value: {evidence_id}") + self.rows.append( + { + "evidence_id": evidence_id, + "value": format(number, ".17g"), + "source_file": source_file, + "source_sha256": self.source_hashes[source_file], + "selector_kind": "archive-json", + "selector_json": json.dumps( + selector, separators=(",", ":"), sort_keys=True + ), + "input_unit": input_unit, + "output_unit": output_unit, + "aggregation": aggregation, + "aggregation_parameters_json": json.dumps( + parameters or {}, separators=(",", ":"), sort_keys=True + ), + "eligibility": self.eligibility, + } + ) + return number + + def csv_bytes(self) -> bytes: + output = io.StringIO(newline="") + fieldnames = [ + "evidence_id", + "value", + "source_file", + "source_sha256", + "selector_kind", + "selector_json", + "input_unit", + "output_unit", + "aggregation", + "aggregation_parameters_json", + "eligibility", + ] + writer = csv.DictWriter(output, fieldnames=fieldnames, lineterminator="\n") + writer.writeheader() + writer.writerows(sorted(self.rows, key=lambda row: row["evidence_id"])) + return output.getvalue().encode("utf-8") + + def registry(self) -> dict[str, Any]: + entries = [] + for row in sorted(self.rows, key=lambda item: item["evidence_id"]): + note = ( + f"{row['evidence_id']}; upstream={row['source_file']}#{row['selector_json']}; " + f"units={row['input_unit']}->{row['output_unit']}; " + f"aggregation={row['aggregation']} {row['aggregation_parameters_json']}; " + f"eligibility={row['eligibility']}" + ) + entries.append( + { + "aggregate": "identity", + "note": note, + "representations": ["raw"], + "selector": { + "column": "value", + "kind": "csv", + "where": {"evidence_id": row["evidence_id"]}, + }, + "source": "numeric-provenance.csv", + "tolerance": 0, + "value": float(row["value"]), + } + ) + return {"entries": entries, "schema_version": EVIDENCE_SCHEMA_VERSION} + + +def _eligibility_banner(disposition: str) -> str: + if disposition == "exploratory": + return ( + "> **INELIGIBLE EXPLORATORY OUTPUT.** This archive is a pilot. " + "Do not copy these values into the manuscript or paper tables." + ) + return ( + "> **FROZEN FINAL CANDIDATE.** Values are archive-derived and may be used only " + "after the remaining paper evidence/build gates pass." + ) + + +def _markdown_table( + headers: Sequence[str], rows: Sequence[Sequence[str]], banner: str, title: str +) -> bytes: + lines = [f"# {title}", "", banner, "", "| " + " | ".join(headers) + " |"] + lines.append("| " + " | ".join("---" for _ in headers) + " |") + lines.extend("| " + " | ".join(row) + " |" for row in rows) + lines.append("") + return "\n".join(lines).encode("utf-8") + + +def _label_for_key(key: tuple[str, Any, int]) -> str: + for _, operation, variant, concurrency, label in PERFORMANCE_ROWS: + if key == (operation, variant, concurrency): + return label + raise GenerationError(f"no table label for cell key {key}") + + +def _validate_protocol_transport( + campaign: Mapping[str, Any], + run: Mapping[str, Any], + expanded: Mapping[str, Any], + protocol_version: str, + disposition: str, +) -> None: + protocol = campaign.get("protocol") + if protocol_version == "v1.0" and protocol is None: + environment = run.get("environment") + _require(isinstance(environment, Mapping), "run environment is invalid") + _require( + environment.get("gateway_endpoint_identity") + in (None, PROTOCOLS["v1.0"]["environment_identity"]), + "legacy v1.0 gateway identity drift", + ) + return + _require(isinstance(protocol, Mapping), "campaign protocol provenance is missing") + expected_version = ( + protocol_version + if protocol_version == "v1.1" or disposition == "final" + else "pre-freeze-exp1" + ) + _require( + protocol.get("version") == expected_version + and protocol.get("id") in (None, PROTOCOLS[protocol_version]["id"]), + "campaign protocol version is invalid", + ) + environment = run.get("environment") + _require(isinstance(environment, Mapping), "run environment is invalid") + if protocol_version == "v1.0": + _require( + environment.get("gateway_endpoint_identity") + in (None, PROTOCOLS["v1.0"]["environment_identity"]), + "legacy v1.0 gateway identity drift", + ) + return + _require( + protocol.get("id") == PROTOCOLS["v1.1"]["id"] + and environment.get("gateway_endpoint_identity") + == PROTOCOLS["v1.1"]["environment_identity"] + and environment.get("gateway_transport") == V11_GATEWAY_TRANSPORT, + "v1.1 named-pipe environment identity is invalid", + ) + policy = run.get("gateway_policy") + _require( + isinstance(policy, Mapping) + and policy.get("protocol_version") == PROTOCOLS["v1.1"]["id"] + and all( + policy.get(key) == value + for key, value in V11_GATEWAY_TRANSPORT.items() + ) + and policy.get("mode") == "isolated" + and policy.get("isolated_runtime_per_execution_block") is True + and policy.get("loopback_only") is False, + "v1.1 named-pipe gateway policy is invalid", + ) + planned = expanded.get("execution_blocks") + launched = run.get("gateway_execution_blocks") + _require( + isinstance(planned, list) + and isinstance(launched, list) + and len(planned) == len(launched), + "v1.1 execution-block endpoint count is invalid", + ) + endpoints: set[str] = set() + for expected, observed in zip(planned, launched): + endpoint = observed.get("endpoint_uri") if isinstance(observed, Mapping) else None + _require( + isinstance(expected, Mapping) + and isinstance(observed, Mapping) + and observed.get("block_id") == expected.get("block_id") + and observed.get("family_id") == expected.get("family_id") + and all( + observed.get(key) == value + for key, value in V11_GATEWAY_TRANSPORT.items() + ) + and isinstance(observed.get("gateway_instance_id"), str) + and bool(observed["gateway_instance_id"]) + and isinstance(endpoint, str) + and SAFE_NPIPE_ENDPOINT.fullmatch(endpoint) is not None + and endpoint not in endpoints, + "v1.1 execution-block endpoint evidence is unsafe", + ) + endpoints.add(endpoint) + + +def _validate_archive( + archive: Path, protocol_version: str = "v1.0" +) -> dict[str, Any]: + _require(protocol_version in PROTOCOLS, "unsupported EXP1 protocol version") + manifest_path = archive / "archive-manifest.json" + _require(manifest_path.is_file(), "archive-manifest.json is required") + manifest = _load_json(manifest_path) + _require( + isinstance(manifest, dict) and manifest.get("schema_version") == 1, + "unsupported archive manifest", + ) + inventory = _verify_archive_inventory(archive, manifest) + required = { + "campaign-manifest.json", + "run-manifest.json", + "expanded-plan.json", + "report.json", + "environment-preflight.txt", + "fixture-manifest.json", + } + _require( + required <= set(inventory), + f"archive is missing required members: {sorted(required - set(inventory))}", + ) + + campaign = _load_json(archive / "campaign-manifest.json") + run = _load_json(archive / "run-manifest.json", envelope=True) + expanded = _load_json(archive / "expanded-plan.json", envelope=True) + report = _load_json(archive / "report.json", envelope=True) + preflight = _load_json(archive / "environment-preflight.txt") + fixture = _load_json(archive / "fixture-manifest.json") + _require( + all(isinstance(item, dict) for item in (campaign, preflight, fixture)), + "invalid archive metadata", + ) + + disposition = manifest.get("disposition") + _require( + disposition in EXPECTED_COUNTS, + f"unsupported table-generation disposition: {disposition!r}", + ) + expected = EXPECTED_COUNTS[disposition] + _require( + manifest.get("protocol_version") + in ((None, "v1.0") if protocol_version == "v1.0" else ("v1.1",)), + "archive protocol identity is invalid", + ) + _validate_protocol_transport( + campaign, run, expanded, protocol_version, disposition + ) + if disposition == "final": + protocol = campaign.get("protocol") + paper_git = campaign.get("paper_git") + _require( + isinstance(protocol, Mapping) + and protocol.get("freeze_state") == "frozen", + "final campaign protocol is not frozen", + ) + _require( + isinstance(paper_git, Mapping) + and _is_git_sha1(paper_git.get("commit")) + and paper_git.get("dirty") is False + and paper_git.get("status_porcelain") == [] + and paper_git.get("freeze_state") == "clean_frozen_commit", + "final paper source freeze provenance is invalid", + ) + _require( + campaign.get("disposition") == disposition, "campaign/archive disposition drift" + ) + _require( + campaign.get("eligibility") == expected["campaign_eligibility"], + "campaign eligibility drift", + ) + _require(run.get("name") == expected["preset"], "run preset/disposition drift") + _require( + campaign.get("state") == run.get("state") == report.get("state") == "completed", + "run state is not completed", + ) + _require( + campaign.get("correctness") + == run.get("correctness") + == report.get("correctness_verdict") + == "pass", + "correctness verdict drift or failure", + ) + _require( + report.get("warnings") == [], "report warnings make table derivation ineligible" + ) + + run_id = campaign.get("run_id") + _require( + isinstance(run_id, str) + and run_id + == manifest.get("run_id") + == run.get("run_id") + == report.get("run_id"), + "run identity drift", + ) + plan_hash = _nested(campaign, "plan", "hash") + _require( + isinstance(plan_hash, str) + and plan_hash + == run.get("plan_hash") + == expanded.get("plan_hash") + == report.get("plan_hash"), + "plan identity drift", + ) + _require( + _nested(campaign, "plan", "client_cohort") == "product_cli", + "non-product CLI cohort", + ) + _require( + _nested(run, "environment", "client_cohort") == "product_cli", + "run client cohort drift", + ) + + design = report.get("design_counts") + _require( + _nested(campaign, "plan", "cells") == expected["cells"] + and _nested(campaign, "plan", "trial_batches") == expected["trial_batches"] + and _nested(campaign, "plan", "issued_operation_requests") + == expected["requests"], + "campaign count contract drift", + ) + _require( + isinstance(design, Mapping) + and design.get("test_combinations") == expected["cells"] + and design.get("trial_batches") == expected["trial_batches"] + and design.get("issued_product_requests") == expected["requests"], + "report count contract drift", + ) + + product_commit = _nested(campaign, "product", "commit") + _require( + _is_git_sha1(product_commit) + and product_commit == _nested(campaign, "cleanup", "product_commit") + and product_commit == _nested(run, "producer", "source_commit") + and product_commit == _nested(run, "treatment", "source_commit") + and product_commit == report.get("source_commit"), + "product source identity drift", + ) + freeze_tag = _nested(campaign, "product", "freeze_tag") + if disposition == "final": + required_tag = PROTOCOLS[protocol_version]["final_tag"] + _require( + isinstance(freeze_tag, Mapping) + and freeze_tag.get("availability") == "available" + and freeze_tag.get("name") == required_tag + and freeze_tag.get("reference") == f"refs/tags/{required_tag}" + and freeze_tag.get("object_type") == "tag" + and _is_git_sha1(freeze_tag.get("tag_object")) + and freeze_tag.get("peeled_commit") == product_commit, + "final product freeze-tag provenance is invalid", + ) + elif freeze_tag is not None: + _require( + isinstance(freeze_tag, Mapping) + and freeze_tag.get("availability") == "unavailable" + and freeze_tag.get("required_final_tag") + == PROTOCOLS[protocol_version]["final_tag"], + "exploratory product freeze-tag provenance is invalid", + ) + _require( + _nested(campaign, "product", "dirty") is False + and _nested(run, "treatment", "source_dirty") is False + and _nested(campaign, "cleanup", "product_status_porcelain") == "", + "dirty product source is ineligible", + ) + binary_pairs = { + "daemon": "daemon_binary_hash", + "gateway": "gateway_binary_hash", + "manager": "manager_cli_binary_hash", + "runtime": "runtime_cli_binary_hash", + "observability": "observability_cli_binary_hash", + } + for binary, treatment_key in binary_pairs.items(): + _require( + _nested(campaign, "product", "binaries", binary, "sha256") + == _nested(run, "treatment", treatment_key), + f"{binary} binary identity drift", + ) + _require( + _nested(campaign, "image", "id") == _nested(run, "environment", "image_digest"), + "container image identity drift", + ) + cleanup = campaign.get("cleanup") + _require( + isinstance(cleanup, Mapping) + and cleanup.get("run_workspace_exists") is False + and cleanup.get("runtime_exists") is False + and all( + cleanup.get(key) == [] + for key in ( + "gateway_labeled_containers", + "gateway_labeled_volumes", + "matching_product_processes", + "run_labeled_containers", + "run_labeled_volumes", + ) + ), + "terminal cleanup proof failed", + ) + _require( + _nested(fixture, "identity", "profile_id") == "paper-100m" + and _nested(fixture, "identity", "fixture", "file_count") == 4000 + and _nested(fixture, "identity", "fixture", "logical_bytes") == 104857600 + and _nested(fixture, "identity", "fixture", "maximum_depth") == 100 + and _nested(fixture, "identity", "seed") == 20260712, + "fixture identity drift", + ) + _require( + fixture.get("fixture_hash") == _nested(campaign, "fixture", "fixture_hash") + and fixture.get("tree_hash") == _nested(campaign, "fixture", "tree_hash"), + "fixture/campaign identity drift", + ) + + plan_cells = expanded.get("cells") + report_cells = report.get("cells") + _require( + isinstance(plan_cells, list) + and isinstance(report_cells, list) + and len(plan_cells) == len(report_cells) == expected["cells"], + "cell count drift", + ) + report_by_id: dict[str, Mapping[str, Any]] = {} + for cell in report_cells: + _require(isinstance(cell, Mapping), "report cell is not an object") + cell_id = cell.get("cell_id") + _require( + isinstance(cell_id, str) and cell_id not in report_by_id, + "duplicate/invalid report cell id", + ) + report_by_id[cell_id] = cell + + cells: dict[tuple[str, Any, int], dict[str, Any]] = {} + for plan_cell in plan_cells: + _require(isinstance(plan_cell, Mapping), "expanded-plan cell is not an object") + cell_id = plan_cell.get("cell_id") + _require(cell_id in report_by_id, f"expanded-plan/report cell drift: {cell_id}") + report_cell = report_by_id[cell_id] + _require( + report_cell.get("operation_id") == plan_cell.get("operation_id"), + f"operation drift: {cell_id}", + ) + key = _cell_key(plan_cell) + _require(key not in cells, f"duplicate EXP1 matrix cell {key}") + factors = _nested(plan_cell, "operation", "cell") + _validate_factor_contract(key, factors) + protocol = plan_cell.get("protocol") + _require( + isinstance(protocol, Mapping) + and protocol.get("warmups") == expected["warmups"] + and protocol.get("measured_trials") == expected["measured"], + f"{key}: trial-count drift", + ) + counts = report_cell.get("counts") + _require( + isinstance(counts, Mapping) + and counts.get("warmup") == expected["warmups"] + and counts.get("measured_attempted") == expected["measured"] + and counts.get("total_attempted") + == expected["warmups"] + expected["measured"] + and counts.get("successful") == expected["measured"] + and all( + counts.get(field) == 0 + for field in ( + "product_failed", + "correctness_failed", + "infrastructure_failed", + "cleanup_invalid", + "missing_primary_latency", + ) + ), + f"{key}: reportable trial/correctness count drift", + ) + checks = report_cell.get("checks") + _require( + isinstance(checks, list) + and bool(checks) + and all( + check.get("attempted") == expected["measured"] + and check.get("passed") == expected["measured"] + and check.get("failed") == 0 + for check in checks + if isinstance(check, Mapping) + ) + and all(isinstance(check, Mapping) for check in checks), + f"{key}: correctness check drift", + ) + latency_metric = _metric(report_cell, "batch_makespan_ns") + throughput_metric = _metric(report_cell, "throughput_ops_s") + _require( + _nested(latency_metric, "identity", "unit") == "nanoseconds", + f"{key}: latency unit drift", + ) + _require( + _nested(throughput_metric, "identity", "unit") == "operations_per_second", + f"{key}: throughput unit drift", + ) + _raw_values(latency_metric, expected["measured"], allow_unavailable=False) + _raw_values(throughput_metric, expected["measured"], allow_unavailable=False) + cells[key] = { + "cell_id": cell_id, + "factors": factors, + "plan_cell": plan_cell, + "report_cell": report_cell, + } + _require(set(cells) == _expected_keys(), "EXP1 19-cell matrix identity drift") + + # If duplicate raw metadata is archived, it must be byte-identical. + for relative in ( + "report.json", + "run-manifest.json", + "expanded-plan.json", + ): + raw_relative = f"raw/{relative}" + if raw_relative in inventory: + _require( + inventory[relative].get("sha256") + == inventory[raw_relative].get("sha256"), + f"root/raw duplicate drift: {relative}", + ) + + source_hashes = { + relative: str(entry["sha256"]) + for relative, entry in inventory.items() + if relative in required + } + return { + "manifest": manifest, + "inventory": inventory, + "campaign": campaign, + "run": run, + "expanded": expanded, + "report": report, + "preflight": preflight, + "fixture": fixture, + "cells": cells, + "expected": expected, + "disposition": disposition, + "protocol_version": protocol_version, + "source_hashes": source_hashes, + } + + +def _table_one( + context: Mapping[str, Any], evidence: Evidence, banner: str +) -> tuple[bytes, dict[str, Any]]: + campaign = context["campaign"] + run = context["run"] + preflight = context["preflight"] + fixture = context["fixture"] + disposition = context["disposition"] + protocol_version = context["protocol_version"] + expected = context["expected"] + run_environment = run.get("environment") + _require(isinstance(run_environment, Mapping), "run environment is invalid") + gateway_transport = ( + run_environment.get("gateway_transport") + if protocol_version == "v1.1" + else V10_GATEWAY_TRANSPORT + ) + _require( + isinstance(gateway_transport, Mapping) + and all( + isinstance(gateway_transport.get(key), str) + and bool(gateway_transport[key]) + for key in ("transport", "scope", "rotation") + ), + "gateway transport provenance is invalid", + ) + gateway_transport = { + key: gateway_transport[key] for key in ("transport", "scope", "rotation") + } + gateway_transport_display = ( + f"{gateway_transport['transport']}; {gateway_transport['scope']}; " + f"{gateway_transport['rotation']}" + ) + product_commit = _nested(campaign, "product", "commit") + freeze_tag = _nested(campaign, "product", "freeze_tag") + if disposition == "final": + product_commit_tag = f"{product_commit}; annotated tag {freeze_tag['name']}" + else: + product_commit_tag = f"{product_commit}; tag unavailable" + host = _first( + preflight, + [ + ("recorded_run_environment", "host"), + ("run_environment", "host"), + ("host",), + ], + ) + if not isinstance(host, Mapping): + host = _nested(run, "environment", "host") or {} + run_host = _nested(run, "environment", "host") or {} + docker = _first( + preflight, [("docker",), ("recorded_run_environment", "host", "docker")] + ) + if not isinstance(docker, Mapping): + docker = campaign.get("docker") or {} + cpu_model = _first(host, [("cpu_model",), ("processor_model",)]) + logical_processors = _first(host, [("logical_processors",), ("logical_cpu_count",)]) + host_os_name = host.get("operating_system") or run_host.get("operating_system") + host_os_caption = host.get("os_caption") or run_host.get("os_caption") + host_os_edition = _first( + host, + [("os_edition",), ("operating_system_edition",)], + ) or _first(run_host, [("os_edition",), ("operating_system_edition",)]) + host_os_build = _first( + host, + [("os_build_number",), ("os_build",), ("build_number",)], + ) or _first( + run_host, + [("os_build_number",), ("os_build",), ("build_number",)], + ) + # The canonical Windows capture stores a full OS caption, so do not prefix it + # with the generic operating-system family (for example, "windows"). + host_os_parts = ( + [host_os_caption] + if host_os_caption is not None + else [host_os_name, host_os_edition] + ) + if host_os_build is not None: + host_os_parts.append(f"build {host_os_build}") + host_os_display = ( + " ".join(str(part) for part in host_os_parts if part) or "unavailable" + ) + if cpu_model is None and logical_processors is None: + cpu_display: Any = "unavailable" + elif logical_processors is None: + cpu_display = str(cpu_model) + elif cpu_model is None: + cpu_display = f"{logical_processors} logical processors" + else: + cpu_display = f"{cpu_model} / {logical_processors} logical processors" + sandbox_limits = _first( + preflight, [("sandbox_limits",), ("recorded_run_environment", "sandbox_limits")] + ) + storage_model = _first(host, [("storage_model",), ("volume_model",)]) + storage_capacity = _first( + host, [("storage_capacity_bytes",), ("volume_capacity_bytes",)] + ) + storage_filesystem = host.get("filesystem") or run_host.get("filesystem") + storage_parts = [] + if storage_model is not None: + storage_parts.append(str(storage_model)) + if isinstance(storage_capacity, (int, float)): + storage_parts.append(f"{_display(float(storage_capacity) / (1024.0**3))} GiB") + if storage_filesystem is not None: + storage_parts.append(str(storage_filesystem)) + storage_display = " / ".join(storage_parts) or "unavailable" + sandbox_limits_display: Any = sandbox_limits + if isinstance(sandbox_limits, Mapping): + limit_parts = [] + cpu_limit = sandbox_limits.get("vcpus", sandbox_limits.get("cpu_count")) + memory_limit = sandbox_limits.get("memory_bytes") + pids_limit = sandbox_limits.get("pids_limit", sandbox_limits.get("pids")) + if isinstance(cpu_limit, (int, float)): + limit_parts.append(f"{_display(float(cpu_limit))} vCPU") + if isinstance(memory_limit, (int, float)): + limit_parts.append(f"{_display(float(memory_limit) / MIB)} MiB") + if isinstance(pids_limit, (int, float)): + limit_parts.append(f"{_display(float(pids_limit))} PIDs") + sandbox_limits_display = " / ".join(limit_parts) or json.dumps( + sandbox_limits, separators=(",", ":"), sort_keys=True + ) + + fields: list[tuple[str, Any, str, Mapping[str, Any], str]] = [ + ( + "Host OS", + host_os_display, + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host"}, + "text", + ), + ( + "Container engine OS", + docker.get("os_type"), + "environment-preflight.txt", + {"pointer": "/docker/os_type"}, + "text", + ), + ( + "Architecture", + host.get("architecture") or run_host.get("architecture"), + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host/architecture"}, + "text", + ), + ( + "CPU", + cpu_display, + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host"}, + "text", + ), + ( + "Memory", + _first(host, [("total_memory_bytes",), ("memory_bytes",)]) or "unavailable", + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host/total_memory_bytes"}, + "bytes", + ), + ( + "Storage", + storage_display, + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host"}, + "text", + ), + ( + "Docker Engine", + docker.get("server_version") or run_host.get("docker_engine_version"), + "environment-preflight.txt", + {"pointer": "/docker/server_version"}, + "text", + ), + ( + "Cgroup", + docker.get("cgroup_version"), + "environment-preflight.txt", + {"pointer": "/docker/cgroup_version"}, + "text", + ), + ( + "Product commit/tag", + product_commit_tag, + "campaign-manifest.json", + {"pointer": "/product"}, + "text", + ), + ( + "Benchmark commit", + _nested(campaign, "paper_git", "commit"), + "campaign-manifest.json", + {"pointer": "/paper_git/commit"}, + "text", + ), + ( + "Sandbox image", + _nested(campaign, "image", "requested"), + "campaign-manifest.json", + {"pointer": "/image/requested"}, + "text", + ), + ( + "Sandbox limits", + sandbox_limits_display or "unavailable", + "environment-preflight.txt", + {"pointer": "/sandbox_limits"}, + "text", + ), + ( + "Workspace", + ( + f"{_nested(fixture, 'identity', 'fixture', 'logical_bytes') // (1024 * 1024)} MiB / " + f"{_nested(fixture, 'identity', 'fixture', 'file_count')} files / " + f"depth {_nested(fixture, 'identity', 'fixture', 'maximum_depth')}" + ), + "fixture-manifest.json", + {"pointer": "/identity/fixture"}, + "text", + ), + ( + "Client", + _nested(campaign, "plan", "client_cohort"), + "campaign-manifest.json", + {"pointer": "/plan/client_cohort"}, + "text", + ), + ( + "Gateway transport", + gateway_transport_display, + "run-manifest.json", + { + "pointer": ( + "/data/environment/gateway_transport" + if protocol_version == "v1.1" + else "/data/environment/gateway_endpoint_identity" + ), + "protocol_version": protocol_version, + }, + "text", + ), + ( + "Seed", + _nested(fixture, "identity", "seed"), + "fixture-manifest.json", + {"pointer": "/identity/seed"}, + "count", + ), + ( + "Trials", + f"{expected['warmups']} warm-up + {expected['measured']} measured", + "expanded-plan.json", + {"pointer": "/data/cells/*/protocol"}, + "text", + ), + ] + + # Missing qualification facts are tolerated only for explicitly ineligible pilot output. + if disposition == "final": + missing = [ + name for name, value, _, _, _ in fields if value in (None, "unavailable") + ] + required_qualification_facts = { + "host OS caption/edition": host_os_caption or host_os_edition, + "host OS build": host_os_build, + "CPU model": cpu_model, + "logical processor count": logical_processors, + "host memory": _first(host, [("total_memory_bytes",), ("memory_bytes",)]), + "storage filesystem": storage_filesystem, + "sandbox limits": sandbox_limits + if isinstance(sandbox_limits, Mapping) + else None, + } + missing.extend( + name + for name, value in required_qualification_facts.items() + if value is None + ) + if ( + not isinstance(storage_filesystem, str) + or storage_filesystem.upper() != "NTFS" + ): + missing.append("storage filesystem must be NTFS") + _require( + not missing, + f"final archive lacks required environment fields: {', '.join(missing)}", + ) + + # Register the numeric components embedded in descriptive Table 1 cells. + embedded_numeric = [ + ( + "table1.workspace_fixture.logical_mib", + _nested(fixture, "identity", "fixture", "logical_bytes") / MIB, + "fixture-manifest.json", + {"pointer": "/identity/fixture/logical_bytes"}, + "bytes", + "MiB", + {"divisor": 1048576}, + ), + ( + "table1.workspace_fixture.file_count", + _nested(fixture, "identity", "fixture", "file_count"), + "fixture-manifest.json", + {"pointer": "/identity/fixture/file_count"}, + "count", + "count", + {}, + ), + ( + "table1.workspace_fixture.maximum_depth", + _nested(fixture, "identity", "fixture", "maximum_depth"), + "fixture-manifest.json", + {"pointer": "/identity/fixture/maximum_depth"}, + "count", + "count", + {}, + ), + ( + "table1.trials.warmups", + expected["warmups"], + "expanded-plan.json", + {"pointer": "/data/cells/*/protocol/warmups", "invariant": "all equal"}, + "count", + "count", + {}, + ), + ( + "table1.trials.measured", + expected["measured"], + "expanded-plan.json", + { + "pointer": "/data/cells/*/protocol/measured_trials", + "invariant": "all equal", + }, + "count", + "count", + {}, + ), + ] + if isinstance(logical_processors, (int, float)): + embedded_numeric.append( + ( + "table1.cpu.logical_processors", + logical_processors, + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host/logical_processors"}, + "count", + "count", + {}, + ) + ) + if isinstance(storage_capacity, (int, float)): + embedded_numeric.append( + ( + "table1.storage.capacity_gib", + storage_capacity / (1024.0**3), + "environment-preflight.txt", + {"pointer": "/recorded_run_environment/host/storage_capacity_bytes"}, + "bytes", + "GiB", + {"divisor": 1073741824}, + ) + ) + if isinstance(sandbox_limits, Mapping): + limit_units = { + "vcpus": "count", + "cpu_count": "count", + "memory_bytes": "bytes", + "pids": "count", + "pids_limit": "count", + } + for field, unit in sorted(limit_units.items()): + value = sandbox_limits.get(field) + if isinstance(value, (int, float)): + output_unit = "MiB" if unit == "bytes" else unit + converted = value / MIB if unit == "bytes" else value + embedded_numeric.append( + ( + f"table1.sandbox_limits.{field}", + converted, + "environment-preflight.txt", + {"pointer": f"/sandbox_limits/{field}"}, + unit, + output_unit, + {"divisor": 1048576} if unit == "bytes" else {}, + ) + ) + for ( + evidence_id, + value, + source, + selector, + input_unit, + output_unit, + parameters, + ) in embedded_numeric: + evidence.add( + evidence_id, + value, + source_file=source, + selector=selector, + input_unit=input_unit, + output_unit=output_unit, + aggregation="identity", + parameters=parameters, + ) + + rows: list[list[str]] = [] + machine_fields: list[dict[str, Any]] = [] + for name, value, source, selector, unit in fields: + displayed = "unavailable" if value is None else str(value) + if unit == "bytes" and isinstance(value, (int, float)): + evidence_id = "table1." + name.lower().replace(" ", "_").replace("/", "_") + if name == "Memory": + evidence.add( + evidence_id + "_bytes", + value, + source_file=source, + selector=selector, + input_unit="bytes", + output_unit="bytes", + aggregation="identity", + ) + displayed = f"{int(value):,} bytes" + else: + mib = float(value) / MIB + evidence.add( + evidence_id, + mib, + source_file=source, + selector=selector, + input_unit="bytes", + output_unit="MiB", + aggregation="identity", + parameters={"divisor": 1048576}, + ) + displayed = f"{_display(mib)} MiB" + elif unit == "count" and isinstance(value, (int, float)): + evidence_id = "table1." + name.lower().replace(" ", "_").replace("/", "_") + evidence.add( + evidence_id, + value, + source_file=source, + selector=selector, + input_unit="count", + output_unit="count", + aggregation="identity", + ) + evidence_source = ( + f"{source}#{json.dumps(selector, separators=(',', ':'), sort_keys=True)}" + ) + rows.append([name, displayed, evidence_source]) + machine_value = value + if name == "Product commit/tag": + machine_value = { + "commit": product_commit, + "freeze_tag": freeze_tag + if freeze_tag is not None + else { + "availability": "unavailable", + "reason": "legacy pre-freeze exploratory archive", + "required_final_tag": PROTOCOLS[protocol_version]["final_tag"], + }, + } + elif name == "Gateway transport": + machine_value = gateway_transport + machine_fields.append( + { + "field": name, + "value": machine_value, + "display": displayed, + "source": source, + "selector": selector, + } + ) + return ( + _markdown_table( + ["Field", "Archived value", "Evidence source"], + rows, + banner, + "EXP1 environment and workload", + ), + {"fields": machine_fields}, + ) + + +def _performance_row( + key: tuple[str, Any, int], + context: Mapping[str, Any], + evidence: Evidence, + table_id: str, +) -> tuple[list[str], dict[str, Any]]: + cell = context["cells"][key] + measured = context["expected"]["measured"] + report_cell = cell["report_cell"] + latency = _raw_values( + _metric(report_cell, "batch_makespan_ns"), measured, allow_unavailable=False + ) + throughput = _raw_values( + _metric(report_cell, "throughput_ops_s"), measured, allow_unavailable=False + ) + assert latency is not None and throughput is not None + cell_id = cell["cell_id"] + label = _label_for_key(key) + concurrency = key[2] + selector_latency = _archive_selector( + "report.json", cell_id=cell_id, metric_id="batch_makespan_ns" + ) + selector_throughput = _archive_selector( + "report.json", cell_id=cell_id, metric_id="throughput_ops_s" + ) + slug = f"{table_id}.{key[0]}.{str(key[1]).lower()}.c{concurrency}" + values: dict[str, float | int] = {"samples": measured} + evidence.add( + slug + ".samples", + measured, + source_file="report.json", + selector={**selector_latency, "field": "available_n"}, + input_unit="count", + output_unit="count", + aggregation="identity", + ) + for label_id, probability in (("p50_ms", 0.50), ("p95_ms", 0.95), ("p99_ms", 0.99)): + derived = _convert( + _quantile(latency, probability), "nanoseconds", "milliseconds" + ) + values[label_id] = evidence.add( + slug + "." + label_id, + derived, + source_file="report.json", + selector=selector_latency, + input_unit="nanoseconds", + output_unit="milliseconds", + aggregation=f"linear_quantile_p{int(probability * 100)}", + parameters={ + "probability": probability, + "position": "(n-1)*q", + "interpolation": "linear", + "unit_divisor": 1000000, + }, + ) + throughput_value = _mean(throughput) + values["throughput_ops_s"] = evidence.add( + slug + ".throughput_ops_s", + throughput_value, + source_file="report.json", + selector=selector_throughput, + input_unit="operations_per_second", + output_unit="operations_per_second", + aggregation="arithmetic_mean", + parameters={"denominator": measured}, + ) + rendered = [ + label, + str(concurrency), + str(measured), + _display(float(values["p50_ms"])), + _display(float(values["p95_ms"])), + _display(float(values["p99_ms"])), + _display(float(values["throughput_ops_s"]), digits=2), + ] + return rendered, { + "cell_id": cell_id, + "key": list(key), + "operation": label, + "concurrency": concurrency, + "values": values, + } + + +def _performance_table( + keys: Sequence[tuple[str, Any, int]], + context: Mapping[str, Any], + evidence: Evidence, + banner: str, + table_id: str, + title: str, +) -> tuple[bytes, dict[str, Any]]: + rows: list[list[str]] = [] + machine_rows: list[dict[str, Any]] = [] + for key in keys: + rendered, machine = _performance_row(key, context, evidence, table_id) + if table_id == "table2": + stage = { + ("create_sandbox", None, 1): "Sandbox create + base mount", + ("create_workspace", None, 1): "Session create to ready", + ("create_workspace", None, 5): "Session create to ready", + ("exec_command", "noop", 1): "First no-op command", + }[key] + rendered = [stage, *rendered[1:]] + machine["stage"] = stage + elif table_id == "table3": + operation, case, size = { + ("exec_command", "noop", 1): ("`exec_command`", "no-op", "--"), + ("exec_command", "noop", 5): ("`exec_command`", "no-op", "--"), + ("exec_command", "fixture_read", 1): ( + "`exec_command`", + "fixture read", + "4 KiB", + ), + ("exec_command", "fixture_read", 5): ( + "`exec_command`", + "fixture read", + "4 KiB", + ), + ("file_read", 4096, 1): ("Read", "snapshot", "4 KiB"), + ("file_read", 4096, 5): ("Read", "snapshot", "4 KiB"), + ("file_read", 262144, 1): ("Read", "snapshot", "256 KiB"), + ("file_read", 262144, 5): ("Read", "snapshot", "256 KiB"), + ("file_write", 4096, 1): ("Write", "session-local", "4 KiB"), + ("file_write", 4096, 5): ("Write", "session-local", "4 KiB"), + ("file_write", 262144, 1): ( + "Write", + "session-local", + "256 KiB", + ), + ("file_write", 262144, 5): ( + "Write", + "session-local", + "256 KiB", + ), + ("file_edit", 4096, 1): ("Edit", "one replacement", "4 KiB"), + ("file_edit", 4096, 5): ("Edit", "one replacement", "4 KiB"), + ("file_edit", 262144, 1): ( + "Edit", + "one replacement", + "256 KiB", + ), + ("file_edit", 262144, 5): ( + "Edit", + "one replacement", + "256 KiB", + ), + }[key] + rendered = [operation, case, size, *rendered[1:]] + machine.update( + operation_label=operation, + case=case, + payload_or_file_size=size, + ) + rows.append(rendered) + machine_rows.append(machine) + if table_id == "table2": + headers = [ + "Stage", + "Concurrent creates", + "Samples", + "p50 (ms)", + "p95 (ms)", + "p99 (ms)", + "Throughput (ready/s)", + ] + else: + headers = [ + "Operation", + "Case", + "Payload/file size", + "Concurrency", + "Samples", + "p50 (ms)", + "p95 (ms)", + "p99 (ms)", + "Throughput (ops/s)", + ] + return _markdown_table(headers, rows, banner, title), {"rows": machine_rows} + + +def _resource_table( + context: Mapping[str, Any], + evidence: Evidence, + banner: str, +) -> tuple[bytes, dict[str, Any]]: + headers = ["Operation/case", "Concurrency"] + [ + column[1] for column in RESOURCE_COLUMNS + ] + rows: list[list[str]] = [] + machine_rows: list[dict[str, Any]] = [] + measured = context["expected"]["measured"] + for key in RESOURCE_KEYS: + cell = context["cells"][key] + cell_id = cell["cell_id"] + operation_case = { + ("create_workspace", None, 1): "Workspace create, 100 MiB/depth 100", + ("create_workspace", None, 5): "Workspace create, 100 MiB/depth 100", + ("exec_command", "noop", 1): "`exec_command`, no-op", + ("exec_command", "noop", 5): "`exec_command`, no-op", + ("file_read", 262144, 5): "Read, 256 KiB", + ("file_write", 262144, 5): "Write, 256 KiB", + ("file_edit", 262144, 5): "Edit, 256 KiB", + }[key] + rendered = [operation_case, str(key[2])] + machine_values: dict[str, Any] = {} + for metric_id, _, input_unit, output_unit, aggregate in RESOURCE_COLUMNS: + metric = _metric(cell["report_cell"], metric_id) + identity_unit = _nested(metric, "identity", "unit") + _require( + identity_unit == input_unit, f"{key}/{metric_id}: resource unit drift" + ) + values = _raw_values(metric, measured, allow_unavailable=True) + unavailable_reason: str | None = None + if metric_id == "upperdir_bytes" and ( + _nested(metric, "identity", "aggregation") != "delta" + or _nested(metric, "identity", "scope") != "workspace" + or _nested(metric, "identity", "source") + != ( + "product_observability.snapshot.workspaces.disk_allocated_bytes.sum" + ) + ): + values = None + unavailable_reason = ( + "archived metric is not the product-reported before/after " + "live-workspace upperdir allocated-space delta" + ) + if values is None: + if unavailable_reason is None: + reasons = _nested(metric, "unavailable", "reasons") + if isinstance(reasons, Mapping) and reasons: + unavailable_reason = "; ".join( + sorted(str(reason) for reason in reasons) + ) + else: + unavailable_reason = "incomplete archived measured samples" + rendered.append("unavailable") + machine_values[metric_id] = { + "value": None, + "display": "unavailable", + "reason": unavailable_reason, + } + continue + aggregate_value = max(values) if aggregate == "max" else _mean(values) + converted = _convert(aggregate_value, input_unit, output_unit) + slug = f"table4.{key[0]}.{str(key[1]).lower()}.c{key[2]}.{metric_id}" + evidence.add( + slug, + converted, + source_file="report.json", + selector=_archive_selector( + "report.json", cell_id=cell_id, metric_id=metric_id + ), + input_unit=input_unit, + output_unit=output_unit, + aggregation="maximum" if aggregate == "max" else "arithmetic_mean", + parameters={ + "denominator": measured if aggregate == "mean" else None, + "unit_divisor": 1048576 if output_unit == "MiB" else 1000000, + }, + ) + display = _display(converted) + rendered.append(display) + machine_values[metric_id] = {"value": converted, "display": display} + rows.append(rendered) + machine_rows.append( + { + "cell_id": cell_id, + "key": list(key), + "operation": operation_case, + "concurrency": key[2], + "values": machine_values, + } + ) + return _markdown_table(headers, rows, banner, "EXP1 resource observations"), { + "rows": machine_rows + } + + +def _build_outputs(context: Mapping[str, Any], generator_sha: str) -> dict[str, bytes]: + disposition = context["disposition"] + eligibility = context["expected"]["output_eligibility"] + banner = _eligibility_banner(disposition) + evidence = Evidence(context["source_hashes"], eligibility) + table1, machine1 = _table_one(context, evidence, banner) + table2, machine2 = _performance_table( + STARTUP_KEYS, + context, + evidence, + banner, + "table2", + "EXP1 startup and workspace creation", + ) + table3, machine3 = _performance_table( + PUBLIC_KEYS, context, evidence, banner, "table3", "EXP1 public CLI operations" + ) + table4, machine4 = _resource_table(context, evidence, banner) + tables = { + "schema_version": SCHEMA_VERSION, + "archive": { + "content_tree_sha256": context["manifest"]["content_tree_sha256"], + "disposition": disposition, + "run_id": context["campaign"]["run_id"], + }, + "eligibility": eligibility, + "protocol_version": context["protocol_version"], + "generator_schema_version": SCHEMA_VERSION, + "tables": { + "environment": machine1, + "startup": machine2, + "public_cli_operations": machine3, + "resources": machine4, + }, + } + outputs = { + "table-1-environment.md": table1, + "table-2-startup.md": table2, + "table-3-cli-operations.md": table3, + "table-4-resources.md": table4, + "tables.json": _json_bytes(tables), + "numeric-provenance.csv": evidence.csv_bytes(), + "numeric-evidence.json": _json_bytes(evidence.registry()), + } + log_lines = [ + "EXP1 deterministic table generation", + f"generator_schema_version={SCHEMA_VERSION}", + f"generator_sha256={generator_sha}", + f"archive_run_id={context['campaign']['run_id']}", + f"archive_disposition={disposition}", + f"archive_eligibility={eligibility}", + f"protocol_version={context['protocol_version']}", + f"archive_content_tree_sha256={context['manifest']['content_tree_sha256']}", + "archive_inventory_verified=true", + "semantic_identity_verified=true", + "correctness_verified=true", + "command=python experiments/analysis/scripts/generate_exp1_tables.py --protocol-version --archive --output ", + "output_path_embedded=false", + "wall_clock_embedded=false", + ] + outputs["generation-log.txt"] = ("\n".join(log_lines) + "\n").encode("utf-8") + manifest_files = [ + { + "bytes": len(data), + "path": relative, + "sha256": _sha256_bytes(data), + } + for relative, data in sorted(outputs.items()) + ] + output_manifest = { + "archive_content_tree_sha256": context["manifest"]["content_tree_sha256"], + "archive_disposition": disposition, + "archive_run_id": context["campaign"]["run_id"], + "eligibility": eligibility, + "protocol_version": context["protocol_version"], + "files": manifest_files, + "generator_schema_version": SCHEMA_VERSION, + "generator_sha256": generator_sha, + "schema_version": SCHEMA_VERSION, + } + outputs["output-manifest.json"] = _json_bytes(output_manifest) + return outputs + + +def generate( + archive: Path, + output: Path, + protocol_version: str = "v1.0", +) -> dict[str, Any]: + archive = archive.resolve() + output = output.resolve() + _require(archive.is_dir(), f"archive directory does not exist: {archive}") + _require( + output != archive and archive not in output.parents, + "output must be outside the immutable archive", + ) + _require(not output.exists(), f"output directory already exists: {output}") + context = _validate_archive(archive, protocol_version) + generator_sha = _sha256_file(Path(__file__).resolve()) + outputs = _build_outputs(context, generator_sha) + # Detect a mutation that raced semantic derivation before writing any output. + _verify_archive_inventory(archive, context["manifest"]) + output.mkdir(parents=True, exist_ok=False) + for relative, data in sorted(outputs.items()): + destination = output / relative + with destination.open("xb") as stream: + stream.write(data) + return { + "archive_run_id": context["campaign"]["run_id"], + "disposition": context["disposition"], + "eligibility": context["expected"]["output_eligibility"], + "protocol_version": protocol_version, + "files": sorted(outputs), + } + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--archive", required=True, type=Path, help="immutable EXP1 archive directory" + ) + parser.add_argument( + "--output", + required=True, + type=Path, + help="new output directory outside the archive", + ) + parser.add_argument( + "--protocol-version", choices=sorted(PROTOCOLS), required=True + ) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = _parser().parse_args(argv) + try: + result = generate(args.archive, args.output, args.protocol_version) + except GenerationError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + print(json.dumps(result, separators=(",", ":"), sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/analysis/tests/test_generate_exp1_tables.py b/ephemeral-sandbox-v1/experiments/analysis/tests/test_generate_exp1_tables.py new file mode 100644 index 0000000..d2f153b --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/analysis/tests/test_generate_exp1_tables.py @@ -0,0 +1,1025 @@ +from __future__ import annotations + +import csv +import hashlib +import importlib.util +import json +import sys +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest + +SCRIPT = Path(__file__).parents[1] / "scripts" / "generate_exp1_tables.py" +SPEC = importlib.util.spec_from_file_location("generate_exp1_tables", SCRIPT) +assert SPEC and SPEC.loader +generator = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = generator +SPEC.loader.exec_module(generator) + + +RUN_ID = "00000000-0000-7000-8000-000000000001" +PLAN_HASH = "sha256:" + "1" * 64 +PRODUCT_COMMIT = "2" * 40 +IMAGE_DIGEST = "sha256:" + "3" * 64 +FIXTURE_HASH = "sha256:" + "4" * 64 +TREE_HASH = "sha256:" + "5" * 64 +BINARY_HASHES = { + "daemon": "sha256:" + "6" * 64, + "gateway": "sha256:" + "7" * 64, + "manager": "sha256:" + "8" * 64, + "runtime": "sha256:" + "9" * 64, + "observability": "sha256:" + "a" * 64, +} + + +def _json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(value, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + newline="\n", + ) + + +def _factors(operation: str, variant: Any, concurrency: int) -> dict[str, Any]: + factors: dict[str, Any] = {"workspace_profile": "paper-100m"} + if operation == "create_sandbox": + factors.update( + network_profile="shared", resolved_isolation="fresh_sandbox_per_trial" + ) + elif operation == "create_workspace": + factors.update(workspace_count=concurrency) + else: + factors["concurrent_requests"] = concurrency + if operation == "exec_command": + factors.update(command_case=variant, session_mode="explicit") + elif operation == "file_read": + factors.update(returned_bytes=variant, source="snapshot") + elif operation == "file_write": + factors.update(content_bytes=variant, destination="session") + elif operation == "file_edit": + factors.update( + file_bytes=variant, destination="session", replacement_count=1 + ) + return factors + + +def _metric( + metric_id: str, + unit: str, + values: list[float], + *, + aggregation: str = "mean", + scope: str | None = None, + source: str | None = None, +) -> dict[str, Any]: + identity: dict[str, Any] = { + "id": metric_id, + "unit": unit, + "aggregation": aggregation, + } + if scope is not None: + identity["scope"] = scope + if source is not None: + identity["source"] = source + return { + "identity": identity, + "attempted_n": len(values), + "failed_n": 0, + "available_n": len(values), + "unavailable": {"count": 0, "reasons": {}}, + "raw_points": [ + { + "trial_id": f"trial-synthetic-measured-{index:06d}", + "request_id": None, + "value": value, + "raw_integer_value": int(value) if value.is_integer() else None, + "outlier": False, + } + for index, value in enumerate(values) + ], + } + + +def _refresh_archive_manifest(root: Path, *, disposition: str = "exploratory") -> None: + files = [] + tree = hashlib.sha256() + total = 0 + for path in sorted( + ( + path + for path in root.rglob("*") + if path.is_file() and path.name != "archive-manifest.json" + ), + key=lambda item: item.relative_to(root).as_posix(), + ): + relative = path.relative_to(root).as_posix() + data = path.read_bytes() + digest = hashlib.sha256(data).digest() + files.append( + { + "bytes": len(data), + "path": relative, + "sha256": "sha256:" + digest.hex(), + } + ) + total += len(data) + tree.update(relative.encode("utf-8")) + tree.update(b"\0") + tree.update(digest) + tree.update(b"\n") + _json( + root / "archive-manifest.json", + { + "archive_bytes": total, + "archive_file_count": len(files), + "content_tree_sha256": "sha256:" + tree.hexdigest(), + "disposition": disposition, + "files": files, + "run_id": RUN_ID, + "schema_version": 1, + }, + ) + + +def _build_archive(root: Path, *, disposition: str = "exploratory") -> Path: + measured = 5 if disposition == "exploratory" else 100 + trial_batches = 133 if disposition == "exploratory" else 1938 + requests = 385 if disposition == "exploratory" else 5610 + eligibility = ( + "exploratory_ineligible" + if disposition == "exploratory" + else "frozen_final_candidate" + ) + preset = "paper-pilot" if disposition == "exploratory" else "paper-good-pass" + freeze_tag = ( + { + "availability": "available", + "name": "paper-v1-freeze", + "reference": "refs/tags/paper-v1-freeze", + "object_type": "tag", + "tag_object": "c" * 40, + "peeled_commit": PRODUCT_COMMIT, + } + if disposition == "final" + else { + "availability": "unavailable", + "reason": "pre-freeze smoke/exploratory archive", + "required_final_tag": "paper-v1-freeze", + } + ) + cells = [] + report_cells = [] + for index, (_, operation, variant, concurrency, _) in enumerate( + generator.PERFORMANCE_ROWS + ): + cell_id = f"sha256:{index:064x}" + factors = _factors(operation, variant, concurrency) + cells.append( + { + "cell_id": cell_id, + "operation_id": operation, + "operation": {"operation": operation, "cell": factors}, + "protocol": {"warmups": 2, "measured_trials": measured}, + } + ) + base = float((index + 1) * 1_000_000) + latency = [base + offset * 100_000.0 for offset in range(measured)] + throughput = [float(concurrency * 10 + offset) for offset in range(measured)] + resources = [ + _metric( + "daemon_rss_bytes", + "bytes", + [1048576.0 * (index + offset + 1) for offset in range(measured)], + ), + _metric( + "sandbox_memory_peak_bytes", + "bytes", + [2097152.0 * (offset + 1) for offset in range(measured)], + ), + _metric( + "sandbox_cpu_time_ns", + "nanoseconds", + [1_000_000.0 * (offset + 1) for offset in range(measured)], + ), + _metric( + "sandbox_block_read_bytes", + "bytes", + [1048576.0 * offset for offset in range(measured)], + ), + _metric( + "sandbox_block_write_bytes", + "bytes", + [1048576.0 * (offset + 1) for offset in range(measured)], + ), + _metric( + "upperdir_bytes", + "bytes", + [1048576.0 * (offset + 1) for offset in range(measured)], + aggregation="delta", + scope="workspace", + source=( + "product_observability.snapshot.workspaces.disk_allocated_bytes.sum" + ), + ), + ] + report_cells.append( + { + "cell_id": cell_id, + "operation_id": operation, + "counts": { + "total_attempted": 2 + measured, + "warmup": 2, + "measured_attempted": measured, + "successful": measured, + "product_failed": 0, + "correctness_failed": 0, + "infrastructure_failed": 0, + "cleanup_invalid": 0, + "missing_primary_latency": 0, + }, + "checks": [ + { + "id": "synthetic", + "attempted": measured, + "passed": measured, + "failed": 0, + } + ], + "metrics": [ + _metric("batch_makespan_ns", "nanoseconds", latency), + _metric("throughput_ops_s", "operations_per_second", throughput), + *resources, + ], + } + ) + + campaign = { + "schema_version": 1, + "run_id": RUN_ID, + "disposition": disposition, + "eligibility": eligibility, + "state": "completed", + "correctness": "pass", + "plan": { + "hash": PLAN_HASH, + "cells": 19, + "trial_batches": trial_batches, + "issued_operation_requests": requests, + "client_cohort": "product_cli", + }, + "product": { + "commit": PRODUCT_COMMIT, + "dirty": False, + "freeze_tag": freeze_tag, + "binaries": { + name: {"sha256": digest} for name, digest in BINARY_HASHES.items() + }, + }, + "cleanup": { + "product_commit": PRODUCT_COMMIT, + "product_status_porcelain": "", + "run_workspace_exists": False, + "runtime_exists": False, + "gateway_labeled_containers": [], + "gateway_labeled_volumes": [], + "matching_product_processes": [], + "run_labeled_containers": [], + "run_labeled_volumes": [], + }, + "image": { + "id": IMAGE_DIGEST, + "requested": "example.invalid/exp1@sha256:" + "3" * 64, + }, + "fixture": {"fixture_hash": FIXTURE_HASH, "tree_hash": TREE_HASH}, + "protocol": { + "id": generator.PROTOCOLS["v1.0"]["id"], + "version": "v1.0" if disposition == "final" else "pre-freeze-exp1", + "freeze_state": "frozen" if disposition == "final" else "pre_freeze", + }, + "paper_git": { + "commit": "b" * 40, + "dirty": False, + "status_porcelain": [], + "freeze_state": ( + "clean_frozen_commit" + if disposition == "final" + else "pre_freeze_worktree" + ), + }, + } + treatment = { + "source_commit": PRODUCT_COMMIT, + "source_dirty": False, + "daemon_binary_hash": BINARY_HASHES["daemon"], + "gateway_binary_hash": BINARY_HASHES["gateway"], + "manager_cli_binary_hash": BINARY_HASHES["manager"], + "runtime_cli_binary_hash": BINARY_HASHES["runtime"], + "observability_cli_binary_hash": BINARY_HASHES["observability"], + } + run = { + "schema_name": "eos_benchmark_run_manifest", + "schema_version": 2, + "data": { + "run_id": RUN_ID, + "name": preset, + "plan_hash": PLAN_HASH, + "state": "completed", + "correctness": "pass", + "producer": {"source_commit": PRODUCT_COMMIT}, + "treatment": treatment, + "environment": { + "client_cohort": "product_cli", + "image_digest": IMAGE_DIGEST, + "host": { + "operating_system": "windows", + "architecture": "amd64", + "filesystem": "NTFS", + "docker_engine_version": "29.0.1", + }, + }, + }, + } + expanded = { + "schema_name": "eos_benchmark_expanded_plan", + "schema_version": 1, + "data": {"plan_hash": PLAN_HASH, "cells": cells}, + } + report = { + "schema_name": "eos_benchmark_report", + "schema_version": 3, + "data": { + "run_id": RUN_ID, + "plan_hash": PLAN_HASH, + "state": "completed", + "correctness_verdict": "pass", + "source_commit": PRODUCT_COMMIT, + "design_counts": { + "test_combinations": 19, + "trial_batches": trial_batches, + "issued_product_requests": requests, + }, + "warnings": [], + "cells": report_cells, + }, + } + preflight = { + "docker": { + "architecture": "x86_64", + "cgroup_version": "2", + "os_type": "linux", + "server_version": "29.0.1", + }, + "recorded_run_environment": { + "host": { + "operating_system": "windows", + "os_edition": "Synthetic Pro", + "os_build": "99999", + "architecture": "amd64", + "cpu_model": "Synthetic CPU", + "logical_processors": 8, + "total_memory_bytes": 16 * 1024 * 1024 * 1024, + "storage_model": "Synthetic NVMe", + "storage_capacity_bytes": 4 * 1024 * 1024 * 1024 * 1024, + "filesystem": "NTFS", + } + }, + "sandbox_limits": { + "vcpus": 1, + "memory_bytes": 512 * 1024 * 1024, + "pids_limit": 256, + }, + } + fixture = { + "schema_version": 2, + "fixture_hash": FIXTURE_HASH, + "tree_hash": TREE_HASH, + "identity": { + "profile_id": "paper-100m", + "seed": 20260712, + "fixture": { + "file_count": 4000, + "logical_bytes": 104857600, + "maximum_depth": 100, + }, + }, + } + _json(root / "campaign-manifest.json", campaign) + _json(root / "run-manifest.json", run) + _json(root / "expanded-plan.json", expanded) + _json(root / "report.json", report) + _json(root / "environment-preflight.txt", preflight) + _json(root / "fixture-manifest.json", fixture) + _refresh_archive_manifest(root, disposition=disposition) + return root + + +def _file_hashes(root: Path) -> dict[str, str]: + return { + path.name: hashlib.sha256(path.read_bytes()).hexdigest() + for path in sorted(root.iterdir()) + if path.is_file() + } + + +def _markdown_table(path: Path) -> tuple[list[str], list[list[str]]]: + table_lines = [ + line + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith("| ") + ] + parsed = [ + [cell.strip() for cell in line.strip("|").split("|")] for line in table_lines + ] + return parsed[0], parsed[2:] + + +def test_generation_is_byte_deterministic_and_marks_pilot_ineligible( + tmp_path: Path, +) -> None: + archive = _build_archive(tmp_path / "archive") + first = tmp_path / "first" + second = tmp_path / "second" + + result = generator.generate(archive, first) + generator.generate(archive, second) + + assert result["eligibility"] == "exploratory_ineligible" + assert _file_hashes(first) == _file_hashes(second) + assert len(_file_hashes(first)) == 9 + for table in ( + "table-1-environment.md", + "table-2-startup.md", + "table-3-cli-operations.md", + "table-4-resources.md", + ): + assert "INELIGIBLE EXPLORATORY OUTPUT" in (first / table).read_text( + encoding="utf-8" + ) + registry = json.loads((first / "numeric-evidence.json").read_text(encoding="utf-8")) + assert registry["schema_version"] == "ai-research-writing/numeric-evidence-v2" + assert registry["entries"] + assert all( + entry["source"] == "numeric-provenance.csv" for entry in registry["entries"] + ) + assert "linear_quantile_p99" in (first / "numeric-provenance.csv").read_text( + encoding="utf-8" + ) + tables = json.loads((first / "tables.json").read_text(encoding="utf-8")) + assert tables["schema_version"] == 2 + assert tables["generator_schema_version"] == 2 + assert len(tables["tables"]["startup"]["rows"]) == 4 + assert len(tables["tables"]["public_cli_operations"]["rows"]) == 16 + assert len(tables["tables"]["resources"]["rows"]) == 7 + + table1_header, table1_rows = _markdown_table(first / "table-1-environment.md") + assert table1_header == ["Field", "Archived value", "Evidence source"] + assert [row[0] for row in table1_rows] == [ + "Host OS", + "Container engine OS", + "Architecture", + "CPU", + "Memory", + "Storage", + "Docker Engine", + "Cgroup", + "Product commit/tag", + "Benchmark commit", + "Sandbox image", + "Sandbox limits", + "Workspace", + "Client", + "Gateway transport", + "Seed", + "Trials", + ] + assert all(row[2] for row in table1_rows) + product_row = next(row for row in table1_rows if row[0] == "Product commit/tag") + assert product_row[1] == f"{PRODUCT_COMMIT}; tag unavailable" + product_machine_field = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Product commit/tag" + ) + assert product_machine_field["value"]["freeze_tag"]["availability"] == "unavailable" + transport_row = next( + row for row in table1_rows if row[0] == "Gateway transport" + ) + assert transport_row[1] == ( + "tcp_loopback; local_only; per_execution_block" + ) + transport_machine_field = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Gateway transport" + ) + assert transport_machine_field["value"] == generator.V10_GATEWAY_TRANSPORT + + table2_header, table2_rows = _markdown_table(first / "table-2-startup.md") + assert table2_header == [ + "Stage", + "Concurrent creates", + "Samples", + "p50 (ms)", + "p95 (ms)", + "p99 (ms)", + "Throughput (ready/s)", + ] + assert table2_rows == [ + ["Sandbox create + base mount", "1", "5", "1.2", "1.38", "1.396", "12"], + ["Session create to ready", "1", "5", "2.2", "2.38", "2.396", "12"], + ["Session create to ready", "5", "5", "19.2", "19.38", "19.396", "52"], + ["First no-op command", "1", "5", "3.2", "3.38", "3.396", "12"], + ] + + table3_header, table3_rows = _markdown_table(first / "table-3-cli-operations.md") + assert table3_header == [ + "Operation", + "Case", + "Payload/file size", + "Concurrency", + "Samples", + "p50 (ms)", + "p95 (ms)", + "p99 (ms)", + "Throughput (ops/s)", + ] + assert [row[:4] for row in table3_rows] == [ + ["`exec_command`", "no-op", "--", "1"], + ["`exec_command`", "no-op", "--", "5"], + ["`exec_command`", "fixture read", "4 KiB", "1"], + ["`exec_command`", "fixture read", "4 KiB", "5"], + ["Read", "snapshot", "4 KiB", "1"], + ["Read", "snapshot", "4 KiB", "5"], + ["Read", "snapshot", "256 KiB", "1"], + ["Read", "snapshot", "256 KiB", "5"], + ["Write", "session-local", "4 KiB", "1"], + ["Write", "session-local", "4 KiB", "5"], + ["Write", "session-local", "256 KiB", "1"], + ["Write", "session-local", "256 KiB", "5"], + ["Edit", "one replacement", "4 KiB", "1"], + ["Edit", "one replacement", "4 KiB", "5"], + ["Edit", "one replacement", "256 KiB", "1"], + ["Edit", "one replacement", "256 KiB", "5"], + ] + + table4_header, table4_rows = _markdown_table(first / "table-4-resources.md") + assert table4_header == [ + "Operation/case", + "Concurrency", + "Peak daemon RSS (MiB)", + "Peak sandbox memory (MiB)", + "Sandbox CPU (ms/trial)", + "Block read (MiB/trial)", + "Block write (MiB/trial)", + "Workspace allocated delta (MiB)", + ] + assert [row[:2] for row in table4_rows] == [ + ["Workspace create, 100 MiB/depth 100", "1"], + ["Workspace create, 100 MiB/depth 100", "5"], + ["`exec_command`, no-op", "1"], + ["`exec_command`, no-op", "5"], + ["Read, 256 KiB", "5"], + ["Write, 256 KiB", "5"], + ["Edit, 256 KiB", "5"], + ] + resource_rows = tables["tables"]["resources"]["rows"] + assert all( + "upperdir_bytes" in row["values"] + and "workspace_allocated_bytes" not in row["values"] + for row in resource_rows + ) + with (first / "numeric-provenance.csv").open( + encoding="utf-8", newline="" + ) as stream: + provenance = list(csv.DictReader(stream)) + upperdir_provenance = [ + row for row in provenance if row["evidence_id"].endswith(".upperdir_bytes") + ] + assert len(upperdir_provenance) == 7 + assert all( + '"metric_id":"upperdir_bytes"' in row["selector_json"] + for row in upperdir_provenance + ) + output_manifest = json.loads( + (first / "output-manifest.json").read_text(encoding="utf-8") + ) + assert output_manifest["schema_version"] == 2 + assert output_manifest["generator_schema_version"] == 2 + for entry in output_manifest["files"]: + data = (first / entry["path"]).read_bytes() + assert entry["bytes"] == len(data) + assert entry["sha256"] == "sha256:" + hashlib.sha256(data).hexdigest() + + +def test_frozen_final_candidate_uses_exact_final_contract(tmp_path: Path) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "environment-preflight.txt", + lambda value: ( + value["recorded_run_environment"]["host"].pop("storage_model"), + value["recorded_run_environment"]["host"].pop("storage_capacity_bytes"), + ), + ) + _refresh_archive_manifest(archive, disposition="final") + output = tmp_path / "output" + + result = generator.generate(archive, output) + + assert result["eligibility"] == "frozen_final_candidate" + table = (output / "table-2-startup.md").read_text(encoding="utf-8") + assert "FROZEN FINAL CANDIDATE" in table + assert "INELIGIBLE EXPLORATORY OUTPUT" not in table + tables = json.loads((output / "tables.json").read_text(encoding="utf-8")) + assert all( + row["values"]["samples"] == 100 for row in tables["tables"]["startup"]["rows"] + ) + product_field = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Product commit/tag" + ) + assert product_field["display"] == ( + f"{PRODUCT_COMMIT}; annotated tag paper-v1-freeze" + ) + assert product_field["value"]["freeze_tag"]["object_type"] == "tag" + assert product_field["value"]["freeze_tag"]["tag_object"] == "c" * 40 + assert product_field["value"]["freeze_tag"]["peeled_commit"] == PRODUCT_COMMIT + + +def test_frozen_final_candidate_accepts_canonical_host_os_fields( + tmp_path: Path, +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + + def use_canonical_host_fields(value: dict[str, Any]) -> None: + host = value["recorded_run_environment"]["host"] + host["os_caption"] = "Synthetic Windows 11 Pro" + host["os_build_number"] = int(host.pop("os_build")) + host.pop("os_edition") + + _mutate_json(archive / "environment-preflight.txt", use_canonical_host_fields) + _refresh_archive_manifest(archive, disposition="final") + output = tmp_path / "output" + + generator.generate(archive, output) + + tables = json.loads((output / "tables.json").read_text(encoding="utf-8")) + host_os = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Host OS" + ) + assert host_os["display"] == "Synthetic Windows 11 Pro build 99999" + + +@pytest.mark.parametrize( + "missing_key,expected_field", + [ + ("os_caption", "host OS caption/edition"), + ("os_build_number", "host OS build"), + ], +) +def test_frozen_final_candidate_requires_canonical_host_os_fields( + tmp_path: Path, missing_key: str, expected_field: str +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + + def use_incomplete_canonical_host_fields(value: dict[str, Any]) -> None: + host = value["recorded_run_environment"]["host"] + host["os_caption"] = host.pop("os_edition") + host["os_build_number"] = int(host.pop("os_build")) + host.pop(missing_key) + + _mutate_json( + archive / "environment-preflight.txt", use_incomplete_canonical_host_fields + ) + _refresh_archive_manifest(archive, disposition="final") + + with pytest.raises(generator.GenerationError, match=expected_field): + generator.generate(archive, tmp_path / "output") + + +def test_frozen_final_candidate_requires_ntfs(tmp_path: Path) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "environment-preflight.txt", + lambda value: value["recorded_run_environment"]["host"].update( + filesystem="ReFS" + ), + ) + _refresh_archive_manifest(archive, disposition="final") + + with pytest.raises( + generator.GenerationError, match="storage filesystem must be NTFS" + ): + generator.generate(archive, tmp_path / "output") + + +@pytest.mark.parametrize( + "mutate", + [ + lambda tag: tag.clear(), + lambda tag: tag.update(object_type="commit"), + lambda tag: tag.update(peeled_commit="d" * 40), + lambda tag: tag.update(tag_object="not-a-git-object"), + ], +) +def test_frozen_final_candidate_requires_valid_annotated_product_tag( + tmp_path: Path, mutate: Callable[[dict[str, Any]], None] +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "campaign-manifest.json", + lambda value: mutate(value["product"]["freeze_tag"]), + ) + _refresh_archive_manifest(archive, disposition="final") + + with pytest.raises( + generator.GenerationError, + match="final product freeze-tag provenance is invalid", + ): + generator.generate(archive, tmp_path / "output") + + +def test_frozen_final_candidate_requires_frozen_protocol_state( + tmp_path: Path, +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "campaign-manifest.json", + lambda value: value["protocol"].update(freeze_state="pre_freeze"), + ) + _refresh_archive_manifest(archive, disposition="final") + + with pytest.raises( + generator.GenerationError, + match="final campaign protocol is not frozen", + ): + generator.generate(archive, tmp_path / "output") + + +@pytest.mark.parametrize( + "mutation", + [ + {"dirty": True}, + {"status_porcelain": [" M experiments/expected_tables.md"]}, + {"freeze_state": "pre_freeze_worktree"}, + {"commit": "not-a-git-object"}, + ], +) +def test_frozen_final_candidate_requires_clean_paper_freeze( + tmp_path: Path, + mutation: dict[str, Any], +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "campaign-manifest.json", + lambda value: value["paper_git"].update(mutation), + ) + _refresh_archive_manifest(archive, disposition="final") + + with pytest.raises( + generator.GenerationError, + match="final paper source freeze provenance is invalid", + ): + generator.generate(archive, tmp_path / "output") + + +@pytest.mark.parametrize( + "mutate,error", + [ + ( + lambda root: _mutate_json( + root / "campaign-manifest.json", + lambda value: value["plan"].update(trial_batches=134), + ), + "campaign count contract drift", + ), + ( + lambda root: _mutate_json( + root / "report.json", + lambda value: value["data"].update(correctness_verdict="fail"), + ), + "correctness verdict drift or failure", + ), + ( + lambda root: _mutate_json( + root / "run-manifest.json", + lambda value: value["data"]["treatment"].update(source_commit="f" * 40), + ), + "product source identity drift", + ), + ( + lambda root: _mutate_json( + root / "campaign-manifest.json", + lambda value: value.update(disposition="final"), + ), + "campaign/archive disposition drift", + ), + ], +) +def test_semantic_drift_fails_closed( + tmp_path: Path, mutate: Callable[[Path], None], error: str +) -> None: + archive = _build_archive(tmp_path / "archive") + mutate(archive) + _refresh_archive_manifest(archive) + with pytest.raises(generator.GenerationError, match=error): + generator.generate(archive, tmp_path / "output") + assert not (tmp_path / "output").exists() + + +def _mutate_json(path: Path, mutate: Callable[[dict[str, Any]], None]) -> None: + value = json.loads(path.read_text(encoding="utf-8")) + mutate(value) + _json(path, value) + + +def test_archive_content_drift_and_archive_output_path_fail_closed( + tmp_path: Path, +) -> None: + archive = _build_archive(tmp_path / "archive") + report = archive / "report.json" + report.write_bytes(report.read_bytes() + b" ") + with pytest.raises( + generator.GenerationError, match="archive (byte-count|content) drift" + ): + generator.generate(archive, tmp_path / "output") + + clean_archive = _build_archive(tmp_path / "clean-archive") + with pytest.raises( + generator.GenerationError, match="outside the immutable archive" + ): + generator.generate(clean_archive, clean_archive / "generated") + + +def test_v11_table_generation_requires_named_pipe_block_evidence() -> None: + campaign = { + "protocol": { + "version": "v1.1", + "id": generator.PROTOCOLS["v1.1"]["id"], + } + } + run = { + "environment": { + "gateway_endpoint_identity": ( + "isolated_windows_named_pipe_per_execution_block" + ), + "gateway_transport": dict(generator.V11_GATEWAY_TRANSPORT), + }, + "gateway_policy": { + "protocol_version": generator.PROTOCOLS["v1.1"]["id"], + "mode": "isolated", + "isolated_runtime_per_execution_block": True, + "loopback_only": False, + **generator.V11_GATEWAY_TRANSPORT, + }, + "gateway_execution_blocks": [ + { + "block_id": "block-1", + "family_id": "runtime", + "gateway_instance_id": "gateway-1", + "endpoint_uri": "npipe://./pipe/eos-exp1-block-1", + **generator.V11_GATEWAY_TRANSPORT, + } + ], + } + expanded = { + "execution_blocks": [{"block_id": "block-1", "family_id": "runtime"}] + } + + generator._validate_protocol_transport( + campaign, run, expanded, "v1.1", "final" + ) + run["gateway_execution_blocks"][0]["endpoint_uri"] = ( + "npipe://server/pipe/not-local" + ) + with pytest.raises( + generator.GenerationError, + match="execution-block endpoint evidence is unsafe", + ): + generator._validate_protocol_transport( + campaign, run, expanded, "v1.1", "final" + ) + + +def test_v11_final_generation_requires_v11_tag_and_discloses_transport( + tmp_path: Path, +) -> None: + archive = _build_archive(tmp_path / "archive", disposition="final") + _mutate_json( + archive / "environment-preflight.txt", + lambda value: ( + value["recorded_run_environment"]["host"].update( + os_caption="Synthetic Windows 11 Pro", + os_build_number=99999, + ), + value["recorded_run_environment"]["host"].pop("os_edition"), + value["recorded_run_environment"]["host"].pop("os_build"), + ), + ) + _mutate_json( + archive / "campaign-manifest.json", + lambda value: ( + value["protocol"].update( + id=generator.PROTOCOLS["v1.1"]["id"], + version="v1.1", + ), + value["product"]["freeze_tag"].update( + name="paper-v1.1-freeze", + reference="refs/tags/paper-v1.1-freeze", + ), + ), + ) + _mutate_json( + archive / "run-manifest.json", + lambda value: ( + value["data"]["environment"].update( + gateway_endpoint_identity=( + "isolated_windows_named_pipe_per_execution_block" + ), + gateway_transport=dict(generator.V11_GATEWAY_TRANSPORT), + ), + value["data"].update( + gateway_policy={ + "protocol_version": generator.PROTOCOLS["v1.1"]["id"], + "mode": "isolated", + "isolated_runtime_per_execution_block": True, + "loopback_only": False, + **generator.V11_GATEWAY_TRANSPORT, + }, + gateway_execution_blocks=[ + { + "block_id": "block-1", + "family_id": "runtime", + "gateway_instance_id": "gateway-1", + "endpoint_uri": "npipe://./pipe/eos-exp1-block-1", + **generator.V11_GATEWAY_TRANSPORT, + } + ], + ), + ), + ) + _mutate_json( + archive / "expanded-plan.json", + lambda value: value["data"].update( + execution_blocks=[{"block_id": "block-1", "family_id": "runtime"}] + ), + ) + _refresh_archive_manifest(archive, disposition="final") + _mutate_json( + archive / "archive-manifest.json", + lambda value: value.update(protocol_version="v1.1"), + ) + + output = tmp_path / "output" + generator.generate(archive, output, protocol_version="v1.1") + + table1_header, table1_rows = _markdown_table(output / "table-1-environment.md") + assert table1_header == ["Field", "Archived value", "Evidence source"] + host_os_row = next(row for row in table1_rows if row[0] == "Host OS") + assert host_os_row[1] == "Synthetic Windows 11 Pro build 99999" + transport_row = next( + row for row in table1_rows if row[0] == "Gateway transport" + ) + assert transport_row[1] == ( + "windows_named_pipe; local_only; per_execution_block" + ) + tables = json.loads((output / "tables.json").read_text(encoding="utf-8")) + transport_field = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Gateway transport" + ) + assert transport_field["value"] == generator.V11_GATEWAY_TRANSPORT + product_field = next( + field + for field in tables["tables"]["environment"]["fields"] + if field["field"] == "Product commit/tag" + ) + assert product_field["value"]["freeze_tag"]["name"] == "paper-v1.1-freeze" + + _mutate_json( + archive / "campaign-manifest.json", + lambda value: value["product"]["freeze_tag"].update( + name="paper-v1-freeze", + reference="refs/tags/paper-v1-freeze", + ), + ) + _refresh_archive_manifest(archive, disposition="final") + _mutate_json( + archive / "archive-manifest.json", + lambda value: value.update(protocol_version="v1.1"), + ) + with pytest.raises( + generator.GenerationError, + match="final product freeze-tag provenance is invalid", + ): + generator.generate( + archive, + tmp_path / "invalid-output", + protocol_version="v1.1", + ) diff --git a/ephemeral-sandbox-v1/experiments/environment_setup.md b/ephemeral-sandbox-v1/experiments/environment_setup.md index 66c8d67..9139492 100644 --- a/ephemeral-sandbox-v1/experiments/environment_setup.md +++ b/ephemeral-sandbox-v1/experiments/environment_setup.md @@ -1,273 +1,295 @@ -# Fast environment setup and first-step verification +# Environment qualification -**Status:** Final-host verification pending -**Protocol:** [`../experiment_inventory.md`](../experiment_inventory.md) -**Preflight:** [`scripts/verify_environment.sh`](scripts/verify_environment.sh) +**Status:** base environment qualified; v1.1 IPC qualification passed on 2026-07-31 -## Principle +**Scope:** environment correctness only; no performance experiment -The measurement host consumes prebuilt artifacts. It is not a build machine. -The experiment begins by verifying the environment, not by compiling the -product, building a container image, installing web assets, or repairing the -host. +**Current qualifier:** [`scripts/qualify_windows_docker_environment.ps1`](scripts/qualify_windows_docker_environment.ps1) -The warm, network-free preflight target is 60 seconds or less. A failure stops -the experiment and is repaired outside the measurement window. +## Active EXP1 v1.1 treatment -## One selected environment +The base host, Docker, image, limits, fixture, and native CLI boundary remain +as qualified below. The active experiment treatment is amended by +[`exp1-v1.1-protocol-amendment.md`](exp1-v1.1-protocol-amendment.md): -| Setting | Required value | +- product `main` candidate + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; +- staged package + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5c48dae1`; +- native Windows named-pipe transport using + `npipe://./pipe/`; +- canonical `--gateway-endpoint` CLI option; +- one isolated endpoint per gateway execution block, with no retry, TCP + fallback, or host network mutation. + +The exact package hashes and preregistered 25,000-invocation gate are in the +amendment. The TCP examples and v0.1.4 package identities below are retained +only as the historical v1.0 environment record; they are not commands or +inputs for v1.1. + +The v1.1 qualifier passed all workload, identity, event-log, owned-TCP, +resource-growth, and cleanup gates. Its retained archive is +`experiments\diagnostics\exp1-v11-ipc-qualification-718cf58dace44dba83bed54601854bc9.zip`, +SHA-256 +`2c4f87dc5bb123157f76e6be58b769bafef8943aba36ee8e9202601b50e62a02`. +The archive is qualification-only and supplies no manuscript performance +value. + +## Historical v1.0 environment record (do not execute for v1.1) + +Everything below this heading records the accepted v1.0 base-environment +qualification. It remains evidence for the unchanged host, Docker, image, and +native-CLI boundary, but its v0.1.4 package, loopback endpoint, and commands are +not active v1.1 inputs. Use the amendment and the active-treatment block above +for v1.1. + +### Historical v1.0 selected environment + +The selected host is the current native Windows workstation. Docker Desktop +provides the Linux container engine, and the pinned Ubuntu image is the sandbox +guest. Ubuntu is not the host operating system. + +| Setting | Qualified value | |---|---| -| Host | Dedicated Ubuntu Server 24.04 LTS, Linux x86-64 | -| Capacity | At least 8 vCPU, 16 GiB RAM, 100 GiB local NVMe-backed storage | -| Filesystem | ext4 for the paper, product, and benchmark-state paths | -| Cgroup | v2 | -| Runtime | Docker Engine with a reachable server | -| Python | CPython 3.13 | -| Product branch | `main`, clean | +| Computer | `DESKTOP-OLP1ADS` | +| Host OS | Native 64-bit Windows, build 26200 | +| Host capacity | 48 logical CPUs, 137,438,953,472 bytes physical memory | +| Host filesystem | NTFS for paper, product, package, workspace, and evidence paths | +| Docker | Docker Desktop client/server 29.0.1 | +| Docker engine | Linux AMD64, `overlayfs`, cgroup v2 | +| Product checkout | clean `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5` | +| Product release | annotated `v0.1.4`, official Windows AMD64 package | +| Product path | `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox` | +| Paper path | `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1` | +| Staged package | `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4` | +| Workspace base repository | `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\workspace-base\ephemeral-sandbox-v0.1.4` | | Sandbox image | `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` | -| Sandbox profile | `standard`: 1 vCPU, 512 MiB maximum, 256 PIDs | -| Network | `shared` | -| Benchmark client | `direct_client` | -| Workspace | `paper-100m`: 4,000 files, 100 MiB, maximum depth 100 | - -Use native Linux for final numbers. Windows, WSL 2, and Docker Desktop may be -used for editing or non-performance checks but are not the selected measurement -environment. - -## Required prebuilt layout - -The product checkout must already contain: - -```text -ephemeral-sandbox/ -|-- target/release/ -| |-- sandbox-gateway -| `-- sandbox-catalog-export -`-- dist/ - |-- sandbox-daemon-linux-amd64 - `-- git/ - |-- linux-amd64.tar - `-- linux-arm64.tar +| Client boundary | Native `sandbox-manager-cli.exe`, `sandbox-runtime-cli.exe`, and `sandbox-observability-cli.exe` | +| Gateway | Native `sandbox-gateway.exe` | +| Sandbox daemon | Linux x86-64 daemon uploaded into Docker containers | +| Python | Not required by the environment qualifier | + +This contract supersedes the earlier native-Ubuntu, +`eos-benchmark-ubuntu24`, ext4, CPython 3.13, SSH, and Linux-transfer-bundle +assumptions. Those assumptions resulted from confusing the Ubuntu sandbox image +with the host. + +### Historical v1.0 execution boundary + +The qualified control flow is: + +1. native Windows PowerShell launches the released Windows gateway; +2. the native Windows manager CLI creates and destroys Docker sandboxes; +3. the native Windows runtime CLI executes commands and file operations; +4. the native Windows observability CLI requests snapshots; +5. Docker Desktop runs the pinned Linux AMD64 Ubuntu sandbox image; +6. the released Linux AMD64 sandbox daemon runs inside each container. + +No direct Python gateway client is used. Docker CLI calls in the qualifier are +limited to engine/image inspection, before/after resource auditing, and removal +of shared-base volumes carrying the qualifier's unique gateway-instance label. +Sandbox lifecycle operations themselves use the product manager CLI. + +### Historical v1.0 sandbox-creation inputs + +The accepted v1.0 qualification used these exact repository-backed inputs: + +```powershell +$package = 'C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4' +$workspaceBaseRepo = 'C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\workspace-base\ephemeral-sandbox-v0.1.4' +$image = 'ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf' +$gatewaySocket = '127.0.0.1:7878' + +$gateway = "$package\bin\sandbox-gateway.exe" +$managerCli = "$package\bin\sandbox-manager-cli.exe" +$runtimeCli = "$package\bin\sandbox-runtime-cli.exe" +$observabilityCli = "$package\bin\sandbox-observability-cli.exe" +$daemon = "$package\dist\sandbox-daemon-linux-amd64" ``` -The benchmark gateway verifies these paths and refuses symlinks or unsafe, -non-executable files. Preserve the product source commit alongside the -prebuilt bundle so the binary hashes can be tied to a revision. +The workspace base repository is a source-only clean clone: -## Off-clock staging +- branch `main`; +- commit `b22862550e0a7cb4fe61ce581831e9244cc492b5`; +- annotated tag `v0.1.4` resolves to the same commit; +- Git status clean; +- no `target` directory. -Complete these actions before the measurement window: +Use this clone—not the build checkout, paper repository, or +`Ephemeral-AI-Lab` parent directory—as the value of +`--workspace-bind-root`. This avoids copying the staged binary package and +other generated artifacts into the sandbox shared-base cache. -1. Provision the Ubuntu host and Docker Engine. -2. Copy or unpack the clean product checkout and its prebuilt release bundle. -3. Copy or clone the paper checkout containing the paper-local benchmark. -4. Pull the exact image digest once: +#### Historical v1.0 gateway launch - ```sh - docker pull \ - ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf - ``` +```powershell +powershell.exe -NoProfile -ExecutionPolicy Bypass ` + -File "$package\bin\start-sandbox-windows-docker-gateway.ps1" ` + -GatewaySocket $gatewaySocket -5. Create the benchmark environment once: +$gatewayToken = ( + Get-Content -LiteralPath "$HOME\.ephemeral-sandbox\gateway.token" -Raw +).Trim() +``` - ```sh - cd /absolute/path/to/research-papers/ephemeral-sandbox-v1 - python3.13 -m venv .venv - . .venv/bin/activate - python -m pip install -e "./benchmark[test]" - ``` +#### Historical v1.0 manager-CLI sandbox creation -6. Confirm the product and paper revisions are the intended revisions. -7. Reboot or otherwise quiesce the host after staging if provisioning caused - material background activity. +```powershell +$created = & $managerCli ` + --gateway-socket $gatewaySocket ` + --gateway-auth-token $gatewayToken ` + create_sandbox ` + --image $image ` + --workspace-bind-root $workspaceBaseRepo | + ConvertFrom-Json -Do not include staging time in benchmark latency. Record it separately if -reproducibility accounting needs it. +$sandboxId = $created.id +``` -## Forbidden during the measurement window +The accepted create response must report a nonempty sandbox ID, state `ready`, +and the workspace root above. All subsequent operations use that sandbox ID: + +```powershell +& $runtimeCli ` + --gateway-socket $gatewaySocket ` + --gateway-auth-token $gatewayToken ` + --sandbox-id $sandboxId ` + exec_command 'pwd && git rev-parse HEAD && git status --short' + +& $observabilityCli ` + --gateway-socket $gatewaySocket ` + --gateway-auth-token $gatewayToken ` + snapshot ` + --sandbox-id $sandboxId +``` -Do not run: +Destroy the sandbox through the manager CLI: -- `cargo build`, `cargo test`, or release packaging; -- `docker build`; -- `npm install`, `npm ci`, or a web build; -- `apt install`, `pip install`, or `uv sync`; -- `git pull`, branch switching, rebasing, or source edits; -- an image pull; -- unrelated workloads or background CI. +```powershell +& $managerCli ` + --gateway-socket $gatewaySocket ` + --gateway-auth-token $gatewayToken ` + destroy_sandbox ` + --sandbox-id $sandboxId +``` -If any prerequisite is missing, stop and repair the staged bundle. Do not let a -measurement run silently include compilation, installation, or network fetches. +The strict environment qualifier intentionally uses two tiny isolated fixture +workspaces under its artifact directory instead of the full base repository. +Those fixtures validate the environment and CLI boundary. The source-only base +repository above is the canonical workspace for repo-backed sandbox creation +after qualification. -## First step: network-free preflight +### Historical v1.0 pinned release package -From the paper directory: +Official archive: -```sh -export PRODUCT_ROOT=/absolute/path/to/ephemeral-sandbox -export PRODUCT_BIN_DIR="$PRODUCT_ROOT/target/release" -export IMAGE_REFERENCE='ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf' +`C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\v0.1.4-windows-release-input\ephemeral-sandbox-windows-amd64.zip` -mkdir -p experiments/runs/PREFLIGHT_RUN_ID +Archive SHA-256: -bash experiments/scripts/verify_environment.sh \ - 2>&1 | tee experiments/runs/PREFLIGHT_RUN_ID/environment-preflight.txt -``` +`9f2327578c186897578f0d502893d894aed52be27306f43f75afa3205eba9fdb` -The script is read-only except for benchmark-owned `.benchmark-state` -initialization performed by `sandbox-benchmark validate`. It does not build, -install, or pull anything. - -## Preflight acceptance - -All items must pass: - -- [ ] Linux x86-64. -- [ ] Ubuntu 24.04. -- [ ] At least 8 logical CPUs. -- [ ] At least 15 GiB reported usable memory. -- [ ] ext4 at the paper and product roots. -- [ ] Cgroup v2 controllers available. -- [ ] Docker server reachable. -- [ ] Exact pinned image already present and `linux/amd64`. -- [ ] Product checkout on clean `main`. -- [ ] `sandbox-gateway` and `sandbox-catalog-export` are executable. -- [ ] Linux AMD64 daemon is executable ELF. -- [ ] Both fixed Git toolchain archives exist and are non-empty. -- [ ] CPython 3.13 paper virtual environment exists. -- [ ] `paper-env-smoke` validates against the prebuilt product catalog. -- [ ] `paper-100m` resolves to 4,000 files, 104,857,600 bytes, depth 100. -- [ ] Warm preflight completes in 60 seconds or less. - -The script prints the product commit, product dirty state, Docker version, -image identity, binary hashes, fixture hash, plan hash, and elapsed time. Archive -the complete output. - -## Second step: minimal live smoke - -Only after preflight passes: - -```sh -. .venv/bin/activate - -sandbox-benchmark run \ - --test-repository-root "$PWD" \ - --product-root "$PRODUCT_ROOT" \ - --product-bin-dir "$PRODUCT_BIN_DIR" \ - --plan paper-env-smoke -``` +| Package artifact | SHA-256 | +|---|---| +| `bin\sandbox-gateway.exe` | `3a96bedcfa9857bd3881155d758ec2d969f6265456ec3b2878eb6dbb26dc9368` | +| `bin\sandbox-manager-cli.exe` | `b43ec520edc2f436adc8aa7e8b2b50680bb9021883fe23d79a85b17afd2e10fe` | +| `bin\sandbox-runtime-cli.exe` | `df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4` | +| `bin\sandbox-observability-cli.exe` | `0e0471e52750805570876a6244868764c44e166ec653627b9ebd490176e2fcbe` | +| `config\windows-amd64.yml` | `0f0efd15e5111851054e0f7c1ce0f3eaebb3b3047c1b9e2322544036f5daf5db` | +| `dist\sandbox-daemon-linux-amd64` | `2da4395cd835e5325bc3e55b9c2f3b67565ea7c698fce5e086167ec4a2092a39` | -The smoke preset uses the small fixture so the environment is verified quickly. -It executes one prepared-session no-op command and one prepared-sandbox session -creation. Smoke results are exploratory and never enter paper tables. +The staged package is under ignored `target/`; the product Git checkout +therefore remains clean. -Accept the live environment only if: +### Historical v1.0 qualification command -- the campaign completes; -- operation verification passes; -- cleanup restores the baseline; -- no benchmark-owned containers, processes, or runtime paths leak; -- wall time is at most 3 minutes. +The accepted v1.0 qualifier used this command shape with a new, empty artifact +directory. It must not be reused for the v1.1 IPC gate: -## Warm-state policy +```powershell +$paper = 'C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1' +$artifact = 'C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-rerun' -The good pass uses a warm, pre-staged environment: +powershell.exe -NoProfile -ExecutionPolicy Bypass ` + -File "$paper\experiments\scripts\qualify_windows_docker_environment.ps1" ` + -ArtifactDirectory $artifact +``` -- image present locally; -- Python environment already installed; -- product binaries already placed; -- fixture cache may be materialized by the exploratory pilot; -- no manual host page-cache drop; -- no source or configuration changes between preflight and run. +The qualifier performs these strict checks before starting a sandbox: -This policy avoids mixing network, package installation, product compilation, -and first-time fixture construction into operation measurements. Record the -policy in Table 1. +- native 64-bit Windows host and exact computer name; +- Windows 11-class build, CPU, memory, NTFS capacity, and free space; +- clean product `main` at the selected commit; +- official archive and per-file hashes; +- x64 PE gateway and CLI binaries plus x86-64 ELF daemon; +- required manager/runtime/observability CLI catalogs; +- Docker Desktop server 29.0.1 with Linux AMD64, `overlayfs`, and cgroup v2; +- exact pinned Ubuntu Linux AMD64 image present locally. -## Good-pass command +It then runs two independent CLI-controlled batches. Each batch validates: -After the protocol and commits are frozen: +1. `list_docker_images`; +2. `create_sandbox`; +3. `exec_command`; +4. `file_write`; +5. `file_read`; +6. `file_edit`; +7. a second `file_read`; +8. observability `snapshot`; +9. `destroy_sandbox`; +10. post-destroy `list_sandboxes`. -```sh -bash experiments/scripts/verify_environment.sh \ - 2>&1 | tee experiments/runs/GOOD_PASS_RUN_ID/environment-preflight.txt +Acceptance requires: -. .venv/bin/activate +- 2/2 completed batches with distinct sandbox IDs; +- 20 successful product-CLI calls; +- strict response and file-content correctness; +- empty product-CLI stderr; +- zero gateway warnings, errors, or panics; +- no authentication-token leak; +- unchanged global EOS-owned resource/process baseline; +- zero containers or volumes owned by the qualifier's gateway instance; +- completion within 180 seconds. -sandbox-benchmark run \ - --test-repository-root "$PWD" \ - --product-root "$PRODUCT_ROOT" \ - --product-bin-dir "$PRODUCT_BIN_DIR" \ - --plan paper-good-pass -``` +### Accepted v1.0 base-environment evidence -Do not run the good pass until every Phase 3 instrumentation item in -[`../experiment_inventory.md`](../experiment_inventory.md) is resolved. +Accepted artifact directory: -## Current verification state +`C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6` -Verified on 2026-07-30: +Summary: -- the registry exposes the selected Linux AMD64 Ubuntu manifest digest; -- Docker Desktop 29.0.1 exposes a Linux AMD64 engine with cgroup v2 and the - `overlayfs` storage driver; -- the exact pinned image is present locally and reports `linux/amd64`; -- a disposable-container probe enforced the planned 1 vCPU, 512 MiB, and - 256-PID limits; -- a disposable privileged probe mounted, copied up, wrote through, and - unmounted nested OverlayFS when given a separate `tmpfs` backing store; -- a container-to-Windows bind-mount write/read round trip passed; -- the paper-local profile loads as 4,000 files, 100 MiB, depth 100; -- the profile's deterministic fixture identity is - `sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7`; -- the two depth-bound tests pass under Python 3.13. +`windows-docker-cli-env-summary.json` -Not verified: +Archived evidence: -- the selected final Ubuntu host; -- local presence of the pinned image on that host; -- final product binaries and hashes; -- ext4, cgroup, and Docker readiness on that host; -- a live smoke or measured campaign. +`C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6.zip` -The current Windows workstation remains a development and capability-check -environment, not the selected measurement host. Docker Desktop now works, but -the checkout is on a WSL `9p` mount, the WSL distribution is Ubuntu 26.04, and -the prebuilt Linux product bundle is incomplete. +Archive SHA-256: -### Current workstation Gate 1 score +`eea981665b031846677046d4c211e71ad144f8a32507c09058923241d4d0f7f9` -After Docker Desktop was started, the 2026-07-30 audit passed 7 of 15 strict -acceptance items (about 47%), not 70%: +Observed qualification result: -| Gate item | Current workstation | -|---|---| -| Linux x86-64 | Partial: Linux AMD64 is available through Docker Desktop/WSL 2, not the selected native host | -| Ubuntu 24.04 | Fail: WSL distribution reports Ubuntu 26.04 | -| At least 8 CPUs | Pass: 48 exposed | -| At least 15 GiB memory | Pass: approximately 62.8 GiB exposed | -| ext4 paper/product roots | Fail: Windows checkout is mounted through WSL `9p` | -| Cgroup v2 | Pass | -| Docker server | Pass: client/server 29.0.1, Linux AMD64, `overlayfs`, cgroup v2 | -| Pinned image local | Pass: exact digest, Linux AMD64 | -| Clean product `main` | Pass | -| Gateway/catalog binaries | Fail: missing | -| Daemon and Git archives | Fail: daemon present; both archives missing | -| Linux CPython 3.13 environment | Fail: WSL Python is 3.14; Python 3.13 exists only in the Windows venv | -| Product-catalog plan validation | Fail: exporter binary missing | -| `paper-100m` profile | Pass | -| Preflight within 60 seconds | Not run because prerequisites fail | - -The Docker substrate is now sufficiently validated to support continued -development: resource controls, the pinned image, nested OverlayFS on an -appropriate backing store, and bind mounts all passed. A direct nested -OverlayFS attempt on the container's existing overlay root failed, as expected -for overlay-on-overlay; therefore the successful `tmpfs`-backed probe does not -replace a live product smoke using the product's real storage layout. - -The benchmark configuration is also validated by profile checks, focused tests, -planner expansion, Python 3.13 imports, and shell syntax. Confidence that the -design can run after staging the required Linux bundle is moderate to high. -Confidence in this workstation as a paper-grade measurement host is still low: -the strict gate is 7/15, and no end-to-end sandbox has been launched. +- target `windows_docker_desktop`; +- client cohort `product_cli`; +- 2/2 completed batches; +- 20 validated operations; +- correctness `pass`; +- zero warnings and failures; +- cleanup `pass`; +- ten seconds elapsed; +- zero resources remaining for gateway instance + `cli-env-windows-20260730T031621Z-34036`. + +The elapsed time is an environment-gate observation only. It is not a +performance result and must not enter a paper table. + +### Base-environment verdict from the v1.0 qualifier + +**GO:** the selected Windows plus Docker Desktop environment is qualified for +CLI-controlled sandbox work. + +No native Ubuntu host, SSH exposure, Linux handoff bundle, CPython 3.13 +environment, pilot, or performance run is required to close this environment +task. diff --git a/ephemeral-sandbox-v1/experiments/exp1-v1.1-protocol-amendment.md b/ephemeral-sandbox-v1/experiments/exp1-v1.1-protocol-amendment.md new file mode 100644 index 0000000..4dd0e05 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/exp1-v1.1-protocol-amendment.md @@ -0,0 +1,201 @@ +# EXP1 v1.1 local-IPC protocol amendment + +**Protocol ID:** `ephemeral-sandbox-v1-practical-performance-v1.1` + +**Status:** authorized pre-measurement treatment; qualification pending + +**Date:** 2026-07-31 + +**Supersedes for new work:** the unexecuted IPv4 dynamic-port-range proposal in +`experiments/analysis/exp1-v1.1-remediation-decision.{md,json}` + +**Does not supersede:** the immutable v1.0 protocol, freeze, sole failed final, +or failed archive + +## Decision and evidence boundary + +The author directed the permanent CLI issue to be fixed and the EXP1 goal to +resume. For v1.1, native Windows product CLIs communicate with the native +gateway through Windows named pipes. TCP remains available only when an +explicit TCP endpoint is requested for compatibility or remote use; it is not +permitted in the v1.1 paper treatment. No host network setting is changed. + +The sole v1.0 final remains permanently `failed_ineligible`. Its archive, +paper freeze, product tag, and partial values are immutable. Those values may +be used only as failure evidence; they must not be pooled with, substituted +for, or numerically compared with v1.1 measurements. + +The transport change is scientifically material. V1.1 therefore requires a +new product identity, package, qualification, smoke, pilot, projection, +paper/source freeze, annotated `paper-v1.1-freeze` tag, and exactly one final +attempt after all preceding gates pass. + +## Locked treatment delta + +Only these treatment facts change from v1.0: + +| Item | Locked v1.1 value | +|---|---| +| CLI-to-gateway transport | `windows_named_pipe` | +| Endpoint scope | `local_only` | +| Endpoint identity | `isolated_windows_named_pipe_per_execution_block` | +| Endpoint rotation | once per gateway execution block | +| Endpoint syntax | `npipe://./pipe/` | +| Canonical CLI option | `--gateway-endpoint` | +| Client behavior | one native CLI process, one named-pipe connection attempt, one request, one validated response, no retry and no fallback | +| Gateway behavior | one isolated released gateway per execution block; bounded pending pipe instances | +| Host network mutation | none | + +Every other scientific choice is retained unchanged: the 19 cells, fixed +image and `paper-100m` fixture, operation definitions, seed, two warmups, +pilot five-sample count, final 100-sample count, concurrency levels, scheduling, +timeouts, resource cadence, correctness rules, exclusions, metrics, and all +numeric/measured table definitions. The sole provenance-table clarification +below does not add or alter a measurement. + +### Provenance-only Table 1 clarification + +Because transport is the scientifically material v1.1 treatment, Table 1 adds +exactly one non-numeric text row named `Gateway transport`. Its value is +derived deterministically from the archived run manifest's transport object; +it is never copied from qualification output or entered manually. The machine +table/output-manifest schema advances to version 2 solely to carry this typed +provenance field and the protocol version. + +Tables 2--4, every numeric column, cell, metric, aggregation, eligibility rule, +and sample count remain unchanged. For legacy v1.0 archive regeneration, the +same provenance field may disclose the historical loopback transport without +changing the frozen v1.0 raw corpus. This is the only table-schema change +authorized by v1.1. + +The primary timing boundary remains immediately before native CLI process +creation through validated JSON response and process exit. Named-pipe +connection establishment remains inside that boundary. Gateway setup, +readiness, endpoint selection, verification, sampling, and teardown remain +outside primary timing. + +## Frozen product candidate + +The prequalification product candidate is clean direct `main` commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, consisting of: + +- `56c676d588fbb704bf3da8f67d22be910453644d`: + `Use local IPC for gateway CLI transport`; +- `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`: + `Preserve TCP endpoint compatibility`. + +The staged package is +`C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5c48dae1`; +its ZIP is `windows-exp1-5c48dae1.zip`. + +| Artifact | Size (bytes) | SHA-256 | +|---|---:|---| +| Package ZIP | 5,739,735 | `11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506` | +| `bin\sandbox-gateway.exe` | 3,480,064 | `42e7642dd025487811abbcd78dcc5513760f2aaa1e6057cfdfa3e74c03748358` | +| `bin\sandbox-manager-cli.exe` | 827,904 | `e1faa2fe0e9f4909fa2d694166784ac65dde40ba82795b7e0c503eb5fea86513` | +| `bin\sandbox-runtime-cli.exe` | 833,536 | `e18827cf765945c958e169748575b89645c730b310ee5ffc1b42c382b44a0e26` | +| `bin\sandbox-observability-cli.exe` | 826,880 | `2b1c13bba36c9486f768824178d1e2ea8d2b1da019bd21cd1f9ea250d5da34c5` | +| `dist\sandbox-daemon-linux-amd64` | — | `f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a` | +| `config\windows-amd64.yml` | — | `987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a` | + +The candidate remains provisional until the qualification, smoke, pilot, +projection, and freeze gates pass. Any source, binary, package, or protocol +drift invalidates the corresponding downstream evidence. + +## EXP1 v1.1 IPC qualification policy preregistration + +This policy is fixed before the first live v1.1 qualifier. Qualification is a +strict engineering and environment gate, not performance evidence. + +### Workload and success criteria + +- Run exactly 25,000 native + `sandbox-manager-cli.exe list_sandboxes` invocations. +- Schedule exactly 5,000 batches at concurrency 5 against one unique named + pipe and one isolated packaged gateway. +- Use the canonical `--gateway-endpoint` option with the exact named-pipe URI. +- Do not use TCP, pacing, sleeps, retries, connection fallback, or an + alternate direct client. +- Give every invocation a unique request ID. +- Require every process to exit 0, write empty stderr, and write exactly one + valid JSON line with the expected list-sandboxes response shape. +- Mark all qualifier artifacts `qualification_only: true` and + `performance_evidence: false`. They are never eligible for manuscript + tables, runtime projection, or numeric claims. +- Preserve per-invocation timestamps, request IDs, exit status, output hashes, + command provenance, and all partial failure evidence. + +Any invocation, response-shape, stderr, process, gateway, cleanup, or evidence +failure fails the qualifier. There is no retry. + +### TCP and Windows event evidence + +- Capture the System event-log cursor immediately before gateway start. +- After gateway cleanup, query the closed interval from that cursor through a + newly captured final cursor for provider `Tcpip` event IDs 4227 and 4231. +- Any newly observed matching event fails qualification conservatively, + regardless of attribution uncertainty. +- At readiness, every 100 batches, immediately before gateway stop, and after + cleanup, capture `Get-NetTCPConnection` evidence. +- The gateway PID must own no TCP listener or TCP connection in every sample. + Any owned TCP endpoint fails qualification and proves treatment drift or + fallback. + +### Gateway resource gates + +Capture gateway process samples at readiness, every 100 batches, and +immediately before stop. Record handles, private bytes, and resident-set bytes +for every sample and evaluate both the peak above readiness and the final +above readiness. + +| Resource | Peak-over-readiness limit | Final-over-readiness limit | +|---|---:|---:| +| Handles | 32 | 32 | +| Private bytes | 16 MiB | 16 MiB | +| Resident-set bytes | 16 MiB | 16 MiB | + +The exact cap passes; one unit or byte above it fails. The 32-handle allowance +is no greater than the gateway listener's maximum pending-instance bound. The +16 MiB memory allowances are conservative fixed qualification limits selected +before evidence is collected; they are not performance results. + +### Provenance and fail-closed cleanup + +The qualification summary and manifest must bind: + +- clean product branch, commit, status, source identities, package directory, + package ZIP, and SHA-256 for every packaged binary/configuration artifact; +- clean prequalification paper commit and status over the archive's frozen + source/protocol/analysis scope and documented generated-file exclusions, + qualifier source and test hashes, protocol-amendment hash, and exact + sanitized commands (pre-existing excluded caches and generated evidence are + preserved, not deleted to manufacture a globally empty worktree); +- host/build identity, package build command, endpoint URI, gateway PID, + gateway executable path, token redaction, timestamps, counters, thresholds, + event cursors/query, TCP samples, and resource samples; +- gateway stop command/result, original PID/path, confirmed process exit, + PID-file removal, output hashes, and cleanup return state. + +The harness must attempt cleanup in `finally`-equivalent control flow. Missing, +ambiguous, or failed stop/cleanup evidence fails the qualifier, while retaining +all evidence already written. + +## Postqualification gate sequence + +If and only if the qualifier passes: + +1. run a fresh complete 19-cell CLI integration smoke and archive/verify it; +2. run one fresh complete 19-cell, five-sample exploratory pilot and + archive/verify it; +3. regenerate exploratory tables twice, require byte identity, and require a + conservative final projection no greater than 1,400 seconds; +4. resolve only pre-freeze instrumentation defects allowed by the original + packet, rerunning affected exploratory gates when necessary; +5. commit the clean paper/source/environment freeze and create the local + annotated `paper-v1.1-freeze` tag without moving `paper-v1-freeze`; +6. run exactly one v1.1 `paper-good-pass` after a read-only preflight; +7. archive and independently verify the attempt whether it succeeds or fails; +8. generate final tables and numeric evidence only from a complete eligible + final archive. + +No v1.1 result is known or implied by this preregistration. diff --git a/ephemeral-sandbox-v1/experiments/expected_tables.md b/ephemeral-sandbox-v1/experiments/expected_tables.md index 4c7d683..271f89c 100644 --- a/ephemeral-sandbox-v1/experiments/expected_tables.md +++ b/ephemeral-sandbox-v1/experiments/expected_tables.md @@ -1,6 +1,6 @@ # Expected experiment tables -**Status:** Schema draft; no measured values exist +**Status:** v1.1 schema preregistered; no eligible measured values exist **Protocol:** [`../experiment_inventory.md`](../experiment_inventory.md) **Rule:** every numeric cell must be generated from archived run data @@ -8,6 +8,11 @@ The focused study produces four tables: one provenance table and three measured result tables. It does not include a baseline-ranking table because the study is intended to characterize practical performance, not claim superiority. +V1.1 adds only the non-numeric `Gateway transport` row to Table 1 and advances +the generated machine/output-manifest schema to version 2 so that the material +transport treatment and protocol version are explicit. Tables 2--4 and every +numeric or measured field remain unchanged. + Verification remains mandatory inside the benchmark, but it is not shown as a column. A row is displayed only when all contributing samples pass the operation, correctness, infrastructure, and cleanup gates. @@ -19,20 +24,21 @@ one host, image, product revision, benchmark revision, and workspace. | Field | Expected value | Evidence source | |---|---|---| -| Host OS | Ubuntu Server 24.04 LTS | preflight record | -| Kernel | `TBD` | `uname -r` | -| Architecture | x86-64 / `amd64` | `uname -m`, image inspect | -| CPU | `TBD model`, 8 vCPU or more | `/proc/cpuinfo`, `nproc` | -| Memory | 16 GiB or more | `/proc/meminfo` | -| Storage | local NVMe-backed ext4 | `findmnt`, `lsblk` | -| Docker Engine | `TBD` | `docker version` | -| Cgroup | v2 | `/sys/fs/cgroup/cgroup.controllers` | +| Host OS | Native Windows build 26200 | Windows preflight record | +| Container engine OS | Docker Desktop Linux engine | `docker info` | +| Architecture | Windows x64 host; Linux AMD64 engine/image | PE/ELF checks and image inspect | +| CPU | 48 logical CPUs | Windows preflight record | +| Memory | 137,438,953,472 bytes | Windows preflight record | +| Storage | NTFS host paths | `Get-Volume` | +| Docker Engine | Docker Desktop 29.0.1 | `docker version`, `docker info` | +| Cgroup | v2 inside the Docker Desktop engine | `docker info` | | Product commit/tag | `TBD final clean main` | Git record | | Benchmark commit | `TBD paper-local commit` | Git record | | Sandbox image | pinned Ubuntu 24.04 digest | plan and image inspect | | Sandbox limits | 1 vCPU, 512 MiB, 256 PIDs | effective configuration | | Workspace | 4,000 files, 100 MiB, depth 100 | fixture manifest | -| Client | `direct_client` | expanded plan | +| Client | native Windows `product_cli` | qualification summary and benchmark run manifest | +| Gateway transport | isolated Windows named pipe per gateway execution block; no retry or TCP fallback | archived run manifest and gateway execution-block manifest | | Seed | `20260712` | expanded plan | | Trials | 2 warmups + 100 measured per cell | expanded plan | @@ -52,11 +58,12 @@ manifest. | Session create to ready | 5 | 100 | -- | -- | -- | -- | | First no-op command | 1 | 100 | -- | -- | -- | -- | -The first row is blocked until the runner preserves an explicit -`create_sandbox` request-to-ready timing. It must be omitted rather than filled -from generic setup time if that instrumentation is not added. "Session create" -uses the existing prepared-sandbox `create_workspace` boundary and must not be -renamed to sandbox creation. +The runner preserves the first row's explicit end-to-end manager-CLI +`create_sandbox` process-launch-to-validated-ready timing. The generator must +still omit the row if that dedicated evidence is absent or invalid; generic +setup time is never an eligible substitute. "Session create" uses a runtime-CLI +`create_workspace_session` invocation against a prepared sandbox and must not +be renamed to sandbox creation. ## Table 3 - Public CLI-operation performance @@ -87,8 +94,11 @@ Failed, non-reportable, and partial samples remain visible in the run manifest and [`experiment_log.md`](experiment_log.md). A row with fewer than 100 reportable samples is not published. -These are public gateway-operation timings, not shell-process startup timings -for launching the `sandbox` CLI executable itself. +These are end-to-end native product-CLI subprocess timings. They include +process launch, CLI-to-gateway transport, operation execution, JSON output +capture and validation, and process exit. Product-reported internal command or +gateway timing may be archived separately but is not substituted into this +table. ## Table 4 - Resource envelope @@ -107,13 +117,20 @@ low latency without its CPU, memory, I/O, or storage context. Use the maximum observed gauge for peak memory, per-trial counter deltas for CPU and block I/O, and the before/after allocated-byte delta for workspace -storage. If the runtime cannot provide a metric, render `unavailable` and -explain why; never substitute a different scope. +storage. The workspace-storage delta is the completed post-operation minus +completed pre-operation product snapshot of allocated bytes summed across all +live workspace-session upperdirs; either boundary is unavailable if any +workspace upperdir walk is incomplete. The immutable host fixture allocation +gauge is a separate scope and is never substituted. If the runtime cannot +provide a metric, render `unavailable` and explain why. ## Aggregation and formatting contract - Latency is reported in milliseconds with enough precision to avoid rounding materially different values together. +- Primary latency begins immediately before native CLI process creation and + ends after validated process exit. Setup, verification, and teardown are + separate untimed operations. - Throughput is completed operation requests divided by batch makespan. - Byte values use binary units: KiB and MiB. - p50/p95/p99 are computed from exactly the reportable measured samples. @@ -128,7 +145,7 @@ explain why; never substitute a different scope. | Table | Required inputs | Blocking acceptance gate | |---|---|---| | Table 1 | Preflight, expanded plan, Git/binary/image/fixture hashes | Gate 1 | -| Table 2 | Workspace lifecycle samples and optional sandbox-create timing | Gate 3 | +| Table 2 | Workspace lifecycle samples and required manager-CLI sandbox-create timing | Gate 3 | | Table 3 | Reportable operation trials | Gate 5 | | Table 4 | Resource samples correlated to reportable trials | Gate 5 | @@ -145,14 +162,14 @@ copied into the manuscript, numeric-evidence records, plots, or analysis tests. | Field | Simulated example value | |---|---| -| Host OS | Ubuntu Server 24.04 LTS | -| Kernel | `6.8.0-example` | -| Architecture | x86-64 | -| CPU | Example 8-vCPU host | -| Memory | 16 GiB | -| Storage | Example local NVMe, ext4 | -| Docker Engine | `29.x (example)` | -| Cgroup | v2 | +| Host OS | Example native Windows 11 host | +| Container engine OS | Example Docker Desktop Linux engine | +| Architecture | Windows x64 / Linux AMD64 | +| CPU | Example 48-logical-CPU host | +| Memory | Example 128 GiB host | +| Storage | Example NTFS host paths | +| Docker Engine | Docker Desktop `29.x (example)` | +| Cgroup | v2 inside the Linux engine | | Product commit/tag | `example-final-commit` | | Benchmark commit | `example-benchmark-commit` | | Sandbox image | pinned Ubuntu 24.04 digest | diff --git a/ephemeral-sandbox-v1/experiments/experiment_log.md b/ephemeral-sandbox-v1/experiments/experiment_log.md index c29f458..51bd93e 100644 --- a/ephemeral-sandbox-v1/experiments/experiment_log.md +++ b/ephemeral-sandbox-v1/experiments/experiment_log.md @@ -21,9 +21,9 @@ | Phase | Status | Evidence | |---|---|---| | 0 - Reproducibility package | In progress | Spec, table schema, environment guide, log, scripts, presets | -| 1 - Final environment verification | Not started | Final Ubuntu host not selected/verified | -| 2 - Minimal live smoke | Not started | Blocked by Phase 1 | -| 3 - Instrumentation and pilot | Not started | Blocked by Phase 2 | +| 1 - Final environment verification | Passed | Native Windows/Docker Desktop strict preflight accepted | +| 2 - Minimal live smoke | Passed | Native Windows product-CLI 2/2 qualification accepted | +| 3 - Instrumentation and pilot | Out of scope | Environment-only task | | 4 - Protocol lock and freeze | Not started | Blocked by Phase 3 | | 5 - Good pass | Not started | Blocked by Phase 4 | | 6 - Analysis and tables | Not started | No measured corpus | @@ -164,6 +164,1995 @@ Ubuntu 24.04/ext4 host, then run `verify_environment.sh` and `paper-env-smoke`. +### 2026-07-30 (time not recorded) - Product installation-guide execution + +- **Entry ID:** `preflight-product-guide-004` +- **Phase:** 1 exploratory check +- **Kind:** preflight +- **Host:** Windows workstation with Docker Desktop 29.0.1 and Ubuntu 26.04 + under WSL 2. +- **Product branch/commit:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Actions:** followed the product's Windows source-build and smoke guide; + separately built the two required Linux release binaries in WSL ext4 + storage. +- **Windows product smoke:** passed image listing, sandbox creation, + `exec_command`, snapshot, and destruction. +- **Linux bundle SHA-256:** gateway + `f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf`; + catalog exporter + `c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba`; + daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Catalog:** schema version 1, three domains, 37,216 bytes, SHA-256 + `5d270c1ac87bad74319f202b127404b5cfd363b185362ceedaece437c88bfef7`. +- **Disposition:** passed for product capability and staging; not a final-host + preflight and not performance evidence. +- **Next action:** run the benchmark's minimal live smoke with the staged Linux + bundle. + +### 2026-07-30 (time not recorded) - Protocol amendment v0.2 + +- **Entry ID:** `amendment-current-product-contract-001` +- **Reason:** current product installation/package scripts and runtime source + no longer use fixed Git toolchain archives; the imported benchmark contract + predated the current product catalog, response schemas, and daemon profile. +- **Decision timing:** before any pilot or paper-eligible measurements. +- **Files changed:** benchmark gateway preflight, catalog/product/observability + models, resource sampling, daemon defaults, focused tests, environment + preflight, product-boundary documentation, and protocol inventory. +- **Changes:** removed the obsolete Git-archive prerequisite; accepted current + float catalog arguments and strict current product response fields; updated + resource sampling for explicitly partial initial cgroup snapshots; replaced + the obsolete daemon thread setting with current standard-profile fields. +- **Verification:** 30 focused unit, contract, and integration tests pass. +- **Prior runs invalidated:** none; no paper-eligible run exists. +- **Required reruns:** environment smoke, five-sample pilot, and good pass. +- **Disposition:** accepted as a pre-measurement compatibility amendment. + +### 2026-07-30T01:19:10Z - End-to-end benchmark environment smoke + +- **Entry ID:** `smoke-wsl-ext4-005` +- **Phase:** 2 exploratory check +- **Kind:** smoke +- **Host identity:** Windows workstation, Docker Desktop Linux AMD64 engine, + Ubuntu 26.04 WSL 2 control environment, isolated WSL ext4 execution + snapshots. +- **Product branch/commit:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Benchmark plan/hash:** `paper-env-smoke`; + `sha256:de517817323c2017b1daa7029dd3403f40f4ec24394c08a7f8c37b88f7afe13e`. +- **Environment fingerprint:** + `sha256:de2011d4dcc196ae6b86f28dcd20aa1db5bf0f3b3a4972ad18867384c6aec12b`. +- **Python:** CPython 3.14 in the isolated WSL virtual environment. +- **Run ID:** `019fb09a-9d36-7686-bb51-ab73803daac6`. +- **Start/end/elapsed:** `2026-07-30T01:19:10.773459Z` / + `2026-07-30T01:19:23.171695Z` / 12.398236 seconds. +- **Result:** completed; correctness passed; 2 of 2 trial batches; two issued + operation requests; zero failures; zero warnings; report ready. +- **Raw artifact path:** + `/root/.cache/ephemeral-sandbox-paper/live-smoke-20260730i/test-root/.benchmark-state/results/019fb09a-9d36-7686-bb51-ab73803daac6`. +- **Cleanup:** benchmark-owned smoke resources cleaned; an unrelated + pre-existing sandbox was intentionally left untouched. +- **Disposition:** passed as an exploratory development/pilot smoke. Final-host + Gate 2 remains open because Gate 1 has not passed on native Ubuntu 24.04. +- **Supported interpretation:** the documented build/staging path and current + benchmark-product integration can complete create, execute, observe, and + cleanup on this computer. +- **Unsafe interpretation:** paper-grade performance, native-host readiness, or + comparison with another sandbox. +- **Next action:** review the Phase 3 measurement boundaries, then reproduce + the same preflight and smoke on the selected native Ubuntu 24.04 host. + +### 2026-07-30 (time not recorded) - Post-amendment good-pass validation + +- **Entry ID:** `planning-good-pass-validation-002` +- **Phase:** 0 +- **Kind:** preflight +- **Plan/hash:** `paper-good-pass`; + `sha256:878e4a68a138d71650a92d381428f71e9e53e0671104a851eadd97c411baeab4`. +- **Result:** runnable; 18 cells, 1,836 trial batches, 5,508 issued operation + requests, zero planner warnings. +- **Focused regression:** 30 tests passed in 0.46 seconds. +- **Shell verification:** `verify_environment.sh` passed `bash -n`. +- **Disposition:** passed as configuration validation only; no good-pass + operation was executed and no performance evidence was produced. + +### 2026-07-30 (time not recorded) - Protocol amendment v0.3 + +- **Entry ID:** `amendment-final-environment-specificity-001` +- **Reason:** the environment choice and handoff boundary needed one + operationally exact interpretation. +- **Decision timing:** before pilot or paper-eligible measurements. +- **Selected environment:** logical host `eos-benchmark-ubuntu24`; native + Ubuntu Server 24.04 x86-64; Docker Engine 29.0.1; ext4; CPython 3.13; product + at `/srv/eos-benchmark/product`; paper at `/srv/eos-benchmark/paper`. +- **Product baseline:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Image:** pinned Linux AMD64 Ubuntu 24.04 manifest already recorded by this + protocol. +- **Decision:** current WSL/Docker Desktop is GO for development smoke and a + diagnostic pilot, but NO-GO for `paper-good-pass` or paper-table numbers. +- **Prior runs invalidated:** none; the successful smoke remains exploratory. +- **Required reruns:** strict preflight and `paper-env-smoke` on + `eos-benchmark-ubuntu24`, followed by the Phase 3 pilot. +- **Disposition:** accepted as a pre-measurement specificity amendment. + +### 2026-07-30T01:47:01Z - Final-host reachability discovery + +- **Entry ID:** `preflight-final-host-discovery-006` +- **Phase:** 1 +- **Kind:** preflight +- **Operator:** Codex environment-qualification handoff +- **Host identity:** Windows development workstation; not the selected final + host. +- **Exact commands/checks:** workspace reference search; Windows hosts-file + lookup; SSH config/effective-target and known-hosts lookup; DNS resolution + for `eos-benchmark-ubuntu24`, `.local`, and the active `ctc` suffix; ICMP + lookup; local virtualization inventory; WSL inventory; Docker context + inventory; Multipass instance inventory. +- **Attempts:** two broad read-only repository/discovery commands timed out + before useful output and were repeated as bounded checks. +- **Result:** no hosts-file, SSH, known-hosts, DNS, Docker-context, Hyper-V, or + Multipass route exists for `eos-benchmark-ubuntu24`. The only Multipass + instance is `ephemeral-sandbox-2204` on Ubuntu 22.04 and is ineligible. WSL + Ubuntu and Docker Desktop are running but remain explicitly ineligible. +- **Disposition:** failed because the selected external host is unreachable; + no system state changed. +- **Supported interpretation:** the final host cannot be qualified from this + machine until it is provisioned and exposed through a reachable SSH route. +- **Unsafe interpretation:** that no suitable host exists outside locally + discoverable configuration. +- **Next action:** provision the documented native Ubuntu 24.04 host and make + `eos-benchmark-ubuntu24` SSH-reachable from this workstation. + +### 2026-07-30T01:47:01Z - Canonical Linux bundle audit + +- **Entry ID:** `planning-canonical-bundle-audit-003` +- **Phase:** 1 staging +- **Kind:** preflight +- **Product branch/commit:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Canonical layout:** gateway and catalog exporter now exist under + `target/release`; the daemon exists under `dist`. +- **Binary verification:** all three files are executable x86-64 ELF under + WSL. SHA-256 values are gateway + `f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf`, + catalog exporter + `c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba`, + and daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Product dirty state:** clean; `target/` remains ignored. +- **Disposition:** passed for off-clock staging readiness; not a final-host + preflight. +- **Next action:** transfer this exact bundle to the selected host. + +### 2026-07-30T02:06:11Z - Final-host handoff automation validation + +- **Entry ID:** `planning-final-host-handoff-validation-004` +- **Phase:** 1 staging +- **Kind:** preflight +- **Operator:** Codex environment-qualification handoff +- **Host identity:** Windows development workstation with WSL used only as an + ineligible negative-control Linux parser/runtime; not the selected final + host. +- **Product branch/commit/tag:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`; annotated tag `v0.1.4` + resolves to that commit. +- **Checks:** PowerShell parsing; `bash -n` for all three Linux scripts; + deliberate execution of the strict verifier on WSL; bundle-manifest + size/hash verification; `git bundle verify` and bundled `main` ref + verification; archive enumeration; archive exclusion checks; archived shell + parsing; and SHA-256 verification of all three canonical Linux ELF payloads. +- **Negative-control result:** the strict verifier passed Linux and x86-64, + then failed closed because the logical hostname was not + `eos-benchmark-ubuntu24`. It did not reach or run the benchmark. +- **Attempts/anomalies:** direct PowerShell script execution was blocked by the + workstation execution policy; the documented command now uses + `-ExecutionPolicy Bypass`. One combined archive-check command and one + negative-control output-capture command had local shell-quoting errors and + were repeated as bounded checks. A temporary clone-check command was rejected + before execution by local destructive-action policy; complete-history bundle + verification and exact ref enumeration passed instead. Three earlier generated + bundles were superseded after documentation and Windows long-path handling + were hardened, then removed; they are reproducible with the recorded bundle + command. +- **Final raw artifact path:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\ephemeral-sandbox-v1-20260730-ready-v2`. +- **Final archive SHA-256:** product Git bundle + `cbb267a93c0575934b5f0571e057fadbc04ce93e340dbd874f26664eaf5e531a`; + product artifacts + `570bbac95bd828fea88d7ce6dfcd03d3f86769126943a4c5e54b25395dff2f21`; + paper snapshot + `724ca7e0c5a25880f2045502074d9c12d32ddb6623d583d48c2aec33c276d1b9`. +- **Archived payload SHA-256:** gateway + `f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf`; + catalog exporter + `c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba`; + daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Failures/exclusions:** CPython 3.13 and the native-host runtime could not be + exercised locally; WSL remains excluded. No benchmark run or measurement was + performed. +- **Disposition:** passed for reproducible transfer readiness; strict final-host + Gate 1 and Gate 2 remain unrun. +- **Next action:** provision the documented native host and make + `eos-benchmark-ubuntu24` SSH-reachable from this workstation. + +### 2026-07-30T02:39:00Z - v0.1.4 product-CLI artifact staging + +- **Entry ID:** `planning-v014-cli-artifacts-005` +- **Phase:** 1 staging +- **Kind:** preflight +- **Product branch/commit/tag:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`; annotated release `v0.1.4`. +- **Release source:** public `v0.1.4` Linux AMD64 archive. +- **Release archive SHA-256:** + `308563ad38bc7a9c5000acd54251db872e2e6a58bf70846d14760fef2b0d713c`. +- **Staged product CLI SHA-256:** manager + `0be4f0c26f8f50b76b175d04cfeec61529a605bcda9ffcd6782a09096ba2983f`; + runtime + `e9ac5f6c7a5f9c07a3de166b320e7d6065fa9480a7f18d6d59114337d15e28e7`; + observability + `6b2dae2369344cbb3960a76f6ccdfa869a7aa9b7a7a255f8a634f4a52d5cfdb5`. +- **Validation:** all three are executable Linux x86-64 ELF files in ignored + `target/release`; product Git remains clean. +- **Attempt anomaly:** the first release-metadata query requested unsupported + `isLatest` JSON output and failed; the bounded retry used supported fields. + One hash loop used the wrong daemon location and one inline WSL loop was + shell-mangled; both were rerun with explicit paths. No product source changed. +- **Disposition:** passed for transfer staging; not a final-host gate. + +### 2026-07-30T02:42:00Z - Protocol amendment v0.4: product-CLI environment gate + +- **Entry ID:** `amendment-cli-only-environment-gate-001` +- **Reason:** the imported `paper-env-smoke` invokes `direct_client`, while the + required public-product boundary is the released manager, runtime, and + observability CLIs. +- **Decision timing:** before any native-host qualification or performance + measurement. +- **Change:** the environment smoke now runs two independent sandbox + lifecycles. Each lifecycle uses ten validated product-CLI calls covering + image listing, create, execute, write/read/edit/read, snapshot, destroy, and + post-destroy listing. +- **Cleanup rule:** product CLI performs sandbox destruction. The harness then + removes only shared-base volumes carrying its unique gateway-instance label + and requires both a clean initial baseline and clean final state. +- **Prior runs invalidated:** WSL run + `019fb09a-9d36-7686-bb51-ab73803daac6` remains historical diagnostic + evidence but is invalid for Gate 2 because it used `direct_client`. +- **Performance boundary:** no performance preset is authorized; the imported + planner still lacks a product-CLI subprocess cohort. +- **Disposition:** accepted as a pre-qualification environment amendment. + +### 2026-07-30T02:49:42Z - Product-CLI smoke automation diagnostics + +- **Entry ID:** `smoke-cli-automation-diagnostic-006` +- **Phase:** 2 diagnostic +- **Kind:** smoke +- **Host identity:** Windows workstation, Docker Desktop Linux AMD64 engine, + Ubuntu 26.04 WSL 2 control environment; ineligible as the final host. +- **Image:** `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf`. +- **Attempts:** + 1. The first inline PowerShell/WSL launcher was shell-mangled and failed + before the smoke script received an artifact directory. + 2. The next attempt failed before gateway startup because the script + incorrectly rejected a symlinked Python interpreter. The check was + corrected while keeping non-symlink enforcement for product artifacts. + 3. A one-lifecycle run passed its CLI correctness checks, but its claimed 2/2 + accounting was rejected as unsupported. + 4. A true two-lifecycle run passed 20 CLI calls, but inspection showed two + shared-base cache volumes owned by that diagnostic gateway. The run was + rejected for the no-leak gate. + 5. The final implementation added exact-owner volume cleanup and passed two + independent lifecycles. +- **Additional command anomalies:** one PowerShell glob was invalid for `rg`; + one Docker volume-inspection loop and the first exact-label cleanup command + were quoting-mangled. Each failed without deletion and was repeated with + explicit JSON inspection. +- **Superseded diagnostic cleanup:** one cache volume for gateway instance + `cli-env-smoke-20260730T024942Z-16705` and two for + `cli-env-smoke-20260730T025229Z-17455` were removed after exact label + verification. Older unrelated EOS resources were left untouched. +- **Final diagnostic result:** `product_cli`; 2/2 completed batches; two + distinct sandboxes; 20 validated product-CLI calls; correctness pass; zero + failures; zero warnings; empty CLI stderr; six seconds; zero containers and + zero volumes remaining for gateway instance + `cli-env-smoke-20260730T025443Z-18897`. +- **Final raw artifact:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\eos-cli-env-diagnostic-cleanup-20260730T120003Z.tar.gz`. +- **Final artifact SHA-256:** + `9d4c16f059dbe24f61cea470715f39afc41673b6261dc5252595adaaccb4acee`. +- **Disposition:** passed as automation diagnostic only; Gate 1 and Gate 2 + remain open because WSL is ineligible. +- **Unsafe interpretation:** native-host qualification, performance evidence, + or a paper-table result. + +### 2026-07-30T02:55:00Z - CLI final-host verifier negative control + +- **Entry ID:** `preflight-cli-final-host-negative-control-007` +- **Phase:** 1 +- **Kind:** preflight +- **Host identity:** Ubuntu 26.04 WSL 2; deliberately ineligible. +- **Exact command:** strict `verify_environment.sh` with canonical product + artifact paths and pinned image. +- **Result:** Linux and x86-64 checks passed, then the verifier failed closed + at the exact logical-hostname check with exit code 1. It did not start a + gateway or sandbox. +- **Log:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\verify-environment-wsl-negative-control-20260730.txt`. +- **Disposition:** passed as a negative control; native-host preflight remains + unrun. + +### 2026-07-30T03:00:00Z - Six-artifact transfer-bundle validation + +- **Entry ID:** `planning-cli-transfer-bundle-validation-006` +- **Phase:** 1 staging +- **Kind:** preflight +- **Bundle attempt:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\ephemeral-sandbox-v1-cli-final`. +- **Archive SHA-256:** product Git bundle + `fc5879964c7e54203c52d19dc3a54ae69f76722597bb0870e25d319afc990459`; + product artifacts + `0b0d386c2a38618163ab0028c446cdc25ef601dee1904ad0b5777595f3d7eddb`; + paper snapshot + `4389e2c9929462880d0722456bb515b3f1c67f2207ebe61eff2beb33c671270e`. +- **Validation:** manifest archive hashes matched; `git bundle verify` reported + complete history and exact `main` ref + `b22862550e0a7cb4fe61ce581831e9244cc492b5`; the six extracted artifacts + matched their pinned hashes; archived Bash and PowerShell scripts parsed; + the 310-entry paper archive excluded `.venv`, `.benchmark-state`, + `experiments/runs`, and `__pycache__`. +- **Attempt anomaly:** an initial combined validator was rejected before + execution because it contained recursive temporary-directory cleanup. The + validator was rerun without deletion, and the extracted audit directory was + retained at + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\bundle-validation-cli-final`. +- **Disposition:** validation passed, but this bundle is superseded solely so + the append-only record of its validation is included in the final paper + snapshot. +- **Final target:** regenerate as + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\ephemeral-sandbox-v1-cli-final-v2`. + Its external `bundle-manifest.json` is the authoritative archive-hash record; + the bundle is not paper or performance evidence. + +### 2026-07-30T03:05:00Z - Protocol amendment v0.5: Windows host correction + +- **Entry ID:** `amendment-windows-docker-host-001` +- **Reason:** user clarification established that the host is native Windows + with Docker Desktop; Ubuntu 24.04 is the sandbox image, not the host OS. +- **Decision timing:** before accepting a final environment result. +- **Superseded contract:** native Ubuntu Server 24.04 host, + `eos-benchmark-ubuntu24`, ext4 host paths, CPython 3.13 qualification + dependency, SSH exposure, and Linux transfer bundle. +- **Corrected contract:** native Windows x64 host `DESKTOP-OLP1ADS`; Docker + Desktop 29.0.1 Linux AMD64 engine with `overlayfs` and cgroup v2; native + Windows gateway and product CLIs; released Linux daemon inside containers; + pinned Ubuntu sandbox image. +- **Prior runs invalidated:** Linux/WSL runs remain diagnostics but cannot + establish the corrected native Windows CLI boundary. +- **Required rerun:** strict native Windows preflight plus two independent + product-CLI sandbox lifecycles. +- **Disposition:** accepted before the final Windows qualification. + +### 2026-07-30T03:06:00Z - Official Windows v0.1.4 package staging + +- **Entry ID:** `planning-v014-windows-package-007` +- **Phase:** 1 staging +- **Kind:** preflight +- **Release archive:** `ephemeral-sandbox-windows-amd64.zip`. +- **Archive SHA-256:** + `9f2327578c186897578f0d502893d894aed52be27306f43f75afa3205eba9fdb`. +- **Staged path:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4`. +- **Artifact hashes:** gateway + `3a96bedcfa9857bd3881155d758ec2d969f6265456ec3b2878eb6dbb26dc9368`; + manager CLI + `b43ec520edc2f436adc8aa7e8b2b50680bb9021883fe23d79a85b17afd2e10fe`; + runtime CLI + `df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4`; + observability CLI + `0e0471e52750805570876a6244868764c44e166ec653627b9ebd490176e2fcbe`; + config + `0f0efd15e5111851054e0f7c1ce0f3eaebb3b3047c1b9e2322544036f5daf5db`; + daemon + `2da4395cd835e5325bc3e55b9c2f3b67565ea7c698fce5e086167ec4a2092a39`. +- **Validation:** four native binaries are x64 PE; daemon is x86-64 ELF; + package is under ignored `target/`; product remains clean `main`. +- **Attempt anomaly:** the first GitHub release metadata request ended with a + transient GraphQL EOF. A bounded retry succeeded before download. +- **Disposition:** passed. + +### 2026-07-30T03:10:00Z - Native Windows qualifier development attempts + +- **Entry ID:** `smoke-windows-docker-attempts-008` +- **Phase:** 1/2 +- **Kind:** preflight and smoke +- **Host:** native Windows x64 `DESKTOP-OLP1ADS`, build 26200. +- **Docker:** Docker Desktop client/server 29.0.1; Linux AMD64, + `overlayfs`, cgroup v2. +- **Attempts:** + 1. `qualification-windows-docker-20260730-final` passed the host/package/ + Docker/image checks, then strict-mode cleanup hit a scalar `.Count` bug. + No result was accepted. + 2. `...-final-2` passed the same preflight, then the native gateway rejected + PowerShell's BOM-bearing generated YAML; the expected startup connection + refusal also terminated instead of retrying. No sandbox was created. + 3. `...-final-3` used BOM-less YAML and reached a ready native gateway. The + container daemon correctly rejected Windows host paths in its in-container + config. The gateway and daemon configs were separated. + 4. `...-final-4` created a ready sandbox, but strict validation rejected the + returned `\\?\` Windows path form before recording the active sandbox ID. + The known sandbox was subsequently destroyed through a restarted gateway + and `sandbox-manager-cli.exe`; its uniquely labeled shared-base volume was + then removed, leaving zero resources for that gateway instance. + 5. `...-final-5` passed create, execute, write, and read. Windows command-line + parsing stripped the inner quotes from the `file_edit` JSON argument; the + active sandbox was destroyed by the product CLI and exact-owner cleanup + ended at zero resources. +- **Disposition:** all attempts preserved; none accepted as the final + qualification. +- **Performance boundary:** no benchmark or performance experiment ran. + +### 2026-07-30T03:16:21Z - Native Windows Docker Desktop qualification + +- **Entry ID:** `qualification-windows-docker-final-009` +- **Phase:** 1 and 2 +- **Kind:** preflight and smoke +- **Host:** native Windows x64 `DESKTOP-OLP1ADS`, build 26200; 48 logical + CPUs; 137,438,953,472 bytes physical memory; qualified paths on NTFS. +- **Product:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`; official `v0.1.4` Windows + package and hashes recorded above. +- **Docker/image:** Docker Desktop 29.0.1, Linux AMD64, `overlayfs`, cgroup v2; + exact pinned Ubuntu 24.04 image present. +- **Client cohort:** native Windows `product_cli`. +- **Result:** completed; correctness pass; 2/2 independent batches; 20 + validated CLI calls; zero failures; zero warnings; empty CLI stderr. +- **Sandbox IDs:** `eos-edab92fa-dbbc-4ec1-b0ac-1a8e35cfdb8c` and + `eos-2311f6bb-328f-4ef7-bc5f-f1b30db6a5b5`. +- **Cleanup:** product CLI destroyed both sandboxes; global EOS-owned baseline + unchanged; zero containers and zero volumes remained for gateway instance + `cli-env-windows-20260730T031621Z-34036`. +- **Elapsed:** ten seconds, environment-gate observation only. +- **Artifact directory:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6`. +- **Archive:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6.zip`. +- **Archive SHA-256:** + `eea981665b031846677046d4c211e71ad144f8a32507c09058923241d4d0f7f9`. +- **Disposition:** passed; the corrected Windows plus Docker Desktop + environment is qualified. +- **Unsafe interpretation:** a performance result or paper-table number. + +### 2026-07-30T03:20:48Z - Windows qualification record validation + +- **Entry ID:** `analysis-windows-qualification-validation-010` +- **Phase:** 1 and 2 +- **Kind:** analysis +- **Checks:** PowerShell parser; paper-state and qualification-summary JSON; + strict summary acceptance; exact gateway-owner Docker resource query; archive + checksum; product branch/commit/dirty state; scoped Git whitespace check. +- **Attempt anomaly:** the first whitespace check found two trailing spaces in + the rewritten environment status header. They were removed and the check was + repeated. +- **Result:** all checks passed; product remains clean `main`; accepted gateway + owner has zero containers and zero volumes; archive SHA-256 remains + `eea981665b031846677046d4c211e71ad144f8a32507c09058923241d4d0f7f9`. +- **Disposition:** passed; no additional environment action is required. + +### 2026-07-30 (time not recorded) - Canonical sandbox workspace base + +- **Entry ID:** `planning-windows-workspace-base-008` +- **Phase:** environment staging +- **Kind:** preflight +- **Path:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\workspace-base\ephemeral-sandbox-v0.1.4`. +- **Source:** local clean product checkout. +- **Branch/commit/tag:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`; `v0.1.4` resolves to the same + commit. +- **Generated-state exclusion:** no `target` directory exists in the clone. +- **Attempt anomaly:** Git emitted non-fatal Windows filename/attribute + warnings for historical report paths containing long or Windows-incompatible + names. Clone exit code, branch, commit, tag resolution, and Git status all + passed. +- **Decision:** this source-only clone is the canonical + `--workspace-bind-root` for repository-backed sandbox creation. The build + checkout, paper repository, and parent lab directory are not workspace bases. +- **Qualification distinction:** the accepted environment gate continues to + use its own tiny isolated fixtures; it does not claim that the full base + repository was an input to the qualification smoke. +- **Disposition:** passed for explicit workspace-base staging; no performance + experiment ran. + +### 2026-07-30T03:39:00Z - EXP1 CLI performance campaign specification + +- **Entry ID:** `planning-exp1-cli-campaign-spec-011` +- **Phase:** 0 and planning for 3--7 +- **Kind:** amendment +- **Reason:** the user requested corrected experiment documentation and a + complete execution specification for the next agent. +- **Decision timing:** all decisions were made before viewing performance + measurements; no performance preset or pilot was run. +- **Environment decision:** retain the already qualified native Windows x64 + host with Docker Desktop's Linux AMD64 cgroup-v2 engine and pinned Ubuntu + image. Do not restore superseded native-Ubuntu-host requirements. +- **Client decision:** every paper sandbox operation must use the released + native Windows manager, runtime, or observability CLI through the canonical + `product_cli` cohort. `direct_client` and `cli_e2e` are prohibited. +- **Timing decision:** primary latency is end-to-end native CLI subprocess + latency from process creation through validated process exit, including + process launch and CLI-to-gateway transport. +- **Workspace decision:** command, lifecycle, read, write, and edit cells all + use a fresh copy of the deterministic `paper-100m` base. +- **Matrix decision:** add a distinct concurrency-1 manager-CLI sandbox-create + + base-mount cell. The final focused matrix is 19 cells; session-create + remains separate. +- **Files changed:** `progress.md`, `experiment_inventory.md`, + `experiments/expected_tables.md`, `benchmark/PAPER_ARTIFACT.md`, + `plan/progress.md`, and + `plan/task-packets/exp1-cli-performance-campaign.md`. +- **Prior runs invalidated:** none. Qualification remains environment evidence; + no performance measurements exist. +- **Required next action:** implement and review the paper-local benchmark's + `product_cli` cohort and pass its deterministic tests before any live + performance preset. +- **Approval boundary:** documentation/specification work was user-authorized; + Gate 0 protocol approval and any later commit, tag, or push remain explicit + gates. +- **Disposition:** passed as a planning amendment; no experimental result. +- **Supported interpretation:** the environment is qualified and the next + experiment work is precisely specified. +- **Unsafe interpretation:** any performance conclusion or claim that the + current `direct_client` presets are eligible. + +### 2026-07-30T03:57:08Z - EXP1-A pre-edit audit and hash baseline + +- **Entry ID:** `audit-exp1-cli-campaign-012` +- **Phase:** EXP1-A / Gate 0 +- **Kind:** preflight +- **Operator:** Codex EXP1 campaign +- **Host identity:** native Windows x64 `DESKTOP-OLP1ADS`, build 26200. +- **Paper Git state before campaign files were added:** clean branch + `agent/complete-pw3-and-final-host-prep` at + `347f32efa3fd19465d43dd40de2c5566404487ff`. +- **Product Git state:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Pre-edit hash artifact:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\exp1-pre-edit-baseline.json`. +- **Pre-edit hash artifact SHA-256:** + `b408147fd6cf4d826c1e03f2beb10137a35e944b702639abc7fb5be79936f959`. +- **Hash coverage:** every tracked file in both repositories; new campaign + files are recorded as absent from the pre-edit tree. +- **Required-source review:** the complete EXP1 packet, every item in its + required read-first sequence, and the research-handoff/artifact references + required by the AI research-writing evidence workflow were read before any + existing file was edited. +- **Released CLI contract audit:** manager, runtime, and observability catalog + and operation help completed successfully. The executables do not implement + `--version`; one bounded `--version` probe exited 2 with a JSON + `invalid_request` envelope and produced no state change. Release identity is + therefore the annotated `v0.1.4` tag plus exact binary hashes. +- **Benchmark audit findings:** the scheduler unconditionally constructs + `ProductAccess(GatewayClient)`; workspace-session creation uses direct daemon + transport and Docker credential lookup; paper presets select + `direct_client`; file operations omit `paper-100m`; sandbox creation is + untimed setup; no `paper-pilot` exists; raw requests omit CLI subprocess + provenance; resource sampling omits available daemon self CPU/RSS; and no + deterministic expected-table generator exists. +- **Native-Windows audit findings:** importing the runner exited 1 because + `os.killpg` is unavailable; executable resolution omits `.exe`; the released + Windows package intentionally has no catalog-export binary; process identity + and runner RSS are Linux-only; the fixture cache uses the Windows-invalid + `sha256:` directory spelling; directory `fsync` is POSIX-only; and metadata + resolves Linux package paths instead of the staged Windows release. +- **Measurement boundary review:** primary timing is fixed as native CLI + process launch through exit, separate stdout/stderr capture, exact single + JSON parse, and response-schema validation. Setup, CLI verification, + observability sampling, and cleanup remain untimed. +- **Disposition:** audit completed; Gate 0 remains in progress until the + implementation, protocol cross-links, and deterministic validation remove + the listed ambiguities. No live performance preset was run. +- **Supported interpretation:** the qualified environment and released CLI + contracts are sufficient to implement a paper-local native-Windows cohort. +- **Unsafe interpretation:** any performance result or claim that the imported + harness is currently runnable on native Windows. +- **Next action:** implement the closed `product_cli` adapter, native-Windows + lifecycle support, exact 19-cell plans, evidence contract, and tests. + +### 2026-07-30T04:02:54Z - Released Windows CLI 256 KiB argv blocker + +- **Entry ID:** `diagnostic-exp1-windows-cli-argv-limit-013` +- **Phase:** EXP1-A / EXP1-B contract validation +- **Kind:** preflight +- **Operator:** Codex EXP1 campaign +- **Host identity:** native Windows x64 `DESKTOP-OLP1ADS`, build 26200. +- **Product branch/commit/tag:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`, release `v0.1.4`. +- **Runtime CLI:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4\bin\sandbox-runtime-cli.exe`. +- **Runtime CLI SHA-256:** + `df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4`. +- **Reviewed released contract:** `file_write` accepts file content only through + `--content TEXT`; the CLI exposes no stdin, response-file, or content-file + alternative. +- **Exact diagnostic:** construct 262,144 ASCII characters in memory and invoke + the released runtime CLI with the required `file_write --content` argument + against the deliberately unused socket `127.0.0.1:1`. +- **Command host exit status:** PowerShell command process exited 0 after + recording the failure because the native child never started and + `$LASTEXITCODE` remained null. +- **Native child exit status:** unavailable; process creation failed before an + executable instance existed. +- **Observed failure:** PowerShell `ResourceUnavailable` / + `NativeCommandFailed`: `The filename or extension is too long`. +- **Evidence artifact:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\diagnostics\exp1-windows-cli-argv-limit-20260730.json`. +- **Evidence artifact SHA-256:** + `41429d69f18de0d0a9426b66cb1dcf841aa7ddd1d9e4dff8dd8ec329b2c19a1c`. +- **Protocol impact:** the fixed 256 KiB `file_write` cells at concurrency 1 + and 5 cannot launch through the released native Windows CLI. Therefore the + exact 19-cell pilot, its Gate 3 acceptance, protocol freeze, and final pass + are impossible with the pinned package. +- **Fallback decision:** none. Direct gateway access, `exec_command`, + host-side mutation, payload reduction, cell removal, or a different cohort + would violate the preregistered operation boundary. +- **Partial paper-local work retained:** pre-edit hash capture plus preliminary + native-Windows/catalog/session/observability adapter scaffolding in the + changed benchmark files. It is intentionally uncommitted and has not passed + EXP1-B tests. +- **Cleanup result:** the runtime CLI process count is zero; no gateway was + started; Docker was not contacted; no sandbox, container, volume, runtime + state, or measurement artifact was created; the product repository remains + clean `main`. +- **Disposition:** failed; actual EXP1 stop condition reached before any live + performance preset. +- **Supported interpretation:** the released Windows CLI invocation contract + cannot carry the preregistered 256 KiB `file_write` payload. +- **Unsafe interpretation:** any performance conclusion, any claim that a pilot + ran, or any estimate of the missing cells. +- **Single required external action:** publish and stage a checksum-pinned + Windows release whose runtime CLI can read `file_write` content from stdin or + a file without placing the payload in the native process command line; then + log a pre-freeze protocol/product-identity amendment and rerun Gates 0--3. + +### 2026-07-30T04:06:43Z - Async benchmark-path reproduction and source-contract audit + +- **Entry ID:** `diagnostic-exp1-asyncio-cli-payload-014` +- **Phase:** EXP1-B contract validation +- **Kind:** preflight +- **Operator:** Codex EXP1 campaign +- **Purpose:** independently reproduce the 256 KiB launch failure through the + exact Python subprocess API required by the benchmark and exhaust released + CLI content-input alternatives. +- **Python:** native Windows CPython 3.13.14, + `asyncio.create_subprocess_exec`, argument array, no shell. +- **Small control:** the released runtime CLI process started for a 4,096-byte + `file_write` payload, received a PID, and exited 1 after the deliberately + unavailable gateway connection. This proves the diagnostic command shape and + executable are otherwise launchable. +- **Required payload:** the same API with 262,144 content bytes raised native + `WinError 206` before assigning a PID. The gateway connection was never + attempted. +- **Async probe artifact:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\diagnostics\exp1-python-subprocess-payload-probe-20260730.json`. +- **Async probe SHA-256:** + `6cefe6eea5876b70e0c72a399d5dd10ddce9d236ce01daef2066cb0ca3269beb`. +- **Source contract:** exact `v0.1.4` source at the required commit copies + string/path values directly from argv. Runtime projection exposes + `file_write` content only as `--content TEXT`; the separately named + `write_command_stdin` operation targets an already-running command and + cannot supply `file_write` content. +- **Source-contract artifact:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\diagnostics\exp1-cli-file-write-contract-20260730.json`. +- **Source-contract SHA-256:** + `cf5c3c315a9a78f138d5cdfd5593798efe15fc3a3dace18f59233647c4884e9f`. +- **Cleanup:** the control runtime-CLI process exited; zero runtime-CLI + processes remain. No released campaign gateway was started and Docker was + not contacted. One debug-build gateway process predating EXP1 remains + running and was intentionally left untouched as unrelated user state. +- **Product state:** clean `main` at the required commit. +- **Disposition:** failed campaign contract; second independent proof of the + same released-product stop condition. No performance preset ran. +- **Supported interpretation:** the exact subprocess mechanism required by the + benchmark cannot create the preregistered 256 KiB `file_write` process. +- **Unsafe interpretation:** an estimate of file-write performance or a claim + about other product versions/interfaces. +- **Next action:** unchanged; a checksum-pinned Windows release needs a + non-argv file-content input before the fixed campaign can resume. + +### 2026-07-30T04:08:01Z - Third external-state blocker audit + +- **Entry ID:** `preflight-exp1-blocker-state-015` +- **Phase:** EXP1-B contract validation +- **Kind:** preflight +- **Operator:** Codex EXP1 campaign +- **Action:** re-inspected the current product repository, staged Windows + packages, runtime-CLI identity, and retained benchmark-path diagnostic before + deciding whether the campaign could resume. +- **Product state:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Available Windows packages:** only `windows-v0.1.4`. +- **Runtime CLI SHA-256:** + `df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4`, + unchanged from both failed launch proofs. +- **External-state change:** none. No new checksum-pinned release or alternate + released `file_write` content-input contract is present. +- **Disposition:** blocked. The same product-contract stop condition has now + recurred across the original campaign turn and two continued blocker audits; + no further safe in-scope benchmark action can make Gate 3 executable. +- **Measurements:** none; no smoke, pilot, final pass, or analysis command ran. +- **Cleanup:** no new process, gateway, sandbox, container, volume, or runtime + state was created by this read-only audit. +- **Single required external action:** publish and stage a checksum-pinned + Windows release whose runtime CLI accepts `file_write` content without + placing the payload in the native process command line. + +### 2026-07-30T04:21:50Z - Protocol amendment v1.1: authorized pre-freeze CLI input fix + +- **Entry ID:** `amendment-exp1-cli-content-input-016` +- **Phase:** EXP1-B contract implementation, before smoke, pilot, protocol + freeze, or final measurement. +- **Kind:** amendment +- **Reason:** the released native Windows runtime CLI accepts `file_write` + content only through `--content TEXT`; the preregistered 262,144-byte payload + therefore fails native process creation with `WinError 206` before a CLI + process exists. +- **Decision timing:** made before viewing any EXP1 smoke, pilot, or final + performance result. No live performance preset has run. +- **Authorization:** the user explicitly authorized checking out from product + `main` and fixing bugs on 2026-07-30. Work will remain on the existing + `ephemeral-sandbox` `main` checkout and will not be pushed. +- **Product starting identity:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`, release baseline `v0.1.4`. +- **Permitted product change:** add a bounded, non-command-line content source + for runtime-CLI `file_write` while preserving the existing `--content` + contract; add focused tests and help text; rebuild the native Windows + executable. +- **Files changed:** to be recorded after implementation. No product source had + changed when this amendment was written. +- **Metrics/cells/rules affected:** no metric, warmup, repetition, concurrency, + payload-size, success, retry, timeout, exclusion, or analysis rule changes. + The two 256 KiB `file_write` cells become launchable through the same required + CLI process boundary. +- **Identity rule:** the rebuilt runtime CLI is a new pre-freeze product + identity, not the original `v0.1.4` binary. Its product commit, package path, + binary SHA-256 values, and CLI contract must be requalified and frozen before + pilot acceptance. +- **Prior runs invalidated:** none; only preflight diagnostics exist. They remain + historical evidence for the original released-binary defect. +- **Required reruns:** focused product tests; exact 262,144-byte Windows + subprocess probe through the new input path; Gates 0--2 for the amended + package; then the full EXP1 smoke and 5-sample pilot Gate 3. +- **Reviewer/author approval:** direct user authorization in the active Codex + task. +- **Unsafe interpretation:** describing the amended binary as unmodified + `v0.1.4`, comparing pre-amendment diagnostics as performance data, or changing + the fixed EXP1 cell matrix. + +### 2026-07-30T04:31:18Z - Authorized runtime-CLI fix and amended Windows package + +- **Entry ID:** `product-fix-exp1-content-file-017` +- **Phase:** EXP1-B product-boundary repair before live smoke or measurement. +- **Kind:** amendment +- **Operator:** Codex EXP1 campaign. +- **Product starting identity:** clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- **Product resulting identity:** clean local `main` at + `46593b10faf945807ee6e4ec9f76fe8338cb1d19`, one commit ahead of + `origin/main`; commit subject `fix(cli): support file-backed write content`. +- **Product change:** runtime `file_write` now accepts + `--content-file FILE` as an alternative value source for the existing + semantic `content` argument. The CLI opens the file after process creation, + caps reads at the 16,777,216-byte gateway request limit, requires UTF-8, and + preserves the original `--content TEXT` behavior. +- **Boundary decision:** the semantic catalog, operation name, request schema, + gateway transport, and runtime handler are unchanged. The new input-source + metadata and help remain owned by `sandbox-cli::projection`. +- **Changed production files:** + `crates/sandbox-cli/src/projection/mod.rs`, + `crates/sandbox-cli/src/projection/runtime.rs`, + `crates/sandbox-cli/src/input.rs`, and + `crates/sandbox-cli/src/help.rs`. +- **Changed verification files:** + `crates/sandbox-cli/tests/runtime.rs`, + `crates/sandbox-cli/tests/projection_integrity.rs`, + `crates/sandbox-cli/tests/compatibility.rs`, + `crates/sandbox-cli/tests/manager.rs`, + `crates/sandbox-cli/tests/observability.rs`, and + `crates/sandbox-cli/tests/fixtures/observability-help.txt`. +- **Ancillary native-Windows test repairs:** exact text fixtures are compared + after CRLF-to-LF normalization; the stale observability help fixture now + matches the already-current semantic catalog wording. +- **Verification commands:** + `cargo fmt --all -- --check`; + `cargo test -p sandbox-cli --all-features`; + `cargo clippy -p sandbox-cli --all-targets --all-features -- -D warnings`. +- **Verification result:** passed; 56 integration tests passed, all feature + combinations compiled, doc tests passed, and strict clippy reported no + warning. +- **Package command:** + `powershell.exe -NoProfile -ExecutionPolicy Bypass -File + .\bin\package-windows-amd64-release.ps1 -PackageName + windows-exp1-46593b10 -OutDir target -Profile release`. +- **Package path:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-46593b10`. +- **Package archive SHA-256:** + `a3550b4464b278ffbc35b398378abd2f649f21ba18e8beb4368c7933d28a892f`. +- **Binary SHA-256 values:** gateway + `5f50012539f92195f74d3474d9c5c52f9bd0389fa565668094846a97e73a5b94`; + manager CLI + `512e93523ddd52e7f2e12e0fe7977dc7b9a4541b40ed9ab25fce844cbad191ac`; + runtime CLI + `1cff6dda6934e2bff4562dc2b51c8d82844ca6129095f7e313791c111d5d1c3a`; + observability CLI + `6bf7c542cb740c85575da015db55ee02fabb805236d21d4d14731df63bbb3043`; + Linux daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Release-label rule:** this local package is the amended EXP1 product + identity and must not be called the unmodified `v0.1.4` release. +- **Push status:** no product commit, branch, tag, or package was pushed. +- **Disposition:** passed focused product verification; exact campaign + requalification still required. + +### 2026-07-30T04:31:00Z - Rebuilt runtime-CLI 256 KiB subprocess proof + +- **Entry ID:** `diagnostic-exp1-content-file-018` +- **Phase:** EXP1-B amended-product contract validation. +- **Kind:** preflight +- **Operator:** Codex EXP1 campaign. +- **Runtime CLI:** amended package at product commit + `46593b10faf945807ee6e4ec9f76fe8338cb1d19`, SHA-256 + `1cff6dda6934e2bff4562dc2b51c8d82844ca6129095f7e313791c111d5d1c3a`. +- **Probe API:** native Windows CPython + `asyncio.create_subprocess_exec`, argument array, no shell. +- **Probe contract:** prepare an exact payload file before starting the clock, + then time process creation, the CLI's file read and request construction, and + its deliberately unsuccessful connection to `127.0.0.1:1`. +- **Payloads:** 4,096 and 262,144 ASCII bytes through + `file_write --content-file PREPARED-PAYLOAD-FILE`. +- **Observed result:** both processes started, received PIDs, constructed their + requests, and exited 1 with the expected `connection_error`. The required + 262,144-byte path therefore crossed the prior native process-creation + boundary and reached gateway transport. +- **Initial diagnostic artifact:** + `experiments/diagnostics/exp1-windows-cli-content-file-probe-20260730.json`, + SHA-256 + `f462ed2f294476267bd3f387e0db22151c158e06e763279b445a9e4a5a09334d`. +- **Initial diagnostic disposition:** failed assertion only. Product behavior + was successful, but the script expected error kind `gateway_error` rather + than the observed stable `connection_error`. The artifact is retained and + was not overwritten. +- **Corrected diagnostic artifact:** + `experiments/diagnostics/exp1-windows-cli-content-file-probe-20260730-v2.json`, + SHA-256 + `7640f479ca5b24dbfa5db7cd6e64732853de64eac92c7fdc0c3551ba6daa43b0`. +- **Corrected probe script:** + `experiments/scripts/probe_exp1_windows_cli_content_file.py`, SHA-256 + `c0a6ddf78257f057c500f1a2369075d6178bd16eb25590d06f3b4d1de3870b1d`. +- **Acceptance:** passed. The exact fixed large-payload subprocess can now + launch without putting content in the native Windows command line. +- **Measurements:** diagnostic elapsed times are not EXP1 performance data and + must not enter pilot, final, or paper tables. +- **Cleanup:** both diagnostic CLIs exited; temporary payload files were + deleted by the probe; no campaign gateway, sandbox, container, volume, or + runtime state was created. +- **Next action:** requalify the amended package and complete the native + Windows CLI-only cohort before smoke and pilot. + +### 2026-07-30T04:55:31Z - Smoke preflight exposed missing manager and observability request IDs + +- **Entry ID:** `smoke-preflight-exp1-cli-request-id-019` +- **Phase:** EXP1-B/EXP1-C boundary, before any successful smoke, pilot, + protocol freeze, or final measurement. +- **Kind:** amendment +- **Operator:** Codex EXP1 campaign. +- **Validated plan:** `paper-env-smoke`, plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + runnable with exactly 19 cells, 19 trial batches, and 55 operation requests. +- **First run ID:** `019fb15d-b8b6-7126-9ecd-637fe75b8e27`. +- **First-run disposition:** failed before any cell or operation request. The + staged Windows configuration omitted the gateway section; the harness + failed closed and terminal artifact finalization initially masked that + startup exception. No performance value was produced. +- **Preserved raw artifacts:** + `.benchmark-state/results/019fb15d-b8b6-7126-9ecd-637fe75b8e27`; + the failed manifest, plan, environment, and event journal were not + overwritten. +- **Harness corrections:** synthesize only the missing per-run gateway keys in + the staged Windows configuration; permit the readiness and campaign CLI + adapters to share one verified evidence directory; permit a failed run with + zero observations to derive a terminal report; register the sandbox + lifecycle report family. +- **Second preflight finding:** after the configuration correction, the gateway + started, but every manager-CLI readiness probe rejected + `--request-id`. Manager and observability CLIs mint UUIDs internally and + expose no explicit request-ID option, whereas the runtime CLI already + forwards one. This prevents exact benchmark-to-product request correlation. +- **Retained diagnostic evidence:** the failed run's `cli-subprocesses` + directory contains each redacted manager invocation and error envelope. + These readiness attempts are diagnostics, not EXP1 measurements. +- **Authorized product amendment:** extend the runtime CLI's existing bounded + explicit request-ID contract to the manager and observability CLIs, without + changing operation semantics, gateway transport, factors, repetitions, or + analysis. The user already authorized product bug fixes on local `main`; + the fix remains pre-freeze and will not be pushed. +- **Prior runs invalidated:** no valid run. The failed smoke preflight remains + negative evidence and is excluded by design because it issued zero planned + operation requests. +- **Cleanup:** the failed startup paths terminated their gateway processes, + removed owned runtime state, and found no campaign containers or sandboxes. +- **Disposition:** partial; exact CLI correlation requires the authorized + product fix, rebuild, and full package requalification before smoke resumes. + +### 2026-07-30T04:59:22Z - Cross-domain CLI request-ID fix and second amended package + +- **Entry ID:** `product-fix-exp1-request-id-020` +- **Phase:** EXP1-B product-boundary repair before successful smoke or pilot. +- **Kind:** amendment +- **Product starting identity:** clean local `main` at + `46593b10faf945807ee6e4ec9f76fe8338cb1d19`. +- **Product resulting identity:** clean local `main` at + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, two commits ahead of + `origin/main`; commit subject + `fix(cli): preserve request IDs across domains`. +- **Product change:** manager and observability CLIs now accept the same + global, bounded `--request-id VALUE` option already implemented by the + runtime CLI and forward it unchanged into the gateway request. Shared + validation permits 1--128 ASCII letters, digits, period, underscore, colon, + or dash. Default UUID generation remains unchanged when the option is + omitted. +- **Changed production files:** `crates/sandbox-cli/src/input.rs`, + `manager.rs`, `observability.rs`, and `runtime.rs`. +- **Changed verification files:** manager and observability integration tests + plus their exact help fixtures. +- **Verification:** `cargo fmt --all -- --check`, + `cargo test -p sandbox-cli --all-features`, and + `cargo clippy -p sandbox-cli --all-targets --all-features -- -D warnings` + all passed; 58 sandbox-CLI integration tests passed. +- **Package path:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-8c79217e`. +- **Package archive:** 5,605,345 bytes, SHA-256 + `a2fe0bfcd9103e30a9116d02785ffa47a1bb4894e7c78614ad2a08d642f2b82c`. +- **Binary SHA-256 values:** gateway + `5f50012539f92195f74d3474d9c5c52f9bd0389fa565668094846a97e73a5b94`; + manager CLI + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`; + runtime CLI + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`; + observability CLI + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`; + daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Plan revalidation:** `paper-env-smoke` remains runnable at plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`, + exactly 19 cells, 19 batches, and 55 operation requests. +- **Scientific scope:** request correlation only. No operation, factor, cell, + payload, repetition, timeout, exclusion, metric, or analysis rule changed. +- **Push status:** neither product commit nor package was pushed. +- **Disposition:** passed focused product verification; live package smoke is + next. + +### 2026-07-30T05:01:55Z - Smoke v2 retained: Windows depth-100 fixture path + +- **Entry ID:** `smoke-exp1-windows-depth100-021` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb164-afe0-7e5b-a3d2-f067c3f53a98`. +- **Plan:** `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the isolated gateway started and manager-CLI readiness + passed with exact request ID. The first sandbox-lifecycle cell then failed + during fixture materialization before any planned operation request. +- **Cause:** the fixed depth-100 fixture uses paths beyond the legacy Windows + `MAX_PATH` boundary under the benchmark's absolute state root. The generator + used ordinary Win32 paths and failed while constructing the cache. +- **Raw artifacts:** + `.benchmark-state/results/019fb164-afe0-7e5b-a3d2-f067c3f53a98`; + terminal report, manifest, event journal, and redacted CLI readiness/cleanup + evidence are retained. Correctness is `fail`; measured batches and planned + operation requests both equal zero. +- **Harness correction:** use the native Windows extended-length path form for + fixture construction, cache traversal/copy, failed-staging cleanup, and + local resource walks. Logical fixture paths and container-visible paths are + unchanged. +- **Verification:** a depth-100 materialization/copy regression test plus + CLI-contract and plan/API tests passed, 31 tests total. +- **Cleanup:** gateway terminated; CLI cleanup confirmed an empty isolated + registry; no campaign container or sandbox remained. The empty failed-run + workspace and invalid partial-cache parent remain owned evidence. +- **Disposition:** failed smoke retained; rerun the same frozen smoke matrix + after the host-path correction. + +### 2026-07-30T05:04:13Z - Smoke v3 retained: equivalent Windows path forms + +- **Entry ID:** `smoke-exp1-windows-path-identity-022` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb166-f100-7482-aac4-a0cdbfe05a8d`. +- **Plan:** `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** depth-100 fixture materialization and copy completed. + The first measured `create_sandbox` CLI invocation returned exit code zero, + a schema-valid response, and a ready sandbox after 59,866,856,700 ns. The + harness then rejected the response before recording a product request. +- **Cause:** the product returned the deep workspace as the native Windows + extended-length alias `\\?\C:\...`; the harness compared it to the same + directory in ordinary `C:\...` form using `Path` equality. Python preserves + those two spellings during resolution, so identical native directories + compared unequal. +- **Raw artifacts:** + `.benchmark-state/results/019fb166-f100-7482-aac4-a0cdbfe05a8d`; + the terminal report, three correctness/trial observations, event journal, + and all redacted CLI stdout/stderr/metadata are retained. Correctness is + `fail`; one trial batch and one planned operation attempt occurred, but no + request observation was admitted. +- **Harness correction:** compare resolved Windows native path identities + after normalizing case and the `\\?\`/`\\?\UNC\` aliases. Apply the same + identity rule at CLI ownership admission and create-sandbox verification. + This changes no workload, product behavior, timing boundary, or analysis + rule. +- **Cleanup:** the isolated gateway terminated and no campaign sandbox or + container remained. The unrelated pre-existing exited sandbox container was + not touched. +- **Disposition:** failed smoke retained; verify the path-identity regression + and rerun the unchanged smoke matrix. + +### 2026-07-30T05:13:57Z - Pre-freeze Windows identity and provenance verification + +- **Entry ID:** `pre-freeze-exp1-path-provenance-023` +- **Phase:** EXP1-B/EXP1-C boundary, before successful smoke, pilot, or freeze. +- **Kind:** amendment +- **Path correction verification:** native Windows extended-path fixture and + CLI tests passed, including the regression proving that `\\?\C:\...` and + `C:\...` workspace responses are admitted as the same resolved directory. +- **Provenance correction:** report derivation revision 4 now identifies + `request_latency_ns` for `product_cli` as the monotonic interval from + immediately before native CLI subprocess creation through process exit, + pipe collection, and one-line JSON schema validation. Report methods also + name the barrier-based batch boundary, the per-invocation + `cli-subprocesses` evidence directory, redaction rule, and all fixed + executable hashes. Numeric calculations did not change. +- **Count guards:** parameterized contract tests now require the complete + 19-cell matrix and exact counts: smoke 19 batches/55 requests, pilot 133/385, + and final 1,938/5,610. Every cell must select `paper-100m`. +- **Verification:** 40 focused unit and contract tests passed after the + changes. One intermediate 13-test invocation failed because the new fake + sandbox response omitted required schema fields; the fixture was completed + and the same suite then passed. One earlier command used the wrong relative + virtual-environment path and did not execute any tests. +- **Plan revalidation:** `paper-env-smoke` is runnable with 19 cells, 19 trial + batches, and 55 requests at unchanged plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Prior runs invalidated:** none were eligible; all three failed smoke runs + remain diagnostic evidence. +- **Disposition:** passed; rerun the unchanged smoke matrix. + +### 2026-07-30T05:16:57Z - Smoke v4 retained: released snapshot CLI shape + +- **Entry ID:** `smoke-exp1-snapshot-shape-024` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb171-e8b7-7b81-8eeb-d80963595865`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the depth-100 fixture copied, `create_sandbox` passed + ownership admission, and the native manager CLI returned a ready sandbox in + 54,015,576,100 ns. Initial cgroup and daemon observability CLI calls also + passed. The snapshot CLI then returned exit code zero and its released + schema, but the harness's shallow pre-parser incorrectly required a `view` + discriminator that snapshot does not define. +- **Raw artifacts:** + `.benchmark-state/results/019fb171-e8b7-7b81-8eeb-d80963595865`; + report derivation revision 4, terminal report, observations, events, and + seven redacted CLI evidence triplets are retained. The one product request + succeeded, but the trial is ineligible because required resource sampling + failed before verification. +- **Harness correction:** validate snapshot's released top-level discriminants + `sandbox_id`, `lifecycle_state`, and `availability`; the existing strict + snapshot parser remains authoritative for the complete response. +- **Cleanup:** the owned sandbox was destroyed through the manager CLI, the + isolated registry was verified empty, the gateway terminated, and no + campaign container remained. +- **Disposition:** failed smoke retained; add the released-shape regression and + rerun the unchanged smoke matrix. + +### 2026-07-30T05:18:54Z - Smoke v5 retained: portable evidence directories + +- **Entry ID:** `smoke-exp1-evidence-path-025` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb175-0d2d-71dc-bcdb-7233c8c33c3d`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the first native CLI create request succeeded in + 55,642,277,100 ns; its request observation was admitted; two complete + 100-ms-contract resource samples were persisted; and manager-CLI inspection + verified ready state and workspace identity. Persistence of the required + bounded operation evidence then failed during verification. +- **Cause:** the evidence layout used the scientific cell ID + `sha256:...` verbatim as a directory name. `:` is legal in the original + POSIX implementation but illegal in a Windows directory component. The run + therefore contains an empty `cells` evidence parent and no operation + evidence record. +- **Raw artifacts:** + `.benchmark-state/results/019fb175-0d2d-71dc-bcdb-7233c8c33c3d`; + the terminal report, 32 observations, 52 events, and 11 redacted CLI + evidence triplets are retained. +- **Harness correction:** store each evidence envelope directly under the + run-level `bounded-evidence` directory using its content digest. Evidence + schema version 2 embeds the original cell and trial IDs in the envelope; + readers continue to accept version 1's nested layout. This keeps paths below + legacy Windows length limits without weakening content addressing or opaque + download IDs. +- **Byte-integrity correction:** the first portability-test attempt exposed + that Windows CRT text mode expanded LF bytes to CRLF in every low-level + `os.open` writer. That made evidence filename digests disagree with the bytes + on disk and could alter multiline CLI content files. All benchmark-owned + low-level byte writers now explicitly use binary mode, including journals, + immutable artifacts, ownership/config/fixture files, gateway logs, and + product CLI payload files. +- **Verification:** the first hashed-nested-directory test still exceeded + Windows path limits; the first flat-layout test then exposed the CRLF + transformation. After both corrections, 43 focused artifact, CLI, fixture, + derivation, planning, and API tests passed, including byte-for-byte + multiline content. +- **Cleanup:** the owned sandbox was destroyed through the manager CLI, the + isolated registry was empty, the gateway terminated, and no campaign + container remained. +- **Disposition:** failed smoke retained; verify portable evidence indexing and + rerun the unchanged smoke matrix. + +### 2026-07-30T13:27:05Z - Smoke v6 retained: deep runtime cleanup path + +- **Entry ID:** `smoke-exp1-runtime-cleanup-026` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb335-074f-7e6d-918c-cbc31b241019`. +- **Start/end:** `2026-07-30T13:27:05.172932Z` to + `2026-07-30T13:28:44.500576Z`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the complete sandbox-lifecycle family passed. The + native manager CLI created a ready sandbox in 58,032,810,900 ns; the + request observation and 58,049,532,100 ns trial-batch latency were + admitted; two full resource samples (28 metric readings) were retained; + strict manager and observability CLI verification passed; schema-v2 + content-addressed operation evidence was written; both correctness checks + passed; and manager-CLI sandbox destruction passed. The family was marked + completed before the run failed while closing its isolated gateway. +- **Cause:** the owned gateway runtime contains the cached depth-100 shared + base, but `OwnershipLedger.remove` passed the ordinary Windows path to + `shutil.rmtree`. Cleanup reached the legacy `MAX_PATH` boundary after the + gateway had terminated and its authentication token had been deleted. +- **Raw artifacts:** + `.benchmark-state/results/019fb335-074f-7e6d-918c-cbc31b241019`; + the failed terminal report, 33 observations, 55 events, one bounded + evidence envelope, and 11 redacted CLI evidence triplets are retained. +- **Cleanup:** the manager-CLI registry check was empty, the sandbox was + destroyed, the gateway process is absent, and no campaign container + remains. The owned runtime directory and marker remain as negative cleanup + evidence; its token is absent. The unrelated pre-existing exited container + from ten days earlier was not touched. +- **Disposition:** failed smoke retained; correct only the host-native + recursive deletion spelling, verify it independently, and rerun the + unchanged smoke matrix. + +### 2026-07-30T13:35:43Z - Pre-freeze deep cleanup correction + +- **Entry ID:** `pre-freeze-exp1-runtime-cleanup-027` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** amendment +- **Harness correction:** after exact marker, active-ledger, role-root, + symlink, device-boundary, and mutable-path validation is complete, + `OwnershipLedger.remove` now gives `shutil.rmtree` the existing native + Windows extended-length spelling. Ownership authority and all scientific + protocol inputs are unchanged. +- **Regression:** a new test constructs an owned directory tree beyond 280 + characters and proves that ledger removal succeeds. The isolated regression + passed. +- **Additional verification:** the 53-test artifact, CLI, fixture, derivation, + planning, API, and cleanup selection produced 52 passes. Its sole failure is + the pre-existing frozen-golden assertion that expects 48 evidence files from + a committed fixture containing none. Running all five safety tests produced + three passes plus the two pre-existing Windows failures that require + symlink privilege; neither failure exercises the cleanup correction. +- **Plan and scientific scope:** no operation, factor, cell, payload, + repetition, timeout, exclusion, metric, timing boundary, analysis rule, or + plan file changed. Plan hash remains + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Prior runs invalidated:** none; all six failed smoke attempts remain + diagnostic evidence. +- **Disposition:** cleanup regression passed; run the unchanged EXP1-C smoke. + +### 2026-07-30T13:37:01Z - Smoke v7 retained: repeated readiness evidence ID + +- **Entry ID:** `smoke-exp1-readiness-evidence-id-028` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Preflight anomaly:** the first validation command selected the package + directory instead of its `bin` child and was rejected before a gateway or + run was created. The corrected command passed with the unchanged binary + hashes, clean product `main`, 548.53 GiB free, 19 cells, 19 batches, 55 + requests, and no plan warnings. +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb33e-21fa-7249-926a-25b57577d649`. +- **Start/end:** `2026-07-30T13:37:01.715826Z` to + `2026-07-30T13:38:28.937442Z`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the sandbox-lifecycle family again passed completely. + Its gateway then closed, emitted its retained redacted-log digest, and + removed the depth-100 runtime tree successfully. The command family reached + `preparing` but its second isolated gateway failed during CLI readiness, + before the family entered `running` or issued a planned request. +- **Cause:** every newly constructed gateway-local CLI adapter restarted its + readiness counter at zero and used + `.gateway.ready.0`. CLI evidence IDs are the SHA-256 of executable + role, operation, and request ID, so the second gateway attempted an + exclusive write to the first gateway's already-retained evidence triplet. + The predicted historical ID + `5e85d67a15b969424eeac5af7bf2150b4ffe89431400fa5c1fa427c6b858dab6` + exactly matches the first readiness metadata filename. +- **Raw artifacts:** + `.benchmark-state/results/019fb33e-21fa-7249-926a-25b57577d649`; + failed report and manifest, 33 observations, 57 events, one bounded + operation envelope, and the first family's 11 CLI evidence triplets are + retained. +- **Cleanup:** both gateways terminated, both owned runtime generations were + removed, the manager registry was empty, and no campaign sandbox, + container, process, volume, or runtime remains. +- **Disposition:** failed smoke retained; make readiness request/evidence IDs + unique per isolated gateway instance, verify, and rerun the unchanged smoke. + +### 2026-07-30T13:41:01Z - Pre-freeze gateway-scoped readiness identity + +- **Entry ID:** `pre-freeze-exp1-readiness-identity-029` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** amendment +- **Harness correction:** CLI readiness request IDs now include the + cryptographically random isolated `gateway_instance_id` before the local + attempt number. Repeated gateways in one run therefore retain distinct + request correlation and immutable CLI evidence while preserving exact + manager request IDs. +- **Verification:** the gateway-scoped identity regression, native deep-tree + cleanup regression, and complete product-CLI unit file passed, 11 tests + total. +- **Scientific scope:** readiness probes are setup-only and excluded from the + 55 planned smoke requests and all primary timing. No operation, factor, + cell, payload, repetition, timeout, exclusion, metric, timing boundary, + analysis rule, preset, or plan hash changed. +- **Prior runs invalidated:** none; the seven failed smoke attempts remain + diagnostic evidence. +- **Disposition:** passed focused verification; rerun the unchanged EXP1-C + smoke. + +### 2026-07-30T13:41:45Z - Smoke v8 retained: command line-window semantics + +- **Entry ID:** `smoke-exp1-command-line-window-030` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb342-77ef-7866-a746-f83f1a66dfb3`. +- **Start/end:** `2026-07-30T13:41:45.921519Z` to + `2026-07-30T13:45:07.847222Z`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** the lifecycle family and first command cell completed + with all checks passing. The gateway-scoped readiness correction also + crossed its prior failure boundary. In the second command cell, all five + concurrent native runtime-CLI calls returned exit code zero, passed JSON + validation, and returned `output: "4096"`. Verification nevertheless + rejected the responses and conservatively marked the cell's three + registered checks failed. +- **Cause:** the harness expected raw terminal bytes `"4096\n"`, while the + released command contract returns a line-window projection whose logical + line values omit delimiter bytes. Product tests and all five retained CLI + responses agree on `"4096"`, with `total_lines: 1`. +- **Raw artifacts:** + `.benchmark-state/results/019fb342-77ef-7866-a746-f83f1a66dfb3`; + the failed report/manifest, 111 observations (including 7 successful + planned requests and 84 resource readings), 163 events, and 40 CLI metadata + records plus stdout/stderr are retained. +- **Cleanup:** both completed gateways terminated and removed their owned + runtime trees; the registry was empty; and no campaign container, volume, + process, or runtime remains. +- **Disposition:** failed smoke retained; align the correctness oracle with + the released line-window representation, verify, and rerun the unchanged + smoke. + +### 2026-07-30T13:47:08Z - Pre-freeze command oracle correction + +- **Entry ID:** `pre-freeze-exp1-command-oracle-031` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** amendment +- **Harness correction:** the fixed `fixture_read` command remains + `wc -c < .eos-benchmark-fixture/command-read.bin`, but its correctness + oracle now expects the released line-window value `"4096"` rather than raw + terminal bytes with a trailing LF. The 64-KiB and no-output cases are + unchanged. +- **Verification:** the explicit line-window regression, command-evidence + test, gateway-scoped readiness regression, deep cleanup regression, and + full product-CLI unit file passed, 13 tests total. +- **Scientific scope:** the command, fixture, concurrency, measured boundary, + response bytes, request count, timing, checks, and operation evidence are + unchanged. This correction was made from failed-smoke raw product responses + before pilot or freeze; no performance value was accepted or excluded. +- **Plan:** no preset or plan field changed; hash remains + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Prior runs invalidated:** none; the eight failed smoke attempts remain + diagnostic evidence. +- **Disposition:** passed focused verification; rerun the unchanged EXP1-C + smoke. + +### 2026-07-30T13:47:56Z - Smoke v9 retained: leading-hyphen auth token + +- **Entry ID:** `smoke-exp1-cli-auth-token-032` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** smoke +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`, package + `windows-exp1-8c79217e`. +- **Run ID:** `019fb348-1e9f-7937-9d6c-d865ca9c0e76`. +- **Start/end:** `2026-07-30T13:47:56.222188Z` to + `2026-07-30T14:03:01.225216Z`. +- **Plan:** unchanged `paper-env-smoke`, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Observed result:** all lifecycle, command, and file cells passed: 17 of 19 + cells, 17 reportable trials, 49 planned native CLI requests, and all 42 + registered checks completed successfully. The workspace-lifecycle family + then failed while preparing its isolated gateway, before entering + `running` or issuing a workspace request. +- **Cause:** this gateway's randomly generated URL-safe auth token began with + a hyphen. The CLI adapter supplied the token as the next argv item after + `--gateway-auth-token`; the released Clap parser interpreted its `-R` + prefix as another option. All 713 attempts from that gateway returned exit + code 2 and the retained `product_error:invalid_request` classification, + after which the 60-second readiness deadline expired. +- **Raw artifacts:** + `.benchmark-state/results/019fb348-1e9f-7937-9d6c-d865ca9c0e76`; + the failed report/manifest, 741 observations (49 requests, 616 resource + readings, 42 checks), 1,045 events, 17 bounded operation envelopes, and + 1,080 redacted CLI metadata/stdout/stderr triplets are retained. No complete + credential appears in the retained error or metadata. +- **Cleanup:** all four gateway processes terminated and all owned runtime + generations were removed; registries were empty; no campaign container, + volume, process, or runtime remains. +- **Disposition:** failed smoke retained; use the parser-safe equals form for + the auth option, fail immediately on non-transient readiness product + rejection, verify, and rerun the unchanged smoke. + +### 2026-07-30T14:06:16Z - Pre-freeze auth argv and readiness correction + +- **Entry ID:** `pre-freeze-exp1-cli-auth-token-033` +- **Phase:** EXP1-C environment smoke, before pilot and freeze. +- **Kind:** amendment +- **Harness correction:** every released CLI invocation now supplies the + arbitrary URL-safe credential as + `--gateway-auth-token=`, which is unambiguous even when the + value begins with `-`. Persisted argv replaces the entire item with + `--gateway-auth-token=[REDACTED]`. +- **Failure correction:** a product-level readiness rejection now fails + gateway startup immediately. Only transport/readiness transients remain + retryable; deterministic CLI usage errors can no longer produce a + timeout-sized evidence flood. +- **Verification:** the product-CLI suite now uses a leading-hyphen token and + proves exact equals-form argv plus redaction. A new readiness regression + proves a product rejection is attempted once. Together with the + gateway-scoped ID, deep cleanup, and command-oracle regressions, 14 focused + tests passed. +- **Scientific scope:** auth and readiness are setup-only and excluded from + planned requests and primary timing. No operation, factor, cell, payload, + repetition, timeout, exclusion, metric, timing boundary, analysis rule, + preset, or plan hash changed. +- **Prior runs invalidated:** none; the nine failed smoke attempts remain + diagnostic evidence. +- **Disposition:** focused verification passed; rerun the unchanged EXP1-C + smoke after completing the read-only host audit. + +### 2026-07-30T14:26:27Z - Pre-freeze full validation and v9 host audit + +- **Entry ID:** `pre-freeze-exp1-full-validation-034` +- **Phase:** EXP1-B/EXP1-C boundary, before pilot and freeze. +- **Kind:** preflight +- **Product identity:** clean local `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`; staged package + `windows-exp1-8c79217e`. +- **v9 host/Docker audit:** retained CLI metadata contains exactly 713 failed + readiness attempts for gateway + `benchmark-gateway-06ae7ec80dd70ceb84daf362eb3ad509`, attempts 0--712 + over 60.0476619 seconds. Every call exited 2 with + `product_error:invalid_request` and stderr SHA-256 + `4517dcba7692d4b49b792eb70b02c6dfd6c9185eee3210b1362f66e1b6ac6b42`; + the parser reported `unexpected argument '-R' found`. Docker recovery and + cleanup queries completed in approximately 2--3 ms. Current inspection + found no campaign process, listener, container, volume, or runtime tree and + ample host disk/memory. Docker/host exhaustion is ruled out. +- **Real-parser probe:** the released manager CLI was invoked with a + leading-hyphen credential using attached + `--gateway-auth-token=-R-exp1-probe`; it parsed the option and reached the + deliberately unavailable endpoint, returning a connection error rather + than `invalid_request`. Bounded logs are retained in + `experiments/diagnostics/leading-hyphen-token.*.log`; no gateway or sandbox + was created and no timing value is eligible for analysis. +- **Focused backend verification:** the product-CLI/readiness/command/deep + cleanup regressions passed 14 tests. Cross-platform safety, catalog, + gateway, squash-evidence, and artifact compatibility tests then passed + 34 tests with 4 Windows WinError 1314 symlink-privilege skips. +- **Complete backend verification:** with + `PYTHONDONTWRITEBYTECODE=1`, + `.\.venv\Scripts\python.exe -m pytest .\benchmark\backend\tests -q` + exited 0: 152 passed and 4 skipped in 36.37 seconds. The skips are limited + to attempted symlink creation rejected by Windows with WinError 1314; other + fail-closed path and ownership checks passed. Test-generated tracked and + untracked bytecode was restored/removed after validation. +- **Web environment and verification:** `npm` was initially blocked by the + PowerShell script policy and `npm.cmd test` then reported the absent local + `vitest` executable. Off-clock `npm.cmd ci` with Node v24.11.1 and npm + 11.6.2 installed the lockfile-pinned dependencies and exited 0. Subsequent + `npm.cmd test` exited 0 (10 files, 43 tests) and `npm.cmd run build` exited + 0. npm reported two high-severity dependency advisories and Vite reported a + large output chunk; no automatic dependency mutation was performed, and + neither warning changes the native CLI measurement path. +- **Plan validation:** an initial preflight accidentally supplied the package + root rather than its `bin` child and failed closed because the required + executables were not located. Reissuing the same validation with + `target\windows-exp1-8c79217e\bin` made all three presets runnable with zero + validation findings, `product_cli`, and only `paper-100m`: + `paper-env-smoke` = 19 cells/19 batches/55 requests, hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + `paper-pilot` = 19/133/385, hash + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + `paper-good-pass` = 19/1,938/5,610, hash + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. +- **Artifact inventory:** `benchmark/PAPER_ARTIFACT.md` now lists the complete + paper-local EXP1 delta, exact staged package, and the three validated plan + expansions. +- **Scientific scope:** no live performance preset ran during this + verification, and no measured value, operation, factor, cell, payload, + repetition, timeout, exclusion, metric, or analysis rule changed. +- **Cleanup:** no campaign-owned runtime state or Docker object remains. The + unrelated old exited `eos-76263...` container and unrelated volumes were + left untouched. +- **Disposition:** EXP1-B deterministic validation passed; recheck exact + environment/package hashes and run the unchanged EXP1-C smoke. + +### 2026-07-30T14:30:21Z - Historical failed-runtime recovery before smoke + +- **Entry ID:** `preflight-exp1-historical-runtime-recovery-035` +- **Phase:** EXP1-C preflight, before smoke v10. +- **Kind:** preflight +- **Finding:** a recursive preflight found four owned runtime directories + retained from failed smoke runs + `019fb166-f100-7482-aac4-a0cdbfe05a8d`, + `019fb171-e8b7-7b81-8eeb-d80963595865`, + `019fb175-0d2d-71dc-bcdb-7233c8c33c3d`, and + `019fb335-074f-7e6d-918c-cbc31b241019`. They contained no live gateway + process or campaign Docker object but had survived the earlier Windows + long-path cleanup defect. This corrects the narrower prior cleanup checks, + which proved no live process/container/volume but did not recursively + enumerate historical runtime directories. +- **Recovery action:** the benchmark's scoped + `sandbox-benchmark cleanup --run-id RUN_ID` command was issued once for each + exact terminal failed run with the staged product `bin` directory. All four + commands exited 0 with `cleaned: true` and `terminalized: false`; all four + runtime directories were removed through their ownership markers. +- **Postcondition:** `sandbox-benchmark recover` exited 0 with + `execution_available: true`, no issues, and no recovered run IDs. No + campaign process, listener, container, volume, or runtime directory remains. +- **Preservation boundary:** immutable failed-run result corpora under + `.benchmark-state/results` were not changed. Failed-run workspace trees + under `.benchmark-state/runs` remain intentionally retained as required by + the runner's failed-run forensic contract and its integration regression; + they are not live gateway/runtime state and carry exact ownership markers. +- **Unrelated process:** a pre-existing debug gateway on loopback port 7878, + started before this campaign with a separate debug binary/configuration, + remains untouched. It has no campaign gateway label or Docker object and is + not bound to any reserved campaign endpoint. +- **Disposition:** historical runtime leakage safely resolved; EXP1-C smoke + preflight may proceed. + +### 2026-07-30T14:55:33Z - Smoke v10 passed and archived + +- **Entry ID:** `smoke-exp1-cli-passed-036` +- **Phase:** EXP1-C live CLI integration smoke. +- **Kind:** smoke +- **Run ID:** `019fb36f-b50f-73c3-a3b9-0d0f861cff9e`. +- **Exact command:** `.\.venv\Scripts\sandbox-benchmark.exe run + --test-repository-root + C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1 + --product-root + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox + --product-bin-dir + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-8c79217e\bin + --plan paper-env-smoke`, with `PYTHONDONTWRITEBYTECODE=1`. +- **Start/end/elapsed:** `2026-07-30T14:31:10.633046Z` to + `2026-07-30T14:47:31.197137Z`; 980.564091 seconds by artifact timestamps. + The outer command exited 0 after 1,059.3 seconds including CLI + validation/startup and terminal projection outside the manifest interval. +- **Plan:** unchanged hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + `product_cli`, `paper-100m`, 19 cells, 19 measured trial batches, 55 planned + requests, no warmups. +- **Product/environment:** clean local product `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c`; package/binary hashes match + entry 034; pinned Linux AMD64 image digest + `sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf`; + Docker Engine 29.0.1, overlayfs, cgroup v2. +- **Acceptance result:** completed with correctness `pass`. All 19 cells and + 19 measured trials are successful and reportable; all 48 registered checks + passed; all 55 planned request IDs are unique; report warnings, product + failures, correctness failures, infrastructure failures, cleanup-invalid + trials, missing primary latencies, retries, and exclusions are zero. +- **CLI evidence:** 400 unique native CLI invocations including setup, + verification, observability, readiness, planned operations, and cleanup; + all 400 exited 0, all response validations passed, total stderr bytes are + zero, and no sanitized argv item contains an unredacted auth token. +- **Resource evidence:** 686 correlated resource observations cover all trial + windows. Required daemon CPU/RSS, host free-space, workspace logical size + and file-count readings are present; sandbox/cgroup and upperdir fields are + present where the product boundary exposes them. Unavailable readings are + explicitly typed `unavailable`, never zero. The report contains the single + standing limitation that fixed-boundary unavailable resources remain + explicit, but contains no warning. +- **Raw corpus:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\.benchmark-state\results\019fb36f-b50f-73c3-a3b9-0d0f861cff9e`; + 827 observations, 1,170 events, 19 bounded operation-evidence envelopes, + and 1,200 CLI stdout/stderr/metadata files. +- **Cleanup proof:** run workspace and runtime directory are absent; no + run-labelled or campaign-gateway-labelled container or volume remains; no + run-labelled product process remains; product checkout remains clean. +- **Immutable archive:** generated once with + `experiments/scripts/archive_exp1_run.py` and independently verified again: + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb36f-b50f-73c3-a3b9-0d0f861cff9e`. + The archive contains 1,254 inventoried files and 18,597,095 inventoried + bytes; deterministic content-tree SHA-256 is + `5f1db2f535e74b434ab63196abbe947186089e4fb1f50587094e52dc711255af`. + It includes the complete raw corpus, exact manifest/plan/report, fixture + manifest, CLI help and hashes, resource-only projection, event log, cleanup + proof, environment preflight, failure/unavailability disclosure, and a + per-file hash manifest. +- **Eligibility:** smoke values are qualification evidence only and are + ineligible for manuscript tables. +- **Disposition:** passed. Gate 2/EXP1-C is accepted; run the single + exploratory `paper-pilot` next without builds, installs, pulls, source + mutation, or environment reconfiguration during its clock. + +### 2026-07-30T14:58:48Z - Smoke archive provenance finalization + +- **Entry ID:** `archive-exp1-smoke-provenance-037` +- **Phase:** EXP1-C evidence handoff before pilot. +- **Kind:** amendment +- **Reason:** post-archive contract review found that the raw benchmark run + manifest records product source/binaries, plan, fixture profile, definition + snapshot, schemas, image, and environment, but does not explicitly identify + the complete paper-local benchmark source tree. The task packet requires a + benchmark identity/hash for every archived attempt. +- **Boundary:** no raw file, metric, result, timing, report, plan, fixture, + CLI record, cleanup proof, or source file used by the smoke was changed. + The preliminary archive content-tree hash from entry 036 is preserved in + the final archive manifest as the superseded candidate hash. +- **Finalization:** the reusable archive script now creates + `benchmark-source-manifest.json` and `campaign-manifest.json`. The benchmark + inventory excludes only generated/cache directories and records all 187 + paper-local benchmark source files, 2,763,849 bytes, per-file hashes, and + content-tree SHA-256 + `22023daa06e6d5f03423eae310e1f7ab136fbbd4fce6759f7320129ead54017f`. + The campaign manifest also records the authoritative protocol files, + analysis/archiving code, paper Git context, exact plan/definition/fixture/ + product/image identities, cleanup proof, and the raw-corpus content-tree + SHA-256 + `d1d134bd80ccbe54c4f29d80c31e921fe0fa4dabac4343bef12cfdf44158ba78`. +- **Final archive:** 1,256 inventoried files, 18,639,493 inventoried bytes, + content-tree SHA-256 + `c73094f43144f32160095b8a2bedf95571d78c39ded2b2bee671745dfc8c33c3`. + A separate verify-only invocation recomputed the same file inventory, + byte count, and content-tree hash. +- **Superseded candidate hash:** + `5f1db2f535e74b434ab63196abbe947186089e4fb1f50587094e52dc711255af`. +- **Disposition:** archive contract passed. The finalized run directory is + immutable from this point; Gate 2 remains passed. + +### 2026-07-30T15:30:44Z - Exploratory pilot passed but runtime gate failed + +- **Entry ID:** `pilot-exp1-cli-runtime-gate-038` +- **Phase:** EXP1-D exploratory pilot before freeze. +- **Kind:** pilot +- **Run ID:** `019fb38a-1b44-77e6-b78a-e52f70fc49a3`. +- **Exact command:** `.\.venv\Scripts\sandbox-benchmark.exe run + --test-repository-root + C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1 + --product-root + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox + --product-bin-dir + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-8c79217e\bin + --plan paper-pilot`, with `PYTHONDONTWRITEBYTECODE=1`. +- **Preflight:** product `main` + `8c79217ee42de327f789696e61ce2b7164a4bf3c` clean; package and pinned image + unchanged; `paper-pilot` runnable at hash + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + 19 cells, 133 batches, 385 requests, `product_cli`, only `paper-100m`; + 584,023,482,368 bytes free; no campaign runtime/container/volume; finalized + smoke archive independently reverified. No build, installation, pull, + source mutation, or environment reconfiguration occurred during the clock. +- **Start/end/elapsed:** `2026-07-30T15:00:00.744923Z` to + `2026-07-30T15:24:01.547559Z`; 1,440.802636 seconds (24.013377 minutes) by + artifact timestamps. The outer command exited 0 after 1,525.9 seconds. +- **Acceptance checks:** completed with correctness `pass`. All 38 warmup + batches and 95 exploratory measured trials completed; all 95 measured + trials are reportable; all 240 measured checks passed; 385 unique planned + requests completed. Product, correctness, infrastructure, cleanup-invalid, + missing-primary-latency, warning, retry, and exclusion counts are zero. +- **CLI/resource evidence:** 2,413 unique native CLI invocations all exited 0 + and validated, with zero stderr bytes and no unredacted token. The corpus + contains 5,831 observations (385 request, 4,844 resource, 336 check, 133 + operation, 133 trial) and 7,746 events. Resource unavailability remains + explicit and the report has no warning. +- **Raw corpus:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\.benchmark-state\results\019fb38a-1b44-77e6-b78a-e52f70fc49a3`. +- **Cleanup:** run workspace and runtime directory absent; no matching product + process or run/campaign-labelled Docker container/volume; product checkout + remains clean. +- **Immutable exploratory archive:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb38a-1b44-77e6-b78a-e52f70fc49a3`; + 7,409 inventoried files, 98,249,024 inventoried bytes, content-tree SHA-256 + `28aaa47ae2bbec9c2e68b7d7cb8b4c340d83090440029c9fc12bed254b193bac`. + Independent verify-only regeneration matched. Archive-manifest SHA-256 is + `70dfe88e79d821f6f054c485929de855d3d1d006c49089accb3321ac995c7686`; + campaign-manifest SHA-256 is + `79a906a0b4084434f77478c8874c44a086c549d981c3b66a806c9c15b025e664`. +- **Runtime projection:** deterministic script + `experiments/scripts/project_exp1_final_runtime.py` (SHA-256 + `88604b889bd480f52a886781bc1ece922a09fc267703dde00476f1a702ad52b6`) + consumed only the immutable smoke/pilot archives and emitted + `experiments/analysis/pilot-final-runtime-projection.json` (SHA-256 + `1609711e8e0306982cf69f12a39586e2525847fcf2e7d58b8c35c900e925a008`). + The final fixed plan is 1,938 batches/5,610 requests. The pilot elapsed time + itself is a monotonic superset lower bound of 1,440.802636 seconds; an + optimistic affine model credits 903.857667 seconds as fixed setup and scales + only the observed 4.037180 seconds per additional batch, projecting + 8,727.912932 seconds (145.465216 minutes). Direct batch-proportional scaling + projects 20,994.552696 seconds. +- **Gate decision:** the protocol limit is 1,200 seconds. Gate 3 runtime + acceptance fails under the lower bound and both projections. Per the task + packet, freeze and `paper-good-pass` are blocked; no final run was issued. +- **Eligibility:** all pilot and projection values are exploratory capacity + evidence only and are ineligible for manuscript tables. +- **Next action:** complete read-only timing, harness, and product-root-cause + audits. Continue only if they demonstrate a semantics-preserving pre-freeze + defect correction that can plausibly satisfy the fixed 20-minute gate; + otherwise stop with the runtime gate as the exact blocker. + +### 2026-07-30T16:15:19Z - Pilot defect correction and create-path optimization + +- **Entry ID:** `pilot-exp1-cli-corrections-039`. +- **Phase:** EXP1-D pilot review and pre-freeze defect correction. +- **Kind:** amendment. +- **Trigger:** exploratory pilot + `019fb38a-1b44-77e6-b78a-e52f70fc49a3` passed correctness and cleanup but + took 1,440.802636 seconds and failed the 1,200-second final-runtime + projection gate. No final-eligible run was issued. +- **Read-only timing audit:** 25 pilot `create_sandbox` CLI invocations consumed + 910.049 seconds (36.402-second mean; 32.592-second median). Eighteen untimed + per-cell creates consumed 662.944 seconds. The seven sandbox-create trials + consumed 247.106 seconds. Pilot cell intervals decomposed to 837.005 seconds + preparation, 530.170 seconds running, and 73.628 seconds family + gateway/finalization residual. +- **Demonstrated product defect:** shared-base lookup copied and hashed the + complete 100 MiB/4,000-file workspace into a 13,619-entry temporary tree + before discovering an existing content-addressed cache entry, then deleted + that tree. The exact paper fixture cache hit took 16.281615 seconds. +- **Product correction:** clean `main` commit + `5751f196556fe837a0cb56c1c4cdba398cd38fc3` + (`perf(layerstack): reuse validated shared bases`) hashes the source and + candidate cached layer read-only, reuses only an intact exact-root match, + and fails closed without mutation on missing or corrupt cached content. The + same validation applies to ordinary and race-reuse paths. Changed content + still builds a new immutable base. +- **Product microbenchmark:** exact integrity-checked hit 1.055585 seconds + versus 16.281615 seconds before (15.424x; 93.517% reduction); cold build + 2.870520 seconds; root hash unchanged at + `93233d2d9deb96b1dc1701079a1ec01b658e3e26bdf35ff2d17cc0114acc3769`. + These are diagnostic pre-freeze values, not manuscript results. +- **Product validation:** LayerStack 46 passed/1 ignored; Docker provider 17 + passed/1 ignored; manager core 18 passed; manager library passed; affected + crate clippy targets passed with warnings denied; `git diff --check` passed. + Broad Windows manager all-target clippy remains unavailable because + pre-existing `manager_export.rs` test code unconditionally imports + `std::os::unix`. +- **Docker lifecycle audit:** direct exact Docker creates with the populated + shared volume were 0.138--0.150 seconds; 30 create/start/remove cycles stayed + at 0.128--0.209 seconds create and 0.840--0.953 seconds start; 25 isolated + full product cycles stayed at 1.212--1.759 seconds create. In contrast, 21 + later pilot-topology Docker create calls were 8.112--16.295 seconds + (12.625-second mean). No provider, Bollard, resource-setting, stats-sampler, + or shared-volume defect reproduced, so no speculative lifecycle change was + made. +- **Harness corrections:** fixed-deadline 100 ms sampling now permits one + expensive collection in flight and emits explicit unavailable records for + every saturated deadline; samples retain scheduled, collection-start, and + collection-completion offsets and persist in order at the trial boundary. + Batch journal writes use one append/fsync per record batch. Waiting and + ready-at-barrier intent is durable before release; makespan is the real + barrier-release-to-last-validated-response interval and excludes event/CLI + evidence persistence. File-write content is created and fsynced before + barrier admission and removed on every exit path. +- **Scheduling decision:** current seeded cell permutation within persisted + sequential family execution blocks is preserved and documented. The packet + does not unambiguously require repetition-level re-randomization, which + would change simultaneous prepared-sandbox load; no post-result scheduling + change was made. +- **Harness validation:** 58 focused tests passed; full backend 161 passed/4 + skipped in 36.22 seconds. Skips are only Windows WinError 1314 symlink + privilege. All plan identities and counts are unchanged: smoke + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + (19/19/55), pilot + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + (19/133/385), final + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + (19/1,938/5,610). +- **Rebuilt package:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5751f196` + and `.zip`; archive 5,607,118 bytes, SHA-256 + `5dbb8cc6f514686987ed63d29342801812201f99496c9015ab3a5384c93c9a9b`. + Gateway SHA-256 + `1da424822b129d82c4a3425ae1840a70e2e2abbc727ebb3b7c9571cb2de75ff0`; + manager + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`; + runtime + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`; + observability + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`; + daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`. +- **Failures/anomalies/unavailability:** resource collectors slower than 100 ms + cannot yield an available reading at every deadline without perturbing the + primary workload; missed ticks are now explicit rather than silently late. + Pilot-topology Docker create slowdown remains unexplained and unreproduced + in isolated diagnostics. Permission bits remain outside the inherited + shared-base root-hash schema; concurrent source mutation after validation is + an inherited TOCTOU limitation. +- **Cleanup:** all owned diagnostic gateways, containers, volumes, processes, + temporary sources, and directories were removed. The unrelated pre-existing + debug gateway PID 62980 was not touched. Product `main` is clean at the new + commit. +- **Eligibility:** audit, microbenchmark, and prior pilot values remain + exploratory and ineligible for paper tables. +- **Disposition:** pre-freeze defects corrected and package rebuilt. A strict + package preflight and new smoke are required before deciding whether one + post-correction exploratory pilot is justified. Freeze and final remain + blocked until Gate 3 demonstrates a <=20-minute final projection. + +### 2026-07-30T16:32:50Z - Post-correction product-CLI smoke + +- **Entry ID:** `smoke-exp1-cli-post-correction-040`. +- **Phase:** EXP1-C requalification after pre-freeze defect correction. +- **Kind:** smoke. +- **Run ID:** `019fb3d1-3735-7170-bf0d-3e83764096e9`. +- **Exact command:** `.\.venv\Scripts\sandbox-benchmark.exe run + --test-repository-root + C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1 + --product-root + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox + --product-bin-dir + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5751f196\bin + --plan paper-env-smoke`, with `PYTHONDONTWRITEBYTECODE=1`. +- **Preflight:** product `main` + `5751f196556fe837a0cb56c1c4cdba398cd38fc3` clean; rebuilt package and + pinned image exact; smoke plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + 19 cells, 19 batches, 55 requests, `product_cli`, only `paper-100m`; no + campaign process, runtime, container, or volume before launch. No build, + installation, pull, source mutation, or environment reconfiguration + occurred during the clock. +- **Start/end/elapsed:** `2026-07-30T16:17:40.975089Z` to + `2026-07-30T16:27:03.258882Z`; 562.283793 seconds (9.371397 minutes) by + artifact timestamps. +- **Acceptance checks:** completed with correctness `pass`. All 19 measured + trials are reportable, all 48 measured checks passed, all 55 planned + requests completed, and product, correctness, infrastructure, + cleanup-invalid, missing-primary-latency, warning, retry, and exclusion + counts are zero. +- **CLI/resource evidence:** 316 unique native CLI invocations all exited 0 + and validated, with zero stderr bytes and passed token redaction. The corpus + contains 1,317 observations: 55 request, 1,176 resource, 48 check, 19 + operation, and 19 trial records. Saturated or inapplicable resource + readings remain correlated and explicitly unavailable rather than encoded + as zero; the report has no warning. +- **Performance qualification:** the measured cold `create_sandbox` trial + took 48.114 seconds. Three additional cold family setup creates took + 46.845--47.692 seconds; the 15 cache-hit setup creates took + 2.818--2.988 seconds. The shared-base correction therefore reproduced + under the complete campaign topology. Smoke timings remain qualification + evidence only. +- **Cleanup:** run workspace and runtime directory absent; no matching + campaign product process or run/campaign-labelled Docker container/volume; + product checkout remains clean at the recorded commit. +- **Immutable smoke archive:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb3d1-3735-7170-bf0d-3e83764096e9`; + 1,004 inventoried files, 20,930,702 inventoried bytes, content-tree SHA-256 + `238803fe0c33e4c397d2d6c8e52e651b08ea308472205de80402f26ecd456fd0`. + Independent verify-only regeneration matched. Archive-manifest SHA-256 is + `0bc9ae261b9a230316213eb59d7fe6c0996663ef20d7a2b85f6b70ef35b55d0b`; + campaign-manifest SHA-256 is + `9994eef3d47300d12253079b033bf1bde5acf8566659625cb5b607d500e00a90`. + The archived benchmark-source tree contains 188 files, 16,012,064 bytes, + with content-tree SHA-256 + `620e9b74941a46373f403d0fb2b134281132d4fc797947a2f88b0eccb6074039`. +- **Eligibility:** smoke values are qualification evidence only and are + ineligible for manuscript tables. +- **Disposition:** passed. The pre-freeze correction is requalified and one + post-correction exploratory `paper-pilot` is justified. Freeze and final + remain blocked until that pilot demonstrates the fixed <=20-minute final + projection gate. + +### 2026-07-30T17:01:46Z - Post-correction pilot exposed Gate 3 blockers + +- **Entry ID:** `pilot-exp1-cli-post-correction-041`. +- **Phase:** EXP1-D exploratory pilot before freeze. +- **Kind:** pilot. +- **Run ID:** `019fb3e0-62d5-7c5d-9846-1a3b2402c0b9`. +- **Exact command:** `.\.venv\Scripts\sandbox-benchmark.exe run + --test-repository-root + C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1 + --product-root + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox + --product-bin-dir + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5751f196\bin + --plan paper-pilot`, with `PYTHONDONTWRITEBYTECODE=1`. +- **Preflight:** recovery found no run; product `main` + `5751f196556fe837a0cb56c1c4cdba398cd38fc3` clean; rebuilt package SHA-256 + `5dbb8cc6f514686987ed63d29342801812201f99496c9015ab3a5384c93c9a9b`; + pinned local image exact; `paper-pilot` runnable at + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + 19 cells, 133 batches, 385 requests, `product_cli`, only `paper-100m`; + 578,394,980,352 bytes free; no campaign process, runtime, container, or + volume. No build, installation, pull, source mutation, or environment + reconfiguration occurred during the clock. +- **Start/end/elapsed:** `2026-07-30T16:34:15.168559Z` to + `2026-07-30T16:45:44.195423Z`; 689.026864 seconds (11.483781 minutes) by + artifact timestamps. +- **Correctness and cleanup:** completed with correctness `pass`. All 38 + warmup batches and 95 exploratory measured trials completed; all 95 + measured trials are reportable; all 336 observed checks, including all 240 + measured checks, passed; all 385 planned requests completed. Product, + correctness, infrastructure, cleanup-invalid, missing-primary-latency, + warning, retry, and exclusion counts are zero. Run workspace and runtime + directory are absent; no matching product process or run/campaign-labelled + Docker container/volume remains; product checkout is unchanged and clean. +- **CLI evidence:** 1,819 unique native CLI invocations all exited 0 and + validated, with zero stderr bytes and passed token redaction. The corpus + contains 9,149 observations: 385 request, 8,162 resource, 336 check, 133 + operation, and 133 trial records. +- **Immutable exploratory archive:** + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb3e0-62d5-7c5d-9846-1a3b2402c0b9`; + 5,627 inventoried files, 111,574,583 inventoried bytes, content-tree SHA-256 + `576be9a442ff52767b8923d1f79d4cb4dc06d6bd4616a0a311f831b8cbe19fae`. + Independent verify-only regeneration matched. Archive-manifest SHA-256 is + `e5db177f138052b3c4c9532255aecf01055abc266bf5d07028fd975f3d03cb46`; + campaign-manifest SHA-256 is + `4c0d4bc79e7ff831ac81be41946b07d7c1c09b9493ccd13469531beadf7a3987`. +- **Runtime projection:** deterministic script + `experiments/scripts/project_exp1_final_runtime.py` (SHA-256 + `88604b889bd480f52a886781bc1ece922a09fc267703dde00476f1a702ad52b6`) + consumed only the matching immutable post-correction smoke/pilot archives + and emitted + `experiments/analysis/pilot-final-runtime-projection-postfix.json` + (SHA-256 + `84ced6c0e065f408f900be3401a9f5c480c88cb2a1bd44fdb1e1fdb5910a509e`). + The optimistic affine model estimates 541.159948 fixed seconds and + 1.111781 seconds per additional batch, projecting 2,695.792155 seconds + (44.929869 minutes). Direct batch-proportional scaling projects + 10,040.105733 seconds. Both exceed the fixed 1,200-second limit. +- **Resource blocker:** the 583 scheduled sample sets contain only 148 actual + collections and 435 explicit concurrency-cap misses. Of 133 trials, 119 + have only one actual collection. Consequently all 95 measured trials lack + derived daemon CPU, sandbox CPU, block-read, and block-write deltas; + workspace allocated bytes are also unavailable for all 95, and every + CPU/latency correlation has support zero. Explicit unavailability is + honest, but the packet's missing-or-uncorrelated-resource stop condition is + met. +- **Determinism blocker:** rebuilding the report twice from the immutable raw + observations produced identical canonical SHA-256 + `ad45938141e4a9dc02d1c21fd1ee3cad413e962713d50d77d1187b7f3d909b84` + and matched the archived report data exactly. However, no four-table + generator or table output exists, so EXP1-D's deterministic table + regeneration item is not yet satisfied. +- **Read-only bottleneck audit:** cell preparation is 383.886 seconds, trial + spans 210.070 seconds, cell-running gaps 17.777 seconds, and other + family/report work 77.294 seconds. Four block-first base-seeding creates + consume 185.806 seconds; the other 21 creates consume 53.740 seconds. + The old Docker API step-up did not recur: all 25 product container-create + API calls are 0.074906--0.101575 seconds. Per measured 19-cell round, + 23.258 seconds elapse while only a small minority is primary operation + time; resource collection and independent correctness/setup/cleanup dominate + the scalable remainder. +- **Gate decision:** Gate 3 fails independently on runtime projection, + resource correlation, and the absent table generator. Per the task packet, + no protocol freeze, tag, or `paper-good-pass` was issued. +- **Eligibility:** every pilot, bottleneck, resource, and projection value is + exploratory capacity/instrumentation evidence only and is ineligible for + manuscript tables. +- **Next action:** correct the demonstrated resource-window defect without + changing primary timing, add the deterministic table/provenance generator, + optimize only independent untimed verification/setup/cleanup work, rerun + deterministic validation and a qualification smoke, and issue another + exploratory pilot only if those corrections plausibly satisfy all Gate 3 + requirements. + +### 2026-07-30T17:15:11.872Z - Pre-freeze resource-window correction validation + +- **Entry ID:** `exp1-pre-freeze-resource-window-correction-20260730` +- **Phase:** EXP1-D defect correction before protocol freeze. +- **Kind:** amendment and deterministic validation; no live benchmark run. +- **Reason:** the exploratory pilot demonstrated that repeated 4,000-file + resource walks made most scheduled samples miss their deadlines, left most + trials with only one actual collection, and allowed resource collection to + extend into correctness verification. +- **Scoped correction:** immutable host-workspace tree metrics are cached per + resolved campaign path; runner RSS and host free space remain dynamically + sampled. Non-create trials now await a completed pre-barrier baseline and + stop immediately after validated product responses, serializing any active + periodic collector before exactly one completed boundary collection. The + fixed one-expensive-collector cap and explicit scheduled saturation records + remain unchanged. Because no sandbox-scoped pre-create baseline can exist, + `create_sandbox` daemon/sandbox counter deltas are explicitly inapplicable; + post-create gauges remain eligible. +- **Safe untimed parallelism:** independent-path correctness reads and + attribution checks may run concurrently only after resource sampling stops; + independent `create_workspace` session destroys run concurrently and all + complete before sandbox destruction. Setup publication remains sequential. +- **Files changed:** + `benchmark/backend/benchmark_lab/resource_sampling.py`, + `benchmark/backend/benchmark_lab/runner.py`, + `benchmark/backend/tests/unit/test_resource_sampling.py`, and + `benchmark/backend/tests/integration/test_runner.py`. +- **Protocol effect:** no cell, trial, request, exclusion, primary-latency, + barrier, makespan, throughput, retry, or reportability rule changed. + Prior exploratory values remain ineligible and are not reinterpreted. +- **Exact focused command:** `$env:PYTHONDONTWRITEBYTECODE='1'; + & .\.venv\Scripts\python.exe -m pytest + benchmark\backend\tests\unit\test_resource_sampling.py + benchmark\backend\tests\integration\test_runner.py -q` +- **Focused result:** exit `0`; `25 passed`. +- **Exact full command:** `$env:PYTHONDONTWRITEBYTECODE='1'; + & .\.venv\Scripts\python.exe -m pytest .\benchmark\backend\tests` +- **Full result:** exit `0`; `176 passed`, `5 skipped` because Windows symbolic + link privilege is unavailable. +- **Product state observed after validation:** clean `main` at + `5751f196556fe837a0cb56c1c4cdba398cd38fc3`; no product source was changed by + this correction. +- **Cleanup/result artifacts:** no gateway, sandbox, image, build, smoke, + pilot, or final preset was started; no run ID or measurement artifact was + created. +- **Disposition:** code validation passed; Gate 3 remains failed until all + independent blockers are corrected and a permitted exploratory rerun passes. +- **Unsafe interpretation:** unit/integration success does not establish live + resource availability, a passing runtime projection, or final evidence. +- **Next action:** complete the other demonstrated pre-freeze corrections, + review the combined diff, then run only the next protocol-permitted + qualification smoke/pilot sequence. + ## Entry template Copy this section for every new action. @@ -213,3 +2202,4704 @@ Copy this section for every new action. An amendment made after viewing final results must be disclosed explicitly and must not be used to selectively remove unfavorable cells. + +### 2026-07-30T17:31:12Z - Pre-freeze fixture materialization acceleration + +- **Entry ID:** `amendment-exp1-fixture-copy-042`. +- **Phase:** EXP1-D pilot review and pre-freeze defect correction. +- **Kind:** amendment and deterministic validation. +- **Trigger:** the retained post-correction pilot attributed 202.024 seconds + across 19 cells to independent fixture materialization. The existing Python + path sequentially traversed 9,618 directories and copied 4,001 files / + 104,858,177 bytes, including the versioned manifest, for every cell. +- **Read-only audit:** a disposable non-live NTFS benchmark measured the + inherited sequential copy at 9.038698 seconds, a 32-worker Python copy at + 7.415077 seconds, and native Robocopy `/MT:32` at 3.763499 seconds. Every + candidate had the same 4,001 files, byte count, and independently recomputed + content digest + `20ade0739acb398399ff944dc01ea390d346552a84a5b76465e8861634a78710`. +- **Safety decision:** hardlinks were rejected because a cell write would + mutate the shared NTFS file record; symlinks and junctions were rejected + because they retain shared-cache mutation authority; and reflink/block clone + was rejected because the qualified NTFS host exposes no supported + copy-on-write clone primitive. The implementation creates ordinary + independent files only. +- **Harness correction:** the cached fixture now receives a full + reparse/plain-entry, layout, manifest, and exact tree-hash validation before + its first process-local reuse. The retained cache key includes the canonical + path, fixture/tree hashes, raw manifest SHA-256 plus file identity, and every + file's relative path, size, modification time, and file identity. Every + materialization revalidates that metadata before and after copying and fails + closed on drift. +- **Windows copy path:** the benchmark invokes the absolute System32 + `robocopy.exe` through an argument array with no shell and options + `/E /COPY:D /DCOPY:D /R:0 /W:0 /MT:32 /XJ`; file/directory listings are + suppressed while the bounded process summary remains required. Exit codes + 0--7 are accepted; missing process output, missing/changed destination + files, source drift, timeout/execution failure, and exit codes 8 or greater + fail closed. Non-Windows hosts retain validated `shutil.copyfile` copies. +- **End-to-end disposable benchmark:** in a fresh Python process against the + exact cached `paper-100m` fixture, the first complete materialization took + 8.716363 seconds including the full source tree-hash gate; the second took + 5.355070 seconds including all per-copy validation. Fixture identity remained + `sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7` + and tree hash remained + `sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f`. + Applied to 19 materializations, this diagnostic projects 105.107617 seconds + versus the 171.735262-second sequential-copy baseline, a fixed saving of + 66.627645 seconds (38.797%). These values are pre-freeze diagnostics and are + ineligible for manuscript tables. +- **Attempts/anomalies:** the first real test used extended-length `\\?\` + roots as Robocopy arguments and failed closed with exit code 16; ordinary + absolute roots were then used because Robocopy itself handles the deep + descendants. An initial per-cell full destination rehash was correct but + took 48.663222 seconds before parallelization and then 11.046363 seconds; + it was replaced by the once-per-process source hash gate plus per-copy + metadata/content-identity checks described above. One concise PowerShell + projection command had a parser error and was immediately repeated with a + stored result collection. No live gateway, sandbox, Docker object, or + performance preset was involved in any attempt. +- **Focused verification:** fixture tests exited 0 with 15 passes and one + Windows WinError 1314 symlink-privilege skip. +- **Complete backend verification:** with + `PYTHONDONTWRITEBYTECODE=1`, + `.\.venv\Scripts\python.exe -m pytest .\benchmark\backend\tests -q` + exited 0 with 184 passes and five WinError 1314 symlink-privilege skips in + 40.36 seconds. +- **Plan stability:** all presets remain runnable with zero findings and + unchanged identities/counts: smoke + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + (19/19/55), pilot + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + (19/133/385), and final + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + (19/1,938/5,610). +- **Files changed:** `benchmark/backend/benchmark_lab/fixtures.py`, + `benchmark/backend/tests/unit/test_fixtures.py`, + `benchmark/PAPER_ARTIFACT.md`, and this append-only log. +- **Cleanup:** every disposable copy root was verified beneath the system temp + directory and removed; no temporary root remains. The shared fixture cache + was read-only throughout. +- **Disposition:** implementation and deterministic validation passed. This + removes only part of the demonstrated fixed setup cost; Gate 3 remains + blocked by the separately recorded resource-correlation and deterministic + table-generation defects and still requires a fresh qualification decision + before any further exploratory pilot. + +### 2026-07-30T17:52:41.474Z - Combined pre-freeze qualification validation + +- **Entry ID:** `exp1-pre-freeze-combined-validation-043`. +- **Phase:** EXP1-D qualification before a fresh smoke/pilot decision. +- **Kind:** amendment review and deterministic validation; no live benchmark + run. +- **Product state:** clean `main` at + `5751f196556fe837a0cb56c1c4cdba398cd38fc3`, three local commits ahead of + `origin/main`; `paper-v1-freeze` does not yet exist. +- **Run-start provenance correction:** each paper preset now captures and + validates the exact Windows host and released sandbox resource limit + authority before starting a gateway or measurement. A real capture passed + for Windows 11 version `10.0.26200`, build `26200`, host + `DESKTOP-OLP1ADS`, AMD Ryzen Threadripper 7960X 24-Cores, 48 logical + processors, 137,438,953,472 bytes of memory, and NTFS `C:\`. The selected + released configuration is + `target/windows-exp1-5751f196/config/windows-amd64.yml`, SHA-256 + `27bee7e26d1d345f08e3842536c2051328d70b89eec75b60ba4c97419d39ae8e`, + with profile `standard`, 1,000,000,000 nano-CPUs / one vCPU, + 536,870,912 memory bytes, 256 PIDs, and no create-request override. +- **Treatment identity rechecked:** source commit + `5751f196556fe837a0cb56c1c4cdba398cd38fc3`, source clean; daemon + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`, + gateway + `1da424822b129d82c4a3425ae1840a70e2e2abbc727ebb3b7c9571cb2de75ff0`, + manager CLI + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`, + runtime CLI + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`, + and observability CLI + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`. + The Ubuntu 24.04 image resolved to the required digest + `sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf`. +- **Resource semantic correction:** mutable workspace allocation is now the + signed boundary delta of product-observed live-session + `upperdir_bytes`, sourced from + `product_observability.snapshot.workspaces.disk_allocated_bytes.sum`, + scoped to the workspace. It is not inferred from the immutable host NTFS + fixture. Immutable fixture allocation remains separately recorded and may + be explicitly unavailable on Windows. Monotonic counter reset rejection is + unchanged; only a gauge delta may be signed. +- **Archival/freeze correction:** the archiver now preserves and verifies the + run-start host and limit evidence, inventories the table generator, excludes + only enumerated generated Python cache artifacts when checking the frozen + paper source scope, and requires an annotated product-repository + `paper-v1-freeze` tag whose peeled commit equals the recorded product + commit for a final archive. Existing cache artifacts were preserved. +- **Deterministic table generator:** the new archive-only generator emits all + four packet schemas, `tables.json`, a strict + `ai-research-writing/numeric-evidence-v2` registry, + `numeric-provenance.csv`, a generation log, and an output manifest. It uses + linear `(n-1)q` p50/p95/p99 quantiles and arithmetic-mean throughput. A + retained exploratory pilot generated nine byte-identical outputs in two + independent new directories; the strict registry loader recomputed all 120 + entries successfully. The output is explicitly + `exploratory_ineligible`. +- **Legacy archive boundary:** the corrected verifier intentionally rejects + retained pilot `019fb3e0-62d5-7c5d-9846-1a3b2402c0b9` with + `recorded run environment lacks final-host or sandbox-limit evidence`. + The immutable archive was not modified or reclassified. +- **Plan validation:** all three plans are runnable, use `product_cli`, + select `paper-100m`, have zero warnings/findings, and retain exact identities + and counts: smoke + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + (19 cells / 19 batches / 55 requests), pilot + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + (19 / 133 / 385), and final + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + (19 / 1,938 / 5,610). +- **Complete validation:** backend tests exited 0 with 190 passes and five + expected Windows symlink-privilege skips; analysis tests exited 0 with 12 + passes; the combined analysis/archive focused run exited 0 with 19 passes; + web unit tests exited 0 with 43 passes in ten files; the production web + build exited 0. Ruff check and format-check passed for the new generator, + its tests, and the archive script after mechanical formatting, import + ordering, use of `capture_output`, and marking an intentionally unused + unpacked report. +- **Attempts/anomalies:** an initial pytest path selected zero tests and was + corrected to the project `backend/tests` path. The host `.venv` did not have + Ruff installed, so the pinned `uv run --project benchmark --with ruff` + invocation was used. A broad Ruff command exposed pre-existing style debt + outside the newly created analysis/archive files; it was not used to rewrite + unrelated campaign code. PowerShell blocked the `npm.ps1` shim under the + host execution policy, after which identical package scripts passed via + `npm.cmd`. An earlier `sandbox-benchmark plan --help` attempt used a + nonexistent subcommand; supported `validate`, `run`, `recover`, and + `cleanup` commands were used thereafter. A request to remove disposable + generator directories was rejected by the command policy, so none was + deleted; they remain outside the frozen source and archive scopes. +- **Cleanup/result artifacts:** no gateway, sandbox, image pull, package + build, smoke, pilot, or final preset was started by this entry. Generator + outputs are disposable validation artifacts outside the immutable run + archives. +- **Disposition:** combined code and deterministic qualification validation + passed. The prior implementation blockers are corrected, but Gate 3 remains + empirically open until a fresh smoke and exploratory pilot demonstrate live + correctness, resource-boundary availability/correlation, cleanup, and a + projected final duration of at most 1,200 seconds. +- **Unsafe interpretation:** these code tests and retained-corpus regeneration + do not establish final performance or repair missing provenance in an older + archive. +- **Next action:** run strict preflight, then one fresh qualification smoke and + one fresh exploratory pilot; freeze only if every Gate 3 item passes. + +### 2026-07-30T18:15:23.025Z - Terminal workspace recovery amendment + +- **Entry ID:** `exp1-terminal-workspace-recovery-044`. +- **Phase:** EXP1-D qualification blocker correction; no live benchmark run. +- **Observed failure:** run + `019fb15d-b8b6-7126-9ecd-637fe75b8e27` had already persisted a terminal + `failed` manifest with `artifact_finalization_failed` after report + finalization, but its exact `runs` ownership marker remained after workspace + removal was interrupted. `recover` ignored every terminal manifest and + explicit `cleanup` recovered only `runtime`, so the owned disposable + workspace remained. +- **Read-only state audit:** the stranded finalization-failure workspace was + distinguished from eight existing terminal `campaign_failed` workspaces. + Every inspected workspace carried an exact run identity; no workspace was + removed during this amendment. +- **Correction:** automatic recovery now removes exact owned `runs`/`runtime` + residue for `completed`, `cancelled`, or + `failed:artifact_finalization_failed` manifests. It continues to retain + ordinary `campaign_failed` workspaces for diagnosis. Explicit cleanup may + remove the exact owned residue for any terminal run. All present roles are + adopted and validated before mutation; marker mismatch, symlink escape, + role-root escape, or device-boundary violation fails closed. Immutable + result artifacts and unrelated owned workspaces are never deletion targets. +- **Files changed:** + `benchmark/backend/benchmark_lab/recovery.py`, + `benchmark/backend/benchmark_lab/service.py`, + `benchmark/backend/tests/integration/test_recovery.py`, + `benchmark/backend/tests/contract/test_api.py`, and this append-only log. +- **Validation:** the focused recovery/service suite passed 24 tests. The full + backend suite passed 196 tests with five expected Windows + symlink-privilege skips. Regression coverage proves automatic recovery, + explicit cleanup, exact marker mismatch refusal, byte-identical preservation + of terminal manifest/report artifacts, preservation of an unrelated exact + owned workspace, and continued automatic retention of a normal failed + diagnostic workspace. Targeted Ruff fatal/import checks and `git diff + --check` passed; pre-existing broad style debt was not rewritten. +- **Disposition:** harness correction passed deterministic validation. The + stranded real workspace remains available for the root campaign controller + to remove through the now-corrected cleanup/recovery command before the next + strict preflight. + +### 2026-07-30T18:26:57.238Z - Fresh-smoke resource-boundary amendment + +- **Entry ID:** `exp1-fresh-smoke-resource-boundary-045`. +- **Phase:** EXP1-C/EXP1-D qualification blocker diagnosis and pre-freeze + instrumentation correction; no live benchmark run. +- **Retained evidence:** immutable smoke archive + `experiments/runs/019fb429-b255-70ae-91d6-5bdf4151a4c8` passed operation + correctness and cleanup but exposed incomplete resource boundaries. Across + its retained observations, `sandbox_cpu_time_ns`, + `sandbox_block_read_bytes`, and `sandbox_block_write_bytes` each had 16 + unavailable non-create observations with reason + `resource ring is not available yet`, plus two deliberately inapplicable + create-sandbox observations. `upperdir_bytes` had 32 unavailable + observations with reason + `workspace upperdir allocation was not completely reported`. The archive + was not modified or reclassified. +- **Exact diagnosis:** affected cgroup CLI responses were valid partial views + with an empty `series`; affected snapshot CLI responses already listed the + live workspace but had `workspace.resources.latest=null`. Product-source + review found two independent two-second periodic samplers: the manager + Docker cgroup ring used a hard-coded + `RESOURCE_SAMPLE_INTERVAL = Duration::from_secs(2)`, while daemon workspace + resource sampling used the default + `observability.resource_stats.sample_interval_ms: 2000`. Immediate + pre/post queries therefore observed startup races rather than legitimate + zero values. +- **Harness correction:** every reportable non-create trial now holds the + pre-operation barrier until product observability returns a complete cgroup + counter baseline and complete upperdir allocation for every live workspace. + After all measured CLI responses exit and validate, but before journal + persistence, verification, or teardown, the sampler waits for cgroup and + workspace sample timestamps strictly newer than that baseline. Polling is + bounded to five seconds at 50-millisecond intervals. Only the ready boundary + enters the resource series; discarded transient queries retain their normal + sanitized CLI subprocess metadata. Unexpected partial responses, missing + counter fields, truncation, or timeout fail closed rather than being + converted to zero. +- **Workspace-create correction:** the runner passes the planned + `workspace_count` to the sampler. Its baseline must contain no live + workspaces, and its post-response boundary is not accepted until exactly the + planned number of created workspaces have complete, post-baseline resource + samples. A newer cgroup sample alone is insufficient. +- **Timing/protocol boundary:** readiness polling is untimed setup or + post-response instrumentation. It does not move the native CLI subprocess + launch-to-validated-response primary clock, alter the operation barrier, + retry an operation, change a cell, change a trial count, or change a metric + definition. Periodic 100-millisecond benchmark sampling remains unchanged. +- **Product cadence correction observed:** product `main` is clean at + `31b1031f2ddc9fc4c702bdc15b7c2743288832eb` and passes the configured + 250-millisecond resource cadence to the manager sampler. Package + `target/windows-exp1-31b1031f` uses configuration SHA-256 + `914d68807d97124a8412963d6f17e7a3914514bb0f0d210ea101ae52ba0ef512`, + gateway SHA-256 + `cb36cc7c1ffba5f97a8216e30d6fb8213d04581c100bd097739981fe4bc0b463`, + unchanged manager/runtime/observability CLI SHA-256 values + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`, + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`, + and + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`, + unchanged daemon SHA-256 + `a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22`, + and archive SHA-256 + `6451e2b72e92a99d5f359ab3d3ff247a0a6aeed80011c99cfb9fd320dfd739ca`. +- **Validation:** focused resource/runner tests exited 0 with 31 passes. The + complete backend suite exited 0 with 201 passes and five expected Windows + symlink-privilege skips in 46.60 seconds. Targeted Ruff import sorting + passed; a targeted fatal/style run passed while ignoring only the already + documented broad-file debt `BLE001`, `F821`, `F401`, and `S110`. + `git diff --check` reported no content errors and only the repository's + existing line-ending warnings. +- **Plan stability:** validation against + `target/windows-exp1-31b1031f/bin` exited 0 for all presets with zero + findings. Identities and counts are unchanged: smoke + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + (19/19/55), pilot + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + (19/133/385), and final + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + (19/1,938/5,610). +- **Attempts/anomalies:** the first package hash query used two incorrect + daemon paths under `bin` and emitted read-only `Resolve-Path` errors. A + recursive filename inspection found the packaged daemon at + `dist/sandbox-daemon-linux-amd64`; its hash matched the prior treatment. + No file was changed by those failed queries. +- **Files changed:** + `benchmark/backend/benchmark_lab/resource_sampling.py`, + `benchmark/backend/benchmark_lab/runner.py`, + `benchmark/backend/tests/unit/test_resource_sampling.py`, + `benchmark/backend/tests/integration/test_runner.py`, and this append-only + log. +- **Disposition:** deterministic correction and validation passed. The prior + code-level resource-boundary blocker is cleared; Gate 3 remains empirically + open until a fresh smoke demonstrates complete first/last cgroup and + workspace boundaries with the 250-millisecond treatment, followed by a + qualifying exploratory pilot. + +### 2026-07-30T18:28:34.154Z - Qualification smoke invalidation and resource-cadence product amendment + +- **Entry ID:** `exp1-smoke-invalidation-product-cadence-046`. +- **Phase:** EXP1-C requalification and pre-freeze product amendment. +- **Invalidated smoke:** run `019fb429-b255-70ae-91d6-5bdf4151a4c8` + used clean product `main` at + `5751f196556fe837a0cb56c1c4cdba398cd38fc3` and package + `target/windows-exp1-5751f196`. It ran `paper-env-smoke` through + `product_cli` at unchanged plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + with 19 cells, 19 measured trials, and 55 requests. +- **Start/end/elapsed:** `2026-07-30T17:54:22.157355Z` to + `2026-07-30T18:02:16.748616Z`; 474.591261 seconds by artifact + timestamps. Correctness passed, all 19 trials were reportable, and the + report recorded zero warnings. +- **Resource blocker:** non-create cgroup CPU/I/O counters and workspace + upperdir allocation were not fully available at the trial boundaries. + These missing readings are preserved as unavailable, never as zero. The + exact diagnosis and harness correction are recorded in Entry 045. +- **Observer and cleanup anomalies:** the initial shell invocation timed out + while the benchmark CLI continued detached. During setup, a recursive + read-only filesystem listing traversed the generated fixture tree. After + `report_ready`, two exact wrapper processes were terminated before the + ownership ledger had finished its deep workspace removal, interrupting + normal cleanup. The exact marker-validated run workspace was subsequently + removed through `OwnershipLedger.adopt/remove`; the cleanup archive then + verified. These actions do not change primary per-operation clocks, but + they make this qualification attempt unsuitable for pairing with a pilot. +- **Immutable archive:** the retained smoke archive at + `experiments/runs/019fb429-b255-70ae-91d6-5bdf4151a4c8` contains 1,211 + inventoried files and 18,638,999 inventoried bytes with content-tree + SHA-256 + `sha256:16ba946fa5bf17163e15dae535a9206348e78b7a3a6e5e65769e8370632c0060`. + Its disposition remains `smoke`; it was not modified. +- **Analysis attempts:** the four-table generator correctly rejected + `smoke` as an unsupported manuscript disposition. A runtime projection + using this smoke and retained older pilot + `019fb3e0-62d5-7c5d-9846-1a3b2402c0b9` correctly rejected the pair because + their identities drifted. No result was promoted. +- **Disposition:** failed as a resource-qualification gate and explicitly + invalidated for smoke/pilot projection because the product treatment was + subsequently amended. It is diagnostic evidence only. +- **Product correction:** product commit + `63cfe81c8` adds explicit Windows resource sampling configuration at 250 ms. + Audit then found the manager Docker cgroup ring still hard-coded to two + seconds. Commit + `31b1031f2ddc9fc4c702bdc15b7c2743288832eb` loads the validated + observability configuration in the gateway and passes its exact cadence to + the manager sampler. No API, benchmark cell, primary clock, fixture, image, + or trial count changed. +- **Product validation attempts:** `cargo test -p sandbox-config` passed 78 + tests. Relevant manager core tests passed 18 tests and manager-router tests + passed 20 tests; one unrelated Unix export test was filtered on Windows. + `cargo check -p sandbox-gateway --bin sandbox-gateway`, manager-library and + gateway-binary Clippy with `-D warnings`, formatting, and diff checks all + passed. An initial broad Windows test attempt exposed existing ungated + Unix-only tests, a pre-existing debug gateway holding a debug executable, + and parallel Cargo target-lock contention. The relevant checks were + repeated successfully in isolated target + `target/exp1-cadence-check`; no unrelated source was rewritten. +- **New staged treatment:** package + `target/windows-exp1-31b1031f` and its ZIP are the only candidates for the + next qualification. ZIP SHA-256 is + `6451e2b72e92a99d5f359ab3d3ff247a0a6aeed80011c99cfb9fd320dfd739ca`; + gateway SHA-256 is + `cb36cc7c1ffba5f97a8216e30d6fb8213d04581c100bd097739981fe4bc0b463`; + configuration SHA-256 is + `914d68807d97124a8412963d6f17e7a3914514bb0f0d210ea101ae52ba0ef512`. + Manager, runtime, observability, and daemon hashes are recorded in Entry + 045. Product `main` is clean at the exact new commit. +- **Next action:** strict preflight and one fresh smoke under the corrected + harness and `31b1031f` package. No pilot is allowed unless every non-create + resource endpoint supplies complete measured boundaries and cleanup is + clean. + +### 2026-07-30T18:31:52.793Z - Corrected-treatment fresh-smoke preflight + +- **Entry ID:** `exp1-corrected-treatment-smoke-preflight-047`. +- **Phase/kind:** EXP1-C requalification; strict preflight only. +- **Harness validation:** an independent full backend run passed 201 tests + with five expected Windows symlink-privilege skips in 51.07 seconds. + Analysis tests passed 12 tests in 1.07 seconds. Both runs set + `PYTHONDONTWRITEBYTECODE=1` and disabled pytest's cache provider. +- **Recovery:** `sandbox-benchmark recover` against the staged package exited + 0 with `execution_available=true`, no issues, and no recovered run IDs. +- **Plan validation:** all presets are runnable through `product_cli`, use + only `paper-100m`, and have zero warnings/findings. Smoke is + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + at 19/19/55; pilot is + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + at 19/133/385; final is + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + at 19/1,938/5,610. +- **Product/package:** product `main` is clean at + `31b1031f2ddc9fc4c702bdc15b7c2743288832eb`. Staged ZIP size is + 5,616,849 bytes and SHA-256 is + `6451e2b72e92a99d5f359ab3d3ff247a0a6aeed80011c99cfb9fd320dfd739ca`. + The gateway, manager, runtime, observability, daemon, and configuration + hashes exactly match Entry 045. +- **Docker/image/disk:** Docker server 29.0.1 reports Linux/AMD64. Local image + `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` + independently inspects to the same image ID and digest on AMD64 Linux. + Drive C has 575,893,774,336 free bytes. +- **Unrelated retained state:** the previously disclosed debug gateway PID + 62980, ten-day-old exited `eos-76263ba1-...` container, its two volumes, + and three shared-base cache volumes remain untouched. None carries a + benchmark run/campaign label for this qualification, and no benchmark CLI + process or corrected-package gateway was active. +- **Disposition:** passed. No build, install, pull, live preset, or final + command occurred. One corrected-treatment `paper-env-smoke` is authorized + next; the pilot and freeze remain blocked on its empirical result. + +### 2026-07-30T18:58:34.866Z - Corrected-treatment smoke exposed missing workspace-disk producer + +- **Entry ID:** `exp1-workspace-disk-producer-smoke-failure-048`. +- **Phase/kind:** EXP1-C requalification; failed smoke. +- **Run/command:** run `019fb44c-887b-79ad-9ae0-5a5c0852696a` issued + `.\.venv\Scripts\sandbox-benchmark.exe run --test-repository-root + C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1 + --product-root + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox + --product-bin-dir + C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-31b1031f\bin + --plan paper-env-smoke` with `PYTHONDONTWRITEBYTECODE=1`. +- **At-run identity:** product was clean `main` at + `31b1031f2ddc9fc4c702bdc15b7c2743288832eb`; package, binary, + configuration, image, host, limits, fixture, and plan identities matched + Entry 047. No build, install, pull, source mutation, or environment + reconfiguration occurred during the run clock. +- **Start/end/elapsed:** `2026-07-30T18:32:25.762441Z` to + `2026-07-30T18:34:52.947894Z`; 147.185453 seconds. +- **Outcome:** terminal state `failed`, correctness `fail`, one + infrastructure failure, zero warnings. The `create_sandbox` cell completed + one reportable measured trial with both checks passed and cleanup restored. + The first `exec_command` trial failed during mandatory pre-operation + resource setup before any operation request was issued; it has null primary + latency, request count zero, and cleanup restored. The campaign stopped + immediately. No pilot or final command was issued. +- **Exact empirical diagnosis:** all 50 cgroup boundary polls were available, + included required CPU, memory, read, and write counters, and advanced at + approximately 250 ms. All 50 snapshots were also available and contained + the same live workspace, but + `workspaces[0].resources.latest` remained null throughout the five-second + readiness window. The reconstructed terminal cause is + `product resource boundary readiness timed out: workspace + 00000118c72503411e4c9b resource sample is not available yet`. +- **Source diagnosis:** the runtime exposes each live workspace ID and + upperdir, the bounded `sample_upperdir` collector exists, and the public + snapshot consumer expects workspace-scoped samples. Production never calls + that collector or emits workspace-scoped disk samples; additionally, + snapshot lookup reads the event reader rather than the dedicated resource + reader. The manager cgroup cadence correction is valid; this is a separate + missing producer/handoff bug. +- **Cleanup:** the benchmark destroyed the exact workspace session and + sandbox; product list calls returned no sandboxes. Explicit terminal + cleanup then removed the marker-validated retained diagnostic workspace in + 21.7 seconds. Runtime and run-workspace paths are absent, and no matching + process, container, or volume remains. +- **Immutable failed archive:** the first archive attempt correctly failed + under the then-existing completed-only rule. After the preservation + amendment in Entry 049, the run archived as explicitly + `failed_ineligible` at + `experiments/runs/019fb44c-887b-79ad-9ae0-5a5c0852696a`: 389 files, + 4,899,939 bytes, content-tree SHA-256 + `sha256:03467746779e381aee6bb58764caa0650a2df262940234d6d8269efcc0a9900e`. + A separate verify-only invocation matched exactly. +- **Disposition:** failed diagnostic evidence only. Gate 3 remains closed. + The strongest pre-freeze correction is to wire bounded workspace-disk + sampling into the daemon's owned cadence and resource reader, rebuild the + treatment, and run a new smoke. A harness fallback that silently accepts an + all-unavailable storage column was rejected because the intended public + producer and consumer already exist. + +### 2026-07-30T18:58:34.866Z - Failed-corpus archival preservation amendment + +- **Entry ID:** `exp1-failed-corpus-archive-amendment-049`. +- **Phase/kind:** EXP1-C evidence-preservation amendment; no live run. +- **Correction:** `archive_exp1_run.py` now accepts an explicit terminal + `--run-status failed` for smoke, exploratory pilot, or final attempts and + assigns immutable eligibility `failed_ineligible`. It preserves partial + observations, negative CLI evidence, warnings, raw report/events, and + terminal failure data without weakening completed-run validation. +- **Fail-closed boundary:** verifier cross-checks run status, correctness, + report copies, manifest eligibility, cleanup proof, at-run treatment, and + post-run checkout provenance. Manifest editing cannot promote a failed + archive. A failed final attempt still requires the annotated freeze tag + against its recorded treatment. Completed and final-candidate archives + still require a clean checkout exactly matching the recorded treatment. +- **Post-run drift disclosure:** a failed archive may record a dirty or + advanced product checkout caused by the next corrective amendment. The + clean at-run treatment from the run manifest, plus exact staged package and + binary hashes, remains authoritative. Process, container, volume, runtime, + run-workspace, branch, package, and binary checks remain strict. Failed-run + benchmark-source provenance explicitly says the archived preservation code + was amended after the run and is not byte-identical run-time source. +- **Files changed:** `experiments/scripts/archive_exp1_run.py`, + `benchmark/backend/tests/unit/test_exp1_archive.py`, and this log. +- **Validation/attempts:** the completed-only real archive attempt failed + with `only completed EXP1 runs can use this archive path`, motivating the + amendment. Focused archive tests passed 13 tests; independent root rerun + passed the same 13. Initial direct `ruff.exe` and + `uv run --frozen ruff` attempts found no installed Ruff; `uvx ruff format + --check` and `uvx ruff check` then passed. +- **Disposition:** passed. The failed smoke corpus is preserved, verified, and + permanently ineligible for manuscript numbers. + +### 2026-07-30T19:23:41.459Z - Bounded create-sandbox second-pass optimization audit + +- **Entry ID:** `exp1-create-sandbox-second-pass-audit-052`. +- **Phase/kind:** EXP1-C pre-freeze diagnostic and optimization audit; no + smoke, pilot, or final preset was issued. +- **Scope:** inspect the complete `create_sandbox` path after the validated + shared-base and native-Windows fixture-copy optimizations, profile only + isolated diagnostic operations, and retain the exact 19-cell protocol, + fixture bytes/tree/hash, native `product_cli` semantics, correctness, + cleanup, timing, and evidence boundaries. +- **Historical evidence inspected:** failed smoke + `019fb44c-887b-79ad-9ae0-5a5c0852696a` recorded one reportable + `create_sandbox` request at 47,812,097,100 ns. Its trial-level `setup_ns` + was 17,300 ns because the independent `paper-100m` materialization is + cell-scoped and outside every primary operation clock. These retained + values remain failed-smoke diagnostic evidence and are ineligible for + manuscript tables. +- **Isolated phase diagnostic:** a unique owned gateway and workspace used + clean product `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`, package + `target/windows-exp1-bc1e6ee0`, the fixed Ubuntu digest, manager CLI + argument arrays, explicit request IDs, and CLI progress timestamps. The + ordinary `paper-100m` materialization took 8.4055315 seconds. The first + create took 47.0135836 seconds: shared-base construction completed at + 3.813 seconds, the first Docker runtime sandbox completed at 45.768 + seconds, and daemon installation/start/readiness plus state publication + completed at 46.999 seconds. After manager-CLI destruction, the same + workspace and gateway produced a second create in 1.5523349 seconds: + validated shared-base reuse completed at 0.353 seconds, runtime sandbox + creation at 0.482 seconds, and readiness/state publication at 1.538 + seconds. The two destroys took 0.9366223 and 1.0019202 seconds. +- **Interpretation boundary:** approximately 41.955 seconds of the cold + create was Docker Desktop's first runtime-container creation for this + shared-base mount, not fixture copying, base hashing, CLI launch, or daemon + readiness. The unchanged pilot and final protocols already execute two + warmup trials before measured trials in every cell. Moving Docker + prewarming into setup, trusting an unvalidated cache, changing the + `create_sandbox` cell, or excluding a cold measured value would alter the + benchmark treatment or evidence boundary and was rejected. +- **Copy-engine matrix:** two independent ordinary-copy samples at each + Robocopy worker count included copy plus all source-drift, destination-file, + and manifest validation. Median seconds were `/MT:32` 8.34082875, + `/MT:16` 10.2005869, `/MT:64` 11.7643670, and `/MT:128` 12.72309195. + `/MT:32` remains the selected bounded setting. Its two samples + (4.5323453 and 12.1493122 seconds) also show that this host path is noisy; + no diagnostic timing is eligible for a paper table. +- **Component profile:** the first process-local full cache validation took + 4.0275112 seconds and a stable-identity validation took 0.5660631 seconds. + Two direct native copies took 3.5675212 and 12.2348150 seconds; their + post-copy source walks took 0.5800985 and 0.5829757 seconds, destination + verification took 0.4260228 and 0.4330685 seconds, and manifest-byte + verification took 0.0089029 and 0.0014380 seconds. Owned diagnostic-tree + cleanup took 8.6437489 and 9.5096523 seconds and was excluded from every + copy timing. +- **Validation-overlap check:** four alternating read-only samples gave a + 1.06716655-second median for sequential post-copy source-drift and + destination verification and a 1.10508360-second median when overlapped. + Parallel overlap was rejected because it was slower and added executor + contention without strengthening correctness. +- **Focused validation:** with `PYTHONDONTWRITEBYTECODE=1` and pytest cache + disabled, + `python -m pytest benchmark/backend/tests/unit/test_fixtures.py` exited 0: + 15 tests passed and one Windows symlink-privilege case skipped in 10.07 + seconds. Repeated full fixture validation retained fixture hash + `sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7` + and tree hash + `sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f`. +- **Cleanup:** both diagnostic sandboxes were destroyed through the manager + CLI. The isolated gateway, runtime/shared-base cache, marker-owned run + workspace, and every copy-profile destination were removed; the final + marker-owned run-tree removal took 18.4590787 seconds. No unrelated + process, container, volume, cache, or generated artifact was deleted. + Read-only terminal checks found the exact diagnostic run/runtime paths + absent, zero remaining probe directories, no matching Docker object, and a + clean product worktree on `main` at the recorded commit. +- **Diff validation:** a scoped `git diff --check` exited 0 with only the + repository's existing Windows line-ending warnings. One read-only timestamp + query initially used the unsupported PowerShell `Get-Date -AsUTC` option; + it changed no state, and the portable `ToUniversalTime()` query succeeded. +- **Files changed:** this append-only experiment log only. +- **Disposition:** no additional safe speedup was demonstrated, so no + benchmark or product source was changed. The existing fixture-copy and + shared-base optimizations remain the fastest validated, protocol-preserving + treatment. Gate 3 remains dependent on the separate fresh resource-complete + smoke and exploratory pilot, not on this diagnostic. + +### 2026-07-31T00:28:06.880Z - Workspace-resource producer corrective treatment and package + +- **Entry ID:** `exp1-workspace-resource-producer-treatment-053`. +- **Phase/kind:** EXP1-C pre-freeze product correction, build, and + qualification; no smoke, pilot, or final preset was issued. +- **Demonstrated defect:** failed smoke + `019fb44c-887b-79ad-9ae0-5a5c0852696a` proved that every live workspace + snapshot exposed `resources.latest: null` because production never invoked + the existing bounded upperdir sampler and the snapshot path did not use the + dedicated resource reader. Entry 048 preserves the exact failure. +- **Corrective treatment:** clean product `main` advanced to + `bc1e6ee04d4df5541290537994a4bf270fcd36b6` + (`fix(observability): sample workspace disk usage`). The daemon resource + sampler now owns one configured blocking cadence, samples each live + workspace upperdir through the existing bounded collector, and emits only + strict-parser-safe logical bytes, allocated bytes, file count, and + truncation fields. Snapshot queries receive the dedicated resource reader; + event and layer readers remain unchanged. Read or budget failure omits the + allocated value and records truncation rather than fabricating zero. +- **Product validation:** isolated WSL target + `/tmp/ephemeral-sandbox-resource-target` passed daemon/query `cargo check`, + all 15 observability-query tests, four focused workspace-resource sampler + tests, one daemon-to-snapshot resource-store test, Clippy with + `--no-deps -D warnings`, `cargo fmt --all --check`, and `git diff --check`. + The full daemon unit binary passed 105 of 106 tests; the sole failure was + unrelated `runner_tests::result_fd_writer_writes_to_fd_peer`, which WSL + rejected with `ENXIO` while opening the result FD. A native-Windows query + test attempt could not compile its Unix-only `rustix::fs` test dependency; + it did not change source. +- **Linux daemon and package build:** the release packaging command + `target\debug\xtask.exe package --builder zigbuild --target + x86_64-unknown-linux-musl` completed in 31 seconds and produced + `dist\sandbox-daemon-linux-amd64` with SHA-256 + `a620d6016fb23c0da82f3f913ac411850d9125dea342886fcbe1edca1ee301b6`. + The first direct PowerShell packaging attempt was blocked by the host + execution policy and misleadingly returned the enclosing shell's exit 0; + the required retry used `powershell.exe -NoProfile -ExecutionPolicy Bypass + -File .\bin\package-windows-amd64-release.ps1 -PackageName + windows-exp1-bc1e6ee0 -OutDir target -Profile release` and succeeded. +- **Candidate package:** `target\windows-exp1-bc1e6ee0` and + `target\windows-exp1-bc1e6ee0.zip`; archive size 5,658,062 bytes and + SHA-256 + `f1bede6d96bdf7907c898c11c6c39865824888086ea6ae9e72518c93fef51240`. + Executable SHA-256 values are gateway + `cb36cc7c1ffba5f97a8216e30d6fb8213d04581c100bd097739981fe4bc0b463`, + manager + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`, + runtime + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`, + and observability + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`. +- **Current read-only recheck:** product is clean `main`, six commits ahead of + `origin/main`, at the recorded commit. Docker client/server remain 29.0.1, + Linux AMD64, overlayfs, and cgroup v2. The fixed package and archive hashes + match. The pre-existing unrelated debug gateway PID 62980 remains untouched. +- **Disposition:** the corrected treatment is technically qualified but not + yet Gate-3 qualified. A one-cell isolated CLI diagnostic must first prove + available, fresh workspace-disk boundaries; then a completely fresh smoke + and exploratory pilot are required. + +### 2026-07-31T00:35:18.777Z - Resource-producer diagnostic exposed strict snapshot-parser gap + +- **Entry ID:** `exp1-workspace-resource-probe-failure-054`. +- **Phase/kind:** EXP1-C pre-freeze diagnostic failure and cleanup; not a + smoke, pilot, or final attempt. +- **Plan validation:** the first temporary diagnostic plan incorrectly wrapped + a preset-shaped default plan and failed closed with Pydantic's + `invalid benchmark default plan`; no run was issued. The corrected raw + experiment plan validated with no warnings as one `product_cli` + `exec_command` cell, one measured batch, one issued request, and plan hash + `sha256:d8d9071f9c5888378c62969e075cd831810d18b6ca7de0df154b28eff05800bf`. +- **Run/command:** run `019fb592-a95f-7dd4-9ce3-8791b991fae2` issued + `.\\.venv\\Scripts\\sandbox-benchmark.exe run --test-repository-root + C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\research-papers\\ephemeral-sandbox-v1 + --product-root + C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox + --product-bin-dir + C:\\Users\\yifan\\code\\Ephemeral-AI-Lab\\ephemeral-sandbox\\target\\windows-exp1-bc1e6ee0\\bin + --plan .\\tmp\\exp1-workspace-resource-probe.yml` with + `PYTHONDONTWRITEBYTECODE=1`. +- **At-run identity:** clean product `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`, corrected package and binary + hashes from Entry 053, fixed image digest, and validated custom-plan hash. + No build, install, pull, source mutation, or environment reconfiguration + occurred during the clock. +- **Start/end/elapsed:** `2026-07-31T00:28:35.833090Z` to + `2026-07-31T00:29:51.145840Z`; the supervised process completed in 81.1 + wall-clock seconds. The CLI process exited 0 because it returned the + persisted terminal run status; the run manifest correctly records state + `failed`, correctness `fail`, and infrastructure failure. +- **Outcome:** the single measured trial failed in mandatory resource setup + before the operation barrier. It issued zero operation requests, recorded + null primary latency, restored trial cleanup, and produced no reportable + sample. No smoke, pilot, or final command was issued. +- **Positive producer evidence:** the first snapshot now exposed the exact + active workspace and a non-null fresh resource record at Unix millisecond + `1785457778639`, with `disk_bytes: 0`, + `disk_allocated_bytes: 4096`, `files: 0`, and + `disk_truncated: false`. The concurrent cgroup boundary was available with + CPU, memory, read, and write counters. This directly proves the Entry-053 + workspace producer and dedicated resource-reader handoff are live. +- **Exact failure:** the paper's strict `SnapshotMetrics`/`SnapshotDeltas` + model was written for the previously hidden workspace-oriented subset. The + now-correct sandbox resource sample also contains the authoritative daemon + fields `metrics_source`, `cgroup_path`, `io_rbytes`, `io_wbytes`, + `pids_cur`, plus read/write deltas. Local replay of the archived response + raised seven `extra_forbidden` Pydantic errors and then + `product snapshot response schema is invalid`. The released Rust sampler + and snapshot serializer explicitly emit these fields; this is a paper-side + strict-schema coverage defect, not missing product data. +- **Evidence preservation:** the exact 35-file, 256,561-byte result corpus is + copied to + `experiments/diagnostics/exp1-workspace-resource-probe-019fb592-a95f-7dd4-9ce3-8791b991fae2` + with content-tree SHA-256 + `sha256:4ef738fd244b1c8fb47ef08c878345cd8b1e1fb846f1d93b208e8fcc00759628`. + It is diagnostic-only and permanently ineligible for paper tables. +- **Cleanup:** explicit benchmark cleanup exited 0 with `cleaned: true`. + The exact run workspace and runtime entry are absent; the exact sandbox has + no matching container or volume; the isolated gateway is gone; and the + product remains clean. The pre-existing unrelated debug gateway PID 62980 + remains untouched. +- **Disposition:** failed diagnostic retained. Gate 3 stays closed. Add exact + strict models for the authoritative sandbox resource sample and workspace + disk sample, retain rejection of unknown keys, validate the archived shape, + then rerun a new one-cell diagnostic before any smoke or pilot. + +### 2026-07-31T00:39:39.934Z - Context-strict snapshot resource parser correction + +- **Entry ID:** `exp1-snapshot-resource-schema-amendment-055`. +- **Phase/kind:** EXP1-C pre-freeze harness correction and validation; no live + preset. +- **Correction:** `observability.py` now models snapshot sandbox resources and + workspace resources as distinct strict types. Sandbox samples require + `metrics_source: sandbox_cgroup`, cgroup identity/availability, and accept + only the released CPU, memory, I/O, PID, truncation, and counter-delta + fields. Workspace samples require the disk-truncation identity and accept + only logical bytes, allocated bytes, file count, truncation, and empty + deltas. +- **Fail-closed boundary:** a root disk metric cannot masquerade as a sandbox + cgroup sample; cgroup metrics or deltas cannot appear in a workspace disk + sample; unknown future keys, mixed-scope shapes, wrong metric sources, and + missing scope identities remain schema errors. +- **Regression evidence:** the exact field values preserved from failed + diagnostic `019fb592-a95f-7dd4-9ce3-8791b991fae2` parse as + `sandbox_cgroup`, sandbox read bytes `98304`, and workspace allocated bytes + `4096`. Negative tests cover unknown keys, scope mixing, wrong source, + missing cgroup identity, and missing workspace disk identity. +- **Validation:** the delegated exact schema suite passed 16 tests; its + resource-sampling plus runner suite passed 32; and its full backend suite + passed 220 tests with five expected Windows symlink-privilege skips. An + independent root focused run passed all 48 tests in 4.66 seconds, and an + independent full backend run passed 220 with the same five expected skips + in 35.23 seconds. Every command used `PYTHONDONTWRITEBYTECODE=1` and disabled + the pytest cache provider. +- **Files changed:** `benchmark/backend/benchmark_lab/observability.py`, + `benchmark/backend/tests/unit/test_observability.py`, and this append-only + log. +- **Disposition:** passed. The false-negative parser defect is resolved + without weakening scope validation. A new isolated one-cell diagnostic must + still prove both mandatory boundaries and cleanup before fresh smoke. + +### 2026-07-31T00:43:07.977Z - Workspace-resource two-boundary diagnostic passed + +- **Entry ID:** `exp1-workspace-resource-probe-pass-056`. +- **Phase/kind:** EXP1-C pre-freeze positive diagnostic; not a smoke, pilot, + or final attempt. +- **Run/identity:** run `019fb59d-2877-700e-8299-bf3042d2da65` used clean + product `main` at `bc1e6ee04d4df5541290537994a4bf270fcd36b6`, + package `target\windows-exp1-bc1e6ee0`, the exact Entry-053 binaries, fixed + image digest, `product_cli`, and the validated one-cell plan hash + `sha256:d8d9071f9c5888378c62969e075cd831810d18b6ca7de0df154b28eff05800bf`. +- **Start/end:** `2026-07-31T00:40:03.751900Z` to + `2026-07-31T00:41:19.011354Z`; the supervised process completed in 82.7 + wall-clock seconds. No build, install, pull, source mutation, or environment + reconfiguration occurred during the clock. +- **Terminal outcome:** state `completed`, correctness `pass`, one measured + batch, one issued operation request, zero failures, zero warnings, and one + reportable trial. The runtime CLI request exited 0 with empty stderr and + passed response validation. All three command checks passed and trial + cleanup restored the baseline. +- **Mandatory boundaries:** the raw corpus contains exactly two persisted + readings for every resource metric. Both `upperdir_bytes` readings are + available at 4,096 allocated bytes. Baseline/post sandbox CPU counters are + 56,616,000/116,673,000 ns; block-read counters are 98,304/3,842,048 bytes; + block-write counters are 16,384/16,384 bytes. The accepted snapshot + workspace timestamps advance from `1785458464483` to `1785458464984`; + accepted cgroup timestamps advance from `1785458464445` to + `1785458464956`. Two earlier post-response polls were correctly rejected as + not yet fresh. +- **Expected explicit unavailability:** the immutable host fixture's allocated + byte count is unavailable because native Windows metadata lacks allocated + block counts; LayerStack total allocated storage is unavailable because the + product did not report it. Neither substitutes for the now-complete live + workspace upperdir metric, and neither was fabricated as zero. +- **Preservation:** the exact 63-file, 442,066-byte result corpus is copied + byte-for-byte to `experiments/diagnostics/resource-probe-019fb59d`. + Source and copy inventories match at content-tree SHA-256 + `sha256:146a883858b8f8f3245d451d765f063a16e24c57c4271fdb23a51cfe1f80f82a`. + An initial ordinary copy missed one overlong bounded-evidence pathname; + the same file was already present through the long-path API, the directory + was moved to the shorter marker path above, and the complete inventories + then matched exactly. +- **Cleanup:** explicit benchmark cleanup exited 0 with `cleaned: true`. The + exact run workspace and runtime entry are absent, the exact sandbox has no + matching container or volume, the isolated gateway is gone, and the product + remains clean. The unrelated debug gateway PID 62980 remains untouched. +- **Disposition:** passed. The resource-producer/parser blocker is resolved. + A fresh `paper-env-smoke` is now permitted; pilot and freeze remain blocked + until that smoke passes its full 19-cell correctness, resource, and cleanup + gate. + +### 2026-07-31T00:44:06.158Z - Fresh smoke qualification and exact-plan recheck + +- **Entry ID:** `exp1-bc1e6ee0-smoke-preflight-057`. +- **Phase/kind:** EXP1-C corrected-treatment qualification; no live preset in + this entry. +- **Test qualification:** independent full backend validation passed 220 tests + with five expected Windows symlink-privilege skips in 35.23 seconds. + Deterministic analysis plus archive-guard validation passed all 25 tests in + 5.01 seconds. The context-strict resource-focused suite passed all 48 tests + in 4.66 seconds. All runs used Python 3.13.14, + `PYTHONDONTWRITEBYTECODE=1`, and no pytest cache provider. +- **Exact expansions:** `paper-env-smoke` is runnable with no warnings or + validation issues at plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`, + 19 cells, 19 batches, and 55 operation requests. `paper-pilot` is runnable + at + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`, + 19 cells, 133 batches, and 385 requests. `paper-good-pass` is runnable at + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`, + 19 cells, 1,938 batches, and 5,610 requests. Every preset selects only + `product_cli` and `paper-100m`. +- **Treatment/environment:** product is clean `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`; all four staged executable + hashes match Entry 053. The local image ID exactly matches the fixed Ubuntu + digest. C: free space is 572,616,560,640 bytes. No benchmark campaign, + candidate gateway, or candidate sandbox process is active. The unrelated + debug gateway PID 62980 remains outside campaign ownership. +- **Diff checks:** product and scoped paper `git diff --check` exited 0; only + existing Git line-ending conversion notices were printed. +- **Disposition:** passed. A single fresh corrected-treatment + `paper-env-smoke` is authorized by the protocol. No pilot or final command + may run unless its complete 19-cell evidence, mandatory resources, and + cleanup audit pass. + +### 2026-07-31T00:58:32.472Z - Corrected-treatment CLI integration smoke passed + +- **Entry ID:** `exp1-bc1e6ee0-smoke-pass-058`. +- **Phase/kind:** EXP1-C live CLI integration smoke. +- **Run/command:** run `019fb5a1-313c-7db7-969d-a064b43afa67` issued the + validated `paper-env-smoke` command against + `target\windows-exp1-bc1e6ee0\bin` with + `PYTHONDONTWRITEBYTECODE=1`. +- **At-run identity:** clean product `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`; package, daemon, all four + native Windows CLIs, image, host, sandbox limits, fixture, analysis source, + and plan identities matched Entry 057. No build, install, pull, source + mutation, or environment reconfiguration occurred during the run clock. +- **Start/end/elapsed:** `2026-07-31T00:44:31.290325Z` to + `2026-07-31T00:52:06.451390Z`; 455.161065 seconds of manifest run time. + The supervised wrapper returned exit 0 after 705.1 seconds; after the + terminal report was written it spent approximately 250 seconds draining + off-clock Python worker activity. No candidate gateway or sandbox remained + during that drain, and no benchmark result uses wrapper elapsed time. +- **Outcome:** terminal state `completed`, correctness `pass`, 19 of 19 cells, + 19 of 19 reportable measured trials, all 55 planned operation requests, all + 48 registered correctness checks, 19 operation-evidence records, zero + failed trials, and zero warnings. Every primary request used the required + manager or runtime CLI; verification and observability used the required + runtime or observability CLI. +- **Mandatory resource audit:** all 72 required + non-`create_sandbox` metric/trial combinations for `upperdir_bytes`, + sandbox CPU time, block reads, and block writes contain exactly two + non-periodic mandatory boundaries and both are available. Sandbox-create's + eight counter readings are explicitly inapplicable because a pre-create + sandbox baseline cannot exist; none is fabricated as zero. The corpus + contains 616 total resource records. +- **Periodic-sampling disclosure:** slower write/edit/create-workspace trials + also emitted 84 scheduled 100-ms readings. Thirteen fields were explicitly + unavailable: the known NTFS host allocated-byte field, the product's + unavailable LayerStack total, and one workspace-create upperdir observation + captured during the transition before the next disk sample. The mandatory + before/after boundaries for that trial are both complete (0 and 20,480 + allocated bytes). No missing periodic value was converted to zero or + removed. +- **Correlation boundary:** each of the 18 post-create cells has one eligible + CPU/latency point; `create_sandbox` has zero because its CPU delta is + inapplicable. Every smoke coefficient and interval is omitted as + `insufficient_n`, as required for one-sample support. +- **Immutable archive:** archive + `experiments/runs/019fb5a1-313c-7db7-969d-a064b43afa67` contains 1,421 + files and 18,963,304 bytes with content-tree SHA-256 + `sha256:6ab9facdab1e7ad75ef5e8a26c30dc121835b10edcc0d3b0ca252098b435e1c0`. + Creation verification and a separate verify-only invocation both matched + exactly. +- **Cleanup:** the archive proof records no run- or gateway-labeled + containers or volumes, no matching product processes, no runtime or run + workspace, clean product `main`, and the exact treatment commit. Explicit + benchmark cleanup also exited 0. Independent terminal checks found zero + running Docker containers, zero candidate sandbox processes, no run/runtime + path, and a clean product. The unrelated debug gateway PID 62980 was not + touched. +- **Disposition:** EXP1-C passed for the corrected treatment. A single fresh + five-sample exploratory `paper-pilot` is now permitted; freeze and final + remain blocked until its full correctness, resources, deterministic + analysis, runtime projection, and cleanup audit pass. + +### 2026-07-31T00:59:13.812Z - Five-sample exploratory pilot preflight + +- **Entry ID:** `exp1-bc1e6ee0-pilot-preflight-059`. +- **Phase/kind:** EXP1-D strict fast preflight; no live pilot in this entry. +- **Treatment:** product remains clean `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`. Gateway, manager, runtime, and + observability hashes remain respectively `cb36cc7...b463`, + `032187d0...eac6`, `3ada794f...349`, and `406ca3fa...dfc3`. The exact image + digest is present locally; no pull was issued. +- **Plan:** `paper-pilot` is runnable with only `product_cli` and + `paper-100m`, zero warnings and validation issues, plan hash + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`, + 19 cells, 133 batches, and 385 operation requests. +- **Environment/cleanup:** C: free space is 572,263,915,520 bytes. No + benchmark process, candidate gateway, or candidate sandbox process is + active. Corrected smoke archive + `019fb5a1-313c-7db7-969d-a064b43afa67` still verifies at + `sha256:6ab9facdab1e7ad75ef5e8a26c30dc121835b10edcc0d3b0ca252098b435e1c0`. +- **Clock boundary:** no build, install, image pull, source mutation, + dependency change, or environment reconfiguration will occur from pilot + start until its terminal manifest and report are written. +- **Disposition:** passed. One exploratory `paper-pilot` command is permitted. + Its values are permanently ineligible for manuscript tables. + +### 2026-07-31T01:19:37.788Z - Exploratory pilot failed and corpus preserved + +- **Entry ID:** `exp1-bc1e6ee0-pilot-failed-060`. +- **Phase/kind:** EXP1-D live five-sample exploratory pilot; permanently + ineligible for manuscript tables. +- **Run/command:** run `019fb5ae-fe8f-796a-ac8a-f7fc0b99e60b` issued the + validated `paper-pilot` command against + `target\windows-exp1-bc1e6ee0\bin` with + `PYTHONDONTWRITEBYTECODE=1`. +- **At-run identity:** clean product `main` at + `bc1e6ee04d4df5541290537994a4bf270fcd36b6`; package, native CLIs, daemon, + image, host, limits, fixture, analysis source, and plan identity matched + Entry 059. No build, install, pull, source mutation, or environment + reconfiguration occurred during the run clock. +- **Start/end:** `2026-07-31T00:59:35.424427Z` to + `2026-07-31T01:07:53.405171Z`. +- **Terminal outcome:** failed correctness after 106 of 133 batches. The + report contains 105 successful trials, 278 issued operation requests, one + failed trial, and zero warnings. Failure occurred in the first warmup of + cell + `sha256:07844b8b3087d14ef328ba48b05955f3d5f9df4e7c572b5bd2e2dd19f87c3e02`: + `file_edit`, concurrency 5, 262,144-byte independent targets, one exact + replacement at density 1.0, destination `session`, fresh session per + trial, and `paper-100m`. +- **Failure evidence:** trial + `trial-d2e2dd19f87c3e02-warmup-000000` is `cleanup_invalid` with zero + reportable requests, `product_succeeded: false`, + `infrastructure_failed: true`, and + `cleanup_baseline_restored: false`. Request 1 passed in 0.108710 seconds. + Requests 0 and 2 each returned `internal_error` after 30.016745 and + 30.016014 seconds with the exact bounded message + `sandbox daemon forwarding failed: daemon request timed out after 30000 + ms`. Requests 3 and 4 were cancelled after 30.016121 and 30.012904 + seconds. The post-operation mandatory snapshot and session destroy then + returned `server_busy`, so no result from this trial is eligible. +- **Resource disclosure:** scheduled periodic sampling saturated its fixed + one-sample concurrency cap while the edit requests were stuck, and one + cgroup sample itself timed out near 30 seconds. These unavailable readings + remain explicit in the corpus; none was removed or converted to zero. +- **Immutable archive:** exploratory failed-run archive + `experiments/runs/019fb5ae-fe8f-796a-ac8a-f7fc0b99e60b` contains 6,304 + files and 83,770,836 bytes with content-tree SHA-256 + `sha256:5640483803335d2139e0ca2467a8145e0601fc379672d0b0ce82812c5f2a4b9a`. + A separate verify-only invocation returned `verified: true`. +- **Preservation-tool note:** the first seal attempt used an incorrect + truncated image digest copied from an intermediate status summary and + failed its Docker-image preflight. The run manifest supplied the correct + pinned digest + `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf`. + The successful retry used that exact value. The failed attempt's + run-specific duplicate staging tree was removed only after the valid + archive independently verified; the raw result corpus remains present. +- **Cleanup:** scoped benchmark cleanup removed the exact run and runtime + scopes. Independent checks found zero candidate benchmark/product + processes and no exact run/runtime path. Archive cleanup verification + passed, and the product remains clean `main` at the exact treatment commit. + Unrelated Docker and debug resources were not touched. +- **Disposition:** failed and preserved. Gate 3 is closed. No treatment + freeze, annotated tag, or `paper-good-pass` command has been issued. + Product and harness diagnosis must identify and qualify a correction before + any replacement exploratory pilot is authorized. + +### 2026-07-31T01:25:33.662Z - Pilot starvation diagnosis and correction boundary + +- **Entry ID:** `exp1-pilot-starvation-diagnosis-061`. +- **Phase/kind:** pre-freeze failed-pilot diagnosis; no live preset in this + entry. +- **Method:** the primary agent and a read-only independent subagent compared + the failed pilot's exact subprocess metadata and event timing with the + successful matching smoke cell, then traced the daemon, namespace-execution, + resource-sampler, and gateway timeout implementations. The subagent changed + no files or runtime state. +- **Matched-cell evidence:** the smoke's concurrency-5, 262,144-byte session + edit completed all five requests in 0.093467--0.177086 seconds. In smoke, + request 1 ended only 0.100 ms before the first scheduled sampler CLI began. + In the failed pilot, the three sampler CLIs began while request 1 was still + active. Payload, factors, fixture profile, operation path, and treatment + were otherwise equivalent. +- **Root cause:** high-confidence Tokio blocking-pool starvation. The standard + daemon has exactly eight blocking workers. Its enabled internal + `ResourceSampler` permanently occupies one. Each ordinary or observability + RPC dispatch occupies another. After request 1 completed, four edit + dispatches plus three periodic observability dispatches occupied all seven + remaining workers. The namespace completion supervisor then attempted to + run `complete_jobs` through the same exhausted blocking pool, so it could + not resolve the file-runner promise that would release an edit dispatch. + This closed dependency cycle explains the two exact 30-second forwarding + timeouts, two cancellations, three 30-second sampler timeouts, and the + subsequent `max_blocking_requests: 8` capacity errors. +- **Correction boundary:** timeout inflation, observation disablement, + concurrency reduction, or retry would only mask the defect and is rejected. + The smallest structural correction is to make namespace completion progress + independent of Tokio's shared blocking pool by using the supervisor's + existing dedicated completion-thread path even when constructed inside a + multi-thread Tokio runtime. The configured worker counts and request + admission limits remain unchanged. +- **Qualification requirement:** add a deterministic saturation regression + that constructs the supervisor inside a Tokio runtime, occupies every + blocking-pool worker, and proves an immediately complete child still + resolves. Then run focused namespace-execution, daemon, full product, package + identity, and a bounded pre-pilot reproduction before any replacement + pilot. +- **Disposition:** correction authorized as a pre-freeze product bug fix. + Gate 3 remains closed, and no freeze or final command is permitted. + +### 2026-07-31T01:44:20.322Z - Namespace-completion correction qualification and treatment rebuild + +- **Entry ID:** `exp1-namespace-completion-correction-062`. +- **Phase/kind:** pre-freeze product correction, qualification, and immutable + treatment-package rebuild; no live benchmark preset in this entry. +- **Product correction:** committed direct to the authorized product `main` as + `06f52dfbadc923b80840dd6d156bb7f026519e19` + (`fix(runtime): isolate namespace completion worker`). Namespace completion + now always uses the existing dedicated `eos-command-reaper` thread instead + of scheduling its completion callback through Tokio's shared blocking pool. + The configured Tokio worker count, daemon blocking-request limit, admission + behavior, resource sampling, benchmark factors, and benchmark harness were + not weakened or changed by this correction. +- **Regression:** the new deterministic supervisor test creates a Tokio runtime + with `max_blocking_threads = 1`, occupies that sole blocking worker, and + proves a completed child resolves before the pool is released. The exact + regression passed. +- **Relevant package qualification (WSL Ubuntu, Rust 1.97.1, isolated + `CARGO_TARGET_DIR=/tmp/eos-exp1-target`):** + `sandbox-runtime-namespace-execution` passed its full test package; + `sandbox-daemon --features jemalloc` passed 112 tests; and + `sandbox-runtime` passed its full test package. Clippy passed with + `--all-targets` for all three affected packages. The only emitted clippy + warning was an unrelated pre-existing Rust 1.97 style suggestion in + `crates/sandbox-observability/telemetry/src/sink.rs:201`. +- **Qualification-only test repairs in the same commit:** an invalid + process-global `active_pty_readers == 0` assertion was removed after both + default and serial package orderings demonstrated dependence on preceding + tests while the exact isolated engine test passed. The daemon result-FD unit + fixture was changed from a Unix socket, which Linux cannot reopen through + `/proc/self/fd` and failed alone with `ENXIO`, to the `rustix` pipe used by + production. Neither repair changes product runtime behavior. +- **Known qualification boundaries:** native Windows compilation of the + Linux-only affected package stops in existing telemetry code because + `rustix::fs` is unavailable on Windows. A workspace-wide + `cargo test --workspace --all-features --quiet` on Rust 1.97.1 reaches an + unrelated pre-existing lifetime error (`E0521`) at + `crates/sandbox-runtime/workspace/tests/holder_lifecycle.rs:361`. These + failures are outside the corrected dependency path; the affected Linux + packages pass in full. +- **Rebuild tool note:** the repository-prescribed + `bin/start-sandbox-docker-gateway --rebuild-binary --help` wrapper exited + before any build or runtime mutation because the Windows checkout exposes + its shebang as `/bin/sh^M`. The documented direct fallback + `target/debug/xtask.exe package --builder zigbuild + --target x86_64-unknown-linux-musl` then succeeded, explicitly recompiling + `sandbox-runtime-namespace-execution`, `sandbox-runtime`, + `sandbox-runtime-workspace`, and `sandbox-daemon`. +- **Treatment package:** Windows release packaging produced + `ephemeral-sandbox/target/windows-exp1-06f52dfb.zip`, SHA-256 + `34f2bf79d59d5c76afcaaf0a8bf8ef032aec9a0d2bc91a8e0019e5442f12d512`. + The archive has exactly the expected nine files. Component identities are: + gateway + `cb36cc7c1ffba5f97a8216e30d6fb8213d04581c100bd097739981fe4bc0b463`; + manager CLI + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`; + runtime CLI + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`; + observability CLI + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`; + corrected Linux daemon + `dfec6818fa5bbece5e26bd88c5be464b2a9027a078bbcfb1e05eaf7e4acca282`; + and config + `914d68807d97124a8412963d6f17e7a3914514bb0f0d210ea101ae52ba0ef512`. +- **State/disposition:** product HEAD and package source identity agree, and + the product worktree is clean. The old failed pilot remains permanently + ineligible and immutable. Gate 3 remains closed pending a bounded live + starvation reproduction, a fresh full smoke, and a complete replacement + five-sample pilot; no freeze or final command is authorized yet. + +### 2026-07-31T01:49:59.712Z - Corrected-daemon bounded-reproduction preflight + +- **Entry ID:** `exp1-namespace-completion-reproduction-preflight-063`. +- **Phase/kind:** pre-freeze diagnostic preflight; no live benchmark command + in this entry. +- **Harness qualification:** the complete benchmark backend suite passed + 220 tests with five expected Windows symlink-privilege skips in 37.95 + seconds under Python 3.13.14, `PYTHONDONTWRITEBYTECODE=1`, and no pytest + cache provider. This includes the native-Windows ordinary-copy fixture + acceleration, strict resource parser, runner, cleanup, product-CLI, + artifact, archive, and deterministic-analysis contracts. +- **Official-plan revalidation:** `paper-env-smoke`, `paper-pilot`, and + `paper-good-pass` are runnable and customized with zero findings, use only + `product_cli` and `paper-100m`, and retain hashes + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`, + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`, + and + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. + Their exact expansions remain 19/19/55, 19/133/385, and 19/1,938/5,610 + cells/trial batches/issued operation requests. Two PowerShell reporting + helpers were discarded before accepting these results: one had a parser + error before invoking validation, and one used an unsupported + `ConvertFrom-Json -Depth` option and therefore printed invalid summaries. + The three CLI validations were then reissued and parsed successfully. +- **Diagnostic-plan audit:** the first one-cell custom plan validated at + `sha256:04c0b6b84d18fdc92f2e6fcdbb58e3053c397952ad4b9dfb40aaf343c72dcf68`, + but inspection rejected it before execution because the planner only + injects the mandatory `paper-100m` profile for a canonical paper-preset + name. Its cell therefore did not match the failed pilot. The corrected + diagnostic deliberately uses canonical name `paper-pilot` while remaining + customized and contains only the failed cell. It validates with zero + findings at + `sha256:fab773f32855fe90061b775781d21ee89fa1f9b7cc8ceb16221cc37b403a4f72`: + one cell, two warmups, five measured trials, seven batches, 35 issued + requests, a 100-ms resource cadence, and a 120-second trial timeout. + Its exact cell ID + `sha256:07844b8b3087d14ef328ba48b05955f3d5f9df4e7c572b5bd2e2dd19f87c3e02` + matches the failed pilot: `file_edit`, concurrency 5, 262,144 bytes, one + exact replacement at density 1.0, session destination, independent targets, + fresh sessions per trial, and `paper-100m`. +- **Treatment/environment recheck:** product is clean `main` at + `06f52dfbadc923b80840dd6d156bb7f026519e19`; package archive remains + `34f2bf79...12d512`, corrected daemon remains `dfec6818...ca282`, the + four native Windows executables and config retain the Entry-062 hashes, + and the fixed local image resolves exactly to + `sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf`. + Docker client/server are 29.0.1 on Linux AMD64, overlayfs, and cgroup v2. + C: has 571,799,793,664 free bytes. +- **Ownership baseline:** zero containers or volumes carry the benchmark + gateway-instance label, zero processes execute from the candidate package, + and the benchmark runtime directory is empty. Older terminal run records + and temporary command logs are retained as pre-existing user evidence and + will not be cleaned. The unrelated debug gateway remains outside this + treatment and is not touched. +- **Disposition:** preflight passed. One bounded diagnostic command is + authorized. Any product error, missing resource boundary, cleanup failure, + identity drift, or daemon instability stops the campaign and remains + ineligible; smoke, pilot, freeze, and final remain blocked until this + diagnostic is preserved and audited. + +### 2026-07-31T01:56:38.634Z - Corrected-daemon starvation reproduction passed + +- **Entry ID:** `exp1-namespace-completion-reproduction-pass-064`. +- **Phase/kind:** pre-freeze bounded live diagnostic; permanently + diagnostic-only and ineligible for manuscript tables. +- **Run/identity:** run `019fb5dd-ff1e-770e-9524-e28bdea4afb3` invoked the + exact Entry-063 custom-plan path against + `target\windows-exp1-06f52dfb\bin`. At-run treatment is clean product + `main` at `06f52dfbadc923b80840dd6d156bb7f026519e19`, corrected daemon + `dfec6818...ca282`, the exact four native Windows executable hashes, + `product_cli`, fixed Ubuntu digest, `paper-100m`, and plan hash + `sha256:fab773f32855fe90061b775781d21ee89fa1f9b7cc8ceb16221cc37b403a4f72`. + No build, install, pull, source mutation, dependency change, or environment + reconfiguration occurred during the run clock. +- **Start/end/elapsed:** `2026-07-31T01:50:56.250746Z` to + `2026-07-31T01:52:21.998213Z`, or 85.747467 seconds of manifest time. +- **Terminal outcome:** state `completed`, correctness `pass`, exact failed + cell ID + `sha256:07844b8b3087d14ef328ba48b05955f3d5f9df4e7c572b5bd2e2dd19f87c3e02`, + seven of seven successful trial batches, two warmups, five reportable + measured trials, all 35 operation requests successful, zero product, + correctness, infrastructure, cleanup, or missing-latency failures, and zero + warnings. All 21 trial check records passed. +- **CLI evidence:** all 430 primary, setup, verification, observability, and + cleanup CLI invocations returned code 0, passed response validation, emitted + zero stderr bytes, had unique request IDs, and preserved authentication + redaction. +- **Mandatory resources:** every trial contains exactly two available + non-periodic boundaries for each required starvation-reproduction metric: + 14 `upperdir_bytes`, 14 sandbox CPU, 14 sandbox block-read, and 14 sandbox + block-write boundary records in total. Across mandatory and scheduled + samples, each of these metrics has 26 available observations. Host + allocated-byte metadata and LayerStack allocated storage remain explicitly + unavailable in 26 readings each for the already disclosed platform/product + reasons; no unavailable value was encoded as zero. +- **Timing discriminator:** the diagnostic is conclusive rather than merely + green. In three measured trials a scheduled resource collection began while + all five edit requests were still in flight: at 97.977 ms before the first + completion at 122.720 ms; at 98.458 ms before 111.481 ms; and at 97.860 ms + before 112.317 ms. These are the exact overlap conditions that deadlocked + the old shared blocking-pool completion path. All requests still completed + normally. +- **Immutable preservation:** the exact raw result first copied and verified + at 1,308 files, 21,887,310 bytes, content-tree SHA-256 + `sha256:4a568a61eb11681e2323d5e694a0e130e87c52acaa193a0db8ebcde4f1ae667e`. + The self-contained diagnostic corpus at + `experiments/diagnostics/starvation-019fb5dd-ff1e` additionally preserves + the validated source plan, outer command result/stderr, exact cleanup proof, + and machine-readable acceptance audit. Its final inventory is 1,313 files, + 21,891,244 bytes, content-tree SHA-256 + `sha256:df031fff42b8b4553c08ead617d029d98f25f73e42c3776dcd57c883047f5357`. +- **Cleanup:** the independent proof found no run- or gateway-labeled + container or volume, no matching product process, no run workspace, no + runtime scope, clean product `main`, and the exact recorded treatment. + Explicit benchmark cleanup then exited 0 with `cleaned: true`. A terminal + result copy remains in `.benchmark-state/results` by benchmark design and + is not a live resource or leak. Rechecks found zero candidate processes and + zero matching Docker resources. +- **Disposition:** passed. The Tokio blocking-pool starvation correction is + qualified under the exact previously failing concurrency/sampling timing. + A fresh full `paper-env-smoke` on treatment `06f52dfb` is now authorized. + Pilot, freeze, and final remain blocked until that smoke is preserved and + passes all 19-cell correctness, resource, and cleanup gates. + +### 2026-07-31T01:57:21.540Z - Fresh 06f52dfb smoke preflight + +- **Entry ID:** `exp1-06f52dfb-smoke-preflight-065`. +- **Phase/kind:** EXP1-C strict fast preflight; no live preset in this entry. +- **Plan:** `paper-env-smoke` revalidated runnable and customized with zero + findings or warnings at + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`, + exactly 19 cells, 19 trial batches, and 55 issued operation requests. It + selects only `product_cli` and `paper-100m`. +- **Treatment:** product remains clean `main` at + `06f52dfbadc923b80840dd6d156bb7f026519e19`. Package archive, daemon, + gateway, manager, runtime, and observability hashes remain respectively + `34f2bf79...12d512`, `dfec6818...ca282`, `cb36cc7c...b463`, + `032187d0...eac6`, `3ada794f...349`, and `406ca3fa...dfc3`. +- **Environment/ownership:** the fixed local image digest matches exactly; + C: has 571,759,853,568 free bytes; the benchmark runtime directory is + empty; and no candidate-package process, gateway-labeled container, or + gateway-labeled volume is active. The unrelated debug gateway remains out + of scope and untouched. +- **Prior diagnostic integrity:** the accepted starvation-reproduction archive + independently re-verifies at 1,313 files, 21,891,244 bytes, content-tree + SHA-256 + `sha256:df031fff42b8b4553c08ead617d029d98f25f73e42c3776dcd57c883047f5357`. +- **Clock boundary/disposition:** no build, install, pull, source mutation, + dependency change, environment reconfiguration, or unrelated cleanup may + occur from smoke start through terminal artifact production. One fresh + `paper-env-smoke` command is authorized. Pilot, freeze, and final remain + blocked. + +### 2026-07-31T02:11:14.819Z - Fresh 06f52dfb CLI integration smoke passed + +- **Entry ID:** `exp1-06f52dfb-smoke-pass-066`. +- **Phase/kind:** EXP1-C live CLI integration smoke. +- **Run/identity:** run `019fb5e4-3f62-7760-bc3f-e7501502ec74` issued the + validated `paper-env-smoke` preset against + `target\windows-exp1-06f52dfb\bin` with + `PYTHONDONTWRITEBYTECODE=1`. At-run product, package, daemon, all four + native Windows executables, image, host, sandbox limits, fixture, analysis + source, and plan identities match Entry 065. No build, install, pull, + source mutation, dependency change, or environment reconfiguration occurred + during the run clock. +- **Start/end/elapsed:** `2026-07-31T01:57:44.888263Z` to + `2026-07-31T02:05:39.454170Z`, or 474.565907 seconds of manifest time. + The outer command flushed its terminal JSON at + `2026-07-31T02:09:47.734Z`, 248.281 seconds later, while off-clock Python + workers drained at near-zero wrapper CPU. No benchmark result uses that + wrapper-drain time. +- **Outcome:** state `completed`, correctness `pass`, all 19 cells, all 19 + measured trial batches, all 55 planned operation requests, all 48 + registered correctness checks, and all 19 operation-evidence records passed. + There are zero product, correctness, infrastructure, cleanup, + missing-primary-latency, or warning counts. +- **CLI evidence:** all 462 primary, setup, verification, observability, and + cleanup CLI invocations returned code 0, passed response validation, + emitted zero stderr bytes, had unique request IDs, and preserved + authentication redaction. +- **Resource audit:** the corpus contains 602 resource records. All 72 + required non-`create_sandbox` metric/trial combinations for + `upperdir_bytes`, sandbox CPU time, block reads, and block writes have + exactly two available non-periodic boundaries. The five scheduled sampling + collections are retained. For `create_sandbox`, both recorded boundaries + for each sandbox counter are explicitly unavailable because a + sandbox-scoped pre-create baseline cannot exist; none is fabricated as + zero. Host allocated bytes and LayerStack allocated storage remain + explicitly unavailable in 43 readings each for the disclosed + platform/product reasons. +- **Correlation boundary:** each of the 18 post-create cells has one eligible + CPU/latency point; `create_sandbox` has zero because its CPU delta is + inapplicable. Every coefficient interval is omitted as `insufficient_n`. + The first read-only audit helper incorrectly assumed the omission was an + object rather than its schema-defined string, raised `TypeError`, and + changed no artifact; the corrected helper then passed all boundary and + correlation assertions. +- **Immutable archive:** archive + `experiments/runs/019fb5e4-3f62-7760-bc3f-e7501502ec74` contains 1,442 + files and 18,936,678 bytes with content-tree SHA-256 + `sha256:35023c597b7218af971bac1e317712d24651d82a00510fc2e6d2907ecb12b9bc`. + Archive creation plus two independent verify-only invocations matched + exactly. +- **Cleanup:** the archive proof and independent terminal recheck found no + run- or gateway-labeled container or volume, matching product process, run + workspace, or runtime scope; product remains clean `main` at the recorded + treatment. Explicit benchmark cleanup exited 0 with `cleaned: true`. +- **Disposition:** EXP1-C passes for the corrected `06f52dfb` treatment. One + fresh exploratory five-sample `paper-pilot` is now authorized. Freeze and + final remain blocked until that pilot is preserved and passes correctness, + resource correlation, deterministic table regeneration, runtime projection, + anomaly review, and cleanup. + +### 2026-07-31T02:12:15.605Z - Replacement five-sample pilot preflight + +- **Entry ID:** `exp1-06f52dfb-pilot-preflight-067`. +- **Phase/kind:** EXP1-D strict fast preflight; no live pilot in this entry. +- **Plan:** `paper-pilot` revalidated runnable and customized with zero + findings or warnings at + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`, + exactly 19 cells, 133 trial batches, and 385 issued operation requests. It + selects only `product_cli` and `paper-100m`, with two warmups and five + measured trials per cell. +- **Treatment:** product remains clean `main` at + `06f52dfbadc923b80840dd6d156bb7f026519e19`; package, corrected daemon, and + all four native Windows executable hashes match Entries 062 and 065. + The fixed local image digest matches exactly. +- **Environment/ownership:** C: has 571,460,431,872 free bytes; the benchmark + runtime directory is empty; and no candidate-package process, + gateway-labeled container, or gateway-labeled volume is active. The + unrelated debug gateway remains outside campaign ownership. +- **Prior-gate integrity:** the corrected-treatment smoke archive independently + re-verifies at 1,442 files, 18,936,678 bytes, content-tree SHA-256 + `sha256:35023c597b7218af971bac1e317712d24651d82a00510fc2e6d2907ecb12b9bc`. +- **Clock boundary/disposition:** no build, install, pull, source mutation, + dependency change, environment reconfiguration, or unrelated cleanup may + occur from pilot start through terminal artifact production. Exactly one + replacement exploratory `paper-pilot` command is authorized. Its values are + permanently ineligible for manuscript tables; freeze and final remain + blocked. + +### 2026-07-31T02:33:19.464Z - Replacement five-sample pilot passed and archived + +- **Entry ID:** `exp1-06f52dfb-pilot-pass-068`. +- **Phase/kind:** EXP1-D live five-sample exploratory pilot; permanently + ineligible for manuscript tables. +- **Run/identity:** run `019fb5f1-d73a-7128-9bab-d75dd229c020` issued the + validated `paper-pilot` preset against + `target\windows-exp1-06f52dfb\bin` with + `PYTHONDONTWRITEBYTECODE=1`. At-run treatment is clean product `main` at + `06f52dfbadc923b80840dd6d156bb7f026519e19`; package, corrected daemon, + native CLIs, image, host, sandbox limits, fixture, benchmark source, and + plan identities match Entry 067. No build, install, pull, source mutation, + dependency change, or environment reconfiguration occurred during the run + clock. +- **Start/end/elapsed:** `2026-07-31T02:12:35.693691Z` to + `2026-07-31T02:22:44.909649Z`, or 609.215958 seconds of manifest time. + The outer command flushed its terminal JSON at + `2026-07-31T02:27:22.644Z`, 277.734 seconds later, during the known + off-clock worker drain. No benchmark result uses wrapper-drain time. +- **Terminal outcome:** state `completed`, correctness `pass`, all 19 cells, + all 133 trial batches, 38 warmups, 95 of 95 reportable measured trials, all + 385 issued operation requests, all 336 trial check records, and all 133 + operation-evidence records passed. Every cell has exactly two warmups and + five successful measured trials. There are zero product, correctness, + infrastructure, cleanup, missing-primary-latency, or warning counts. +- **CLI evidence:** all 2,966 primary, setup, verification, observability, and + cleanup CLI invocations returned code 0, passed response validation, + emitted zero stderr bytes, had unique request IDs, and preserved + authentication redaction. +- **Resource/correlation audit:** the archive has 4,298 resource records. + All 504 required non-`create_sandbox` metric/trial pairs for upperdir + allocation, sandbox CPU, block reads, and block writes have exactly two + available non-periodic boundaries. All 21 create-sandbox counter/trial + pairs have exactly two explicitly inapplicable boundaries because no + sandbox-scoped pre-create baseline can exist. The 41 periodic sample + collections are retained. Six transition-time periodic upperdir readings + are explicitly unavailable as incompletely reported; every corresponding + mandatory boundary is available. Six create-sandbox memory boundary + readings explicitly report that the resource ring is not available yet; + the measured report still has complete post-create memory values. Host + allocated bytes and LayerStack allocation remain explicitly unavailable as + already disclosed. Each post-create cell has five aligned CPU/latency + points and zero correlation exclusions; `create_sandbox` has zero by the + counter-inapplicability rule. +- **Immutable archive:** archive + `experiments/runs/019fb5f1-d73a-7128-9bab-d75dd229c020` contains 9,068 + files and 102,646,127 bytes with content-tree SHA-256 + `sha256:80658973aec8449bca7a3b8880a1b25ad15ee38ee1a9229b450aaca0fa62536e`. + The first supervised archive invocation exceeded its 60-second shell + timeout after the archiver had atomically installed the complete target; + it left no staging duplicate and no archiver process. No archive retry was + issued. Two subsequent verify-only invocations matched the inventory and + hash exactly. +- **Cleanup:** the archive proof and independent terminal recheck found no + run- or gateway-labeled container or volume, matching product process, run + workspace, or runtime scope; product remains clean at the exact treatment. + Explicit benchmark cleanup exited 0 with `cleaned: true`. +- **Disposition:** functional, correctness, resource, correlation, evidence, + and cleanup portions of EXP1-D passed. Pilot data remain exploratory and + ineligible. Gate 3 still requires deterministic table regeneration, + runtime projection at or below 20 minutes, and anomaly review before freeze. + +### 2026-07-31T02:33:19.464Z - Pilot tables deterministic; runtime projection blocks Gate 3 + +- **Entry ID:** `exp1-06f52dfb-pilot-analysis-blocker-069`. +- **Phase/kind:** EXP1-D immutable exploratory analysis and Gate-3 capacity + decision; no live preset. +- **Deterministic tables:** generator + `sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3` + regenerated all four expected tables twice from only the immutable pilot + archive. Outputs + `experiments/analysis/pilot-06f52dfb-tables-a` and `-b` are byte-identical: + nine files, 228,981 bytes, content-tree SHA-256 + `sha256:7e142a8ee03477f807b1c29e68b5adfb00c03835d73e5eff92e0237dcbe9db24`. + Their numeric-evidence v2 registry and numeric provenance are explicitly + `exploratory_ineligible`; no pilot value may be copied into the manuscript. +- **Table component hashes:** environment + `28028724...f752`, startup `a677b019...0fd7`, CLI operations + `ad280978...3c8b`, resources `895bf303...de8d`, tables JSON + `085fdcc4...20ed`, numeric evidence `e994a991...b771`, and provenance CSV + `39a8eb6a...33c9`. +- **Runtime projection:** the immutable + `project_exp1_final_runtime.py` source is + `sha256:88604b889bd480f52a886781bc1ece922a09fc267703dde00476f1a702ad52b6`. + Projection output + `experiments/analysis/pilot-final-runtime-projection-06f52dfb.json` is + SHA-256 + `6a4c0e3f7133aee4c138ac5f99d0ce0cf1ee6d05fbddd2eebe5efe4f29e98552`. + Smoke and pilot identities match exactly. The completed pilot is a + 609.215958-second monotonic lower bound; the affine model estimates + 452.124232 fixed seconds plus 1.181141 seconds per trial batch, yielding + 2,741.175099 seconds (45.686252 minutes) for the fixed 1,938-batch final. + Direct proportional scaling yields 8,877.146817 seconds. Even the optimistic + model exceeds the fixed 1,200-second limit. +- **Decision/blocker:** `gate_3_runtime_pass: false`, + `block_freeze_and_final_runtime_projection_exceeds_20_minutes`. This is a + mandatory protocol stop, not a manuscript result. Freeze, tag, and the + exactly-once final command remain prohibited. Read-only subagents are + decomposing phase/gateway/CLI costs and auditing the product + `create_sandbox` plus harness setup paths for a protocol-preserving speed + correction. Any source correction will require focused tests, a new + treatment package, bounded qualification, fresh smoke, and a fresh + exploratory pilot before Gate 3 can be reconsidered. + +### 2026-07-31T02:58:13.077Z - Pre-freeze CLI evidence I/O acceleration + +- **Entry ID:** `amendment-exp1-cli-evidence-io-070`. +- **Phase/kind:** EXP1-D pilot review and pre-freeze harness correction; no + live preset. +- **Trigger:** every retained CLI invocation re-read and hashed the same + checksum-pinned executable and independently flushed immutable stdout, + stderr, and metadata files. The accepted pilot retained 2,966 such + invocations, so this paper-local evidence path was a repeated fixed-cost + component outside the primary operation timing. +- **Executable identity correction:** `ProductCliAccess` now computes each of + the manager, runtime, and observability executable SHA-256 values exactly + once at instance construction and reuses the immutable values in every + invocation record. The fixed campaign forbids product-package mutation + during a run, so this preserves the existing executable identity contract. +- **Durable evidence contract:** schema-v2 metadata losslessly embeds the + already-redacted stdout and stderr byte streams as base64, with their exact + lengths and SHA-256 values. The normal on-disk layout remains exactly three + immutable files per invocation: `.stdout`, `.stderr`, and `.json`. + Stdout/stderr are closed immutable projections written without individual + flushes; the self-contained metadata file is written last and fsynced as + the sole durable `metadata-packed-payload-fsync-v1` commit marker. POSIX + additionally syncs the evidence directory; the Windows helper remains a + deliberate no-op because this optimization targets the qualified NTFS + host. +- **Crash/failure behavior:** a failed metadata write or flush removes the + marker, retains any redacted immutable projection orphans as failure + evidence, and an identical request cannot overwrite them because all three + files use exclusive creation. If an NTFS crash loses or tears an unflushed + projection after the marker commits, the marker still contains the exact + redacted bytes needed for reconstruction. Archive verification decodes and + authenticates the packed streams, rejects malformed markers, unsafe or + missing projection paths, projection/packed mismatches, and any size or + digest mismatch. It never silently accepts a partial record. +- **Bounded-evidence size control:** packed byte fields exist only in the + durable `.json` marker. They are explicitly omitted from the compact + `transport_evidence` returned to request observations, so the packed payload + is not duplicated into journals or bounded trial evidence. +- **Artifact journal audit:** `ArtifactStore.append_records` already writes + each caller-supplied record batch with one append and one fsync, while + recovery quarantines a partial tail and durably truncates to the last + complete newline. No additional proven trial-level barrier exists, so + journal durability, `artifacts.py`, `runner.py`, and + `resource_sampling.py` were not changed. +- **Focused verification:** product-CLI and archive evidence tests passed + 31/31 in 1.36 seconds with `PYTHONDONTWRITEBYTECODE=1`. Coverage includes + once-per-instance executable hashing, one NTFS fsync for the three-file + record, exact layout and redaction, packed-stream reconstruction, compact + transport evidence, failed marker flush cleanup, immutable orphan retry + refusal, and marker/projection/path/hash corruption rejection. +- **Complete backend verification:** all 233 tests passed with five expected + Windows WinError 1314 symlink-privilege skips in 36.62 seconds under + `PYTHONDONTWRITEBYTECODE=1`. +- **Disposable NTFS diagnostic:** five alternating rounds of 40 invocations + per method compared the inherited three-fsync layout with the new packed + metadata commit using the same three files. Median latency was 9.980020 + milliseconds per invocation versus 4.317100 milliseconds, a diagnostic + saving of 5.662920 milliseconds and 2.312x speedup. The system temporary + directory was removed automatically. These values are pre-freeze + engineering diagnostics and are ineligible for manuscript tables. +- **Protocol stability:** no preset, definition, cell, repetition, warmup, + concurrency, metric, success, retry, exclusion, timeout, or analysis rule + changed. The existing official identities/counts therefore remain smoke + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + (19/19/55), pilot + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f` + (19/133/385), and final + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b` + (19/1,938/5,610). +- **Files changed:** `benchmark/backend/benchmark_lab/product_cli.py`, + `benchmark/backend/tests/unit/test_product_cli.py`, + `experiments/scripts/archive_exp1_run.py`, + `benchmark/backend/tests/unit/test_exp1_archive.py`, and this append-only + log. +- **Source identities:** `product_cli.py` + `sha256:145c13646af0bcd35196da18c638707fbd5a9f1a087ff435a946367a5dcab19d`, + its unit test + `sha256:d6040ca0436fd7c1a77a560e00f4d55f53f2a35e5d6c8ed140c8ef108672e295`, + archive verifier + `sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e`, + and its unit test + `sha256:a2a0fb0e0d91bea42c25d3f946ead02eeb854bac514191f6521a09313339e250`. +- **Cleanup/scope:** no live gateway, sandbox, Docker object, fixture, product + checkout, preset, or measurement corpus was touched. No live performance + command ran. +- **Disposition:** implementation and offline validation passed. This removes + a demonstrated fixed CLI-evidence cost but does not by itself clear Gate 3; + the combined corrected harness still requires bounded qualification, a + fresh smoke, and a fresh exploratory pilot before freeze or final can be + reconsidered. + +### 2026-07-31T03:06:50.782Z - Pre-freeze sampling and independent-work acceleration + +- **Entry ID:** `amendment-exp1-sampling-independent-work-071`. +- **Phase/kind:** EXP1-D pilot review and pre-freeze harness correction; no + live preset. +- **Trigger:** the accepted exploratory pilot showed mandatory resource + boundaries, independent fixture setup, and paged correctness reads as + repeated untimed costs. Mandatory resource collection alone increased by + 47.450 seconds over the 114 smoke-to-pilot incremental batches. +- **Sampling correction:** the mandatory product boundary, host/workspace + metrics, and daemon observation now start concurrently. Boundary failure + remains fatal, while the established unavailable-field behavior is + preserved for dynamic and daemon collectors. The boundary poll interval is + 100 milliseconds, matching the newly qualified Windows product ring and + avoiding repeated CLI polls against one generation. +- **Independent work correction:** fixture writes for independent read, + write, and edit targets execute concurrently. Verification reads and + attribution checks for independent targets also execute concurrently. + Exact reads still establish the bounded page contract from the first page, + then fetch all remaining deterministic pages concurrently and validate + every offset, line count, byte count, total, and payload before assembly. +- **Timing/protocol stability:** setup, verification, resource collection, and + teardown remain outside the primary operation interval. No operation, + preset, cell, repetition, warmup, concurrency, payload, metric, success, + retry, exclusion, or analysis rule changed. +- **Focused verification:** 37 runner/resource tests passed in 5.26 seconds + under `PYTHONDONTWRITEBYTECODE=1`. +- **Complete verification:** the subsequent combined backend suite, including + the run-scoped cache tests below, passed 238 tests with five expected + Windows symlink-privilege skips in 40.48 seconds. +- **Files/source identities:** `resource_sampling.py` + `sha256:a9a9c420d0726907b51eb2a28371f5967f210eaefd35ccb130fd7968594befdd`; + `runner.py` + `sha256:768d0eb2b8c9f7f3c51034788cecc88181cf3962d8ab6553cdea718d4aacaeed`; + resource tests + `sha256:76c98df12c0a33eae58ffc22f618e1532ddd53bf79780cb2509a592e12676d46`; + runner tests + `sha256:eab9c477e803927aec777cacab9cda95a5417dd03d3ecfbf86f54d689d7a6f41`. +- **Cleanup/scope:** no live gateway, sandbox, Docker object, fixture, + package, or preset was run or mutated. +- **Disposition:** offline correction passed. Gate 3 remains blocked pending a + rebuilt exact product package, bounded qualification, fresh smoke, and a + fresh exploratory pilot. + +### 2026-07-31T03:06:50.782Z - Run-scoped shared-base cache for create_sandbox + +- **Entry ID:** `amendment-exp1-run-scoped-shared-base-cache-072`. +- **Phase/kind:** EXP1-D pilot review and pre-freeze `create_sandbox` + correction; no live preset. +- **Trigger:** the first sandbox created under each isolated gateway family + spent about 42 seconds reseeding the same content-addressed Docker shared + base after intermediate cleanup removed both the host cache and volume. + Warm measured sandbox creation was approximately 1.8--2.0 seconds. +- **Correction:** every isolated gateway in one campaign receives the same + owned run-scoped `EOS_SHARED_BASE_CACHE`. Intermediate cleanup may retain + only a volume whose content-addressed name and complete gateway/root/target/ + readonly label proof validate. Ordinary volumes and all containers are + still removed. +- **Final cleanup proof:** the runner records every unique gateway identity and + performs a mandatory final Docker sweep across all of them on success, + operation failure, cleanup failure, and cancellation. A retained volume is + removed through its original owner identity even after later gateways reuse + it. Ambiguous ownership fails closed. Stale-gateway recovery and failed + startup always request full cleanup and never retain volumes. +- **Cross-gateway verification:** two distinct gateway identities for the same + run receive the identical cache directory; the first close retains the + validated shared volume, a later non-owner sweep cannot claim it, and final + cleanup removes it through the original owner. Runner success, failure, and + cancellation paths all assert finalization. +- **Focused verification:** 38 tests passed with one expected Windows symlink + skip. Scoped Ruff formatting/import checks and `git diff --check` passed. +- **Complete verification:** 238 backend tests passed with five expected + Windows symlink-privilege skips in 40.48 seconds under + `PYTHONDONTWRITEBYTECODE=1`. +- **Files/source identities:** `gateway.py` + `sha256:f8e1e55699e351f76f64ffb7079a7852596ee52e02f209452543edde5025f85d`; + `runner.py` + `sha256:768d0eb2b8c9f7f3c51034788cecc88181cf3962d8ab6553cdea718d4aacaeed`; + gateway lifecycle tests + `sha256:cd5dbf28570111c282733809a33bd607485d531338945c0c1c722393264c791d`; + runner tests + `sha256:eab9c477e803927aec777cacab9cda95a5417dd03d3ecfbf86f54d689d7a6f41`. +- **Cleanup/scope:** no live sandbox, gateway, Docker command, preset, fixture, + or measurement corpus was run or mutated. +- **Disposition:** offline correction passed. It removes repeated fixed + create-sandbox cold work but cannot clear Gate 3 without fresh measurements. + +### 2026-07-31T03:06:50.782Z - Product cadence and termination-aware teardown + +- **Entry ID:** `amendment-exp1-product-cadence-holder-073`. +- **Phase/kind:** EXP1-D pilot-discovered product performance corrections + before freeze; no live preset. +- **Resource-ring evidence:** corrected pilot cgroup samples had a 256 + millisecond median generation delta and workspace snapshots had a 251 + millisecond median, while the fixed campaign requires 100 millisecond + resource sampling. Mandatory boundaries therefore waited and repolled a + 250-millisecond product ring. +- **Cadence correction:** the committed Windows release configuration now uses + a 100-millisecond resource sampling interval. The observability + configuration's documented/validated minimum is 100 milliseconds, its edge + test rejects 99 milliseconds, and the committed Windows configuration guard + asserts the exact 100-millisecond value. +- **Recorded failed validation:** after changing only the Windows release + value, `cargo test -p sandbox-config --all-features` failed 1 of 79 tests + because the then-current validator still rejected values below 250 + milliseconds. This exposed the incomplete correction; no result was hidden. +- **Corrected validation:** after aligning the validator and edge tests, + `cargo test -p sandbox-config --all-features` passed 79/79, + `cargo fmt --all --check` passed, and + `cargo clippy -p sandbox-config --all-targets --all-features -- -D warnings` + passed. Commit `4a05a9563c1d2920caec79da85b6d88a74cec2c0` + (`perf(observability): align Windows resource cadence`) contains this + correction. +- **Holder-teardown evidence:** exploratory phase decomposition showed 84 + incremental session destroys consuming 11.53 subprocess-seconds. The + holder supervisor's 50-millisecond idle cadence could delay observation + after termination had already started. +- **Holder correction:** idle polling remains 50 milliseconds. While any + owned holder has an active termination attempt, the supervisor caps its + poll interval at 5 milliseconds. Sole-child ownership, wait/reap, grace + deadline, escalation, and finalization proof semantics are unchanged. +- **Holder validation:** all 30 holder tests passed under WSL; the new focused + regression, library build/test, scoped Clippy with `-D warnings --no-deps`, + `cargo fmt --check`, and `git diff --check` passed. Native Windows telemetry + still has the previously known unrelated Unix-only `rustix::fs` build + failure; the broader WSL package test still has the existing + `tests/holder_lifecycle.rs:361` E0521 failure; unscoped Clippy still has the + existing telemetry `byte_char_slices` lint. Commit + `57b437ba60a2304e896e7345f13e6bc43f431f55` + (`perf(runtime): accelerate pending holder teardown`) contains this + correction. +- **Product source identities:** Windows config + `sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`; + observability validator + `sha256:b709d3054d6ab06a803778f48749f4fd31cf869e2ff05e290655964ce3a1641e`; + validator tests + `sha256:7505b5455c72d9476318a3dcd02a385e7a89084b13e3848461b69e2bcf40f729`; + holder supervisor + `sha256:424d13b90d282a78dcfd7928e38088ff90c2c3f9de26111e01255be0073af301`; + holder tests + `sha256:f5a741b463fc1e6740fc4ba775e49522cd1a7e5e3e8efbe1e2db65cc98500a17`. +- **Repository state:** product remains direct clean `main`; no branch, + worktree, tag, or push was created. The new treatment commit is + `57b437ba60a2304e896e7345f13e6bc43f431f55`. +- **Cleanup/scope:** no live sandbox, gateway, Docker command, preset, fixture, + or measurement corpus ran during either product correction. +- **Disposition:** product corrections passed their proportional offline + validation. Release binaries must be rebuilt from the exact new commit + before bounded qualification, smoke, or pilot. Freeze and final remain + blocked. + +### 2026-07-31T03:10:49.371Z - Exact 57b437ba treatment package rebuild + +- **Entry ID:** `validation-exp1-package-57b437ba-074`. +- **Phase/kind:** EXP1-D pre-freeze off-clock treatment build; no live preset. +- **Precondition:** product was clean direct `main` at + `57b437ba60a2304e896e7345f13e6bc43f431f55`. +- **Recorded failed attempt:** the first direct + `target\debug\xtask.exe package --builder zigbuild --target + x86_64-unknown-linux-musl` invocation was issued with an insufficient + command-runner timeout and exited 124 after approximately five seconds. A + filtered process check found no surviving `xtask`, `cargo`, or `rustc` + process. This was a build-orchestration failure, not a live benchmark + attempt. +- **Linux daemon build:** the same exact xtask command was reissued off-clock + under a hidden process with bounded log capture and exited 0. It produced + `dist\sandbox-daemon-linux-amd64`, 7,171,600 bytes, + `sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`. +- **Windows package build:** `bin\package-windows-amd64-release.ps1 + -PackageName windows-exp1-57b437ba -OutDir target -Profile release` + exited 0. The release gateway build took 1 minute 1 second; the three CLI + target checks completed from cache. +- **Candidate package:** absolute directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-57b437ba`; + ZIP + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-57b437ba.zip`, + 5,651,981 bytes, + `sha256:0d6a784a5447ef012d03d4b8742beac0693f30766e14c2bd9dc4c53e9c4cb359`. +- **Packaged identities:** gateway 3,320,832 bytes, + `sha256:526ba4e1b90b38d8605475b4ad1e53ded12eeef396cb533acfe0d1c90854dfaa`; + manager CLI 819,712 bytes, + `sha256:032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`; + runtime CLI 825,344 bytes, + `sha256:3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`; + observability CLI 819,200 bytes, + `sha256:406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`; + packaged daemon identity as above; packaged Windows config 1,997 bytes, + `sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Config guard:** the packaged Windows config contains + `sample_interval_ms: 100`. +- **Postcondition:** product remained clean direct `main` at the same commit. + No tag or push was created. +- **Cleanup/scope:** no live gateway, sandbox, Docker command, preset, fixture, + or measurement corpus was run. Build logs remain under `target` as + off-clock diagnostics. +- **Disposition:** exact treatment package build passed. The package is not + frozen and cannot be used for final measurement until the remaining + paper-source corrections pass, bounded qualification and fresh smoke/pilot + pass, and Gate 3 clears. + +### 2026-07-31T03:19:47.510Z - Trial hot-path consolidation and shared daemon observation + +- **Entry ID:** `amendment-exp1-trial-hot-path-consolidation-075`. +- **Phase/kind:** EXP1-D pilot review and pre-freeze paper-harness correction; + no live preset. +- **Session mutation baseline:** session-destination `file_write` and + `file_edit` cells now publish their deterministic baseline paths once during + cell setup. Every trial still creates and destroys a fresh shared-network + no-op session and begins from the unchanged published snapshot. Independent + and same-target semantics, measured operation timing, verification, + attribution, and cleanup are unchanged. Publish-destination cells retain + per-trial preparation because they do not have a reusable cell context. +- **Daemon observation consolidation:** product `cgroup` responses already + contain the same daemon process metrics under + `topology.daemon`. A shared strict `DaemonProcessMetrics` model now validates + both the standalone shape and extraction from the exact cgroup response. + Mandatory and periodic samplers make no redundant standalone daemon CLI + subprocess. Missing or malformed embedded metrics preserve the prior + unavailable-field behavior with an explicit observability error. Metric + sources now disclose + `product_observability.cgroup.topology.daemon`. +- **Trial journal group commit:** trial phase, resource, request observation, + correctness, operation, and trial records are buffered and committed with + at most one append/fsync per journal at each trial durability boundary. + Setup records flush before request admission. Waiting and ready barrier + states remain independently durable before release, and deferred request + terminal records retain their existing durable post-batch append. Trial + preparation is durable before the transaction; success, failure, and + cancellation flush the grouped records before the terminal trial state. + Public run-state/cancellation transitions force a preceding flush and remain + immediately durable. Sequence numbers and event-sink delivery preserve + commit order. +- **Crash/recovery behavior:** a crash during setup leaves the durable + preparing state; a crash after request admission leaves the durable + waiting/ready barrier. A partial grouped append is still handled by the + existing journal tail quarantine/truncation path. Missing terminal trial + evidence fails archive completeness and is never silently accepted. +- **Focused validation:** session baseline tests passed 8/8 and the then-current + runner integration file passed 29/29. Observability/resource tests passed + 39/39, metric derivation passed 6/6, and the mock runner path passed. After + journal grouping, the complete runner integration file passed 32/32. + The combined runner/observability/resource/derivation selection passed + 77/77. +- **Complete backend validation:** 254 tests passed with five expected Windows + symlink-privilege skips in 38.93 seconds under + `PYTHONDONTWRITEBYTECODE=1`. +- **Recorded validation failures/corrections:** the first formatting command + attempted `.venv\Scripts\python.exe -m ruff`, which failed because Ruff is + not installed in that virtual environment. Cached + `uvx --offline ruff 0.16.1` was used thereafter. The first 32-test runner + attempt had one failed test because the new synthetic failure-path test + wrote an intentionally underspecified schema-v5 trial observation and then + invoked the strict decoder. The invalid synthetic observation was removed; + the production implementation was unchanged, and all 32 tests passed. + An unscoped Ruff invocation without the repository's intended rule + selection reported 39 existing/scoped diagnostics, including broad + `BaseException` policy, Python-target inference for `BaseExceptionGroup`, + and one unused import; it was not used as a qualification result. The + intended scoped import check then exposed one unsorted import block in the + new observability test, fixed it mechanically, and passed. Final scoped + import checks and `git diff --check` passed. +- **Files/source identities:** runner + `sha256:a3c2b26bbc1a095eb8c6a8abc8b64ed8abb86d3c2403c2b2c589ba2cf9c4fd79`; + observability models + `sha256:dfc25aa2efce2405ed12d3d73d240fd143aa0506535cbf6ddae56de11c1e1500`; + resource sampler + `sha256:434ea0b49f96871ec83fcc474ed5a033b1697fd866ca37c2a7cf6d9d38168e89`; + runner tests + `sha256:214ffb815e0580c265fd83fa0e26f3f507e088ae352ec1d5830aa1ce9a81955b`; + observability tests + `sha256:a23d8094dd34bda65408189ff0b39d282688542e32c0032a0ab75e8fea91363c`; + resource tests + `sha256:5f9633aad13175d1ef26371cc8ec9b2cabd4ad0edb8734b4c251a35247b81ea3`. +- **Timing/protocol stability:** no primary timing boundary, preset, cell, + repetition, warmup, concurrency, payload, success, retry, exclusion, metric, + or final analysis rule changed. The daemon metric source disclosure changed + to its actual containing response before freeze. +- **Cleanup/scope:** no live gateway, sandbox, Docker command, fixture, + package mutation, preset, or measurement corpus was run. +- **Disposition:** combined offline validation passed. The corrected source + still requires the preregistered structural runtime projection, bounded + qualification, fresh smoke, and fresh exploratory pilot before Gate 3 can + clear. + +### 2026-07-31T03:19:47.510Z - 57b437ba packaged CLI help-contract check + +- **Entry ID:** `validation-exp1-package-help-076`. +- **Phase/kind:** EXP1-D off-clock static package validation; no gateway. +- **Package:** `target\windows-exp1-57b437ba`, ZIP identity + `sha256:0d6a784a5447ef012d03d4b8742beac0693f30766e14c2bd9dc4c53e9c4cb359`. +- **Results:** packaged gateway `--help` exited 0, wrote 93 stdout bytes with + `sha256:1a01a7c60c1ee1507b47b39c47ffab5d843184e9f94b5d49ecc89ab424f57934`, + and wrote zero stderr bytes. Manager `help` exited 0, wrote 802 stdout bytes + with + `sha256:b8c3a96951784a67b8dfd021cc754ec1e58ddd5e3ba83512424721709c576fe5`, + and zero stderr. Runtime `help` exited 0, wrote 1,137 stdout bytes with + `sha256:2b2bb48142678f0418d6b01a528e0c79b8de6ab89fafc843d4d660ccd0d7dc5e`, + and zero stderr. Observability `help` exited 0, wrote 1,026 stdout bytes with + `sha256:090d943792ace5873bf7099e3f185030474efd73a59fa876a56fa46878f56243`, + and zero stderr. +- **Security/scope:** the checks required no gateway socket or authentication + token; only exit status, byte counts, and digests were printed. No live + sandbox, Docker command, preset, or measurement corpus was touched. +- **Disposition:** static packaged help contracts passed. Live CLI behavior + remains subject to bounded qualification and fresh smoke. + +### 2026-07-31T03:33:17.889Z - Structural Gate 3 runtime projection preregistration + +- **Entry ID:** `amendment-exp1-structural-runtime-projection-077`. +- **Phase/kind:** EXP1-D pre-freeze analysis correction and preregistration; + no live preset. +- **Trigger/defect:** the prior script estimated one slope from two whole-run + elapsed totals and scaled the smoke-to-pilot delta by final batch count. + That delta confounded run-, family-, and cell-level fixed setup/cleanup + variance with trial work, assigned the smoke's one cold trial to steady + state, verified only four provenance fields, and selected the minimum of two + candidate projections as its gate. The 20-minute limit was fixed, but this + projection formula was not preregistered. +- **Structural inputs:** the replacement verifies the complete archive + inventory/tree hash; terminal disposition, eligibility, correctness, and + cleanup; raw event/observation/plan hashes; benchmark, fixture, product, + binaries, config, image, Docker, host, sandbox limits, artifact schema, + lifecycle, gateway, protocol, and analysis identities; a separately + supplied reviewed final expanded plan; contiguous schema-v1 event and + schema-v5 observation journals; and exact product-CLI cell/request/trial + counts. +- **Semantic matching:** smoke, pilot, and final cells are matched by their + canonical comparison semantics rather than `cell_id`, because the fixed + repetition policy changes cell identities. Family/block order and stable + canonical/effective environment values must agree. +- **Decomposition:** integer monotonic event times partition each run into a + run residual, family residuals, per-cell leading/trailing fixed work, trial + active spans, and between-trial gaps. Every interval must be nested, + ordered, non-overlapping, and nonnegative. Trial phase sums may not exceed + their event span, and every trial must be successful, reportable when + measured, checked, cleanup-restored, and matched to its exact request and + operation observations. +- **Central model:** retain the componentwise maximum smoke/pilot run, family, + cell-leading, and cell-trailing fixed work once; retain the pilot's two + observed warmups and their transitions; scale the exact mean of five + measured trial spans to 100; and scale the exact mean of the four + measured-to-measured gaps to 99. `Fraction` arithmetic is exact and gating + uses the ceiling in integer nanoseconds. +- **Observed envelope:** retain the same fixed maxima; use the larger of pilot + warmups and the smoke cold trial for both final warmups; use the largest + pilot measured span for all 100 measured trials; and use the largest pilot + transition gap for all 101 final transitions. +- **Frozen pass rule:** the runtime component passes only if strict + provenance/structure validation succeeds and all three are at most exactly + 1,200,000,000,000 nanoseconds: completed pilot elapsed, central structural + projection, and observed-envelope projection. There is no candidate-model + `min()` selection. Other Gate 3 correctness, resource, determinism, + anomaly, and cleanup conditions remain independent. +- **Required CLI:** the script now requires `--smoke-archive`, + `--pilot-archive`, `--final-plan`, and a new nonexisting `--output`. It + records the final-plan file SHA-256, internal plan hash, script SHA-256, + exact source archive identities, all decomposition components, rational + numerators/denominators, integer ceilings, and deterministic display + seconds. Output is deterministic sorted JSON. +- **Synthetic verification:** the fixed/family/cell decomposition test + produced exact central `226773/4` nanoseconds (ceiling 56,694), + envelope 82,861 nanoseconds, and proved each fixed component is retained + once. +- **Immutable negative control:** strict parsing of accepted corrected smoke + `019fb5e4-3f62-7760-bc3f-e7501502ec74` and corrected exploratory pilot + `019fb5f1-d73a-7128-9bab-d75dd229c020` produced pilot elapsed + 609.215958000 seconds, central 2460.664462900 seconds, and envelope + 2745.098529500 seconds. Conditions were true/false/false and Gate 3 + remained FAIL. These are engineering/gate diagnostics and remain + ineligible for manuscript tables. +- **Scientific decision:** this is a logged post-pilot instrumentation + amendment with explicit gate-gaming risk. It is accepted before the next + corrected pilot because it fixes structural confounding, is more + conservative than the former optimistic selector, and does not reverse the + observed FAIL decision. The formula and pass rule may not be changed after + viewing the next pilot without a new logged protocol amendment and another + pilot. +- **Focused tests:** two projection tests passed first in 4.28 seconds and + again after independent review/formatting in 3.94 seconds under + `PYTHONDONTWRITEBYTECODE=1`. +- **Recorded validation failures/corrections:** the first ad hoc importlib + inspection omitted registering the temporary module in `sys.modules`, so + Python dataclass processing raised `AttributeError`; the corrected + inspection registered it and strictly loaded both archives. A broad + recursive search for an existing final plan timed out after 14 seconds; + it made no changes. The first Ruff format-check correctly reported the two + new files would be reformatted; formatting was applied. The next scoped + import check reported both new import blocks; mechanical import sorting + fixed them, after which format, import, tests, and `git diff --check` + passed. +- **Files/source identities:** structural projection + `sha256:d6126902dac9bddd868be57425fb1f29d0e1ec95cf4a9c8aa14cdfbaeb694afc`; + tests + `sha256:f31ecb3a89d5066fad342c952d95bb8b8186acf7b3dce1cdb6684a8263428c69`. +- **Cleanup/scope:** no live gateway, sandbox, Docker command, fixture, + package mutation, preset, or measurement corpus was run or changed. +- **Disposition:** the structural projection is now the fixed Gate 3 runtime + rule for the next corrected smoke/pilot pair. The old 06f52dfb projection + remains immutable historical failure evidence and cannot qualify the new + treatment. + +### 2026-07-31T03:33:17.889Z - Final combined offline backend validation before live qualification + +- **Entry ID:** `validation-exp1-pre-live-backend-078`. +- **Phase/kind:** EXP1-D pre-live complete backend validation; no live preset. +- **Recorded failed attempt:** the first complete-suite invocation used a + five-second shell timeout and exited 124 before completion. It was an + orchestration timeout, not a test failure and not a live benchmark attempt. +- **Corrected command/result:** with + `PYTHONDONTWRITEBYTECODE=1`, + `.venv\Scripts\python.exe -m pytest benchmark\backend\tests -q` exited 0: + 256 tests passed and five expected Windows symlink-privilege tests skipped + in 41.83 seconds. +- **Formatting/static result:** the two projection files pass cached + `uvx --offline ruff 0.16.1` format and scoped import checks. The complete + modified runner/gateway/observability/resource file set passed scoped import + checks after the recorded mechanical correction, and `git diff --check` + passed with only expected line-ending warnings. +- **Product postcondition:** product remains clean direct `main` at + `57b437ba60a2304e896e7345f13e6bc43f431f55`; exact package + `windows-exp1-57b437ba` remains unchanged. +- **Cleanup/scope:** no live gateway, sandbox, Docker command, fixture, + package mutation, preset, or measurement corpus was run. +- **Disposition:** all current paper-harness and analysis corrections pass + offline validation. The next authorized step is bounded live qualification + of the exact package/source, followed by a new smoke and exploratory pilot; + freeze/final remain prohibited until the new structural Gate 3 rule passes. + +### 2026-07-31T03:35:05.829Z - Exact plan validation and structural negative-control CLI + +- **Entry ID:** `validation-exp1-plans-structural-cli-079`. +- **Phase/kind:** EXP1-D pre-live static plan and analysis validation; no live + preset. +- **Recorded failed attempt:** the first three validation commands invoked + `python -m benchmark_lab.cli`. That module defines `main` but is not the + executable module, so each process exited 0 with zero stdout/stderr and did + not validate a plan. The six zero-byte files with the unsuffixed + `*-57b437ba-expanded` names are retained as failed orchestration evidence + and are ineligible inputs. +- **Corrected command:** the executable module is + `.venv\Scripts\python.exe -m benchmark_lab validate`, with the absolute + paper root, clean product root, and exact + `target\windows-exp1-57b437ba\bin` package. All three corrected `-v2` + commands exited 0 and wrote zero stderr bytes. +- **Smoke expansion:** 19 cells, 19 batches, 55 requests, zero warmups, one + measured trial per cell, `product_cli`, runnable with zero validation + findings. Plan hash + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + generated file + `experiments\analysis\paper-env-smoke-57b437ba-expanded-v2.json`, + `sha256:c3812cc033c067617a985282615e9672e8d9e9ec606046c68f1b4160bc3031fc`. +- **Pilot expansion:** 19 cells, 133 batches, 385 requests, two warmups and + five measured trials per cell, `product_cli`, runnable with zero validation + findings. Plan hash + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + generated file + `experiments\analysis\paper-pilot-57b437ba-expanded-v2.json`, + `sha256:3cacb95b7a8e0adaf52067db63c9741a727264b18dde3e966595a6770a129f6e`. +- **Final expansion:** 19 cells, 1,938 batches, 5,610 requests, two warmups and + 100 measured trials per cell, `product_cli`, runnable with zero validation + findings. Plan hash + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`; + generated reviewed-plan file + `experiments\analysis\paper-good-pass-57b437ba-expanded-v2.json`, + `sha256:6307cc2f485bdbef356d3b370f0eec4f9aa1534636493cb1216843cf7c5347f5`. +- **Plan stability:** the three canonical plan hashes and counts are unchanged + by the protocol-preserving speed corrections. No plan admits or falls back + to `direct_client` or `cli_e2e`. +- **End-to-end structural negative control:** the production projection CLI + consumed the immutable corrected smoke/pilot archives plus the reviewed + final expansion, exited 0, and emitted the already reviewed FAIL decision: + pilot 609.215958000 seconds, central 2460.664462900 seconds, observed + envelope 2745.098529500 seconds; true/false/false. Output + `experiments\analysis\pilot-final-runtime-structural-06f52dfb-negative.json`, + 44,498 bytes, + `sha256:5ac734611e301d964bc2de74ec768bca49ea2166f13aa0839fc5bf261f6ee20b`. +- **Eligibility:** these old smoke/pilot values and the projection are + engineering negative-control evidence only and remain ineligible for + manuscript tables. They do not qualify commit 57b437ba or its package. +- **Cleanup/scope:** plan validation refreshed only paper-local catalog/state + metadata and wrote the listed analysis artifacts. No live gateway, sandbox, + Docker command, fixture, package mutation, smoke, pilot, or final run + occurred. +- **Disposition:** static plan/product-boundary and structural CLI validation + passed. Proceed to bounded live qualification of the exact new package and + source; freeze/final remain prohibited. + +### 2026-07-31T03:38:29.404Z - Optimized-package live preflight + +- **Entry ID:** `preflight-exp1-optimized-package-080`. +- **Phase/kind:** EXP1-C pre-live qualification; no live preset or measurement + clock started. +- **Command-surface checks:** `python -m benchmark_lab --help`, + `python -m benchmark_lab run --help`, and + `experiments\scripts\archive_exp1_run.py --help` all exited 0 with + `PYTHONDONTWRITEBYTECODE=1`. The exact runner requires the paper root, + product root, package `bin` directory, and preset ID. +- **Qualified host:** native Windows AMD64 host `DESKTOP-OLP1ADS`, build + 26200, 48 logical CPUs, 137,438,953,472 bytes physical memory, and + 571,251,810,304 bytes free on `C:` at the check. +- **Qualified Docker/image:** client/server 29.0.1; Linux AMD64 server, + `overlayfs`, cgroup v2. Exact image + `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` + is present locally with the same image ID; no pull occurred. +- **Product/package:** clean direct `main` at + `57b437ba60a2304e896e7345f13e6bc43f431f55`. Exact package archive remains + 5,651,981 bytes with + `sha256:0d6a784a5447ef012d03d4b8742beac0693f30766e14c2bd9dc4c53e9c4cb359`. + Gateway, manager, runtime, and observability binary SHA-256 values remain, + respectively, + `526ba4e1b90b38d8605475b4ad1e53ded12eeef396cb533acfe0d1c90854dfaa`, + `032187d0d4b3827ddfc37594ddf3571e6d3510ad52a2d37d222cbbe7db63eac6`, + `3ada794fea0c6c1c1ed1705bf308d7f60f5fe45242e7ac93d12091c6c011d349`, + and + `406ca3fa7c74f9cccb5228fc20cb4c54374dc5472b15685bfc891728e0c5dfc3`. +- **Preserved baseline:** host gateway PID 62980 and the pre-existing stopped + `eos-gateway` container + `5ce30657bf836cc332141baf2697192d48ec4553cf41ecf103ef4f1b05175f85` + remain untouched. Five pre-existing `eos-gateway` volumes remain untouched: + two sandbox volumes for + `eos-76263ba1-97f9-475c-9b85-229ab45f87d4` and three validated + content-addressed shared-base volumes. +- **Recorded diagnostic failures:** two read-only Docker inspection attempts + used unsupported or incorrectly quoted Go-template expressions and emitted + template parse errors. They changed no state. A first timestamp command used + the unavailable Windows PowerShell `Get-Date -AsUTC` parameter and exited 1; + the corrected UTC conversion exited 0. +- **Corrected ownership inspection:** Docker JSON was parsed in memory and + projected to non-secret ID/name/status/ownership fields only. It confirmed + the single labeled container is exited and the five labeled volumes are the + preserved baseline above. Authentication labels were neither printed nor + logged. +- **Disposition:** the exact optimized source/package and qualified + environment pass the no-build/no-install/no-pull preflight. Freeze and final + remain prohibited until a fresh smoke and exploratory pilot pass Gate 3. + +### 2026-07-31T03:46:35.303Z - Optimized-package smoke failed at retained shared-base cleanup + +- **Entry ID:** `smoke-exp1-optimized-shared-base-cleanup-failure-081`. +- **Phase/kind:** EXP1-C live `paper-env-smoke`; failed/partial and permanently + ineligible. +- **Run identity:** `019fb640-f59a-71cb-a3dd-a6a0bd23d86e`; exact plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + exact clean product `main` commit + `57b437ba60a2304e896e7345f13e6bc43f431f55`; exact + `windows-exp1-57b437ba` package and qualified image from entry 080. +- **Clock/result:** started `2026-07-31T03:39:01.368982Z`, ended + `2026-07-31T03:40:02.983296Z`, elapsed 61.614314 seconds. The terminal + manifest state is `failed`, correctness is `fail`, and only one of 19 trial + batches and one product request completed. This attempt is not a passed + smoke and no value is eligible for a paper table or Gate 3 projection. +- **Successful product work:** the measured `create_sandbox` request exited 0 + with empty stderr; both observability boundaries, sandbox inspection, and + manager-CLI destroy exited 0. The single trial is internally recorded as + reportable/success with product success, two passed correctness checks, + correlated resources, and restored per-trial cleanup baseline. No automatic + retry occurred. +- **Failure boundary/root evidence:** `sandbox_lifecycle` and its first cell + reached `completed`, then campaign block cleanup failed before the next + family. A single new content-addressed shared-base volume, + `eos-shared-base-0be2ebdcdc1e0d39996e614a76461bd21a28e90b688ad3dd502dea995ca2b63d`, + remained. Its exact gateway label is + `benchmark-gateway-6ab02f5eb81af0965b7085aa0a964a28`, matching this + run's PID-46800 gateway ownership record; no container referenced it. This + is an infrastructure cleanup failure introduced by the pre-freeze + cross-gateway cache optimization, not a product request or correctness + failure. +- **Clock discipline:** no build, install, pull, test, source edit, + experiment-log edit, or environment reconfiguration occurred during the + live clock. A read-only recursive progress probe enumerated fixture paths + noisily and returned exit 1 after its bounded output; it changed no state + and was not repeated. +- **Cleanup:** the run-owned gateway exited. After strict name, gateway-label, + root-hash, readonly-label, and zero-container-reference checks, only the + exact leaked shared-base volume above was removed. The benchmark's exact + `cleanup --run-id` command then exited 0, removed the owned run workspace and + runtime directory, and preserved the immutable result store. The post-run + baseline is again only gateway PID 62980, one pre-existing exited + `eos-gateway` container, and five pre-existing `eos-gateway` volumes. + Product remains clean direct `main`. +- **Archive attempts:** archive attempts 1 and 2 each exited 1 with + `post-run cleanup proof failed`: the first captured the retained volume and + owned directories, and the second still captured the owned directories + before the benchmark cleanup command. After cleanup, attempt 3 exited 0 with + zero stderr and verified the failed corpus. +- **Retained failed archive:** absolute directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb640-f59a-71cb-a3dd-a6a0bd23d86e`, + 65 files, 4,385,878 bytes, content-tree + `sha256:8d9049af62474e93da6d525abac22fe6e790a5936c761c32ee3ecec8cb840eda`. + The initial live stdout/stderr and all three archive-attempt captures remain + under `tmp`; live stderr was empty. +- **Recorded diagnostic mistakes:** two safe event/observation shape probes + selected an incorrect nesting level before corrected reads, a gateway-log + probe tried to JSON-decode the product's `cli_log(...)` text and failed, and + one post-archive inspection mistakenly passed the archive directory to + `Get-FileHash` and then called a method on the null result. None changed + evidence or external state. +- **Disposition:** stop condition enforced. The failed smoke is preserved and + the final remains untouched. A paper-harness fix and tests are required + before a wholly new smoke; the pre-registered cells, trials, metrics, + stopping rules, and projection model remain unchanged. + +### 2026-07-31T03:49:53.652Z - Shared-base Docker label cleanup amendment + +- **Entry ID:** `amendment-exp1-shared-base-dotted-labels-082`. +- **Phase/kind:** EXP1-C pre-freeze paper-harness correction after failed smoke + 019fb640-f59a-71cb-a3dd-a6a0bd23d86e; no live preset. +- **Demonstrated cause:** the product's Docker provider and the retained live + volume use `eos.shared_base.root_hash`, `eos.shared_base.target`, and + `eos.shared_base.readonly`. The newly added paper-harness cleanup validator + instead looked for underscore spellings + `eos.shared_base_root_hash`, `eos.shared_base_target`, and + `eos.shared_base_readonly`. It therefore failed closed on the valid + content-addressed volume during intermediate retention and again during the + final sweep. +- **Correction:** changed only those three paper-harness label literals in + `benchmark/backend/benchmark_lab/gateway.py`. Updated the existing + cross-gateway final-cleanup and ambiguous-volume fixtures in + `benchmark/backend/tests/integration/test_gateway_lifecycle.py` to use the + released product's exact dotted label contract. The regression now would + fail with the old harness constants and proves that intermediate cleanup + retains the validated shared base while the final sweep removes it through + its original gateway owner. +- **Protocol impact:** none. Product source/package, plan cells/counts/order, + trials, timing boundaries, metrics, exclusions, resource sampling, fixture, + image, stopping rules, and the pre-registered structural projection model + are unchanged. This is a justified pre-freeze cleanup-contract defect fix. +- **Focused validation:** gateway-lifecycle plus runner integration tests + exited 0: 45 passed and one expected Windows symlink-privilege skip in + 5.92 seconds. +- **Full validation:** complete backend suite exited 0: 256 passed and five + expected Windows symlink-privilege skips in 42.67 seconds. +- **Formatting/static validation:** the first scoped Ruff format check exited + 1 because the three patched lines needed mechanical newline normalization; + import checks and `git diff --check` passed. Cached offline Ruff 0.16.1 then + formatted the file, and both format and import checks exited 0. The + post-format gateway-lifecycle suite exited 0: 13 passed and one expected + Windows skip in 4.38 seconds. +- **Independent blocker audit:** a read-only subagent independently traced the + product constants in `sandbox-provider-docker/src/labels.rs` and their + runtime emission, confirmed the old harness literals made both retain-mode + cleanup and final cleanup reject the same valid volume before `volume rm`, + and found no second cleanup defect. It recommended no retry or backoff + change; the strict ownership/name/root/target/readonly checks remain + fail-closed. +- **Disposition:** offline correction is green. A wholly new unique smoke, + followed by a wholly new exploratory pilot if smoke passes, is required. + The failed smoke remains archived and ineligible; freeze/final remain + prohibited. + +### 2026-07-31T03:51:32.784Z - Corrected-smoke fast preflight + +- **Entry ID:** `preflight-exp1-corrected-smoke-083`. +- **Phase/kind:** EXP1-C fast preflight after the dotted-label amendment; no + live clock started. +- **Plan validation:** `benchmark_lab validate` exited 0 with zero stderr and + zero findings. `paper-env-smoke` remains runnable `product_cli`, 19 cells, + 19 trial batches, 55 issued requests, and plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. + A first read of the successful JSON selected nonexistent nested properties + and therefore displayed null/one placeholders; corrected direct top-level + selectors confirmed the values above. The retained validation output is + `tmp\paper-env-smoke-after-label-fix.validate.json`. +- **Environment drift check:** native host `DESKTOP-OLP1ADS` build 26200; + Docker client/server 29.0.1, Linux AMD64, `overlayfs`, cgroup v2; exact + qualified image ID; 570,993,184,768 bytes free on `C:`. +- **Treatment drift check:** product remains clean direct `main` at + `57b437ba60a2304e896e7345f13e6bc43f431f55`; package archive remains + `sha256:0d6a784a5447ef012d03d4b8742beac0693f30766e14c2bd9dc4c53e9c4cb359`; + staged config remains + `sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Cleanup baseline:** only the protected host gateway PID 62980, one + pre-existing exited labeled container, and five pre-existing labeled + volumes exist. No resource from failed run + 019fb640-f59a-71cb-a3dd-a6a0bd23d86e remains. +- **Disposition:** all corrected-smoke prerequisites pass. Start exactly one + new unique smoke; freeze, pilot, and final remain prohibited until its + terminal cleanup and archive pass. + +### 2026-07-31T04:01:59.138Z - Corrected optimized-package smoke passed + +- **Entry ID:** `smoke-exp1-corrected-optimized-pass-084`. +- **Phase/kind:** EXP1-C live `paper-env-smoke`; accepted smoke evidence but + ineligible for manuscript tables. +- **Run identity:** `019fb64d-4841-742e-9e70-115c7f8d378c`; exact clean + product commit `57b437ba60a2304e896e7345f13e6bc43f431f55`; exact package, + image, fixture protocol, and unchanged plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Terminal result:** manifest `completed`, correctness `pass`, report ready; + 19/19 trial batches, 55 issued product operation requests, zero failures, + zero warnings. Every product subprocess record has return code 0 and empty + required stderr; all operation-specific checks, correlations, and + per-trial cleanup proofs passed. +- **Campaign timing:** manifest start + `2026-07-31T03:52:28.413517Z`, end + `2026-07-31T03:56:29.401572Z`, exact manifest elapsed 240.988055 seconds. + The redirected stdout file existed from + `2026-07-31T03:52:25.3093494Z` until its terminal write at + `2026-07-31T04:00:25.5517553Z`, a 480.2424059-second file-lifetime bracket. + The post-manifest portion was the required recursive deletion of 19 deep + NTFS `paper-100m` cell copies. This outer cleanup overhead is engineering + evidence only and is not inserted into an operation-latency distribution. +- **Outer CLI observation:** the process watcher acquired the already-running + Windows process and confirmed termination, but Windows `Get-Process` + returned a null `ExitCode` for that externally reopened handle. Terminal + stdout is valid JSON, stderr is zero bytes, and the CLI's persisted terminal + state is completed. This outer-wrapper capture limitation does not affect + the recorded return code of any product CLI subprocess; an explicit + exit-status sidecar wrapper will be used for the pilot and final. +- **Cleanup:** after the report was ready, the runner removed the complete + owned run workspace and runtime directory. Post-run state is exactly the + protected baseline: gateway PID 62980, one pre-existing exited labeled + container, and five pre-existing labeled volumes. Product remains clean + direct `main`; no run-owned process, container, volume, registry state, or + workspace remains. +- **Archive:** `archive_exp1_run.py` exited 0 with zero stderr and verified + the corpus at + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb64d-4841-742e-9e70-115c7f8d378c`. + The archive has 1,175 files, 30,036,537 bytes, and content-tree + `sha256:746b71ded2733c2ad7ce1572ee8731c590e432bf199e9684b9f2aab3384c3471`. +- **Clock discipline:** no build, install, pull, source edit, test, log edit, + or environment reconfiguration occurred during the live clock. Read-only + progress checks inspected only committed event records and process state. +- **Disposition:** EXP1-C smoke passes for the corrected pre-freeze source. + Proceed to one wholly new five-sample exploratory pilot after a strict fast + preflight. Freeze/final remain prohibited pending Gate 3 projection and + anomaly review. + +### 2026-07-31T04:03:13.782Z - Five-sample pilot fast preflight + +- **Entry ID:** `preflight-exp1-five-sample-pilot-085`. +- **Phase/kind:** EXP1-D strict fast preflight; no pilot clock started. +- **Outer exit capture:** added the paper-local temporary orchestration helper + `tmp\run-benchmark-with-exit-capture.ps1`. It starts the exact Python CLI + with argument arrays, waits on its own process handle, emits the benchmark + exit code in a sidecar, and exits with that code. PowerShell syntax parsing + passed; helper + `sha256:7091ecdcee6b69ef471e0322aae4f28e1a5e61e776b80c0f04f84fabb9b6aa4f`. + It changes no benchmark behavior, timing boundary, plan, or raw evidence and + writes only outer orchestration stdout/stderr after the campaign command. +- **Pilot plan:** `benchmark_lab validate` exited 0 with zero stderr and zero + findings. `paper-pilot` remains runnable `product_cli`, 19 cells, 133 trial + batches, 385 requests, two warmups plus five measured trials per cell, no + warnings, and exact plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Smoke prerequisite:** independent archive verification exited 0 with zero + stderr and reconfirmed corrected-smoke content-tree + `sha256:746b71ded2733c2ad7ce1572ee8731c590e432bf199e9684b9f2aab3384c3471`. +- **Environment/treatment:** native host `DESKTOP-OLP1ADS` build 26200; + Docker 29.0.1 Linux AMD64, `overlayfs`, cgroup v2; exact qualified image; + 570,920,165,376 bytes free on `C:`. Product is clean direct `main` at + `57b437ba60a2304e896e7345f13e6bc43f431f55`; package archive remains + `sha256:0d6a784a5447ef012d03d4b8742beac0693f30766e14c2bd9dc4c53e9c4cb359`. +- **Cleanup baseline:** only the protected gateway PID 62980, one pre-existing + exited labeled container, and five pre-existing labeled volumes exist. +- **Disposition:** every EXP1-D start condition passes. Run one new complete + exploratory pilot. It is ineligible for manuscript tables regardless of + outcome; freeze/final remain prohibited until its cleanup, archive, + deterministic regeneration, structural projection, and anomaly review pass. + +### 2026-07-31T04:17:54.579Z - Fresh pilot passed; structural projection still blocks Gate 3 + +- **Entry ID:** `exp1-57b437ba-pilot-analysis-blocker-086`. +- **Phase/kind:** EXP1-D live exploratory pilot, immutable archival, + deterministic regeneration, and preregistered structural Gate-3 decision. + Pilot evidence is permanently ineligible for manuscript tables. +- **Run identity:** `019fb657-802d-78d4-9245-518de0de50d7`; clean product + `main` commit `57b437ba60a2304e896e7345f13e6bc43f431f55`; exact optimized + package/image; plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Terminal result:** outer benchmark exit code 0 captured by the exact + sidecar wrapper; terminal manifest `completed`, correctness `pass`, report + ready; 19/19 cells, 133/133 trial batches, 385 product requests, 95 measured + trials, zero failures, and zero warnings. All request, correctness, + observability, correlation, and per-trial cleanup requirements passed. +- **Timing:** manifest start `2026-07-31T04:03:38.098552Z`, end + `2026-07-31T04:08:56.546172Z`, exact elapsed 318.447620 seconds. The + redirected command-output file lifetime, including the complete recursive + NTFS workspace deletion, was 580.2188861 seconds. Both observed bounds are + under 1,200 seconds, but pilot duration alone is not the final projection. +- **Cleanup:** the runner removed the entire run workspace and runtime + directory. Product remains clean direct `main`; only protected gateway PID + 62980, the one pre-existing exited labeled container, and five pre-existing + labeled volumes remain. No pilot-owned process, container, volume, registry + state, or workspace remains. +- **Immutable archive:** archiver exited 0 with zero stderr and verified + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb657-802d-78d4-9245-518de0de50d7`. + It contains 6,605 files and 181,314,655 bytes with content-tree + `sha256:4a2fc3c3c2a79f33661ddc1ec6085d1589b7af99f582ae188e35e58f9ebd9362`. +- **Deterministic tables:** generator + `sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3` + exited 0 twice with zero stderr using only the immutable pilot archive. + Outputs + `experiments\analysis\pilot-57b437ba-019fb657-tables-a` and `-b` are + byte-identical: nine files, 228,902 bytes, content-tree + `sha256:1ea56e99abc227a8a0319091f5df4548f70dd4a56d96d5ce936010c9cd1cee72`. + Both explicitly record `exploratory_ineligible` and + `ai-research-writing/numeric-evidence-v2`. +- **Table component hashes:** environment + `9741d1de397004ad249dbd0e0b6270dd91663999e81aaa127ce0a6c79e5a07a0`; + startup + `ee336c13149c0ec064921eac48a91859763ac2f455a1b5f02e068b7d9a4644f1`; + CLI operations + `b3fca8615ec9359c872f6a58bae620e81e5bdd2f0e7518b9fa29c747712c2bd2`; + resources + `e41cb7a37a4953c87db6e34f6140493bd6547a5ca2b0023b59b6c653ddcef5e7`; + tables JSON + `c97fa723fb24fed3b8f25143faa6e47b9ef8bbbf71f2153b8b02cafae78d62cf`; + numeric evidence + `8ddd30b60810648db1741ec19180e0079010fcd3c01cf0394938256e7cca8cad`; + numeric provenance + `491040b0a4f3d74b2a964dcf69d16a7c3989e7165ea93f8da8a7a6cf8835043c`. +- **Recorded analysis mistake:** the first canonical tree-comparison attempt + passed quoted Python source through native PowerShell argument parsing; + quotes were stripped and it exited 1 with `NameError`. No output directory + or evidence changed. The corrected temporary helper invoked the existing + `archive_inventory` function and proved the two complete output inventories + identical. +- **Structural projection:** preregistered script + `sha256:d6126902dac9bddd868be57425fb1f29d0e1ec95cf4a9c8aa14cdfbaeb694afc` + consumed only the fresh verified smoke/pilot archives and the reviewed exact + final plan, exited 0 with zero stderr, and emitted + `experiments\analysis\pilot-final-runtime-structural-57b437ba-019fb657.json`, + 44,406 bytes, + `sha256:4450cda5eabbc534163cd38909d30b42b3d865adde04fccde478a6190329259c`. +- **Gate-3 result:** pilot elapsed 318.447620000 seconds is within the limit, + but the exact central structural projection is 1455.634228525 seconds and + the observed envelope is 1604.045005800 seconds. Pass conditions are + true/false/false; `gate_3_runtime_pass: false`; decision + `block_freeze_and_final_runtime_projection_exceeds_20_minutes`. +- **Clock discipline:** no build, install, pull, test, source edit, log edit, + or environment reconfiguration occurred during the pilot clock. All + analysis and log mutation began only after command exit and cleanup. +- **Disposition/blocker:** Gate 3 remains FAIL. Freeze, local commit/tag, and + the exactly-one eligible final pass remain untouched and prohibited. + Three parallel read-only audits are decomposing exact contributors and + protocol-preserving harness/product speed options. Any adopted correction + requires offline tests and another entirely fresh smoke/pilot before Gate 3 + can be reconsidered. + +### 2026-07-31T04:31:11.025Z - Gate-3 decomposition and one-shot CLI runtime correction + +- **Entry ID:** `exp1-gate3-cli-current-thread-correction-087`. +- **Phase/kind:** post-pilot read-only decomposition, offline microbenchmark, + and pre-freeze product-source correction. No smoke, pilot, freeze, tag, or + final clock was active. +- **Exact blocker decomposition:** the immutable pilot remains functionally + healthy, but the frozen structural model remains 1455.634228525 seconds + central and 1604.045005800 seconds envelope, requiring reductions of + 255.634228525 and 404.045005800 seconds. Exact central components are + 173.6189496 seconds cell-fixed, 84.0605484 seconds warmup active, + 1189.184050 seconds measured active, 4.480688825 seconds gaps, and + 4.2899917 seconds run/family fixed. The final remains untouched. +- **Resource-boundary evidence:** the archived pilot contains 457 cgroup and + 457 snapshot CLI calls. Across the 126 non-create trials, 92 baselines + completed in one attempt and 34 required two; only four post boundaries + completed in one attempt and 122 required two. Each paired attempt is about + 37--43 milliseconds median, while the product ring advances about 106 + milliseconds. Mandatory baseline plus post-boundary wall therefore offers + 539.521980 seconds of final-scaled central opportunity, but a true fresh + synchronous product boundary is required for a robust 50--75 millisecond + target; merely changing the fixed 100-millisecond retry sleep is not enough. +- **Other exact opportunities:** mutation verification has 41--48 seconds of + final-scaled independent-read overlap; repeated fresh-copy cell preparation + consumed 83.848695 seconds of pilot leading wall while still requiring 19 + distinct fresh copies; per-invocation paper CLI evidence durability has a + diagnostic upper opportunity of about 121 final seconds but needs a focused + microbenchmark. No protocol decision was changed. +- **Parallel product audit:** all three released one-shot CLIs used default + multi-thread `tokio::main` on this 48-logical-processor host even though each + process parses and sends one request. Final-mode evidence projects about + 30,170 such launches. The narrow product correction changes only + `sandbox-manager-cli`, `sandbox-runtime-cli`, and + `sandbox-observability-cli` to Tokio's `current_thread` flavor. Request + schemas, CLI timing, transport, concurrency across subprocesses, resource + cadence, and lifecycle behavior are unchanged. +- **Offline before/after diagnostic:** 60 release `--help` launches per binary + against the immutable `57b437ba` package had medians + 9.869/9.882/9.581 milliseconds and p95 + 10.927/11.046/10.724 milliseconds for manager/runtime/observability. + Locally rebuilt corrected binaries had medians + 6.277/6.534/6.388 milliseconds and p95 + 6.954/7.482/7.429 milliseconds. Means had unrelated host outliers and are + not used as a performance claim. This diagnostic is exploratory and + ineligible for paper tables. +- **Validation:** `cargo fmt --check`, `cargo test -p sandbox-cli`, and + `cargo clippy -p sandbox-cli --all-targets -- -D warnings` all exited 0. + Feature-specific locked release builds for all three corrected executables + exited 0. +- **Recorded analysis/validation mistakes:** initial post-pilot factor joins + addressed `plan.cells` instead of the envelope's `plan.data.cells` and were + rerun against the correct schema. A first PowerShell resource aggregation + had a trailing comma in `Sort-Object` and did not parse; the corrected + aggregation produced the counts above. The first offline launch benchmark + used a direct `foreach` pipeline that PowerShell rejected as an empty pipe + element; the array-wrapped rerun passed. A one-second combined Cargo command + timed out before validation and was rerun with an adequate bound. A generic + `cargo build --release -p sandbox-cli --bins` was a no-op because every + binary is feature-gated; the three exact feature-specific builds succeeded. + `Get-Date -AsUTC` was unsupported in this PowerShell version; the timestamp + was captured with `DateTime.UtcNow`. None of these failed attempts changed + evidence, product state beyond the three intended source lines, or the live + environment. +- **Disposition:** retain the three-line product optimization, complete + proportional product validation, commit it directly on authorized local + `main`, and produce a new immutable Windows package. Because product source + changed, the prior smoke and pilot remain archived exploratory evidence only; + an entirely fresh smoke and pilot are required before Gate 3 can be + reconsidered. Freeze, tag, and final remain prohibited. + +### 2026-07-31T04:38:35.158Z - Current-thread CLI commit and immutable Windows package + +- **Entry ID:** `exp1-product-cli-current-thread-package-088`. +- **Phase/kind:** offline product validation, authorized direct-`main` local + commit, and immutable Windows package construction. No live benchmark clock, + freeze, tag, or final attempt was active. +- **Product commit:** the three-line one-shot CLI runtime correction was + committed directly on clean local `main` as + `37fdbcfeab2400e2b3741fd9993392d3ecfe7537` + (`Reduce one-shot CLI runtime startup cost`). No branch, worktree, push, or + unrelated product file was created or changed. Product status is clean. +- **Focused validation:** all-feature `sandbox-cli` tests passed 58 tests + across compatibility, help, manager, observability, projection, request + building, and runtime suites. All-feature `sandbox-cli` Clippy with warnings + denied passed. The earlier no-feature format/test/Clippy checks and all three + feature-specific locked release builds also passed. +- **Platform-scoped workspace check:** `cargo test --workspace` was attempted + proportionally and failed during compilation because + `sandbox-observability-telemetry` imports `rustix::fs`, which is deliberately + gated out by `rustix` on Windows. This pre-existing cross-platform workspace + incompatibility is unrelated to the three changed binary attributes; no + test executed or product file changed. The exact applicable Windows CLI + feature suites above pass. +- **Package:** exact directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-37fdbcfe` + and archive `windows-exp1-37fdbcfe.zip`; 5,623,169 archive bytes; + archive + `sha256:3d2d876ea881fc529acf1c7cc80d549f03bcb7803e52143d81b413b549a56d66`. + The package script rebuilt the Windows gateway and the three exact + feature-gated release CLIs from the committed tree. +- **Packaged component hashes:** gateway + `526ba4e1b90b38d8605475b4ad1e53ded12eeef396cb533acfe0d1c90854dfaa`; + manager CLI + `564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed`; + runtime CLI + `c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8`; + observability CLI + `25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26`; + Linux daemon + `f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`; + configuration + `987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Recorded packaging mistakes:** direct invocation of the repository + PowerShell package script was rejected by the host execution policy before + it ran; the exact rerun used `powershell.exe -NoProfile -ExecutionPolicy + Bypass` and passed. A first component-hash display used a direct `foreach` + pipeline that PowerShell rejected before evaluation; the array-wrapped + read-only rerun passed. No partial package existed before the successful + script, and both exact targets were confirmed absent before its scoped + cleanup/create step. +- **Disposition:** this package is qualified for the next smoke only if no + additional product source correction is adopted. Paper-harness source is + still being optimized before that smoke, so no live run has begun. Freeze, + tag, and final remain prohibited. + +### 2026-07-31T04:48:19.638Z - Protocol-preserving harness runtime amendments validated + +- **Entry ID:** `exp1-gate3-harness-runtime-amendments-089`. +- **Phase/kind:** offline pre-freeze paper-harness optimization and validation. + No benchmark, freeze, tag, or final clock was active. +- **Resource-boundary correction:** mandatory post-response product-boundary + sampling now waits one qualified 100-millisecond product-ring cadence before + its first CLI poll. Periodic sampling remains exactly 100 milliseconds and + the existing fail-closed retry/readiness contract remains intact. This + removes the pilot's almost-certain stale first post-boundary observation + without weakening freshness. +- **Mutation verification overlap:** independent content reads and product + attribution checks now execute concurrently behind the existing complete + failure-gathering boundary. Tests prove the independent reads overlap and + that simultaneous branch failures are both retained. +- **Fixture materialization:** all contextual cell workspaces are still fresh, + distinct, fully verified copies, but they are materialized once in a bounded + four-worker batch before the first execution block. The source inventory is + validated once before the batch and again after it; exact destination + inventory, manifest, inode independence, and the eight-worker hard cap are + tested. The existing single-workspace API is unchanged. +- **CLI evidence durability:** paper `product_cli` now exposes an explicit + per-trial evidence transaction. Redacted stdout/stderr projections are + staged without per-command durable metadata writes; exact schema-2 marker + bytes are sorted and durably committed with bounded concurrency before the + trial journal commits. Immediate durable behavior remains unchanged outside + a trial, flush failures stay pending and fail closed, cleanup after the + boundary is immediate, and archive compatibility is unchanged. +- **Diagnostic microbenchmark:** 12 repetitions over a 256-KiB evidence case + measured 61.652 milliseconds median for immediate durability versus 26.895 + milliseconds stage plus 9.999 milliseconds boundary commit, or 36.894 + milliseconds total and 1.67x exploratory speedup. The resulting evidence + trees were byte-identical. This diagnostic is ineligible for paper tables. +- **Exact amended implementation hashes:** fixtures + `sha256:5edd3f2ed1c23fd0305d519a6066a48de77720156d4a64937595d03d75cf8903`; + product CLI + `sha256:44dc001cdd51b137d084412431f6f9f3d7981e196788acd2418a7dd737ceda05`; + resource sampling + `sha256:b45c7f5063a60aa052056adcf7d403f1677847348cc01c1b59470614a80cdbf7`; + runner + `sha256:453e7dcdd11e9a35d097e8369341f5edd8231c87dcbe08e458afdae315994396`. +- **Validation:** focused product-CLI and runner tests passed 54 tests; fixture + tests passed 24 with one expected Windows symlink-privilege skip; resource + tests passed 16; archive compatibility passed six. After integrating all + four amendments, the complete backend suite passed 275 tests with five + expected Windows symlink-privilege skips in 48.28 seconds. Ruff 0.16.1 + formatting and import checks pass on all eight touched implementation/test + files. +- **Recorded validation mistakes:** a first direct `uv run --offline ruff` + attempt failed before linting because Ruff is not installed in the project + virtual environment. The cached exact Ruff 0.16.1 executable was located. + A first inspection command used paths relative to the paper backend while + spelling them from the benchmark root and therefore found no files; the + corrected read-only inspection passed. A whole-backend Ruff invocation then + exposed the repository's broad pre-existing unformatted/lint baseline; it + changed nothing. The exact touched-file invocation identified four files + needing format normalization and three import blocks; the mechanical + correction followed by repeated exact checks passed. No live evidence or + environment state changed. +- **Disposition:** retain these amendments. Product `create_sandbox` transport + optimization is completing separate validation; after its exact commit and + immutable package, run an entirely fresh smoke and pilot. Gate 3 remains + blocked until that new pilot's preregistered structural projection passes. + Freeze, tag, and final remain prohibited. + +### 2026-07-31T04:50:55.035Z - Gateway daemon-forwarding optimization committed and packaged + +- **Entry ID:** `exp1-product-daemon-forwarding-package-090`. +- **Phase/kind:** offline product correction, validation, authorized + direct-`main` commit, and immutable Windows package construction. No live + benchmark, freeze, tag, or final clock was active. +- **Product correction:** `TcpSandboxDaemonClient` no longer creates and joins + one operating-system thread and constructs/tears down one current-thread + Tokio runtime for every forwarded daemon request. It performs the same + authenticated newline-framed exchange synchronously with `std::net` inside + the manager router's existing `spawn_blocking` boundary. +- **Preserved protocol/safety:** the 16-MiB response bound, authentication, + newline termination, response decoding, error surface, and request timeout + remain fail closed. One monotonic total deadline now spans numeric endpoint + resolution, connect attempts, partial writes, shutdown, and partial reads; + remaining read/write timeout is reset before every syscall. Synchronous DNS + itself cannot be interrupted, but elapsed time is checked immediately after + it and the qualified endpoint is the numeric `127.0.0.1`. +- **Tests/review:** six focused tests cover successful authenticated framing, + a stalled response, a drip-fed response that must not reset the total + deadline, missing newline, oversized response, and eight independent + concurrent calls. Root independently reran `cargo fmt --all -- --check`, + those six tests, focused library/test Clippy with warnings denied, and + `git diff --check`; all exited 0. The exact forwarding call site was + independently confirmed to run under `tokio::task::spawn_blocking`. +- **Broader pre-existing checks:** `cargo test -p sandbox-gateway` reaches the + existing `gateway_streams_create_sandbox_progress_before_final_response` + assertion failure, which uses `RecordingDaemonClient`, not this TCP client. + All-target gateway Clippy reaches unchanged unused items in + `tests/local_daemon_installer.rs`. A running protected gateway locks the + default debug executable, so the focused checks used separate scoped Cargo + target directories and did not stop or alter it. +- **Product commit:** two scoped files were committed directly on authorized + local `main` as + `214cea7a9926abfc5c5e3d681946949eda040ab6` + (`Reduce gateway daemon forwarding overhead`). Product is clean and + `main` is eleven commits ahead of `origin/main`; there was no branch, + worktree, push, or unrelated product change. +- **Immutable package:** exact directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-214cea7a` + and 5,615,826-byte archive `windows-exp1-214cea7a.zip`; archive + `sha256:0d8dc9802048d2db99f5d47c2e4d136a55130f749042d7d4f71fac6e6b828852`. + The repository package script rebuilt the exact locked release gateway and + three feature-gated one-shot CLIs from the committed tree. +- **Packaged component hashes:** gateway + `6f40fefd527cac17212470373678a69def55b4f89f4741388454c69b069494a7`; + manager CLI + `564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed`; + runtime CLI + `c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8`; + observability CLI + `25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26`; + Linux daemon + `f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`; + configuration + `987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Recorded packaging mistake:** the first read-only package-script lookup + guessed a nonexistent `scripts` directory and produced an `rg`/file-not-found + diagnostic. The correct tracked script was then located under `bin`; exact + stage/archive absence was confirmed before its scoped create step, and the + package and checksum verification passed. +- **Disposition:** use this exact clean product commit and immutable package + for a fresh post-correction smoke. If smoke and archive checks pass, run one + new exploratory five-sample pilot and the preregistered structural + projection. Gate 3 remains blocked until that projection passes; freeze, + tag, and final remain prohibited. + +### 2026-07-31T04:53:21.433Z - Post-optimization smoke fast preflight passed + +- **Entry ID:** `preflight-exp1-214cea7a-smoke-091`. +- **Phase/kind:** EXP1-C strict fast preflight; no live benchmark clock + started. +- **Offline qualification:** the combined paper backend passed 275 tests with + five expected Windows symlink-privilege skips in Entry 089. Exact touched + files pass Ruff 0.16.1 formatting/import checks and paper `git diff --check` + passes with only the repository's expected line-ending notices. +- **Plans:** fresh `benchmark_lab validate` subprocesses exited 0 with empty + stderr. `paper-env-smoke` is runnable/customized with zero findings and + warnings, only `product_cli` and `paper-100m`, exactly 19 cells, 19 trial + batches, 55 issued requests, and plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. + `paper-pilot` remains runnable/customized with zero findings and warnings, + the same 19 cells/cohort/profile, 133 batches, 385 requests, and plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Command surface:** Python is 3.13.14. The exact packaged gateway, manager, + runtime, and observability `--help` invocations all exited 0 with empty + stderr. Outer exit-capture helper remains + `sha256:7091ecdcee6b69ef471e0322aae4f28e1a5e61e776b80c0f04f84fabb9b6aa4f`; + archive script + `sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e`. +- **Treatment:** product is clean direct local `main` at + `214cea7a9926abfc5c5e3d681946949eda040ab6`. The exact package remains + 5,615,826 bytes and + `sha256:0d8dc9802048d2db99f5d47c2e4d136a55130f749042d7d4f71fac6e6b828852`; + gateway/manager/runtime/observability hashes remain + `6f40fefd527cac17212470373678a69def55b4f89f4741388454c69b069494a7`, + `564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed`, + `c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8`, + and + `25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26`. +- **Qualified host:** `DESKTOP-OLP1ADS`, Windows build 26200, 64-bit, 48 + logical processors, 137,438,953,472 physical-memory bytes, and + 565,626,413,056 free bytes on `C:`. +- **Qualified Docker/image:** client/server 29.0.1, Linux AMD64, + `overlayfs`, cgroup v2. Exact pinned Ubuntu image ID/digest + `sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` + is present locally; no pull was issued. +- **Preserved baseline:** protected debug gateway PID 62980 remains the only + sandbox-named process. The one pre-existing labeled container remains + exited with exact ID + `5ce30657bf836cc332141baf2697192d48ec4553cf41ecf103ef4f1b05175f85`; + the two preserved sandbox volumes and three content-addressed read-only + shared-base volumes remain the only five labeled volumes. Benchmark runtime + is empty. Eight older terminal run records are retained unchanged as + pre-existing evidence. +- **Recorded inspection mistakes:** the first successful plan-summary display + selected nonexistent nested fields and PowerShell's null-array conversion + misleadingly printed one finding/warning. Direct schema inspection proved + `validation` is an empty top-level array and warnings are the empty + `estimates.warnings` array. A first corrected two-plan display used a bare + `foreach` pipeline that this PowerShell rejects as an empty pipe element; + the array-wrapped read-only rerun printed the exact zero/count values above. + Neither attempt changed plans, evidence, or environment state. +- **Clock discipline/disposition:** from the next smoke command start through + its terminal artifact/cleanup production, no build, test, install, pull, + source/log edit, dependency mutation, or environment reconfiguration is + permitted. Exactly one fresh `paper-env-smoke` is authorized. Pilot, freeze, + tag, and final remain prohibited until smoke is archived and audited. + +### 2026-07-31T05:04:27.313Z - Post-optimization CLI integration smoke passed + +- **Entry ID:** `smoke-exp1-214cea7a-pass-092`. +- **Phase/kind:** EXP1-C live `paper-env-smoke`; accepted qualification + evidence and permanently ineligible for manuscript tables. +- **Run/identity:** run `019fb685-9524-7555-ba91-7dac3008ae7e`; clean product + commit `214cea7a9926abfc5c5e3d681946949eda040ab6`; exact package/binaries, + pinned image, fixture protocol, and plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937` + from Entry 091. +- **Captured process result:** the exact sidecar wrapper recorded benchmark + exit code 0 and completed at `2026-07-31T05:02:09.3739052Z`. Benchmark and + wrapper stderr are both zero bytes. The outer wrapper lifetime from launch + timestamp was 494.5649052 seconds and includes complete recursive deletion + of all 19 fresh 100-MiB workspace trees. +- **Terminal corpus:** manifest start + `2026-07-31T04:53:58.660663Z`, end + `2026-07-31T04:57:56.944424Z`, exact elapsed 238.283761 seconds. State is + `completed`, correctness `pass`, report ready/non-provisional; 19/19 cells + and trial batches, 55/55 issued product requests, zero failures, and zero + warnings. +- **CLI evidence:** 339 exact schema-2 subprocess records have 339 unique + request IDs, return code 0, passed response validation, zero stderr bytes, + and `metadata-packed-payload-fsync-v1`. The independent archive CLI summary + agrees, and authentication redaction passed. +- **Correctness/resources:** every requested correctness check passed. All + unavailable resource fields remain explicitly unavailable rather than zero; + the expected Windows allocated-block-count and unreported LayerStack + limitations are retained. Mandatory correlated resource windows completed. +- **Cleanup:** the exact run workspace and runtime scope are absent; archive + cleanup proof reports no run/gateway-labeled container or volume and no + matching process, with clean exact product `main`. Independent terminal + checks found only the one pre-existing exited labeled container, five + pre-existing labeled volumes, and protected debug gateway PID 62980. + Explicit benchmark cleanup exited 0 with `cleaned: true`. +- **Immutable archive:** archive creation and an independent verify-only + invocation both exited 0 with empty stderr and exact agreement at + `experiments\runs\019fb685-9524-7555-ba91-7dac3008ae7e`: 1,073 files, + 29,744,883 bytes, content-tree + `sha256:35336d8e6112d294e4c888c645f837ecc2d6a7ff16fee9782c77a6a59a20b023`. + Disposition is `smoke` and eligibility `qualification_only`. +- **Clock discipline:** no build, test, install, pull, source/log edit, + dependency mutation, or environment reconfiguration occurred during the + live command. Progress inspection read only event/manifest/process state. + One early progress probe recursively counted the still-materializing run + tree and therefore added avoidable disk reads; it was immediately + discontinued. Because this is non-quantitative smoke evidence, it does not + affect a paper estimate, but no such probe is permitted during pilot/final. +- **Recorded inspection mistakes:** a transient process exited while a + read-only process list sorted by `StartTime`, producing one harmless + property-read diagnostic. After archival, an inspection accidentally + serialized the complete archive manifest and produced oversized console + output; no file changed. A first independent CLI metadata aggregation + selected nonexistent `.data.return_code` fields and thus misclassified null + values; direct schema inspection and the corrected top-level selectors + proved zero nonzero exits or validation failures. +- **Disposition:** EXP1-C passes for the exact `214cea7a` treatment. One fresh + exploratory five-sample `paper-pilot` is authorized after a strict fast + preflight. Freeze, tag, and final remain prohibited until pilot archival, + deterministic regeneration, preregistered structural runtime projection, + anomaly review, and cleanup pass Gate 3. + +### 2026-07-31T05:05:15.538Z - Post-optimization five-sample pilot preflight passed + +- **Entry ID:** `preflight-exp1-214cea7a-pilot-093`. +- **Phase/kind:** EXP1-D strict fast preflight; no live pilot clock started. +- **Smoke prerequisite:** independent verify-only invocation exited 0 with + empty stderr and reconfirmed accepted smoke + `019fb685-9524-7555-ba91-7dac3008ae7e`: 1,073 files, 29,744,883 bytes, + content-tree + `sha256:35336d8e6112d294e4c888c645f837ecc2d6a7ff16fee9782c77a6a59a20b023`. +- **Plan:** fresh `paper-pilot` validation exited 0 with empty stderr. It is + runnable/customized with zero findings/warnings, only `product_cli` and + `paper-100m`, 19 cells, two warmups plus five measured trials per cell, + 133 batches, 385 issued product requests, and plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Treatment:** product remains clean direct local `main` at + `214cea7a9926abfc5c5e3d681946949eda040ab6`; package remains + `sha256:0d8dc9802048d2db99f5d47c2e4d136a55130f749042d7d4f71fac6e6b828852`. + No source, package, dependency, fixture, image, or protocol identity drifted + after the smoke. +- **Environment/cleanup baseline:** exact pinned image remains local. There is + one pre-existing exited labeled container, five pre-existing labeled + volumes, zero benchmark-runtime entries, and protected gateway PID 62980. + No candidate package process or smoke-owned resource remains. `C:` has + 565,559,267,328 free bytes. +- **Clock discipline/disposition:** no recursive workspace inventory or other + avoidable disk probe will run during the pilot. From pilot command start + through terminal artifact and recursive cleanup production, only small + event/manifest/process reads are allowed; no build, test, install, pull, + source/log edit, dependency mutation, or environment reconfiguration may + occur. Exactly one fresh exploratory `paper-pilot` is authorized. It remains + ineligible for manuscript tables; freeze, tag, and final remain prohibited. + +### 2026-07-31T05:20:04.169Z - Fresh pilot passed; Gate 3 projection remains above limit + +- **Entry ID:** `exp1-214cea7a-pilot-analysis-blocker-094`. +- **Phase/kind:** EXP1-D live exploratory pilot, immutable archival, + deterministic regeneration, exact-final-plan revalidation, and fixed + structural Gate-3 decision. All pilot/projected values remain ineligible for + manuscript tables. +- **Run/identity:** run `019fb690-7dac-72c0-88fc-7f81898b0f8c`; clean product + commit `214cea7a9926abfc5c5e3d681946949eda040ab6`; exact package/image; + pilot plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Terminal result:** captured benchmark exit code 0, empty benchmark/wrapper + stderr, terminal `completed`, correctness `pass`, report-ready and + non-provisional. All 19 cells, 133 trial batches, 385 product requests, and + 95 measured trials completed with zero failures and warnings. +- **Timing:** manifest start `2026-07-31T05:05:53.021835Z`, end + `2026-07-31T05:10:44.994593Z`, exact elapsed 291.972758 seconds. The outer + launch-to-wrapper-completion bracket was 566.0244449 seconds including + complete recursive workspace deletion. The prior healthy pilot elapsed was + 318.447620 seconds, so this treatment reduced manifest elapsed by + 26.474862 seconds. +- **CLI/correctness audit:** 1,897 schema-2 subprocess records have unique + request IDs, return code 0, passed response validation, zero stderr bytes, + packed evidence identity, and passed authentication redaction. Every + correctness and cleanup check passed; unavailable resource fields remain + explicit. +- **Cleanup:** exact run workspace/runtime are absent, archive proof records + no run/gateway-labeled resource or matching process, and explicit benchmark + cleanup exited 0. Independent checks found only the protected PID 62980, + one pre-existing exited labeled container, and five pre-existing volumes; + product remains clean direct `main`. +- **Immutable archive:** creation and independent verify-only invocation + exited 0 with empty stderr and exact agreement at + `experiments\runs\019fb690-7dac-72c0-88fc-7f81898b0f8c`: 5,861 files, + 177,596,758 bytes, content-tree + `sha256:cebbe121541420f9c940aeb9b891055b722c50e16c68d47dc22ee3aa1eaff0a8`. +- **Deterministic tables:** generator + `sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3` + exited 0 twice with empty stderr using only the immutable archive. Outputs + `pilot-214cea7a-019fb690-tables-a` and `-b` are byte-identical: nine + files, 228,904 bytes, content-tree + `sha256:b7e1d21a0c31665c23eaba2c8e559dc38eac0d02940df7883329e753ab7ef245`; + both are explicitly `exploratory_ineligible`. +- **Table component hashes:** environment + `e1e000cc50cc6fb281676cc5424045c2dd5d53fb576891f7fb0ff9364efadea5`; + startup + `5606b9cf54b0c27aee87503cca6540e66fad96567824e51fb967a52c24da90f6`; + CLI operations + `19353b4283556df9e48bd85ab790ac859f61d9a9006dc1355fec39a5a7b5171c`; + resources + `bf7932d47bafe05121f1fd1ac6958eb961d03e9d5054c3482190d4e7eb7ecdc7`; + tables JSON + `fd2e9b6f60796ee6aadfc7008829f722fd5b681e13ea3841eeac551dfaf18528`; + numeric evidence + `57cc819c65a4376cfe0830b845f76d7037ffaa0060fb010d6c29b96a53d14fd0`; + provenance + `0e722d7ae842973ac90dcc455e17908d4586162ef2ebe6ee94355306e2bc20b6`. +- **Exact final-plan revalidation:** `paper-good-pass` is runnable/customized + with zero findings/warnings, only `product_cli`/`paper-100m`, exactly 19 + cells, 1,938 batches, 5,610 issued requests, and plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. + Reviewed expansion + `paper-good-pass-214cea7a-expanded-v2.json`, 42,157 bytes, + `sha256:625bd7c4924607fecdf4f8069082719f6d4d19da05bff8211ce103111f6ae0a1`. +- **Structural projection:** fixed script + `sha256:d6126902dac9bddd868be57425fb1f29d0e1ec95cf4a9c8aa14cdfbaeb694afc` + exited 0 with empty stderr. Output + `pilot-final-runtime-structural-214cea7a-019fb690.json`, 44,385 bytes, + `sha256:201a2c85b34f707053107614e4c46346cc88dd5caa7f840cd3e406cbb1578b5c`. +- **Gate-3 result:** pilot elapsed condition passes, but central structural + projection is 1,246.701123050 seconds and observed envelope is + 1,388.753949600 seconds. They exceed the 1,200-second limit by + 46.701123050 and 188.753949600 seconds; decision remains + `block_freeze_and_final_runtime_projection_exceeds_20_minutes`. + Versus the prior projection, central/envelope fell by 208.933105475 and + 215.291056200 seconds. +- **Decomposition:** new run-fixed cost is 89.974739 seconds (primarily the + batched 19-copy fixture boundary), family-fixed 1.1799336 seconds, cell + projected central 1,155.54645045 seconds, cell-fixed 74.6359862 seconds, + and cell envelope 1,297.599277 seconds. Family central/envelope totals are: + command 116.60083145/121.3424404; files + 613.486186575/674.8308374; workspace lifecycle + 82.9191334/85.4662777; sandbox lifecycle + 342.540299025/415.9597215 seconds. +- **Clock discipline:** no builds, tests, pulls, installs, source/log edits, + recursive workspace scans, or environment mutations occurred during the + pilot clock; progress reads touched only small event/manifest/process state. +- **Recorded analysis mistake:** a first post-projection PowerShell component + aggregation passed script blocks to `Measure-Object -Property`, which this + host treated as nonexistent literal property names and emitted diagnostics. + The corrected explicit decimal loops produced the decomposition above. +- **Disposition/blocker:** Gate 3 remains FAIL. Freeze, commits/tag, and the + exactly-one eligible final remain untouched and prohibited. Launch focused + parallel audits of the new immutable pilot for protocol-preserving fixture, + sandbox-lifecycle, and CLI/file-family savings; any adopted correction + requires offline validation and an entirely fresh smoke/pilot pair. + +### 2026-07-31T05:33:50.721Z - Gate-3 bottleneck audit remains pre-amendment + +- **Entry ID:** `exp1-214cea7a-gate3-bottleneck-audit-095`. +- **Phase/kind:** EXP1-D read-only analysis of immutable pilot + `019fb690-7dac-72c0-88fc-7f81898b0f8c` plus parallel implementation + research. No live benchmark clock, protocol amendment, source edit, product + build, package, freeze, tag, or final run occurred in this entry. +- **Authoritative blocker:** the accepted Entry-094 structural projection + remains 1,246.701123050 seconds central and 1,388.753949600 seconds observed + envelope, respectively 46.701123050 and 188.753949600 seconds above the + fixed 1,200-second Gate-3 limit. +- **Sandbox lifecycle audit:** the projected family is 342.540299025 seconds + central and 415.959721500 seconds envelope. Archived trial/CLI evidence + attributes approximately 95% of steady create-sandbox trial span to the + product manager create and destroy calls: five measured active spans are + 2.612771300--3.025206400 seconds, measured create CLI calls are + 1.577--1.999 seconds, verification is 16--23 milliseconds, and teardown + destroy is 0.907--0.926 seconds. The first cold create is 54.493 seconds; + the second warmup create is 1.574 seconds. Product source inspection + confirmed the create path builds/reuses the shared base, creates a Docker + container and runtime volumes, uploads seed and daemon archives, starts and + authenticates the daemon, while destroy gracefully stops, force-removes the + container, and removes runtime volumes. No product change is yet accepted. +- **CLI evidence audit:** all 1,897 archived CLI records were parsed by the + delegated audit: 1,807 are trial-scoped and 90 are outside trials, with + 82.8 MiB metadata and 59.6 MiB stdout. File trials contain 5--35 CLI calls + and at most 45 buffered calls, 2.62 MiB stdout, and 3.56 MiB marker bytes. + Although file active-minus-recorded-phase overhead is approximately + 193--358 milliseconds per trial, the exact post-teardown grouped + marker/journal gap is only approximately 10--29 milliseconds. Marker fsync + alone therefore cannot close Gate 3. Deferring successful-output projection + and base64/JSON materialization to the existing bounded flush is being + replay-benchmarked; it is not accepted or source-modified yet. +- **Fixture audit:** current batch materialization uses four outer Robocopy + workers, each with `/MT:32`. A same-sweep, four-destination raw-copy + comparison measured `/MT:4` at 23.81 seconds, `/MT:8` at 49.59 seconds, + `/MT:16` at 47.39 seconds, and `/MT:32` at 54.79 seconds. `/MT:4` was + 56.6% below `/MT:32` in that exploratory sweep. Applying that ratio to the + measured 82.7-second materialization component would optimistically save + approximately 46.8 seconds, but this is not yet an end-to-end result and is + explicitly not accepted as a projection input. Direct Win32 copy and + end-to-end API comparisons remain in progress. Ordinary PowerShell + traversal of the depth-100 research copies encountered `MAX_PATH`; validated + extended-length cleanup succeeded and the research temp roots were removed. +- **Resource-boundary audit:** a fresh read-only aggregation of every archived + resource observation formed collection groups by trial and monotonic + collection offsets. There are 133 trials: 98 have two groups and 35 have + three because one periodic 100-millisecond collection was admitted. The + final boundary span across all trials has minimum 31.1981 ms, median + 133.5581 ms, mean 128.278330827 ms, p95 135.9063 ms, and maximum + 142.2804 ms. Exactly seven create-sandbox trials are below 80 ms + (31.1981--33.8030 ms); all 126 non-create trials exceed 100 ms. This matches + the intentional 100-millisecond post-response wait plus approximately + 30--42 ms of concurrent product-observability/local collection. Removing or + overlapping that wait is not accepted: the final cgroup/workspace boundary + must be newer than the validated product response and verification, + journaling, or teardown must not contaminate the resource window. +- **Current source identities:** resource sampler + `sha256:b45c7f5063a60aa052056adcf7d403f1677847348cc01c1b59470614a80cdbf7`; + fixture module + `sha256:4cc1d1e29e1574014890b9eb57c41844c18ee1cfe29bc6346f0506ac10f57c9c`; + CLI adapter + `sha256:44dc001cdd51b137d084412431f6f9f3d7981e196788acd2418a7dd737ceda05`. +- **Recorded read-only diagnostics:** one `rg --files | Select-Object` + inventory stopped with exit 1 after the requested first 80 paths because + the downstream pipeline closed; it returned the intended archive layout. + A broad two-root resource-cadence search used directories that exist only + in one of the two repositories and emitted missing-directory diagnostics + plus truncated output; the narrower rerun established the packaged Windows + product cadence as 100 milliseconds. Several `rg | Select-Object` source + inspections likewise returned exit 1 after the downstream result cap while + returning the requested matches. A timestamp command first used unsupported + `Get-Date -AsUTC`; the compatible `ToUniversalTime()` rerun produced this + entry timestamp. The earlier broad raw-observation `rg` attempt also ended + at its output cap. None of these read-only attempts changed the archive, + source, package, Docker state, or protected gateway PID 62980. +- **Disposition:** continue the three bounded audits. Adopt only changes whose + semantics preserve the fixed measurement and evidence boundaries and whose + end-to-end validation supports enough central and envelope savings. Any + accepted amendment invalidates Entry-092/094 treatment evidence and requires + a complete offline validation followed by an entirely fresh smoke and + five-sample pilot. Freeze and final remain prohibited. + +### 2026-07-31T05:40:42.936Z - Bound Windows fixture-copy oversubscription + +- **Entry ID:** `exp1-fixture-robocopy-mt4-amendment-096`. +- **Phase/kind:** pre-freeze EXP1-D harness implementation amendment and + offline exploratory microbenchmark. No live smoke, pilot, final, Docker, or + protected-gateway operation occurred. +- **Amendment:** changed only the native Windows Robocopy option from + `/MT:32` to `/MT:4` in + `benchmark/backend/benchmark_lab/fixtures.py` and the exact command-vector + assertion in `benchmark/backend/tests/unit/test_fixtures.py`. The four + outer destination workers, native Robocopy boundary, source/destination + validation, exact manifest/inventory checks, independent-file identity, + no-link policy, fixture bytes, fixture hash, 19 fresh per-cell copies, and + failure semantics are unchanged. This caps the intended copy batch at 16 + Robocopy threads instead of 128 for the metadata-heavy depth-100 fixture. +- **Representative fixture/protocol:** NTFS; exact `paper-100m` fixture with + 4,000 payload files, 104,857,600 logical bytes, maximum depth 100, 9,619 + directories plus the fixture manifest; four distinct destinations and four + outer workers. +- **Exploratory comparisons:** a same-sweep raw Robocopy comparison measured + `/MT:4` 23.805 seconds, `/MT:8` 49.59 seconds, `/MT:16` 47.39 seconds, and + `/MT:32` 54.79 seconds. A direct Win32 + `CopyFile2(FAIL_IF_EXISTS)` implementation was rejected after 67.892 + seconds for four destinations because Python-driven creation of the 9,619 + directories dominated. Full validated API `/MT:4` measured 20.229 seconds; + an initial clean validated `/MT:32` baseline measured 20.991 seconds, while + an adjacent load/noise-sensitive `/MT:32` repeat measured 54.922 seconds. +- **Interpretation:** Defender/cache/order noise makes the large paired + improvement unsuitable as a claim. The defensible observed floor is 0.762 + seconds per four-copy batch, or approximately 3.62 seconds when linearly + scaled to 19 destinations; an optimistic application to the prior + 82.7-second materialization component is approximately 52.25 seconds. Both + are exploratory capacity estimates only. A fresh smoke/pilot is required to + establish the actual run-fixed central and envelope effect. +- **Validation:** with `PYTHONDONTWRITEBYTECODE=1`, Ruff 0.16.1 format and + selected import checks passed. The exact focused fixture suite passed 24 + tests with one expected Windows symlink-privilege skip. Scoped + `git diff --check` passed with only the repository's expected CRLF notice. + Fixture module hash is + `sha256:4cc1d1e29e1574014890b9eb57c41844c18ee1cfe29bc6346f0506ac10f57c9c`; + focused test hash is + `sha256:bda49e6e1942fee6eea2674788bc0600d644cdcce5d032f7c413f5d8f6dde1da`. +- **Cleanup:** all `fixture-copy-bench-*` research roots were removed through + the validated extended-length path cleanup. Ordinary PowerShell enumeration + had encountered `MAX_PATH` during research, but no temp root remains. +- **Disposition:** retain the minimal `/MT:4` amendment for combined offline + validation. Entries 092--094 no longer qualify the current harness + treatment. Freeze and final remain prohibited; a new complete smoke/pilot + pair and fixed structural projection are mandatory after all accepted + amendments settle. + +### 2026-07-31T05:42:32.476Z - Further CLI-evidence deferral rejected + +- **Entry ID:** `exp1-cli-evidence-deferral-nochange-097`. +- **Phase/kind:** EXP1-D read-only immutable-pilot replay and design audit. + No source, evidence protocol, archive, product, Docker, preset, freeze, tag, + or live-run state changed. +- **Corpus:** parsed all 1,897 CLI invocations from immutable pilot + `019fb690-7dac-72c0-88fc-7f81898b0f8c`: 1,807 trial-scoped and 90 outside + trials, approximately 82.8 MiB of schema-2 metadata and 59.6 MiB stdout. + File trials contain 5--45 calls and the largest trial buffer remains bounded + at approximately 2.62 MiB stdout and 3.56 MiB packed metadata. +- **Candidate:** defer successful-call projection writes and base64/JSON + materialization to the existing bounded trial flush, reuse the already + computed stdout digest, and skip redundant successful-output redaction work + while retaining outside-trial and failure durability. +- **Valid replay result:** a disk-free archived-payload replay sampled one + measured trial from each of the 12 file cells, repeated seven times. Its + CPU-only difference linearly projects to only approximately 1.13 seconds + across the final run. +- **Ceiling:** the complete archived post-teardown gap, which contains all CLI + marker commits plus observation and event journal commits, scales to only + approximately 21.6 seconds across the 12 final file cells. CLI evidence is + only a subset of this interval, so even its defensible absolute ceiling is + below 20 seconds. +- **Excluded attempt:** a full disk replay ran concurrently with the fixture + copy audit and encountered multi-second, sign-inconsistent NTFS/Defender + interference. That replay is invalid, is explicitly discarded, and is not + used in any estimate or decision. +- **Decision:** reject the candidate without editing. Its approximately + 1.13-second CPU benefit is disproportionate to delayed failure evidence, + raw-output memory lifetime, partial-write recovery complexity, and risk of + moving stdout I/O into the primary timing boundary. Preserve the already + validated `metadata-packed-payload-fsync-v1` grouped-marker implementation. + Gate 3 remains open on fixture, resource-boundary, and product-lifecycle + work; freeze and final remain prohibited. + +### 2026-07-31T05:46:10.000Z - Batch pre-release evidence uses one durability transaction + +- **Entry ID:** `exp1-request-barrier-grouped-commit-amendment-098`. +- **Phase/kind:** pre-freeze EXP1-D harness implementation amendment and + immutable-pilot replay analysis. No live smoke, pilot, final, Docker, + protected-gateway, product-package, freeze, tag, or manuscript-eligible + operation occurred. +- **Amendment:** `_run_request_batch` now combines already-buffered + setup/phase records, `waiting_at_barrier` records, and the complete + `ready_at_barrier` set into one ordered durable transaction immediately + before the operation gate is released. A failed transaction leaves the + operation coroutines behind the barrier. In-flight and terminal records + remain post-release records and retain their prior order and failure + semantics. `benchmark/PAPER_ARTIFACT.md` now states this exact durability + boundary. +- **Archived evidence:** in immutable pilot + `019fb690-7dac-72c0-88fc-7f81898b0f8c`, the two removed historical + pre-release commit intervals have minimum 4.5383 ms, median 4.9895 ms, mean + 6.8522 ms, p95 18.4573 ms, and maximum 23.8374 ms across 1,938 projected + final batches. Replaying the complete operation-running-to-ready interval + gives an observed structural upper saving of 13.0379455 seconds central and + 13.3770556 seconds envelope; because that interval also contains small + non-fsync setup/scheduling work, it is explicitly an upper, not a guaranteed + saving. Multiplying the strict historical minimum by 1,938 gives an + 8.7952254-second floor for the removed intervals only. +- **Validation:** three existing focused barrier tests first passed. The new + injected-commit-failure regression then failed twice only because its + assertions inspected the outer `ExceptionGroup` string rather than its + direct leaf exception; after correcting the test inspection, all four + focused tests passed. The complete runner integration file passed 36 tests. + A subsequent full shared backend run, after this amendment and the resource + amendment below, passed 276 tests with five expected skips. The regression + proves a failed pre-release journal commit cannot launch an operation. +- **Recorded implementation/analysis attempts:** the first combined patch + failed atomically because the documentation context differed from the + expected text; exact source inspection and a narrower patch succeeded. A + first inline projection import omitted the dynamically loaded module from + `sys.modules` and raised a dataclass import error; the corrected import then + addressed a nonexistent `identity` key before exact semantic inspection + established `semantics["operation"]["cell"]`. A PowerShell attempt to find + the first create stdout double-prefixed an already-relative + `cli-subprocesses` path and returned no result; the corrected path lookup + succeeded, while a later request-ID search legitimately found no match. + These failed read-only/atomic attempts changed no evidence or runtime state. +- **Source identities:** runner + `sha256:464e2caeac12fe00d5d0c160dd26cdefaa37e80c7ada9b95965a50edade72357`; + runner integration test + `sha256:8305f3fa3033f3595bee0fec2ea337f5eed38f74310fc661467b92013872c7c5`; + artifact contract + `sha256:90385fda5fc65e7cbe47bd3cea1ad2f97307b248f510ea021aa9b20c2bed9ecb`. +- **Disposition:** retain the protocol-preserving amendment. Prior smoke/pilot + treatment evidence no longer qualifies the current harness; a fresh + complete smoke/pilot pair remains mandatory before freeze. + +### 2026-07-31T05:49:05.000Z - Post-response freshness overlaps admitted sampler drain + +- **Entry ID:** `exp1-resource-boundary-overlap-amendment-099`. +- **Phase/kind:** pre-freeze EXP1-D resource-instrumentation amendment and + immutable-pilot replay. No live benchmark, Docker, product, package, freeze, + tag, or final operation occurred. +- **Amendment:** `TrialResourceSampler.stop()` now fixes both its wall-clock + post-response freshness threshold and monotonic 100-ms not-before deadline + immediately at stop entry. It stops the cadence loop, drains any periodic + collector admitted during the primary window, and waits only the remainder + of that same fixed interval before launching the mandatory final product + boundary query. Product queries remain serialized, the final query cannot + start before 100 ms, strict product-ring timestamp validation and retry + remain active, samples retain scheduled order, and verification/journaling/ + teardown remain outside the resource window. +- **Regression:** a deterministic test holds an admitted periodic collector + beyond 100 ms, then proves the mandatory final boundary starts at least + 100 ms after stop entry and less than 75 ms after the collector is released. + This distinguishes overlap from weakening or removing the freshness wait. +- **Archived capacity bound:** only 20 archived periodic collections completed + after the last validated response. Their affected overlap is approximately + 11.7072--25.8808 ms. The maximum immutable-pilot replay benefit is + 4.9117802 seconds central and 5.1694866 seconds envelope; actual benefit may + be zero when no collector overlaps. Early polling, product-ring phase + prediction, and reuse of a pre-stop snapshot were rejected because they + would weaken the post-response evidence boundary. +- **Validation:** the focused resource suite passed 17 tests; resource plus + runner suites passed 52; the full shared backend passed 276 with five + expected skips. Ruff format/import checks and scoped `git diff --check` + passed. +- **Source identities:** sampler + `sha256:c051089fa2730e00962aebcf26d2be0777e6c5e00df25867fb95a933222a07b3`; + unit test + `sha256:200b11197edcd784c4c8509d828c6369dc78f1d0806f06aa19caf2229e1c55ec`. +- **Disposition:** retain the amendment. As with every current pre-freeze + change, it requires fresh smoke/pilot treatment evidence. + +### 2026-07-31T05:52:16.998Z - User-authorized Gate-3 limit is 1,400 seconds + +- **Entry ID:** `exp1-gate3-limit-1400-amendment-100`. +- **Phase/kind:** explicit user-authorized pre-freeze protocol amendment and + offline validation. The user stated that 1,400 seconds is allowed. No live + run, Docker operation, evidence deletion, final, freeze, commit, or tag + occurred. +- **Fixed amended rule:** the pilot elapsed time, central structural + projection, and observed-envelope projection must each be no more than + 1,400,000,000,000 ns (1,400 seconds; 23 minutes 20 seconds). All other Gate + 3 correctness, cleanup, resource, determinism, anomaly, and protocol + requirements remain unchanged. The full 19-cell matrix, two warmups, 100 + measured trials, 1,938 batches, and 5,610 issued requests are unchanged. +- **Implementation:** updated the current experiment inventory, EXP0 protocol + note, EXP1 campaign packet, deterministic projection constant and stable + failure decision, plus its exact unit expectation. Historical log entries + and historical analysis JSON remain immutable records of the former + 1,200-second rule. +- **Immediate interpretation:** immutable pilot + `019fb690-7dac-72c0-88fc-7f81898b0f8c` had elapsed 291.972758 seconds, + central 1,246.701123050 seconds, and observed envelope 1,388.753949600 + seconds, so those three values would satisfy the amended limit. They do not + qualify the current harness because Entries 096, 098, and 099 changed the + treatment after that pilot. A fresh smoke, five-sample pilot, deterministic + table regeneration, and projection are still mandatory before freeze. +- **Validation:** the first unit-test command incorrectly selected a + nonexistent backend-local virtual environment, fell back to an unrelated + Python, and failed with `No module named pytest`; it changed no file. The + corrected repository virtual environment passed both projection tests in + 4.23 seconds. +- **Current identities:** experiment inventory + `sha256:2c7eaa10d395c6460da07f6ef4cab2b5f40f6b431f50c2df82685b109d7467ea`; + EXP0 packet + `sha256:401223fdb65c43423121c2f640b77a80f101383e1833cd680ed4ace001c4de33`; + EXP1 packet + `sha256:c021ecdd47cc80b1601171c7c175ed4f42d12dfe51229aa4c36b603b8d0a18ce`; + projection script + `sha256:444eafef866c9d8be5a00f63a80eeb3727b6dfddd0faa0128d2f54deb161319f`; + unit test + `sha256:25d111a6c9c77852e247b6d79116fea3b3d8647ca13941e38c9223f0ebdf18be`. +- **Disposition:** the former runtime blocker is removed in principle, but + freeze remains prohibited until the current complete harness and whichever + product optimization is retained pass offline validation and a fresh + qualification pair. + +### 2026-07-31T05:55:42.000Z - Combined paper amendments pass offline validation + +- **Entry ID:** `exp1-paper-amendments-offline-validation-101`. +- **Phase/kind:** pre-freeze EXP1-D offline validation and formatting only. No + live preset, Docker operation, product package, evidence deletion, freeze, + tag, or final operation occurred. +- **Full suite:** with `PYTHONDONTWRITEBYTECODE=1`, the complete backend suite + passed 277 tests with five expected Windows symlink-privilege skips in + 45.71 seconds. After line-ending/import normalization, the combined resource, + runner, and projection suites passed 55 tests in 8.78 seconds. +- **Formatting/import/diff:** offline Ruff 0.16.1 reports the seven changed + backend Python files and the projection script formatted; selected import + checks pass. Ruff mechanically normalized the newly inserted runner and + runner-test lines to their files' existing CRLF convention and fixed one + resource-test import separator. Scoped `git diff --check` passes with only + expected checkout line-ending notices. +- **Recorded validation attempts:** the first full-suite command was + accidentally given a one-second command-runner timeout. The exact + workspace-scoped pytest process was verified, terminated by PID, and the + full suite was rerun to a captured terminal result. A first Ruff command + addressed a nonexistent venv-local `ruff.exe`. The corrected offline Ruff + command initially checked the complete historically unformatted backend and + reported 43 baseline files that would be reformatted; it changed nothing. + The scoped format check then exposed only mixed line endings introduced in + the two changed runner files, which were mechanically normalized. A broad + all-rule scoped lint also surfaced pre-existing project-wide policy findings + such as intentional `BaseException` handling and an old unused import; the + contract-relevant format and import checks were rerun narrowly and passed. +- **Current amendment identities:** fixture + `sha256:4cc1d1e29e1574014890b9eb57c41844c18ee1cfe29bc6346f0506ac10f57c9c`; + resource sampler + `sha256:c051089fa2730e00962aebcf26d2be0777e6c5e00df25867fb95a933222a07b3`; + runner + `sha256:b08270d8d4e00f412dc29ea16acc3c9669a36eb9a7b2916856a8b455472a149f`; + resource test + `sha256:39c6425652aacc6cad93c30caa0a368d2986c2e485d7e3b78267c080ea15bb73`; + runner test + `sha256:76c7e91317b1df3ddbd7f413649065d5725a84c7bc7db3b59df9efcb8487d22e`. +- **Disposition:** all current paper-side treatment amendments pass offline + validation. They remain pre-freeze and require a fresh smoke/pilot pair. + +### 2026-07-31T06:01:31.188Z - Create-sandbox product optimization committed and packaged + +- **Entry ID:** `product-exp1-create-sandbox-optimization-102`. +- **Phase/kind:** user-requested, user-authorized pre-freeze product + optimization, offline validation, local direct-`main` commit, and release + packaging. No push, tag, live benchmark, Docker command, protected-gateway + action, or baseline-resource mutation occurred. +- **Retained product changes:** ordinary Docker bridge calls now share one + lazily initialized one-worker Tokio runtime and Bollard client while each + synchronous caller waits for its own typed result. The daemon installation + tar is encoded with fast gzip and cached by byte-exact daemon/config + contents; every install still rereads and compares both source files, so a + byte change rebuilds the archive. Docker volume mutation ordering, graceful + stop behavior, readiness cadence, daemon semantics, and external CLI + contracts are unchanged. +- **Rejected/pruned candidates:** speculative readiness polling/configuration + changes were reverted byte-for-byte before commit because archived evidence + did not quantify failed-poll frequency. Concurrent volume mutations were + rejected because they would change failure ordering. Bollard still opened + one test TCP connection per request, so executor savings are not claimed + before live qualification. +- **Offline payload evidence:** the exact daemon install archive decreased + from 7,176,192 to 2,918,377 bytes, a reduction of 4,257,815 bytes or 59.333% + per create. Five-run construction averages were 23.343 ms plain and + 228.119 ms gzip. Because the cache pays compression once per content + identity, 102 lifecycle creates would transfer approximately 434.3 MB fewer + archive bytes; realized seconds remain unclaimed until the fresh pilot. +- **Validation:** all 17 non-live provider tests passed and the one + Docker-required test remained explicitly ignored. The new archive test + verifies gzip magic and exact extracted paths, 0755/0644 modes, and bytes; + the runtime regression exercises repeated executor calls from inside a + separate Tokio runtime. `cargo fmt --all -- --check`, provider all-target + Clippy with warnings denied, `cargo check -p sandbox-gateway`, and + `git diff --check` passed. Root independently repeated the offline provider + suite and all four checks successfully. +- **Commit:** exact local product commit + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3` + (`perf(docker): reuse executor and compress daemon archive`) on direct + `main`, parent `214cea7a9926abfc5c5e3d681946949eda040ab6`. + Product is clean and 12 commits ahead of `origin/main`; no branch, worktree, + push, or tag was created. +- **Source identities:** archive + `sha256:98a0aad398c52fccb42a856d4f2a66e28832be71a33f0aa8f5a6dd56cbd3d751`; + engine + `sha256:14c737fc9276f4bc788a498216f65b102b397a296a24a5776e48f286de7bd5bf`; + installer + `sha256:07df5ef2963ea47b76441fd0b0e0b2add7c775887c1b05628e7ac1b0246e3571`; + archive test + `sha256:650c0b8bb10752e3adcb7a602ab841edfa8df858099f75526b82addd0ee7f923`; + runtime test + `sha256:8129e7a4efe119aa7f354c8ea554d2c38a10e66ab38ddd9868e0d4300646cf5d`. +- **Package build attempts:** a fresh Linux daemon xtask rebuild failed before + output because the newly cleared cross-target build needed `sh` for + jemalloc's configure script and no Windows `sh.exe` is installed. The + daemon crate/source was untouched by this commit, so the exact prior verified + 7,171,600-byte daemon binary was retained with unchanged + `sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`. + The first Windows package-script launch was blocked before execution by the + host PowerShell execution policy; the explicit repository-local + `-ExecutionPolicy Bypass -File` rerun exited 0. +- **Immutable candidate package:** exact directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-0392b299`; + 5,685,130-byte ZIP + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-0392b299.zip`; + ZIP + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`. + Component hashes are gateway + `9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285`, + manager + `564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed`, + runtime + `c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8`, + observability + `25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26`, + daemon as above, and config + `987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Cleanup:** the subagent's exact 50-MiB offline archive-benchmark directory + was resolved beneath `C:\Users\yifan\AppData\Local\Temp`. A first native + `Remove-Item` command was blocked before execution by shell policy; the same + exact validated path was deleted through the PowerShell-hosted .NET + directory API and confirmed absent. No other path was removed. +- **Disposition:** this exact clean commit/package is the candidate treatment. + It requires fast preflight, complete smoke, archive verification, and a + fresh five-sample pilot before any freeze decision. + +### 2026-07-31T06:03:35.829Z - Candidate 0392b299 smoke preflight passed + +- **Entry ID:** `preflight-exp1-0392b299-smoke-103`. +- **Phase/kind:** EXP1-C strict fast preflight after all current paper/product + amendments. No live benchmark clock, source edit during measurement, Docker + mutation, pull, install, freeze, tag, or final operation occurred. +- **Plans:** fresh validation subprocesses exited 0 with empty stderr, + `runnable: true`, `is_customized: true`, and zero findings/warnings. + `paper-env-smoke` has 19 cells, 19 batches, 55 issued requests, and plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + `paper-pilot` has 19/133/385 and + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + `paper-good-pass` has 19/1,938/5,610 and + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. + All plans select only `product_cli`, the pinned image, and `paper-100m`. +- **Exact treatment:** clean product direct `main` at + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; exact package ZIP + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`. + Gateway/manager/runtime/observability `--help` each exited 0 with empty + stderr and hashes + `9f2102d13e9643e57e0988644f9ea61534c469e26f4c705b5b8913e892ff7285`, + `564876d4c11040a254d149622d28d3c01259c997ae6472f247dab05e9e1e72ed`, + `c73a5b3bf4b04df5326391bc77b233904074bde72eabc20f393a0f7031d5c1c8`, + and + `25906c1910119a52d98580c748cdad5886aa118d59eccef1f7aa87265c754f26`. +- **Host/environment:** `DESKTOP-OLP1ADS`, Windows build 26200, NTFS, 48 + logical processors, 137,438,953,472 physical-memory bytes, and + 563,732,828,160 free bytes on `C:`. Docker client/server remains 29.0.1, + Linux AMD64, `overlayfs`, cgroup v2. Exact image + `sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` + is local; no pull was issued. +- **Preserved baseline:** protected debug gateway PID 62980 is the only + sandbox-named process. There is one pre-existing exited labeled container, + two sandbox volumes, three shared-base volumes, an empty benchmark runtime, + and the same eight older terminal run records. Nothing was started, stopped, + removed, or relabeled. +- **Recorded diagnostic mistake:** one read-only Docker preflight used the + complete JSON formatter and locally printed the stopped protected baseline + container's full label set, which included its authentication label. The + value is intentionally not reproduced here. It was not written to campaign + artifacts or user-facing reporting, and no resource changed. All future + baseline probes are restricted to ID/name/state fields. +- **Clock discipline/disposition:** outer exit-capture helper remains + `sha256:7091ecdcee6b69ef471e0322aae4f28e1a5e61e776b80c0f04f84fabb9b6aa4f`. + From the next smoke command start through its terminal artifact and recursive + workspace cleanup, no build, test, install, pull, source/log edit, + dependency mutation, recursive workspace probe, or environment + reconfiguration is permitted. Exactly one fresh smoke is authorized; pilot, + freeze, tag, and final remain prohibited pending archival and audit. + +### 2026-07-31T06:13:21.440Z - Candidate 0392b299 CLI integration smoke passed + +- **Entry ID:** `smoke-exp1-0392b299-pass-104`. +- **Phase/kind:** EXP1-C live `paper-env-smoke`, immutable archival, + independent archive verification, and explicit cleanup. This is accepted + qualification evidence and permanently ineligible for manuscript tables. +- **Run/identity:** run `019fb6c5-dab0-7958-b7ba-94f2a9eda944`; clean product + commit `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; candidate package ZIP + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`; + pinned image; smoke plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`. +- **Captured result:** supervised wrapper and benchmark both exited 0. + Benchmark stderr and wrapper stderr are empty. Manifest start + `2026-07-31T06:04:10.415013Z`, end + `2026-07-31T06:07:51.671728Z`, exact elapsed 221.256715 seconds. The outer + launch-to-wrapper-completion bracket is 448.5456188 seconds and includes + complete recursive deletion of all 19 deep fixture copies. +- **Terminal corpus:** state `completed`, correctness `pass`, report ready and + non-provisional; all 19 cells, 19 trial batches, 55 issued requests, 48 + correctness checks, and 19 operation-evidence records passed with zero + failures and report warnings. The cold create-sandbox request was + 52.0341268 seconds, proving Docker accepted and extracted the compressed + daemon archive. The total manifest elapsed is 17.027046 seconds below the + prior accepted smoke, but smoke values remain qualification-only and the + combined treatment prevents attribution to a single amendment. +- **CLI/resource evidence:** all 339 schema-2 CLI invocations have unique + request IDs, return code 0, passed response validation, zero stderr bytes, + and passed authentication redaction. There are 602 resource records with + complete mandatory windows. Expected Windows allocated-byte and LayerStack + limitations remain explicit unavailable values; nothing is coerced to zero. +- **Cleanup:** the owned run workspace/runtime are absent. Archive cleanup proof + reports no run/gateway-labeled candidate container or volume, no matching + process, and a clean exact product checkout. Independent restricted probes + found only protected gateway PID 62980, the one pre-existing exited + container, and the five pre-existing volumes. Explicit benchmark cleanup + exited 0 with `cleaned: true`. +- **Immutable archive:** creation and independent verify-only invocations exited + 0 with empty stderr and exact agreement at + `experiments\runs\019fb6c5-dab0-7958-b7ba-94f2a9eda944`: 1,073 files, + 29,730,028 bytes, content-tree + `sha256:3ad6e3aba681cfdf257939df79a0561a5b73b710a73d46686146bc5780fe8a6b`. + Archive-manifest SHA-256 is + `6fc6924f5bf775b4572093d6f5d619c249e2362273b69dc5d1626ec2bb8e21d1`; + campaign-manifest SHA-256 is + `bf301771c5c2fff3ee82687d8ee8982a6c6d8451635664b411a2140afd99a325`. + Disposition is `smoke`, eligibility `qualification_only`, and protocol state + `pre_freeze`. +- **Clock discipline:** no build, test, install, pull, source/log edit, + dependency mutation, Docker inspection, or recursive filesystem probe + occurred from command start through wrapper cleanup. Progress reads were + restricted to the exact wrapper process and small run-manifest fields. +- **Disposition:** EXP1-C passes for the exact `0392b299` treatment. One fresh + exploratory five-sample pilot is authorized after a strict fast preflight. + Freeze, tag, and final remain prohibited until pilot archival, deterministic + table regeneration, the amended 1,400-second structural projection, anomaly + review, and cleanup pass Gate 3. + +### 2026-07-31T06:14:06.087Z - Candidate 0392b299 pilot preflight passed + +- **Entry ID:** `preflight-exp1-0392b299-pilot-105`. +- **Phase/kind:** EXP1-D strict fast preflight; no live pilot clock started. +- **Smoke prerequisite:** an additional independent verify-only invocation + exited 0 with empty stderr and reconfirmed smoke + `019fb6c5-dab0-7958-b7ba-94f2a9eda944`: 1,073 files, 29,730,028 bytes, + content-tree + `sha256:3ad6e3aba681cfdf257939df79a0561a5b73b710a73d46686146bc5780fe8a6b`. +- **Plan:** fresh `paper-pilot` validation exited 0 with empty stderr. It is + runnable/customized with zero findings/warnings, only `product_cli` and + `paper-100m`, exactly 19 cells, two warmups plus five measured trials per + cell, 133 batches, 385 issued requests, and plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Treatment/environment:** product remains clean direct local `main` at + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; package remains + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`. + `C:` has 563,676,590,080 free bytes. Benchmark runtime is empty and the + smoke workspace is absent. Restricted probes found only protected gateway + PID 62980, the one pre-existing exited container, and the same five + pre-existing volumes. +- **Clock discipline/disposition:** from the next pilot command start through + terminal artifact and recursive cleanup production, no build, test, install, + pull, source/log edit, dependency mutation, Docker inspection, recursive + workspace scan, or environment reconfiguration is permitted. Exactly one + fresh exploratory pilot is authorized. It is permanently ineligible for + manuscript tables; freeze, tag, and final remain prohibited pending its + complete Gate-3 audit. + +### 2026-07-31T06:28:19.016Z - Fresh pilot passes amended Gate 3 + +- **Entry ID:** `exp1-0392b299-pilot-gate3-pass-106`. +- **Phase/kind:** EXP1-D live exploratory pilot, immutable archival, + independent archive verification, deterministic regeneration, exact + final-plan validation, preregistered structural runtime projection, anomaly + review, and cleanup. Every pilot/projected value remains ineligible for + manuscript tables. +- **Run/identity:** run `019fb6cf-6021-76d5-ab4f-c6ed53e1d293`; clean product + commit `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; candidate package/image; + pilot plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Terminal result:** supervised benchmark exit 0, empty benchmark/wrapper + stderr, terminal `completed`, correctness `pass`, report-ready and + non-provisional. All 19 cells, 133 trial batches, 385 product requests, 336 + checks, 133 operation-evidence records, and 95 measured trials completed with + zero failures, exclusions, or report warnings. +- **Timing:** manifest start `2026-07-31T06:14:34.194294Z`, end + `2026-07-31T06:19:10.698722Z`, exact elapsed 276.504428 seconds. The outer + launch-to-wrapper-completion bracket is 563.5771511 seconds including + complete recursive deletion of all deep fixture copies. Relative to the + prior healthy current-protocol pilot, manifest elapsed decreased + 15.468330 seconds. +- **CLI/correctness/resources:** all 1,901 schema-2 CLI invocations have unique + request IDs, return code 0, passed response validation, zero stderr bytes, + and passed authentication redaction. There are 4,256 resource records and + 385 correlated request IDs. One optional periodic collector in a warmup + explicitly recorded its fixed-concurrency-cap saturation; its records are + marked `sampled: true`, the mandatory baseline/final boundaries completed, + and no measured request/correctness result is missing. Expected Windows + allocated-byte and unreported LayerStack fields remain explicit unavailable + values. +- **Cleanup/archive:** exact run workspace/runtime are absent; cleanup proof + records no run/gateway-labeled candidate resource or matching process and a + clean exact product checkout. Explicit benchmark cleanup exited 0. Creation + and independent verify-only invocations agree at + `experiments\runs\019fb6cf-6021-76d5-ab4f-c6ed53e1d293`: 5,873 files, + 177,619,234 bytes, content-tree + `sha256:f81cf711bbb734f04201c7fdc09652e0d59c7cbdecd50fe3676370b1df77b93c`. + Archive-manifest SHA-256 is + `c2810ac9014f835c676dce294ba4a164c7cc979c4e9cd49be4c3239bdf9bdbc6`; + campaign-manifest SHA-256 is + `de60036c6fa2bb5d715315e61c36909a50701462a030d9f2af649a079b1033cb`. +- **Deterministic tables:** generator + `sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3` + exited 0 twice with empty stderr using only the immutable archive. The two + nine-file outputs are byte-identical: 228,922 bytes, content-tree + `sha256:4196b53dac63db96053375ca779bf60321849bd14e66366f4390854e85877cd8`, + and both say `exploratory_ineligible`. Component hashes are environment + `325a9c8ac9326f0b4b5bc4009df89453bed4033802a4269f1cb5fb593c9fe11d`; + startup + `3b413b5a12e2f4d5ec35cc3017cdfc7577638456acc47a27fd4043b4814a9556`; + CLI operations + `7eb9709e4d9870190f2ab6d7d66b6a2f895bd188b4c674a9e16216803d624fa0`; + resources + `7c9ed30898589abf5d726d3582c2e157d94f6460cddafa156edd6dfb87c56d75`; + tables JSON + `2697a52ecd376dc573784c8ab5525b002f8a60a376da1f81c1be384b78909794`; + numeric evidence + `92478bd426e10ba1b58ecbd50f10b904be3922dd75fed0a17aecd287bcc524e6`; + provenance + `aaab8dc3584bba274c3d8d2651a62bfa386c0a0d490bf8ef7bbbcbcdb8052072`. +- **Exact final plan:** `paper-good-pass` remains runnable/customized with zero + findings/warnings, only `product_cli`/`paper-100m`, 19 cells, 1,938 batches, + 5,610 requests, plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`; + reviewed expansion SHA-256 + `4d2389e1782a5075bf4130b3a158b55cc39cc697a1f53593f8c93ff2388e7c92`. +- **Structural projection:** amended fixed script + `sha256:444eafef866c9d8be5a00f63a80eeb3727b6dfddd0faa0128d2f54deb161319f` + exited 0 twice with empty stderr and byte-identical output + `experiments\analysis\pilot-final-runtime-structural-0392b299-019fb6cf-limit1400.json`, + `sha256:8b02470b98a8ff2374d140398e6a7bb00abb469fa96810db4a680ab00079a4a8`. + Pilot elapsed is 276.504428 seconds, central structural projection is + 1,170.818322650 seconds, and observed envelope is 1,307.100411100 seconds. + All are below 1,400 seconds; central margin is 229.181677350 seconds and + envelope margin is 92.899588900 seconds. Decision is + `pass_runtime_projection`. +- **Projection decomposition:** run-fixed 80.922308300 seconds; family-fixed + 1.154963500; cell central 1,088.741050850; cell envelope 1,225.023139300. + Family central/envelope totals are command + 112.089687425/116.950502100; files + 583.998037100/634.833464900; workspace lifecycle + 79.595985150/82.700570800; sandbox lifecycle + 313.057341175/390.538601500 seconds. Versus the prior current-protocol + projection, central/envelope decreased 75.882800400/81.653538500 seconds. +- **Clock discipline:** no build, test, pull, install, source/log edit, + dependency mutation, Docker inspection, recursive workspace read, or + environment reconfiguration occurred during the pilot or its wrapper + cleanup. Progress reads touched only exact process and small manifest fields. +- **Gate-3 decision:** PASS. Correctness, cleanup, timing separation, resource + correlation, deterministic regeneration, anomaly review, and all three + amended runtime conditions pass. Freeze is now authorized under the user's + prior explicit local commit/tag permission; no final run has occurred. + +### 2026-07-31T06:34:57.974Z - EXP1 protocol v1.0 and treatment frozen + +- **Entry ID:** `exp1-protocol-v1-freeze-107`. +- **Phase/kind:** EXP1-E scoped local source commit, annotated product tag, + machine-readable freeze record, and independent clean-scope verification. + No final measurement clock has started. +- **Paper source:** exact staged review covered 62 campaign source, test, + preset, protocol, analysis, and experiment-log files; `git diff --cached + --check` exited 0. Local commit + `eb10c26d1bfd632772baf1bc331c985d0231f52d` + (`feat(benchmark): freeze EXP1 CLI protocol v1.0`) was created on the + existing paper branch. The archive script's final-mode provenance check + independently reports `dirty: false`, empty scoped status, and + `clean_frozen_commit`. Excluded caches and non-frozen evidence remain + preserved. +- **Benchmark identity:** 207 non-excluded files, 16,290,860 bytes, + content-tree + `sha256:1efeff548dd664580dcb452829d86e1ae114477828a1af65589b7e34cc311b67`. + The frozen protocol and expected-table specifications hash to + `sha256:c021ecdd47cc80b1601171c7c175ed4f42d12dfe51229aa4c36b603b8d0a18ce` + and + `sha256:7c1ae3d78a9ca7a5b6cbb4a9cc80bac555d676c5833a40d2846597884f8f50da`. +- **Product treatment:** direct clean product `main` remains + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`. Annotated local tag + `paper-v1-freeze` was created; tag object + `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d` has object type `tag` and + peels exactly to the measured commit. Nothing was pushed. +- **Package/image/fixture:** candidate ZIP is 5,685,130 bytes, + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`; + the Ubuntu 24.04 digest is unchanged. Fixture hash is + `sha256:9484b132c8a35afd18bc37383759d0fe6d45dd4700b42a99336aed535e651cc7` + with tree + `sha256:d4c2fefbf94a30352f39d701ececaeeb8fad35603e4fb721dd5cf21296258c9f`. +- **Final plan/analysis:** the reviewed final plan remains + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`, + seed 20260712, 19 cells, two warmups plus 100 measured trials, 1,938 + batches, and 5,610 requests, exclusively `product_cli` and `paper-100m`. + Archive, table-generation, and projection scripts hash to + `sha256:25cca63e4acc035a60384dce11221d6571a10ccd1bdf5b3a39e0e7a0d027426e`, + `sha256:37f2dddec2684b6eb2682c65ebdb4750b3a4f795596045ec05da827f06dff7e3`, + and + `sha256:444eafef866c9d8be5a00f63a80eeb3727b6dfddd0faa0128d2f54deb161319f`. +- **Freeze artifact:** the complete machine-readable record is + `experiments\analysis\exp1-freeze-record-eb10c26-0392b299.json`. It freezes + protocol version, Git identities, binary/package/image/fixture identities, + seed, trials, plan, table schema, metrics, exclusions, stopping rules, and + analysis identities. +- **Gate-4 decision:** PASS. Every scientific and executable identity is now + fixed. The sole eligible `paper-good-pass` is authorized after one final + read-only preflight; any source/protocol/treatment mutation from this point + would invalidate the freeze and block final execution. + +### 2026-07-31T06:38:09.609Z - Frozen final preflight passed + +- **Entry ID:** `preflight-exp1-0392b299-final-108`. +- **Phase/kind:** EXP1-F last read-only preflight for the sole eligible final + attempt. No final benchmark clock had started while this record was written. +- **Freeze/source:** the machine-readable freeze JSON parses successfully and + hashes to + `sha256:38608c306476ce19cd63f1e42808aaa45cd030e3e160a88dd6defd1189aa3429`. + Frozen paper scope remains clean at + `eb10c26d1bfd632772baf1bc331c985d0231f52d`. Product remains clean direct + `main` at `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; annotated tag object + `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d` still peels exactly to that + commit. Candidate package SHA-256 remains + `2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`. +- **Fresh final-plan validation:** `benchmark_lab validate` exited 0 with empty + stderr. The UTF-16 PowerShell capture hashes to + `sha256:fc12b1e75fb9716369a0c63743b074277252121f08b7de28d143d6418d1fd534`. + Corrected structural parsing confirms `runnable: true`, no findings or + warnings, 19 cells, 1,938 batches, 5,610 requests, `product_cli`, and exact + plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. +- **Uniqueness/state:** there are zero prior `paper-good-pass` result manifests + and zero final archives. Benchmark runtime and campaign workspace are + absent; `C:` has 563,278,090,240 free bytes. +- **Protected baseline:** restricted probes that exposed no labels found only + protected gateway PID 62980, one pre-existing exited container + `5ce30657bf83`, and the same five pre-existing volumes. No protected resource + was changed. +- **Inspection corrections retained:** one wrapper-sidecar read used a + nonexistent timestamped filename before the exact existing filename was + read; one combined no-match `rg` probe propagated exit 1 before a corrected + no-match-aware probe proved both final counts are zero; one malformed + exploratory `rg` regular expression was replaced by fixed-string searches. + The first PowerShell view misleadingly counted absent JSON properties as one, + and a first Python parse assumed UTF-8 for a UTF-16 PowerShell redirect. + Corrected UTF-16 structural parsing produced the accepted values above. + These were read-only orchestration/inspection failures and did not mutate + the frozen treatment or consume a final attempt. +- **Authorization/clock discipline:** exactly one `paper-good-pass` launch is + now authorized. From its command start through terminal wrapper cleanup, no + build, test, install, pull, source/log edit, dependency change, Docker + inspection, recursive workspace scan, archive, or environment + reconfiguration may occur. Progress reads are limited to the exact wrapper + process and small run-manifest fields. + +### 2026-07-31T07:19:03.383Z - Sole frozen final fails and is preserved ineligible + +- **Entry ID:** `exp1-0392b299-final-failed-handoff-109`. +- **Phase/kind:** EXP1-F sole frozen final attempt, terminal failure + preservation, exact cleanup, independent archive verification, post-run + diagnosis, claim-boundary handoff, and final Gate 0--7 report. No second + final was launched; no frozen source, treatment, environment, plan, metric, + exclusion, or trial count was changed. +- **Run/identity:** `paper-good-pass` run + `019fb6e5-c00b-7b02-8a3c-d76bd1346eb4`; frozen paper commit + `eb10c26d1bfd632772baf1bc331c985d0231f52d`; clean product commit + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`; annotated product tag object + `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d`; package + `sha256:2d487a7d42bfb85058ce0f9a2336229e1bda112b940a6854ee25fbd2e604920e`; + final plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. +- **Terminal result:** benchmark/wrapper process exit 0 and empty stderr do not + override the run manifest: terminal state is `failed`, correctness `fail`, + eligibility `failed_ineligible`, and infrastructure failure is true. The + manifest started `2026-07-31T06:39:01.112787Z`, ended + `2026-07-31T06:50:13.449078Z`, and records 672.336291 seconds. The run + completed 853/1,938 trial batches and issued 2,077/5,610 product requests + before its single failure; these are completion/provenance counts, not + performance results. +- **Failed trial:** family `files`, operation `file_read`, cell + `sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16`, + measured trial + `trial-242749a59d59cc16-measured-000034`. The product request succeeded, but + the mandatory post-response snapshot boundary failed before verification. + Only one of two 14-metric resource boundary batches was durably recorded. + The trial is `infrastructure_failed`, `reportable: false`, and records + `cleanup_baseline_restored: true`. +- **Direct connection evidence:** observability request + `trial-242749a59d59cc16-measured-000034.observe.snapshot.1.boundary.0`, + invocation record + `raw/cli-subprocesses/de511434eb741316c53eba28b2e8586c402560f3d346b593c3f37c36e4547311.json`, + returned 1 with + `transport_error:gateway transport failed (connection_error)` and Windows + WSAEADDRINUSE 10048. Record SHA-256 is + `3e188c6e284c0f65e2a909df854ac18a5f550ea1daffb86f2eb3d02ebc26ab0f`; + its 187-byte stderr hashes to + `b61e0c2f370d165f3aee73407c216ca82db0bed41112c5462f1257f9888bab5c`. + The concurrent post-boundary cgroup sibling returned valid data, so the + gateway was still live and the product file-read was not the proximate + failure. +- **System evidence/root cause:** a read-only post-run query retrieved exact + Windows System/Tcpip event 4227, record 88385, timestamp + `2026-07-31T06:50:09.9652511Z`. Its message says the selected local endpoint + had recently been used for the same remote endpoint and identifies high-rate + connection open/close churn as the typical cause. The primary event capture + is + `experiments/analysis/exp1-final-system-event-4227.json`, + `sha256:b6eac476b6ecf8c20de529be5c5ca8de297874ae9273c65a4baaf6ffc34ac89d`; + message SHA-256 is + `535f882dfc5e90d98a9ffb429ef7a2c3d5eeb45d71892b54eb0af2bb1bc548dc`; + XML SHA-256 is + `864d02daffa7ea1c6030f520ccbe7983279229ea8f6ff0e2a5d468bc0a134c34`. + High-confidence scoped diagnosis is Windows local TCP endpoint-reuse + pressure at the snapshot connection. The evidence does not establish that + all 16,384 dynamic ports were simultaneously occupied, an exact TIME_WAIT + count, or a universal Windows ceiling. +- **Connection-pressure context:** before stop, the archive retained 7,992 + committed CLI invocation records: 7,991 return-code-zero and exactly one + return-code-one snapshot. Of these, 6,760 were produced in the preceding + 240 seconds; roles are manager 328, runtime 3,829, and observability 3,835. + The host reported IPv4 and IPv6 dynamic TCP ranges beginning at 49,152 with + 16,384 ports each. These contextual counts do not prove instantaneous port + occupancy. +- **Evidence-group disclosure:** the raw CLI projection directory has 7,993 + basenames, one more than the 7,992 committed invocation metadata records. + Basename + `766ed434998f2dc7c002bac6dd08c3d642cf1ec416eb2d2cc7df65a6906471a0` + has a 20,165-byte valid cgroup JSON stdout + (`sha256:0c53f4064938e550a540f0b7004f3ee0f131e097bfabb39a3874e34ff03e4c5b`) + and empty stderr, but no metadata commit marker. It is consistent with the + concurrent cgroup sibling completing while the snapshot exception unwound + the boundary gather; the exact persistence race is not logged and is not + claimed as proven. +- **Archival attempts and cleanup:** the first archive invocation failed + closed because the exact owned + `.benchmark-state/runs/019fb6e5-c00b-7b02-8a3c-d76bd1346eb4` + workspace still existed. Explicit exact-run benchmark cleanup then exited 0 + with `cleaned: true`; the owned workspace and benchmark runtime are absent. + The second archive creation and an independent verify-only invocation both + exited 0 and agree at + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb6e5-c00b-7b02-8a3c-d76bd1346eb4`: + 24,867 files, 620,311,242 bytes, content-tree + `sha256:7efa643b12aba09f0ba5ecfbed5b5692a166a5c12931490402d3992d92f3ae6a`. + Archive-manifest SHA-256 is + `5e0a3c4f7c864df8070a668d2f373b75bece3c2a57cc4340cd89ece292cc7927`; + campaign-manifest SHA-256 is + `8eefbec9772406943bb1baa2476b181c7436a6fafd7a6d7984874e8889f96982`; + raw-tree SHA-256 is + `1cc85e7883136ede15e342aa6f2ac50d72bdf6d4eace340dc0e6dba9e992f5b5`; + verify output SHA-256 is + `13b4b44f9a6f2e8835fd85b81d73de0bba7a65ac0eb7c110a57ad21168f43630`. +- **Independent final cleanup proof:** no benchmark runtime or exact run + workspace exists; no matching process remains. Protected gateway PID 62980 + remains live. The one pre-existing exited container and five pre-existing + volumes remain. Product is clean direct `main` at `0392b299`, and + `paper-v1-freeze^{}` still resolves exactly to that commit. No protected + resource was changed and nothing was pushed. +- **Inspection/orchestration attempts:** the initial background wrapper launch + returned blank output and the exact process/run ID was recovered without a + relaunch. Early progress reads addressed absent or nested manifest + properties and were corrected against the actual envelope. An initial final + archive failed cleanup proof as described above. A combined tracker patch + failed before any partial edit because of a quote-encoding mismatch; smaller + patches then succeeded. The first post-run `git diff --check` found four + Markdown trailing-space defects, which were removed. One PowerShell tag-peel + expression was parsed as a script block; quoting `paper-v1-freeze^{}` fixed + it. One orphan-group inspection searched for `.stdout.txt` instead of the + actual `.stdout` suffix and was corrected. The host PowerShell/.NET version + lacks static `SHA256.HashData`; a disposable instance `ComputeHash` + calculation then reproduced the accepted message/XML hashes. These + attempts did not mutate the frozen treatment or consume another final. +- **Validation and independent audit:** `paper_state.json`, the freeze record, + the failure diagnostic, and the System-event capture parse as JSON. + The first final-report marker check required literal `Gate 0` through + `Gate 7`, while the table initially used bare numbers for Gates 1--7; the + labels were made explicit and the complete JSON/diff/encoding/marker check + then exited 0. Post-format scoped `git diff --check` passes. A + user-requested independent read-only subagent audit reproduced the failed + trial classification, WSAEADDRINUSE connection error, still-live gateway + evidence, frozen-code control flow, archive count discrepancy, and rerun + prohibition. It made no edit or Docker call. +- **Post-run documents:** failure diagnostic + `experiments/analysis/exp1-final-failure-diagnostic.json`, claim handoff + `experiments/analysis/exp1-final-handoff.md`, primary Event 4227 capture + `experiments/analysis/exp1-final-system-event-4227.json`, and required final + report `experiments/analysis/exp1-gate0-7-final-report.md` were created. + `claim_evidence_map.md`, `experiment_inventory.md`, `progress.md`, + `plan/progress.md`, `paper_state.json`, and `benchmark/PAPER_ARTIFACT.md` + were updated. These post-run records do not alter the immutable freeze or + make partial values eligible. +- **Gate/protocol decision:** Gates 0--4 PASS. Gate 5 FAILS. Gates 6 and 7 + FAIL because partial-final aggregation and numeric paper handoff are + prohibited. No final tables or numeric-evidence v2 record are generated. + Smoke/pilot values remain qualification/exploratory only; all partial-final + performance values are excluded. Frozen v1.0 cannot be relaunched. Any + future attempt requires a scientifically documented protocol amendment, a + new source/environment freeze, and explicit author authorization; it cannot + replace this immutable failed archive. + +### 2026-07-31T07:39:16.055Z - Read-only v1.1 endpoint-pressure remedy audit + +- **Entry ID:** `exp1-v1.1-endpoint-remediation-proposal-110`. +- **Phase/kind:** post-failure archive forensics, frozen source-path audit, + primary Microsoft documentation review, host TCP configuration read, and + protocol-remediation decision package. This is diagnostic planning only: + no source, product, network setting, registry value, Git object, Docker + resource, protected process, frozen archive, or live benchmark was changed. +- **Independent audits:** three user-authorized read-only subagents separately + reviewed Windows primary sources, the immutable final archive, and + product/paper remedy options. Two recommended connection reuse as the + long-term engineering fix. One recommended per-cell gateway blocks as the + smallest paper-only amendment; root extended that audit with the accepted + pilot's cells that the failed final never reached and found a 4,532-call + projected worst cell, slightly above the failed endpoint's 4,398 committed + calls. Per-cell rotation is therefore retained as a fallback requiring a + stronger qualifier, not accepted as sufficient by itself. +- **Exact connection inventory:** the archive identifies 7,993 + client-to-gateway TCP attempts: 7,992 committed schema-2 metadata records + plus the exact valid cgroup orphan projection. Manager accounts for 328; + runtime 3,829; observability 3,836. Exact operations are create/destroy/ + inspect/list sandbox 110/110/102/6; create/destroy session 208/208; + exec 1,224; file read 1,567; file write 622; cgroup 1,918; snapshot 1,918. + The committed trace spans 589.946 seconds; the diagnostic average is 13.549 + attempts/second. +- **Failure-centered rates:** exact 10/30/60/120/240-second windows contain + 186/657/1,349/3,145/6,760 attempts, or + 18.600/21.900/22.483/26.208/28.167 attempts/second. Maximum rolling rates + were 43.700/42.767/41.667/33.100/28.404 attempts/second. The failure + occurred near the maximum sustained 240-second count, not the short-window + rate peak. These are failure-diagnostic counts, not performance results or + a universal threshold. +- **Resource split:** 3,626 connections were mandatory and 210 periodic; + mandatory traffic is 94.53% of observability and periodic traffic only + 2.63% of all attempts. There are 1,705 mandatory and 105 periodic retained + 14-metric batches. The absent 1,706th mandatory batch is the failed final + snapshot. All 1,917 complete cgroup/snapshot pairs overlapped; the 1,918th is + the failed snapshot plus its valid cgroup sibling. Removing periodic cadence + cannot solve the mandatory failure and would weaken the frozen resource + construct. +- **Endpoint and projected demand:** committed endpoints carried 2,878 + command, 4,398 files, and 716 sandbox-lifecycle calls. Proportional + trial-scoped extrapolation from the accepted seven-trial pilot to 102 trials + gives a diagnostic 26,392 total calls: files 20,578; command 2,856; + workspace lifecycle 2,244; sandbox lifecycle 714. The largest single + projected cell is the concurrency-5 256-KiB file-edit cell at 4,532 calls. + Periodic/freshness polls make these structural estimates rather than exact + final counts. +- **Frozen mechanism:** product + `crates/sandbox-operations/client/src/client.rs`, + `sha256:61ed900ae92a6911649390966b7dc47aec9eff98898f1942305c77ef4740edcc`, + creates a fresh `TcpStream::connect` and shuts down its write side per + request. Gateway + `crates/sandbox-gateway/src/gateway/connection.rs`, + `sha256:0cd590e9e3bc3db0c3c4eb0cd878bda44b7085487f3a91c5a514604b49be8a8b`, + reads one request and shuts down after its response. Retaining a client + object cannot pool this protocol; actual persistence requires product and + CLI redesign and is a separate treatment. +- **Read-only host state:** all reported TCP profiles use dynamic range + 49,152 plus 16,384 and auto-reuse range 0 plus 0. IPv4 and IPv6 `netsh` + ranges agree. `TcpTimedWaitDelay`, `MaxUserPort`, and + `StrictTimeWaitSeqCheck` are not explicitly configured in the inspected + registry key; no default duration is asserted. A transport-filter query + returned Windows access denied, and the read-only Windows principal check + reports `IsAdministrator: false`. +- **Candidate decisions:** removing all periodic sampling saves only 210 + connections and is rejected. Removing observed baseline repolls plus all + periodic sampling saves 424, 5.30%, and is not evidence of safety. Combining + cgroup/snapshot into one product operation would save 1,918 observed calls, + 24.00%, and removes the incomplete-pair shape but requires a new product + schema and has no validated safe threshold. Serializing the pair saves zero + calls. Persistent multiplexing is the preferred product fix but changes the + frozen timing construct if used for measured calls. TIME_WAIT reduction, + `SO_REUSEADDR`, pacing, sleeps, retries, and reboot-only recovery are + rejected for the first amendment. +- **Recommended campaign path:** pending explicit author authorization, use an + elevated active-store-only IPv4 range of 1,025 plus 64,511 through the + current boot. This preserves every measured one-shot subprocess/transport + boundary and changes no product operation, metric, cell, payload, trial, + exclusion, or analysis schema. It is a disclosed experiment workaround, + not the long-term product fix. The exact proposed command, not run, is + `netsh interface ipv4 set dynamicportrange protocol=tcp startport=1025 + numberofports=64511 store=active`; exact rollback, also not run, is + `startport=49152 numberofports=16384 store=active`. IPv6, TIME_WAIT, + auto-reuse, routing, firewall, Docker, and protected resources would remain + unchanged. +- **Required qualification before freeze:** capture elevated pre-state and + event cursor; apply/verify only the active IPv4 range; wait for prior state + to quiesce; run an ineligible strict same-endpoint native-CLI qualifier for + at least 700 seconds and 20,000 successful connections at no less than the + failed run's sustained 240-second rate; capture BOUND/TIME_WAIT and Event + 4227/4231 evidence; fail on any transport error/event; clean up and quiesce; + then repeat complete smoke/pilot, deterministic exploratory regeneration, + cleanup, and the no-more-than-1,400-second projection. Only then may v1.1 + source/environment identities be frozen and exactly one new final + separately authorized. The active range must be restored after archival, + including on failure. +- **Primary documentation:** Microsoft port-exhaustion/Event-4227 guidance, + dynamic-range support, `netsh` active/persistent store, SQL connection-pool + guidance, Winsock error definitions, TCP-setting WMI documentation, and + `SO_REUSEADDR` safety guidance are linked in the decision package. Microsoft + characterizes range expansion as mitigation and recommends addressing + connection churn/pooling for the long term. +- **Recorded read-only attempts:** one source inspection requested a + nonexistent `sandbox-operations/client/src/transport.rs` beside the actual + `client.rs`; the missing read changed nothing. `Get-NetTransportFilter` + failed with access denied and establishes the external elevation boundary. + A `netsh ... set dynamicport tcp ?` help probe returned a missing-parameter + diagnostic plus authoritative syntax; no setting argument was supplied and + no mutation occurred. One broad request-ID categorization initially counted + create-sandbox mandatory samples and freshness repolls as periodic; the + independent resource-observation audit corrected the accepted split to + 3,626 mandatory and 210 periodic. +- **Artifacts/blocker:** + `experiments/analysis/exp1-v1.1-remediation-decision.json` and `.md` retain + the complete machine/human decision. Both JSON state files parse; scoped + diff and encoding checks pass; operation, resource-split, range-arithmetic, + and required-marker consistency checks exit 0. The exact remaining blocker + is one author decision approving or rejecting the host-wide active-store + IPv4 mutation and the complete qualifier/refreeze/one-final sequence. No + step of that sequence is implied by earlier v1.0 authorization. + +### 2026-07-31T07:59:50.209Z - Author authorized permanent CLI transport fix and EXP1 v1.1 resumption + +- **Author direction:** after reviewing the local TCP endpoint-reuse failure + and the temporary dynamic-range workaround, the author directed: “fix the + issue then resume the goal.” This authorizes the demonstrated released-CLI + transport defect to be corrected in product `main` and authorizes the EXP1 + campaign to resume as a new protocol/treatment version after qualification. +- **Protocol decision:** EXP1 v1.1 will replace one-shot CLI-to-gateway + loopback TCP with direct OS-local IPC to the existing gateway: Windows named + pipes on the fixed host, with Unix-domain sockets implemented for + cross-platform product parity and TCP retained only as an explicit + compatibility/remote endpoint. Each measured request remains one fresh + native CLI subprocess and one request/response exchange. No persistent CLI, + broker, retry, pacing, cell, payload, trial, sampling, exclusion, metric, or + table change is authorized. +- **Why this supersedes the prior proposal:** independent native CLI processes + cannot share an in-process TCP pool. Direct local IPC removes Windows + ephemeral TCP allocation from the measured CLI-to-gateway path without an + additional broker process, while preserving the native subprocess timing + boundary. The existing gateway request handler is already transport-neutral; + only endpoint discovery/connect/listen plumbing needs to change. +- **Evidence boundary:** the sole v1.0 final and its archive remain immutable, + failed, and ineligible. Its partial values will not be rerun, replaced, + pooled, or compared numerically with v1.1. Local IPC is a scientifically + material transport treatment and therefore requires a new product identity, + package hashes, protocol/source/environment freeze, smoke, five-sample + pilot, projection no greater than 1,400 seconds, and exactly one new final + only if all preceding gates pass. +- **Host state:** the proposed IPv4 active-store range expansion was never + applied and is no longer the selected v1.1 path. IPv4, IPv6, TIME_WAIT, + auto-reuse, routing, firewall, Docker, and protected resources remain + unchanged. +- **Implementation/qualification gate:** before live benchmark smoke, require + product transport parity and failure tests, bounded gateway lifecycle and + cleanup, an ineligible high-churn native-CLI named-pipe qualification of at + least 25,000 successful invocations including concurrency-5 bursts, zero + transport failures, no new TCP/IP 4227/4231 event attributable to the + qualifier, no silent TCP fallback, and bounded process handle/RSS growth. + Any failure remains a pre-freeze blocker. +- **Git boundary:** work remains directly on product `main`; no branch, + worktree, push, or movement of the existing `paper-v1-freeze` tag is + authorized. Any successful v1.1 freeze will use new immutable commit/tag + identities and preserve all v1.0 identities. + +### 2026-07-31T08:42:08.878Z - Permanent local-IPC product fix committed and packaged + +- **Product identity:** direct product `main` commit + `56c676d588fbb704bf3da8f67d22be910453644d` (`Use local IPC for gateway + CLI transport`). No push occurred and the existing `paper-v1-freeze` tag was + not moved. +- **Implemented treatment:** typed `tcp://`, `npipe://`, and `unix://` + endpoints; Windows named-pipe and Unix-domain listeners; Windows named-pipe + client; Windows local-IPC defaults; explicit `--gateway-endpoint` with the + released `--gateway-socket` compatibility alias; and updated Windows + launcher/package documentation. The client performs one connection attempt, + one request, and one response with no retry or transport fallback. +- **Product verification:** formatting and diff checks passed. The complete + changed-crate suite for config, gateway, operation client, all three CLIs, + and MCP passed, including real named-pipe request/response and concurrency-5 + tests. Warnings-denied all-target Clippy passed for the changed crates. + A fresh release-binary integration launched five native manager CLI + processes concurrently against one named-pipe gateway; all five exited 0, + wrote one valid `{"sandboxes":[]}` JSON line, and had empty stderr. +- **Portability repairs discovered by validation:** the existing gateway + progress test compared an unescaped Windows path inside a JSON-framed log; + it now compares the serialized message. The POSIX local-daemon installer + test is now explicitly Unix-only. Both previously failing gateway tests + pass. +- **Whole-workspace boundary:** `cargo test` was attempted on native Windows. + It first exposed the pre-existing Unix-only telemetry `rustix::fs` import. + A temporary diagnostic portability experiment was fully reverted and was + not committed; the next attempt reached the pre-existing Linux namespace + process crate, which cannot compile `std::os::fd`, `std::os::unix`, or + `libc::pause` on Windows. This is not on the changed Windows + gateway/client/CLI path; changed-crate tests and strict Clippy are the + applicable native-Windows gates. +- **Package build:** the first direct `.ps1` invocation was rejected by the + host execution policy before the script ran. The repository-documented + `powershell.exe -NoProfile -ExecutionPolicy Bypass -File` invocation then + completed successfully as + `target/windows-exp1-56c676d5`. The package ZIP is 5,706,464 bytes, + `sha256:bba3376de85a80c1664cbc12114666cf00a2d9f45cc4f8cf09477cc5c6df9b1d`. +- **Packaged identities:** gateway + `sha256:80948557a439e4072a7acbd29fa7bd52824ac7e7ebdcade53656cacc1ca11c68`; + manager CLI + `sha256:9ee4ad89c1be3f9461991d4ee468b0ac5be1b8f97bd393d90b1d844c0977fa16`; + runtime CLI + `sha256:3b66c73bfe3661ec9a1c71e4d7b74925f6b0ab1957ccb6a33891c710f762082a`; + observability CLI + `sha256:3197990555825ee17a2224cad62933ff3eb720f411661015b5604360cc7c1a5b`; + Linux daemon + `sha256:f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`; + Windows config + `sha256:987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Package smoke:** the packaged gateway and manager CLI repeated the + concurrency-5 native-process named-pipe check with five exit-0, empty-stderr, + single-JSON responses. Earlier ad hoc PowerShell result-check wrappers + mis-handled empty files and process exit-code properties; those wrapper + errors did not indicate product failures and the final + `System.Diagnostics.Process` verifier passed. +- **Host/protected state:** no host network setting changed. Protected gateway + PID 62980 remained alive and untouched. The high-churn qualifier, live + smoke, pilot, freeze, and final have not started. + +### 2026-07-31T08:52:59.986Z - Final local-IPC candidate and qualifier policy preregistered + +- **Entry ID:** `exp1-v1.1-ipc-preregistration-113`. +- **Phase:** prequalification protocol/provenance freeze; no live benchmark or + qualifier result. +- **Candidate correction:** product follow-up commit + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` (`Preserve TCP endpoint + compatibility`) supersedes `56c676d588fbb704bf3da8f67d22be910453644d` + as the prequalification candidate. It preserves legacy DNS TCP endpoints, + retains the strict Windows named-pipe default, and rejects ambiguous Unix + endpoint syntax. Product `main` is clean; neither commit nor tag was pushed. +- **Final candidate package:** staged directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-exp1-5c48dae1`; + ZIP size 5,739,735 bytes and + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`. + Gateway SHA-256 is + `42e7642dd025487811abbcd78dcc5513760f2aaa1e6057cfdfa3e74c03748358`; + manager CLI + `e1faa2fe0e9f4909fa2d694166784ac65dde40ba82795b7e0c503eb5fea86513`; + runtime CLI + `e18827cf765945c958e169748575b89645c730b310ee5ffc1b42c382b44a0e26`; + observability CLI + `2b1c13bba36c9486f768824178d1e2ea8d2b1da019bd21cd1f9ea250d5da34c5`; + Linux daemon + `f5a71c3c3fe05345958b1d4d4561c64dec298022d80d3595bb0397c9b15f3c2a`; + Windows config + `987776d700108c8a9a9c1a3ed42b9155a4db46e7dde20765a79ef6df6e13677a`. +- **Package validation:** packaged gateway plus five simultaneous native + manager CLI processes completed on one unique named pipe. Each process + exited 0, had empty stderr, and emitted exactly one valid + `{"sandboxes":[]}` JSON line. No retry or TCP fallback was present. +- **Preregistration:** created + `experiments/exp1-v1.1-protocol-amendment.md` before any live qualifier + result. The stable + `EXP1 v1.1 IPC qualification policy preregistration` section fixes exactly + 25,000 `list_sandboxes` invocations as 5,000 concurrency-5 batches; one + unique named pipe; unique request IDs; strict exit/stdout/stderr validation; + no TCP, retry, fallback, or pacing; event 4227/4231 coverage through + post-cleanup; gateway-owned TCP checks at readiness, every 100 batches, + pre-stop, and post-cleanup; gateway process sampling at readiness, every + 100 batches, and pre-stop; peak and final growth caps of 32 handles, 16 MiB + private bytes, and 16 MiB RSS; exact-cap pass and cap-plus-one failure; + complete source/package/command/host provenance; and fail-closed cleanup. + The qualifier is explicitly `qualification_only` and cannot supply + performance evidence. +- **Rationale fixed before evidence:** 32 handles is no greater than the + gateway listener pending-instance maximum. The two 16 MiB allowances are + conservative fixed engineering-qualification bounds, not measured + performance results. +- **Supersession:** the historical active-store IPv4 range proposal is marked + superseded without execution. No IPv4, IPv6, TIME_WAIT, auto-reuse, + firewall, routing, Docker, or other host network setting changed. +- **Next gate:** finish harness/archive/table hardening and the complete + backend test suite, create a clean paper prequalification commit, then run + the preregistered live qualifier. Smoke, pilot, freeze, and final remain + prohibited until that gate passes. + +### 2026-07-31T12:33:58.622Z - Provenance-only Table 1 schema clarification preregistered + +- **Phase:** prequalification protocol/documentation hardening; no live + qualifier, smoke, pilot, or performance evidence was collected. +- **Clarification:** the earlier prohibition on a table change continues to + lock Tables 2--4 and every numeric/measured definition. Because gateway + transport is the scientifically material v1.1 treatment, Table 1 now adds + exactly one non-numeric text row named `Gateway transport`. +- **Schema boundary:** the deterministic table and output-manifest schema is + version 2 solely to bind the protocol version and typed transport provenance. + The value is derived from archived run-manifest evidence, never entered + manually or copied from qualifier results. Legacy v1.0 regeneration may + disclose its loopback transport without changing the frozen raw corpus. +- **Unchanged design:** the 19 cells, trials, seed, timing boundary, metrics, + aggregations, eligibility, resource cadence, correctness gates, exclusions, + and every numeric table field remain unchanged. This is the only authorized + v1.1 table-schema change. +- **Freeze-scope correction:** authoritative root `progress.md` is now included + in the archived protocol identity and scoped clean-freeze check alongside + `plan/progress.md`; tracker drift can no longer escape final provenance. + +### 2026-07-31T12:41:43.385Z - V1.1 prequalification source gate passed + +- **Entry ID:** `exp1-v1.1-prequalification-source-gate-115`. +- **Phase:** independent integrated review before the clean paper commit and + live IPC qualifier; no qualifier invocation or performance measurement. +- **Integrated tests:** with `PYTHONDONTWRITEBYTECODE=1`, the complete + `benchmark\backend\tests` plus `experiments\analysis\tests` run passed: + 354 passed, five expected Windows-symlink-privilege skips, exit 0 in + 48.88 seconds. The final qualifier-only rerun passed 46/46 in 5.28 seconds. +- **Qualifier lint/format:** offline Ruff format-check and lint over + `ipc_qualification.py` and `test_ipc_qualification.py` both exited 0. + `git diff --check` exited 0 apart from informational LF-to-CRLF checkout + warnings. +- **Recorded non-mutating tool failures:** a broader Ruff format-check over + the imported benchmark subtree exited 1 because 49 legacy files do not + match the currently cached Ruff formatter; check mode changed no file. A + broader Ruff lint invocation also exited 1 on pre-existing exception-policy, + Python-target, import-order, and style findings across legacy runner/gateway + sources. The scoped qualifier files pass both checks, and the full runtime + suite passes. No dependency was installed and no bulk reformat was applied. +- **Independent parity correction:** direct comparison found that the + qualifier's scoped paper path tuple redundantly named files already covered + by `benchmark`, while the archive tuple did not. The redundant entries were + removed; qualifier and archive `PAPER_FROZEN_SCOPE` and generated-file + exclusions now compare exactly equal. The 46 qualifier tests pass after the + correction. +- **Exact plan validation:** all three plans are runnable, select only + `product_cli`, and use only `paper-100m`. Smoke remains 19 cells, 19 + batches, 55 requests, + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + pilot remains 19/133/385, + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + final remains 19/1,938/5,610, + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. +- **Gate decision:** source, protocol, qualifier, plan, archive, projection, + and table hardening are ready for a scoped clean paper commit. The live + qualifier remains prohibited until that commit exists and the qualifier's + own product-global and paper-scoped clean checks pass. + +### 2026-07-31T12:48:54.009Z - EXP1 v1.1 local-IPC qualifier passed + +- **Entry ID:** `exp1-v1.1-ipc-qualification-pass-116`. +- **Disposition:** `qualification_only`; `performance_evidence: false`. No + value from this attempt is eligible for a manuscript table, projection, or + performance claim. +- **Bound identities:** paper prequalification commit + `50d068d3ed55a7dfd8a9be9f8a3d52f064610866`; product clean direct `main` + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; package ZIP + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`; + fixed host `DESKTOP-OLP1ADS`, Windows build 26200, x64, 48 logical + processors, 137,438,953,472 bytes physical memory. +- **Qualification identity:** `718cf58dace44dba83bed54601854bc9`; + endpoint + `npipe://./pipe/ephemeral-sandbox-exp1-ipc-718cf58dace44dba83bed54601854bc9`; + isolated gateway PID 12132. The protected v1.0 gateway PID 62980 remained + alive and untouched. +- **Strict workload result:** exactly 25,000 attempted and successful native + manager-CLI `list_sandboxes` processes in exactly 5,000 concurrency-5 + batches; zero failed invocations. Independent NDJSON verification found + 25,000 lines, 25,000 unique request IDs, 25,000 unique batch/slot pairs, and + zero records that deviated from exit 0, empty stderr, the expected 17-byte + stdout hash, or the qualification-only flags. +- **No-TCP/event gates:** all 53 readiness/cadence/pre-stop/post-cleanup TCP + samples found zero gateway-owned endpoints. The conservative System/Tcpip + query from record 88541 through 88548 found zero new 4227/4231 events. + `tcp_used: false`; no fallback, retry, or pacing was permitted. +- **Resource gates:** all 52 required process samples were present. Peak and + final handle growth above readiness were both 0 against the fixed cap 32. + Peak and final private-byte growth were both 389,120 bytes, and peak/final + RSS growth were both 475,136 bytes, against fixed 16,777,216-byte caps. +- **Cleanup:** isolated gateway process exit and PID-file removal both + validated; termination mode `terminate`; gateway return code 1 is the + recorded Windows termination status and is accepted by the preregistered + stop schema. The isolated PID is no longer alive; protected PID 62980 + remains alive. CLI stderr capture is empty. +- **Primary evidence:** active directory + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\.benchmark-state\results\qualification-only\exp1-ipc-718cf58dace44dba83bed54601854bc9`. + `summary.json` SHA-256 + `5219f09cfe58627a44dd46443c148758483b02b56125fc4a5126a30387e1d286`; + manifest + `599c8054ea1ca8517acb778648c86deca70ae20228f94fb5cc73b79fc6b7e880`; + host evidence + `83ce2faee1126340798747ce54b14d5c2ea2f541a1f7dba509e72e88fd7842ce`; + invocations NDJSON + `1605d176d6bb791ffa726ef5474e70537571719b26798356fc81bfb4de5fb9f8`. +- **Retained copy:** four files, 18,405,217 total bytes, copied with zero + SHA-256 mismatch to + `experiments\diagnostics\exp1-v11-ipc-qualification-718cf58dace44dba83bed54601854bc9`; + compressed archive size 951,312 bytes, SHA-256 + `2c4f87dc5bb123157f76e6be58b769bafef8943aba36ee8e9202601b50e62a02`. +- **Wrapper evidence limitation:** the detached `Start-Process` object was not + retained after process termination, so its OS-level wrapper exit code is + unavailable. The exact CLI stdout object byte-for-byte parses equal to + `summary.json`, reports `status: passed` with no gate failure, and its + stderr file is zero bytes. All 25,000 scientifically relevant child process + exit codes are directly present and equal to zero. +- **Next gate:** commit this append-only result/tracker update, then run one + fresh complete 19-cell v1.1 smoke. Pilot, projection, freeze, and final + remain prohibited until smoke passes. + +### 2026-07-31T13:02:42.753Z - EXP1 v1.1 CLI integration smoke passed + +- **Entry ID:** `smoke-exp1-v1.1-5c48dae1-pass-117`. +- **Phase/kind:** EXP1-C live `paper-env-smoke`, immutable archival, and + verify-only replay. This is qualification-only evidence and is permanently + ineligible for manuscript tables, runtime projection, or performance claims. +- **Bound identities:** run `019fb83a-54bc-79db-b6ac-6189fb28f5f2`; clean + paper commit `09795c39a2e4ab92ef57470064a6313985568037`; clean product direct + `main` commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; package ZIP + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`; + smoke plan + `sha256:1a8364a4612ac16834747d7619be1f63da3857d0223ec181c8a93c6851793937`; + protocol `ephemeral-sandbox-v1-practical-performance-v1.1`; pinned image + unchanged. +- **Captured result:** the supervised wrapper and benchmark both exited 0; + wrapper and benchmark stderr are empty. Manifest start + `2026-07-31T12:51:02.121558Z`, end + `2026-07-31T12:54:41.773028Z`, exact corpus elapsed 219.651470 seconds. + The launch-to-wrapper-completion bracket was 455.068 seconds and includes + complete recursive deletion of all 19 deep fixture copies. +- **Terminal corpus:** state `completed`, correctness `pass`, report ready and + non-provisional; all 19 cells, 19 trial batches, 55 issued requests, 48 + correctness checks, and 19 operation-evidence records passed with zero + failures or report warnings. The cold create-sandbox request was + 47.4741525 seconds. +- **Transport/CLI evidence:** the manifest records exactly four execution + blocks, all `windows_named_pipe`, `local_only`, and + `per_execution_block`, with `npipe` endpoint URIs. All 339 schema-2 CLI + invocations have unique request IDs, return code 0, passed response + validation and authentication redaction, and produced zero stderr bytes. + All 602 resource records and mandatory observation windows are retained. +- **Cleanup:** wrapper cleanup removed the owned run workspace and runtime. + Archive cleanup proof records no matching product process, run- or + gateway-labeled container or volume, and a clean exact product checkout. + Independent PID/name/state checks found only protected gateway PID 62980, + which remained alive and untouched. +- **Immutable archive:** creation and verify-only replay both exited 0 and + agreed at + `experiments\runs\019fb83a-54bc-79db-b6ac-6189fb28f5f2`: 1,073 files, + 29,778,180 bytes, content-tree + `sha256:c8e0e872d42c0df2ce2c19c4b030a29b615a7d250c95097dac9bff66fa4405e4`. + Archive-manifest SHA-256 is + `66736f3c23ebe372bb350071f531491a01c038d2137eb9170921ba805286d8aa`; + campaign-manifest SHA-256 is + `e2df50964644bc22abfbafffe034c71ee32569c64bb1b5789a3a01feafef6e72`. + Disposition is `smoke`, eligibility `qualification_only`, and protocol + freeze state `pre_freeze`. +- **Runtime-budget interpretation:** the inventory's three-minute acceptance + budget belongs to the already accepted Phase-2 two-lifecycle/20-call + environment qualifier, not this packet's 19-cell EXP1-C integration smoke. + EXP1-C acceptance is correctness, warnings, and leak based and specifies no + runtime cap. The 219.651470-second duration is retained as an anomaly and is + 1.605245 seconds below the previously accepted v1.0 treatment smoke + `019fb6c5-dab0-7958-b7ba-94f2a9eda944`; it does not block the pilot. +- **Diagnostic evidence-handling correction:** one read-only progress + diagnostic inspected live process command-line metadata and caused a + transient gateway credential to appear in diagnostic tool output. The + literal is intentionally not reproduced, was not copied into campaign + artifacts, and became unusable when the isolated gateway exited. All + subsequent live checks are restricted to PID/name/state and exact artifact + fields. +- **Disposition:** EXP1-C passes for the exact v1.1 treatment. One fresh + exploratory five-sample pilot is authorized after a strict fast preflight. + Freeze, tag, and final remain prohibited until pilot archival, + deterministic table regeneration, the conservative no-more-than-1,400 + second projection, anomaly review, and cleanup pass Gate 3. + +### 2026-07-31T13:11:50.680Z - EXP1 v1.1 pilot preflight passed + +- **Entry ID:** `preflight-exp1-v1.1-5c48dae1-pilot-118`. +- **Phase/kind:** EXP1-D strict read-only fast preflight; no pilot clock, + gateway, sandbox, source mutation, Docker mutation, pull, build, + installation, cleanup, freeze, tag, or final operation occurred. +- **Independent smoke review:** a read-only independent audit confirmed that + the three-minute inventory budget names the earlier two-lifecycle/20-call + environment qualifier, not the 19-cell EXP1-C smoke. It decomposed current + smoke elapsed into 81.281300 seconds fixture materialization, 52.190791 + seconds cell setup, 81.999392 seconds cell-active time, and 2.711728 seconds + terminal cleanup. The 47.474461-second cold create dominates primary + operation time, is protocol-required, and must not be prewarmed or reordered. + No amendment or rerun is warranted. +- **Smoke prerequisite:** two additional verify-only invocations, one by the + primary workflow and one independent, exited 0 and reconfirmed run + `019fb83a-54bc-79db-b6ac-6189fb28f5f2`: 1,073 files, 29,778,180 bytes, + content-tree + `sha256:c8e0e872d42c0df2ce2c19c4b030a29b615a7d250c95097dac9bff66fa4405e4`. +- **Plan:** fresh `paper-pilot` validation exited 0. It is runnable and + customized with zero findings/warnings, only `product_cli` and + `paper-100m`, exactly 19 cells, 133 trial batches, 385 issued product + requests, four execution blocks, two warmups, and five measured trials per + cell. Plan SHA-256 is + `e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`. +- **Treatment/source:** paper frozen scope was clean at commit + `d427df75e62751c99b99f42d5eebfc20bff35ff7`; product was clean direct + `main` at `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`. Package ZIP remained + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`. + Gateway, manager, runtime, observability, daemon, and configuration hashes + matched the preregistered v1.1 values. +- **CLI contract preflight:** gateway, manager, runtime, and observability help + exited 0 with empty stderr. Their stdout SHA-256 values were + `1a01a7c60c1ee1507b47b39c47ffab5d843184e9f94b5d49ecc89ab424f57934`, + `b8c3a96951784a67b8dfd021cc754ec1e58ddd5e3ba83512424721709c576fe5`, + `2b2bb48142678f0418d6b01a528e0c79b8de6ab89fafc843d4d660ccd0d7dc5e`, + and + `090d943792ace5873bf7099e3f185030474efd73a59fa876a56fa46878f56243`. +- **Host/environment:** `DESKTOP-OLP1ADS`, Windows build 26200, NTFS, 48 + logical processors, 137,438,953,472 physical-memory bytes, and + 544,201,162,752 free bytes on `C:`. Docker client/server remained 29.0.1, + Linux x86_64, `overlayfs`, cgroup v2. The exact pinned image remained local; + no pull was issued. +- **Preserved baseline:** the same eight old terminal benchmark-runtime + records, one pre-existing exited gateway-labeled container, and five + pre-existing gateway-labeled volumes remained. The only sandbox-named + process was protected gateway PID 62980. Nothing was started, stopped, + removed, or relabeled. +- **Preflight wrapper corrections:** three fail-closed helper defects were + retained. A help attempt from the backend subdirectory used a nonexistent + relative `.venv` path; a subsequent inventory command requested a + nonexistent backend-local `pyproject.toml`; neither started a product or + benchmark process. The first aggregate preflight looked for the Linux daemon + under nonexistent `libexec` instead of package `dist`; the second + incorrectly required the historical runtime-record directory to be empty + rather than equal to its known eight-entry baseline. A third aggregate + attempt disposed its help process object before retaining the exit code and + therefore falsely reported gateway-help failure. Each wrapper stopped + before the pilot. Corrected read-only checks then passed; no artifact, + source, process, Docker object, or environment state was changed by these + helper failures. +- **Clock discipline/disposition:** after committing this preflight log, one + final minimal identity/cleanliness check may run. From pilot command start + through terminal artifact and recursive workspace cleanup, no build, test, + install, pull, source/log edit, dependency mutation, Docker inspection, + recursive workspace scan, or environment reconfiguration is permitted. + Exactly one fresh exploratory v1.1 `paper-pilot` is authorized. It remains + ineligible for manuscript tables; freeze, tag, and final remain prohibited + pending complete Gate-3 audit. + +### 2026-07-31T13:37:01.222Z - EXP1 v1.1 pilot and Gate 3 passed + +- **Entry ID:** `pilot-exp1-v1.1-5c48dae1-gate3-pass-119`. +- **Phase/kind:** EXP1-D live five-sample exploratory pilot, immutable + archival, deterministic exploratory analysis, anomaly review, runtime + projection, and pre-freeze checker correction. Pilot and projected values + remain permanently ineligible for manuscript tables or performance claims. +- **Bound identities:** run `019fb84e-aef1-7fdc-9a56-1adbe712f30d`; clean + paper commit `4b962228813c43e57bfcad6d2dae7bde0e71d9d5`; clean product direct + `main` commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; package ZIP + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`; + pilot plan + `sha256:e142322153e5beec84c72994ce0da20fb78b2e418174324b96d954b6e8b6631f`; + reviewed final plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`; + protocol `ephemeral-sandbox-v1-practical-performance-v1.1`; pinned image + unchanged. +- **Live result:** the supervised wrapper and benchmark exited 0 with empty + stderr. Manifest start `2026-07-31T13:13:14.786824Z`, end + `2026-07-31T13:17:50.880871Z`, exact corpus elapsed 276.094047 seconds, + within the five-minute pilot budget. The wrapper completed recursive owned + cleanup at `2026-07-31T13:22:21.9399141Z` and retained benchmark exit 0. +- **Terminal corpus:** state `completed`, correctness `pass`, report ready and + non-provisional; exactly 19 cells, 133 trial batches, 38 warmups, 95 + successful measured trials, and 385 issued product requests. Raw evidence + contains 133 trial, 133 operation, 385 request, 336 passed correctness-check, + and 4,214 resource observations. All measured cells have exactly five + reportable trials; product, correctness, infrastructure, cleanup, and + missing-primary-latency failure counts are zero. Report warnings are empty. +- **CLI/transport evidence:** four execution blocks are exclusively + `windows_named_pipe`, `local_only`, and `per_execution_block`; no TCP + endpoint or fallback appears. All 1,903 captured native CLI invocations have + unique request IDs, return code 0, passed response validation and + authentication redaction, and emitted zero stderr bytes. +- **Resource correlation and unavailable fields:** all 4,214 resource records + carry a trial ID and cover all 133 trial IDs; all 385 request records have + unique request IDs. Eighteen cells have five aligned measured CPU/latency + pairs. The create-sandbox cell has zero CPU pairs because a sandbox-scoped + pre-create counter baseline is inapplicable, recorded explicitly rather + than zero-filled. Of 4,214 resource observations, 678 are explicitly + unavailable: 301 `layerstack_bytes` readings because the product does not + report LayerStack allocation, 301 `workspace_allocated_bytes` readings + because host metadata lacks allocated block counts, 14 each for daemon CPU, + sandbox CPU, block-read, and block-write deltas where the pre-create + baseline is inapplicable, and 10 each for current/peak sandbox memory before + the resource ring exists. These are preregistered boundary limitations, not + missing or uncorrelated samples. +- **Cleanup:** the owned run worktree and runtime were removed. Archive proof + records no matching product process, run- or gateway-labeled container or + volume, and an exact clean product checkout. Protected gateway PID 62980 + remained alive and untouched. +- **Immutable pilot archive:** creation and verify-only replay exited 0 and + agreed at + `experiments\runs\019fb84e-aef1-7fdc-9a56-1adbe712f30d`: 5,879 files, + 177,813,974 bytes, content-tree + `sha256:e951342a73d94b2f21aec76d1926bd2b1fc196303fabde5ced01d6c3ab5a4da9`. + Archive-manifest SHA-256 is + `eb7f674dabd09491de4d6ad44aa8476d4e9f9e7393033aa416cd2415e2222dd9`; + campaign-manifest SHA-256 is + `f102bd2f0f5e6a5802f8460836780900f3d3cdcb523abdb254573dfab40109eb`. + Disposition is `pilot`, eligibility `exploratory_ineligible`, and freeze + state `pre_freeze`. +- **Deterministic exploratory tables:** generation into + `experiments\analysis\pilot-v11-019fb84e-tables-a` and + `experiments\analysis\pilot-v11-019fb84e-tables-b` succeeded. Each output + contains nine files and 229,557 bytes; outputs are byte-identical with + content-tree + `sha256:f3e2e0c4d6f39622f23251c2661eb270689cd980cd00730c53ae8a600067858e`. + They are validation artifacts only and cannot enter the manuscript. +- **Projection attempts retained:** the first attempt incorrectly supplied + preset YAML to a command requiring expanded-plan JSON and failed with a + JSON parse error before producing output. Fresh final validation then + produced + `tmp\validate-paper-good-pass-v11-20260731T1327Z.json` (42,157 bytes, + `sha256:7c4d2fc4f086e06971146dcdf38028ecd4dff9865745b16b238305594e51138c`). + The second attempt failed closed because four mandatory evidence/status + files evolved between smoke and pilot. No projection output was produced by + either failed attempt. +- **Projection defect and correction:** an independent read-only recursive + provenance audit proved that every scientific identity matched and only + `progress.md`, `experiments/experiment_log.md`, `paper_state.json`, and + `plan/progress.md` changed. The six immutable scientific/protocol files and + every benchmark, analysis/archive, schema, definition, Docker, fixture, + gateway/transport, host, image, lifecycle, product/package/binary, limit, + and treatment identity were byte-identical. The checker now normalizes only + byte count and SHA-256 for those exact four pre-freeze status paths, retains + their original identities in output, requires `pre_freeze` and + `pre_freeze_worktree`, requires the exact ten-file v1.1 protocol set, and + rejects missing, extra, duplicate, malformed, frozen, or scientific drift. + This does not claim general append-prefix verification; Git history proves + the exact observed status evolution. +- **Projection validation:** the initial narrowly corrected output + `experiments\analysis\pilot-v11-019fb84e-final-runtime.json` is retained but + superseded by the tightened-checker outputs + `pilot-v11-019fb84e-final-runtime-prefreeze.json` and + `pilot-v11-019fb84e-final-runtime-prefreeze-repeat.json`. The final pair are + byte-identical at 48,311 bytes and + `sha256:a7b4eda8cd1f15e59bca2e6495cb8b8c36914619e4c812143ac361bdb1803822`; + analysis-script SHA-256 is + `d6d73f75f3a2186eef3f92ae8ef8f176f60133e366d89f6ae69a78ad5aa47781`. + Exact exploratory scheduling values are pilot 276.094047000 seconds, + central structural projection 1179.784426150 seconds, observed envelope + 1303.732241600 seconds, and limit 1400.000000000 seconds. Every condition is + true and the decision is `pass_runtime_projection`. +- **Pre-freeze identity correction:** the first draft of this entry copied a + non-authoritative candidate final-plan hash from orchestration notes. The + fresh validated expansion, frozen preset inventory, prior protocol records, + and both tightened projection outputs agree on final plan hash + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. + The incorrect transcription was corrected before freeze preflight or final + launch; no plan, preset, benchmark, archive, projection value, or treatment + changed. +- **Tests/checks:** the first real-archive regression run failed one assertion + because the test expected an invented decision prefix instead of the + implementation's exact `pass_runtime_projection`; the assertion was + corrected without changing behavior. Targeted projection tests then passed + 26/26. Full benchmark plus analysis tests passed 377 with five expected + Windows-symlink-privilege skips in 52.24 seconds. Offline Ruff 0.16.1 + formatting, format check, and scoped `E,F,I` lint passed; `git diff + --check` passed with only line-ending notices. Earlier attempts to invoke a + nonexistent venv Ruff executable/module failed without mutation; a broad + Ruff policy run exposed three pre-existing UP035/RUF007 findings, so no + unrelated policy rewrite was made. +- **Read-only helper failures retained:** monitoring once queried + `StartTime.ToUniversalTime()` across processes lacking accessible start + times and emitted null-value errors; a directory-listing `foreach` pipeline + had a PowerShell syntax error; neither changed state. During this audit, + Windows PowerShell rejected unsupported `ConvertFrom-Json -Depth`, and a + follow-up initially indexed `cells` above the report envelope's `data` + object; corrected read-only queries produced the counts above. No live + gateway, sandbox, Docker object, archive, source identity, or protected PID + was altered by these helper errors. +- **Disposition:** Gate 3 passes for the exact v1.1 treatment. No unresolved + scientific or instrumentation decision remains. Gate 4 may proceed by + committing the scoped checker/evidence update and creating the authorized + annotated local `paper-v1.1-freeze` identity. The sole eligible final run + remains prohibited until that freeze and strict final preflight pass. + +### 2026-07-31T13:40:38.895Z - EXP1 v1.1 measurement freeze and final preflight passed + +- **Entry ID:** `freeze-preflight-exp1-v1.1-1680b59-5c48dae1-120`. +- **Phase/kind:** EXP1-E Gate-4 measurement freeze and strict read-only + final preflight. No final benchmark clock began during this entry. +- **Paper freeze:** measurement source is clean frozen commit + `1680b599129532f72e706b6acb12ef62c63759e2` on + `agent/complete-pw3-and-final-host-prep`. Its benchmark source inventory is + 213 files, 16,451,598 bytes, content-tree + `sha256:c060e397ce3511a7839c71e13506dd4db99c9ad774464d0a0555f6949319dabd`. +- **Product freeze:** clean direct `main` commit + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; annotated local + `paper-v1.1-freeze` tag object + `834c84534359f37653fb25ac45304091e82c37a6`, peeled to the same commit. + Historical `paper-v1-freeze` tag object + `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d` remained unchanged. Nothing was + pushed. +- **Package/treatment:** package ZIP remained 5,739,735 bytes at + `sha256:11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`. + Gateway, manager, runtime, observability, daemon, Windows configuration, + named-pipe transport, image, fixture, sandbox limits, and host identities + all matched the accepted pilot and preregistration. +- **Freeze record:** created + `experiments\analysis\exp1-v11-freeze-record-1680b59-5c48dae1.json`, 6,888 + bytes, + `sha256:5b8ca3962f479f1776be0298acbbe7620b683a334c1964889b031122a0ffdc32`. + It verifies the exact ten protocol files, product/package/binaries/tag, + paper source, image, fixture, final plan, archive/projection scripts, and + frozen table generator + `sha256:7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7`. +- **Strict final plan:** fresh expanded output + `tmp\validate-paper-good-pass-v11-frozen-20260731T1342Z.json` is 42,157 + bytes, + `sha256:b5dd979532151f0964b031072a54c6e73f00e74f94a3948b2fc2eda0633a1880`. + Validation exited 0 with empty stderr, zero findings/warnings, exactly 19 + cells, 1,938 batches, 5,610 requests, two warmups plus 100 measured trials, + only `product_cli` and `paper-100m`, and plan + `sha256:391b521b406f0f221a7a342b822cfa8d459e339fee6c53b4a60a913a2cb0089b`. +- **Host/baseline:** host identity, Docker Desktop 29.0.1 Linux AMD64 engine, + `overlayfs`, cgroup v2, pinned local image, 543,051,427,840 free bytes, one + pre-existing exited container, five pre-existing volumes, and protected + gateway PID 62980 matched the frozen baseline. No earlier v1.1 final + existed. +- **Read-only helper failures retained:** two `rg` free-space searches + returned exit 1 despite producing the needed read-only context; one + PowerShell `rg` command had an unterminated quote; and one Docker Go-template + `join` expression failed to parse before corrected JSON inspection. None + started or changed a process, Docker object, source file, archive, package, + host setting, or protected resource. +- **Disposition:** Gate 4 passes. Exactly one `paper-good-pass` v1.1 final is + authorized under the frozen identities and no other final attempt is + permitted. + +### 2026-07-31T14:12:41.290Z - Sole EXP1 v1.1 final and immutable archive passed + +- **Entry ID:** `final-exp1-v1.1-019fb86c-gate5-pass-121`. +- **Phase/kind:** EXP1-F sole frozen final, fail-closed terminal cleanup, + immutable archive creation, and independent verify-only replay. The final + was launched once and was not rerun. +- **Run/timing:** run `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b`; manifest start + `2026-07-31T13:45:18.307991Z`; end + `2026-07-31T14:05:29.554460Z`; exact corpus elapsed 1,211.246469 seconds, + below the fixed 1,400-second envelope. The supervisor completed at + `2026-07-31T14:12:41.2897460Z` after owned deep-workspace cleanup. Benchmark + and supervisor exited 0; both stderr captures are empty. +- **Terminal corpus:** state `completed`, correctness `pass`, report ready and + non-provisional, warnings empty. Exactly 19 cells, 1,938 attempted batches, + 38 warmups, 1,900 measured attempts, 1,900 successful/reportable measured + trials, and 5,610 issued requests. Product, correctness, infrastructure, + cleanup, and missing-primary-latency failure counts are all zero. All + 4,800 correctness checks passed. +- **CLI/transport:** four execution blocks are exclusively + `windows_named_pipe`, `local_only`, and `per_execution_block`. All 26,692 + native CLI invocation records have unique request IDs, return code 0, + passed response validation and authentication redaction, and zero stderr + bytes. No TCP fallback, retry, or pacing appears. +- **Observations:** 76,276 retained observations: 1,938 trial, 1,938 + operation, 5,610 request, 4,896 check, and 61,894 resource observations. + Raw events are 45,659,053 bytes and raw observations are 67,045,771 bytes. +- **Unavailable fields:** the final report preserves LayerStack allocated + storage and host workspace allocated-block counts as unavailable for all + 1,900 reportable trials. Create-sandbox counter deltas are unavailable for + 100 trials because a sandbox-scoped pre-create baseline cannot exist; + sandbox current/peak memory is unavailable for 50 trials before its ring + exists. No unavailable value is encoded as zero. The complete raw + observation availability counts remain in `failures.md` and + `resources/resource-summary.json`. +- **Cleanup:** the owned workspace and benchmark runtime were removed. No + matching product process, run- or gateway-labeled container or volume + remains. Product is clean direct `main` at the frozen commit. Protected + gateway PID 62980 remained alive and untouched; the pre-existing container + and volume baseline remained unchanged. +- **Archive:** creation completed successfully after 872 seconds at + `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\runs\019fb86c-096e-7589-a0a4-a6d6ef5d7f8b`. + The archive contains 82,051 files and 3,139,214,747 bytes, content-tree + `sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. + Its raw subtree contains 82,025 files and 2,964,792,876 bytes, tree + `sha256:561dd3bd8ac1a7106fcf970acdcd6972a76da24fa07da147e4f19d49c83f3981`. + Archive-manifest SHA-256 is + `239dbedb781f2e427fb61b316629ea57393d1a92a3be56a45bd107e998d9131c`; + campaign-manifest SHA-256 is + `93dd241e38c48b2a3f337d66065492f101d8945f7189c2f6402a32fa0fd7e7cf`. +- **Independent verification:** immediate verify-only replay exited 0 in 36 + seconds with the identical file count, byte count, run ID, and content tree. + A later post-analysis replay again returned the same identity. The archive + remained immutable. +- **Monitoring/read-only attempts:** one progress helper queried the + observations file before it existed and emitted a `Get-Item` error; it did + not change state. The first post-final report aggregate exceeded a 10-second + helper timeout; an identical read-only aggregation with a 60-second limit + succeeded. Neither condition affected the benchmark, corpus, archive, or + final-attempt count. +- **Disposition:** Gate 5 passes with one complete provenance-rich corpus. + The final cannot be relaunched; Gate 6 may analyze only the immutable + archive. + +### 2026-07-31T14:44:57.843Z - Post-freeze analysis erratum, deterministic tables, and Gates 6--7 passed + +- **Entry ID:** `analysis-handoff-exp1-v1.1-019fb86c-pass-122`. +- **Initial fail-closed result:** the frozen generator's first final invocation + exited 1 before creating its requested output directory: + `ERROR: final archive lacks required environment fields: host OS edition, + host OS build`. The archive already contained canonical verified + `os_caption` and `os_build_number` in both campaign and preflight evidence; + the reader accepted only synthetic legacy aliases `os_edition` and + `os_build` for final eligibility. The pilot did not expose this because + exploratory output tolerates missing qualification fields. +- **Protocol decision:** an independent read-only blocker audit concluded + that the only defensible path was a narrow post-final analysis-only schema + compatibility correction. Rerunning would violate the exactly-one-final + rule, and editing the archive would violate immutability. The correction + does not change a metric, selector, aggregate, exclusion, eligibility rule, + raw value, or Tables 2--4. It is explicitly an erratum and is not described + as the frozen generator passing unchanged. +- **Correction identity:** local paper commit + `538f6c98233863957082620329203348ddaa781c` accepts canonical caption/build, + retains legacy aliases, avoids duplicated OS-family presentation, and + fails closed if caption/edition or build evidence is absent. Corrected + generator SHA-256 is + `ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b`; + frozen identity remains + `7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7`. +- **Tests:** targeted analysis tests passed 22/22. The final complete + `experiments\analysis\tests` plus `benchmark\backend\tests` run passed 380 + with five expected Windows-symlink-privilege skips in 59.86 seconds. + Python compilation and scoped `git diff --check` passed. +- **Numeric-neutrality proof:** corrected-generator replay on immutable pilot + `019fb84e-aef1-7fdc-9a56-1adbe712f30d` produced byte-identical + `numeric-evidence.json`, `numeric-provenance.csv`, Table 2, Table 3, and + Table 4 relative to the frozen-generator pilot output. Parsed Tables 2--4 + structures are identical; only intended non-numeric Table 1 host display + changed. +- **Final generation:** fresh outputs + `experiments\analysis\final-v11-019fb86c-tables-a` and `...-tables-b` + each contain nine files and 231,047 bytes. All paths, byte counts, and + SHA-256 values are identical; output content-tree is + `sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. + Output-manifest SHA-256 is + `a77b30469c3d46c26ed045f748f09313a646ef13b571f40c45a7a31f7a72505e`; + tables JSON is + `c8aaa13c58d0dad900f3d08d6a926d7736c281bacdb665e3976da269cbdab3dd`. +- **Numeric provenance:** schema + `ai-research-writing/numeric-evidence-v2` contains 153 entries and the CSV + contains 153 rows, with 153 unique IDs, no duplicate, missing ID, or numeric + mismatch. All four archive source hashes, all report selectors' + `reportable_measured` scope, and all output-manifest hashes independently + verified. +- **Archive immutability after analysis:** verify-only replay exited 0 and + reproduced 82,051 files, 3,139,214,747 bytes, and original tree + `sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. +- **Post-final helper failures retained:** the venv has no Ruff module, so a + requested format/lint invocation failed without changing files; Python + compile, tests, and diff checks supplied the applicable validation. A + PowerShell output-tree helper used unavailable `Convert.FromHexString` and + `Convert.ToHexString`; a compatible byte converter then produced the output + tree above. Two preliminary PowerShell numeric audits falsely reported 153 + and then five mismatches because of single-object `.Count` behavior and + PowerShell JSON floating-point coercion; an exact Python CSV/JSON audit + verified 153/153 values. Other read-only inspections included an `rg` + no-match exit, a nonexistent `backend` directory query, a path-separator + pattern miss, and an initially unquoted PowerShell tag-peel expression; + corrected checks succeeded. None mutated the archive, source treatment, + Docker, host settings, or protected PID, and no final was rerun. +- **Host-caption disclosure:** the canonical archived caption is literally + `Microsoft Windows 11 ???`; the localized edition suffix was not preserved + legibly by the frozen capture. Build 26200, version, architecture, computer + name, and the remaining host identity are present. A post-run read-only + registry query reported `EditionID=Core`, `CompositionEditionID=Core`, and + build 26200. That external context was not inserted into the immutable + archive or substituted into Table 1, which preserves the exact archived + caption. +- **Handoff:** `claim_evidence_map.md`, `experiment_inventory.md`, + `progress.md`, `plan/progress.md`, `paper_state.json`, + `benchmark/PAPER_ARTIFACT.md`, `exp1-final-handoff.md`, and the required + Gate report now bind the final run, outputs, supported wording, unsafe + wording, exclusions, unavailable fields, cleanup, and erratum. No numeric + result was manually inserted into LaTeX. +- **Generated-byte preservation:** repository `core.autocrlf=true`, so the + freeze record and deterministic final table trees are explicitly marked + `-text` in the root `.gitattributes`. They are versioned at their reported + paths with their generated bytes unchanged, preserving the recorded + SHA-256 identities. Large run archives, temporary diagnostics, exploratory + projections, and Python caches remain local and are excluded by the root + `.gitignore`. +- **Disposition:** Gates 6 and 7 pass. EXP1 v1.1 and the complete focused + CLI-only performance campaign are finished. There is no remaining EXP1 + execution or analysis blocker. The next external action is author review of + generated tables and bounded wording before LaTeX import; broader paper + work remains outside this campaign. diff --git a/ephemeral-sandbox-v1/experiments/final-host-requirements.in b/ephemeral-sandbox-v1/experiments/final-host-requirements.in new file mode 100644 index 0000000..e3fb951 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/final-host-requirements.in @@ -0,0 +1,8 @@ +setuptools>=77 +fastapi>=0.116,<1 +pydantic>=2.11,<3 +PyYAML>=6,<7 +uvicorn>=0.35,<1 +httpx>=0.28,<1 +pytest>=8.4,<10 +pytest-asyncio>=1.1,<2 diff --git a/ephemeral-sandbox-v1/experiments/final-host-requirements.lock b/ephemeral-sandbox-v1/experiments/final-host-requirements.lock new file mode 100644 index 0000000..c551955 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/final-host-requirements.lock @@ -0,0 +1,305 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile experiments/final-host-requirements.in --python-version 3.13 --python-platform linux --generate-hashes --output-file experiments/final-host-requirements.lock +annotated-doc==0.0.5 \ + --hash=sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101 \ + --hash=sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb + # via fastapi +annotated-types==0.8.0 \ + --hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \ + --hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0 + # via pydantic +anyio==4.14.2 \ + --hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \ + --hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f + # via + # httpx + # starlette +certifi==2026.7.22 \ + --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ + --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 + # via + # httpcore + # httpx +click==8.4.2 \ + --hash=sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6 \ + --hash=sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76 + # via uvicorn +fastapi==0.141.1 \ + --hash=sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3 \ + --hash=sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1 + # via -r experiments/final-host-requirements.in +h11==0.16.0 \ + --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ + --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 + # via + # httpcore + # uvicorn +httpcore==1.0.9 \ + --hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55 \ + --hash=sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8 + # via httpx +httpx==0.28.1 \ + --hash=sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc \ + --hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad + # via -r experiments/final-host-requirements.in +idna==3.18 \ + --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ + --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 + # via + # anyio + # httpx +iniconfig==2.3.0 \ + --hash=sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730 \ + --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 + # via pytest +packaging==26.2 \ + --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e \ + --hash=sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661 + # via pytest +pluggy==1.6.0 \ + --hash=sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3 \ + --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746 + # via pytest +pydantic==2.13.4 \ + --hash=sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba \ + --hash=sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6 + # via + # -r experiments/final-host-requirements.in + # fastapi +pydantic-core==2.46.4 \ + --hash=sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0 \ + --hash=sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262 \ + --hash=sha256:041bde0a48fd37cf71cab1c9d56d3e8625a3793fef1f7dd232b3ff37e978ecda \ + --hash=sha256:0c563b08bca408dc7f65f700633d8442fffb2421fc47b8101377e9fd65051ff0 \ + --hash=sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e \ + --hash=sha256:0ce40cd7b21210e99342afafbd4d0f76d784eb5b1d60f3bdc566be4983c6c73b \ + --hash=sha256:0e96592440881c74a213e5ad528e2b24d3d4f940de2766bed9010ab1d9e51594 \ + --hash=sha256:10e17cbb10a330363733efc4d7c4d0dd827ac0909b8f6a6542298fed1ea62f29 \ + --hash=sha256:133878133d271ade3d41d1bfb2a45ec38dbdbda40bc065921c6b04e4630127e2 \ + --hash=sha256:14d4edf427bdcf950a8a02d7cb44a08614388dd6e1bdcbf4f67504fa7887da9c \ + --hash=sha256:14f4c5d6db102bd796a627bbb3a17b4cf4574b9ae861d8b7c9a9661c6dd3362d \ + --hash=sha256:17299feefe090f2caa5b8e37222bb5f663e4935a8bfa6931d4102e5df1a9f398 \ + --hash=sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d \ + --hash=sha256:18e5ceec2ab67e6d5f1a9085e5a24c9c4e2ac4545730bfe668680bca05e555f3 \ + --hash=sha256:19e51f073cd3df251856a8a4189fbdf1de4012c3ebacfb1884f94f1eb406079f \ + --hash=sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb \ + --hash=sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7 \ + --hash=sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5 \ + --hash=sha256:228ee9bae8bef5b1e97ec58302f80357c37199e0d0a99174e138d28e6957b9d9 \ + --hash=sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462 \ + --hash=sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4 \ + --hash=sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b \ + --hash=sha256:2f84c03c8607173d16b5a854ec68a2f9079ae03237a54fb506d13af47e1d018d \ + --hash=sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df \ + --hash=sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2 \ + --hash=sha256:3447661d99f75a3683a4cf5c87da72f2161964611864dbbeac7fbb118bb4bfc0 \ + --hash=sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519 \ + --hash=sha256:395aebd9183f9d112f569aeb5b2214d1a10a33bec8456447f7fbdfa51d38d4cd \ + --hash=sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7 \ + --hash=sha256:3be77f45df024d789a672ae34f8b06fb346c4f9f46ea714956660ea4862e89ac \ + --hash=sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6 \ + --hash=sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565 \ + --hash=sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898 \ + --hash=sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb \ + --hash=sha256:432c179df7874eeb73307aad2df0755e1ae0efa61ff0ea89b93e194411ae3928 \ + --hash=sha256:4a05d69cba51d852c5c3e92758653245a50c0b646ced0cf05bd793ed592839d6 \ + --hash=sha256:4c63ebc82684aa89d9a3bcbd13d515b3be44250dc68dd3bd81526c1cb31286c3 \ + --hash=sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a \ + --hash=sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596 \ + --hash=sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987 \ + --hash=sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e \ + --hash=sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d \ + --hash=sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712 \ + --hash=sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008 \ + --hash=sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd \ + --hash=sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1 \ + --hash=sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be \ + --hash=sha256:66ce7632c22d837c95301830e111ad0128a32b8207533b60896a96c4915192ea \ + --hash=sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292 \ + --hash=sha256:6f2eeda33a839975441c86a4119e1383c50b47faf0cbb5176985565c6bb02c33 \ + --hash=sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3 \ + --hash=sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4 \ + --hash=sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b \ + --hash=sha256:7bfb192b3f4b9e8a89b6277b6ce787564f62cfd272055f6e685726b111dc7826 \ + --hash=sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac \ + --hash=sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7 \ + --hash=sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d \ + --hash=sha256:8358a950c8909158e3df31538a7e4edc2d7265a7c54b47f0864d9e5bae9dcebf \ + --hash=sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4 \ + --hash=sha256:86e1a4418c6cd97d60c95c71164158eaf7324fae7b0923264016baa993eba6fc \ + --hash=sha256:8b9bab013d1c7a79d3501ff86d0bc9c31bf587db4551677b96bec07df78c6b15 \ + --hash=sha256:8c5dac79fa1614d1e06ca695109c6105923bd9c7d1d6c918d4e637b7e6b32fd3 \ + --hash=sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b \ + --hash=sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914 \ + --hash=sha256:9037063db01f09b09e237c282b6792bd4da634b5402c4e7f0c61effed7701a04 \ + --hash=sha256:905a0ed8ea6f2d61c1738835f99b699348d7857379083e5fc497fa0c967a407c \ + --hash=sha256:90884113d8b48f760e9587002789ddd741e76ab9f89518cd1e43b1f1a52ec44b \ + --hash=sha256:91a06d2e259ecfbd8c901d70c3c507900458498142b3026a296b7de4d1322cc9 \ + --hash=sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce \ + --hash=sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4 \ + --hash=sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a \ + --hash=sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f \ + --hash=sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424 \ + --hash=sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894 \ + --hash=sha256:9bc519fbf2b7578398853d815009ae5e4d4603d12f4e3f91da8c06852d3da3e9 \ + --hash=sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76 \ + --hash=sha256:9f444c499b3eefd3a92e348059471ea0c3a6e303d9c1cec09fa748fd9f895201 \ + --hash=sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb \ + --hash=sha256:a0f62d0a58f4e7da165457e995725421e0064f2255d8eccebc49f41bbc23b109 \ + --hash=sha256:a396dcc17e5a0b164dbe026896245a4fa9ff402edca1dff0be3d53a517f74de4 \ + --hash=sha256:aaa2a54443eff1950ba5ddc6b6ccda0d9c84a364276a62f969bdf2a390650848 \ + --hash=sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526 \ + --hash=sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0 \ + --hash=sha256:b078afbc25f3a1436c7a1d2cd3e322497ee99615ba97c563566fdf46aff1ee01 \ + --hash=sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458 \ + --hash=sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e \ + --hash=sha256:bb63e0198ca18aad131c089b9204c23079c3afa95487e561f4c522d519e55aba \ + --hash=sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a \ + --hash=sha256:c1747f85cee84c26985853c6f3d9bd3e75da5212912443fa111c113b9c246f39 \ + --hash=sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c \ + --hash=sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000 \ + --hash=sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b \ + --hash=sha256:c7a7bd4e39e8e4c12c39cd480356842b6a8a06e41b23a55a5e3e191718838ddf \ + --hash=sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4 \ + --hash=sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd \ + --hash=sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28 \ + --hash=sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9 \ + --hash=sha256:d396ec2b979760aaf3218e76c24e65bd0aca24983298653b3a9d7a45f9e47b30 \ + --hash=sha256:d51026d73fcfd93610abc7b27789c26b313920fcfb20e27462d74a7f8b06e983 \ + --hash=sha256:d80ee3d731373b24cebbc10d689ca4ee1875caf0d5703a245db18efd4dd37fc1 \ + --hash=sha256:d995260fdf4e1db774581b4900e0f832abe3c7c84996726bbc161b19c8f29e76 \ + --hash=sha256:da4b951fe36dc7c3a1ccb4e3cd1747c3542b8c9ceede8fc86cae054e764485f5 \ + --hash=sha256:daa27d92c36f24388fe3ad306b174781c747627f134452e4f128ea00ce1fe8c4 \ + --hash=sha256:db06ffe51636ffe9ca531fe9023dd64bdd794be8754cb5df57c5498ae5b518a7 \ + --hash=sha256:e0d65b8c354be7fb5f720c3caa8bc940bc2d20ce749c8e06135f07f8ed95dd7c \ + --hash=sha256:e68b7a074f65a2fd746c52a7ce6142ab7006074ac269ace0c25cd8ba171f8066 \ + --hash=sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3 \ + --hash=sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02 \ + --hash=sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89 \ + --hash=sha256:ea793e075b70290d89d8142074262885d3f7da19634845135751bd6344f73b50 \ + --hash=sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76 \ + --hash=sha256:f13a646d65d09fbf1bc6b3a9635d30095c8e7e5cc419ff35ecc563c5fd04cd49 \ + --hash=sha256:f47286a97f0bc9b8859519809077b91b2cefe4ae47fcbf5e466a009c1c5d742b \ + --hash=sha256:f747929cf940cddb5b3668a390056ddd5ba2e5010615ea2dcf4f9c4f3ab8791d \ + --hash=sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7 \ + --hash=sha256:f9fa868638bf362d3d138ea55829cefb3d5f4b0d7f142234382a15e2485dbec4 \ + --hash=sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c \ + --hash=sha256:fc010ab034c8c7452522748bf937df58020d256ccae0874463d1f4d01758af8e \ + --hash=sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff \ + --hash=sha256:fd8b3d9fd264be37976686c7f65cd52a83f5e84f4bfd2adf9c1d469676bbb6ae + # via pydantic +pygments==2.20.0 \ + --hash=sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f \ + --hash=sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176 + # via pytest +pytest==9.1.1 \ + --hash=sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313 \ + --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c + # via + # -r experiments/final-host-requirements.in + # pytest-asyncio +pytest-asyncio==1.4.0 \ + --hash=sha256:933ca923a23075a87fb7070c0ec272a6848489824d887c85c812670932835aa1 \ + --hash=sha256:c6c0d2259945122819f171a32ecea2c349ead889ee28176caaf492143424be42 + # via -r experiments/final-host-requirements.in +pyyaml==6.0.3 \ + --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \ + --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \ + --hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \ + --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ + --hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \ + --hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \ + --hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \ + --hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \ + --hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \ + --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ + --hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \ + --hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \ + --hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \ + --hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \ + --hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \ + --hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \ + --hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \ + --hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \ + --hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \ + --hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \ + --hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \ + --hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \ + --hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \ + --hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \ + --hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \ + --hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \ + --hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \ + --hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \ + --hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \ + --hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \ + --hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \ + --hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \ + --hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \ + --hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \ + --hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \ + --hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \ + --hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \ + --hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \ + --hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \ + --hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \ + --hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \ + --hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \ + --hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \ + --hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \ + --hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \ + --hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \ + --hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \ + --hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \ + --hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \ + --hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \ + --hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \ + --hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \ + --hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \ + --hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \ + --hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \ + --hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \ + --hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \ + --hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \ + --hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \ + --hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \ + --hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \ + --hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \ + --hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \ + --hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \ + --hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \ + --hash=sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f \ + --hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \ + --hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \ + --hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \ + --hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \ + --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ + --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ + --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 + # via -r experiments/final-host-requirements.in +setuptools==83.0.0 \ + --hash=sha256:025bccbbf0fa05b6192bc64ae1e7b16e001fd6d6d4d5de03c97b1c1ade523bef \ + --hash=sha256:29b23c360f22f414dc7336bb39178cc7bcbf6021ed2733cde173f09dba19abb3 + # via -r experiments/final-host-requirements.in +starlette==1.3.1 \ + --hash=sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0 \ + --hash=sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6 + # via fastapi +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via + # fastapi + # pydantic + # pydantic-core + # typing-inspection +typing-inspection==0.4.2 \ + --hash=sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7 \ + --hash=sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464 + # via + # fastapi + # pydantic +uvicorn==0.52.0 \ + --hash=sha256:3d887809810b89ed33501bcf0a9aba469b06ecd608158efce04bd6b48d8c9b08 \ + --hash=sha256:ca8876ad6c1983f394157c168b39d52f6dd56dabf5602fa0982751cffc2293ae + # via -r experiments/final-host-requirements.in diff --git a/ephemeral-sandbox-v1/experiments/scripts/archive_exp1_run.py b/ephemeral-sandbox-v1/experiments/scripts/archive_exp1_run.py new file mode 100644 index 0000000..5f00411 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/archive_exp1_run.py @@ -0,0 +1,1807 @@ +#!/usr/bin/env python3 +"""Archive and verify one terminal EXP1 run without altering raw evidence.""" + +from __future__ import annotations + +import argparse +import base64 +import binascii +import collections +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any + +import yaml + +DISPOSITIONS = { + "smoke": { + "name": "paper-env-smoke", + "cells": 19, + "batches": 19, + "requests": 55, + "warmups": 0, + "measured": 1, + }, + "exploratory": { + "name": "paper-pilot", + "cells": 19, + "batches": 133, + "requests": 385, + "warmups": 2, + "measured": 5, + }, + "final": { + "name": "paper-good-pass", + "cells": 19, + "batches": 1938, + "requests": 5610, + "warmups": 2, + "measured": 100, + }, +} + +CLI_HELP = { + "gateway": ("sandbox-gateway.exe", "--help"), + "manager": ("sandbox-manager-cli.exe", "help"), + "observability": ("sandbox-observability-cli.exe", "help"), + "runtime": ("sandbox-runtime-cli.exe", "help"), +} + +BENCHMARK_EXCLUDED_PARTS = { + ".pytest_cache", + ".venv", + "__pycache__", + "dist", + "node_modules", + "playwright-report", + "test-results", +} +BENCHMARK_GIT_EXCLUSIONS = [ + *[ + f":(exclude,glob)benchmark/**/{part}/**" + for part in sorted(BENCHMARK_EXCLUDED_PARTS) + ], + ":(exclude,glob)benchmark/**/*.pyc", +] +PAPER_PROTOCOL_PATHS = ( + "progress.md", + "plan/task-packets/exp1-cli-performance-campaign.md", + "experiment_inventory.md", + "experiments/exp1-v1.1-protocol-amendment.md", + "experiments/environment_setup.md", + "experiments/expected_tables.md", + "experiments/experiment_log.md", + "benchmark/PAPER_ARTIFACT.md", + "paper_state.json", + "plan/progress.md", +) +PAPER_ANALYSIS_PATHS = ( + "benchmark/backend/benchmark_lab/derivation.py", + "benchmark/backend/benchmark_lab/reports.py", + "experiments/scripts/archive_exp1_run.py", + "experiments/scripts/project_exp1_final_runtime.py", + "experiments/analysis/scripts/generate_exp1_tables.py", +) +PAPER_FROZEN_SCOPE = ( + "benchmark", + *( + path + for path in PAPER_PROTOCOL_PATHS + if not path.startswith("benchmark/") + ), + *( + path + for path in PAPER_ANALYSIS_PATHS + if not path.startswith("benchmark/") + ), +) +EXP1_EXPECTED_HOST = { + "computer_name": "DESKTOP-OLP1ADS", + "operating_system": "windows", + "architecture": "x64", + "os_build_number": 26200, + "logical_processors": 48, + "total_memory_bytes": 137_438_953_472, + "filesystem": "NTFS", +} +EXP1_EXPECTED_SANDBOX_LIMITS = { + "profile": "standard", + "nano_cpus": 1_000_000_000, + "vcpus": 1, + "memory_bytes": 536_870_912, + "pids_limit": 256, +} +RUN_STATUSES = {"completed", "failed"} +CLI_EVIDENCE_COMMIT_PROTOCOL = "metadata-packed-payload-fsync-v1" +PROTOCOLS = { + "v1.0": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.0", + "final_tag": "paper-v1-freeze", + "gateway_endpoint_identity": "isolated_loopback_per_execution_block", + }, + "v1.1": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.1", + "final_tag": "paper-v1.1-freeze", + "gateway_endpoint_identity": ( + "isolated_windows_named_pipe_per_execution_block" + ), + }, +} +V11_GATEWAY_TRANSPORT = { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", +} +SAFE_NPIPE_ENDPOINT = re.compile( + r"npipe://\./pipe/[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z" +) + + +class ArchiveError(RuntimeError): + """The source corpus or archive contract failed validation.""" + + +def archive_eligibility(disposition: str, run_status: str) -> str: + if run_status == "failed": + return "failed_ineligible" + if run_status != "completed": + raise ArchiveError("archive run status is invalid") + if disposition == "smoke": + return "qualification_only" + if disposition == "exploratory": + return "exploratory_ineligible" + if disposition == "final": + return "frozen_final_candidate" + raise ArchiveError("archive disposition is invalid") + + +def protocol_version_from_campaign( + campaign: dict[str, Any], *, disposition: str +) -> str: + protocol = campaign.get("protocol") + if not isinstance(protocol, dict): + raise ArchiveError("campaign protocol provenance is missing") + version = protocol.get("version") + if version == "v1.1": + if protocol.get("id") != PROTOCOLS["v1.1"]["id"]: + raise ArchiveError("campaign v1.1 protocol identity is invalid") + return "v1.1" + expected = "v1.0" if disposition == "final" else "pre-freeze-exp1" + if version == expected and protocol.get("id") in (None, PROTOCOLS["v1.0"]["id"]): + return "v1.0" + raise ArchiveError("campaign protocol version is invalid") + + +def validate_protocol_transport( + *, + protocol_version: str, + environment: dict[str, Any], + manifest: dict[str, Any], + plan: dict[str, Any], + completed: bool, +) -> None: + if protocol_version not in PROTOCOLS: + raise ArchiveError("unsupported EXP1 protocol version") + if protocol_version == "v1.0": + identity = environment.get("gateway_endpoint_identity") + if identity not in (None, PROTOCOLS["v1.0"]["gateway_endpoint_identity"]): + raise ArchiveError("legacy v1.0 gateway endpoint identity drift") + return + + if ( + environment.get("gateway_endpoint_identity") + != PROTOCOLS["v1.1"]["gateway_endpoint_identity"] + or environment.get("gateway_transport") != V11_GATEWAY_TRANSPORT + ): + raise ArchiveError("v1.1 named-pipe environment identity is invalid") + policy = manifest.get("gateway_policy") + if ( + not isinstance(policy, dict) + or policy.get("protocol_version") != PROTOCOLS["v1.1"]["id"] + or any(policy.get(key) != value for key, value in V11_GATEWAY_TRANSPORT.items()) + or policy.get("mode") != "isolated" + or policy.get("isolated_runtime_per_execution_block") is not True + or policy.get("loopback_only") is not False + ): + raise ArchiveError("v1.1 named-pipe gateway policy is invalid") + blocks = plan.get("execution_blocks") + launched = manifest.get("gateway_execution_blocks") + if not isinstance(blocks, list) or not isinstance(launched, list): + raise ArchiveError("v1.1 execution-block endpoint evidence is missing") + if (completed and len(launched) != len(blocks)) or len(launched) > len(blocks): + raise ArchiveError("v1.1 execution-block endpoint count is invalid") + endpoints: set[str] = set() + for expected, observed in zip(blocks, launched): + endpoint = observed.get("endpoint_uri") if isinstance(observed, dict) else None + if ( + not isinstance(expected, dict) + or not isinstance(observed, dict) + or observed.get("block_id") != expected.get("block_id") + or observed.get("family_id") != expected.get("family_id") + or any( + observed.get(key) != value + for key, value in V11_GATEWAY_TRANSPORT.items() + ) + or not isinstance(observed.get("gateway_instance_id"), str) + or not observed["gateway_instance_id"] + or not isinstance(endpoint, str) + or SAFE_NPIPE_ENDPOINT.fullmatch(endpoint) is None + or endpoint in endpoints + ): + raise ArchiveError("v1.1 execution-block endpoint evidence is unsafe") + endpoints.add(endpoint) + + +def benchmark_source_capture_boundary(run_status: str) -> str: + if run_status == "completed": + return ( + "captured after terminal cleanup; no benchmark source file " + "changed during or between the run clock and this capture" + ) + if run_status == "failed": + return ( + "captured after terminal cleanup and after the failed-corpus " + "archival-tool amendment; archived benchmark source is post-run " + "preservation code and must not be interpreted as byte-identical " + "run-time source" + ) + raise ArchiveError("archive run status is invalid") + + +def product_checkout_capture_boundary(run_status: str) -> str: + if run_status == "completed": + return "post-run checkout is clean and exactly matches the recorded treatment" + if run_status == "failed": + return ( + "post-run checkout HEAD and status were captured after terminal cleanup; " + "they may reflect a subsequent corrective amendment and are not the " + "at-run treatment identity" + ) + raise ArchiveError("archive run status is invalid") + + +def validate_cleanup_proof( + cleanup: dict[str, Any], + *, + run_status: str, + recorded_product_commit: str, +) -> None: + if ( + cleanup.get("run_workspace_exists") is not False + or cleanup.get("runtime_exists") is not False + or cleanup.get("matching_product_processes") != [] + or cleanup.get("run_labeled_containers") != [] + or cleanup.get("gateway_labeled_containers") != [] + or cleanup.get("run_labeled_volumes") != [] + or cleanup.get("gateway_labeled_volumes") != [] + or cleanup.get("product_branch") != "main" + ): + raise ArchiveError("post-run cleanup proof failed") + status = cleanup.get("product_status_porcelain") + if not isinstance(status, str): + raise ArchiveError("post-run product status proof is invalid") + if run_status == "completed": + if ( + status + or cleanup.get("product_commit") != recorded_product_commit + or cleanup.get("product_checkout_policy") + not in (None, "clean_exact_recorded_treatment") + ): + raise ArchiveError("completed archive product checkout drift") + elif run_status == "failed": + if ( + cleanup.get("product_checkout_policy") + != "post_run_drift_recorded_failed_ineligible" + or not _is_git_sha1(cleanup.get("product_commit")) + ): + raise ArchiveError("failed archive product checkout proof is invalid") + else: + raise ArchiveError("archive run status is invalid") + + +def sha256_bytes(content: bytes) -> str: + return hashlib.sha256(content).hexdigest() + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def load_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def envelope_data(path: Path) -> dict[str, Any]: + value = load_json(path) + data = value.get("data") + if not isinstance(data, dict): + raise ArchiveError(f"artifact is not a JSON envelope: {path}") + return data + + +def write_json(path: Path, value: Any) -> None: + path.write_text( + json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n", + encoding="utf-8", + newline="\n", + ) + + +def validate_exp1_environment(environment: dict[str, Any]) -> None: + if not isinstance(environment, dict): + raise ArchiveError("recorded run environment is not an object") + host = environment.get("host") + limits = environment.get("sandbox_limits") + if not isinstance(host, dict) or not isinstance(limits, dict): + raise ArchiveError( + "recorded run environment lacks final-host or sandbox-limit evidence" + ) + required_host_text = ( + "os_caption", + "os_version", + "cpu_model", + "volume_root", + "captured_at", + "capture_source", + ) + if any( + not isinstance(host.get(field), str) or not host[field].strip() + for field in required_host_text + ): + raise ArchiveError("recorded final-host text evidence is incomplete") + if host.get("logical_processors") != host.get("processor_logical_processors"): + raise ArchiveError("recorded logical processor counts disagree") + host_mismatches = [ + field + for field, expected in EXP1_EXPECTED_HOST.items() + if ( + str(host.get(field)).casefold() != expected.casefold() + if isinstance(expected, str) + else host.get(field) != expected + ) + ] + if host.get("capture_boundary") != "run_start_before_gateway_and_measurement": + host_mismatches.append("capture_boundary") + if host_mismatches: + raise ArchiveError( + "recorded final-host identity drift: " + + ", ".join(sorted(set(host_mismatches))) + ) + limit_mismatches = [ + field + for field, expected in EXP1_EXPECTED_SANDBOX_LIMITS.items() + if limits.get(field) != expected + ] + authority = limits.get("authority") + if ( + not isinstance(authority, dict) + or authority.get("kind") != "released_gateway_configuration" + or not isinstance(authority.get("path"), str) + or not authority["path"] + or not isinstance(authority.get("sha256"), str) + or len(authority["sha256"]) != 71 + or not authority["sha256"].startswith("sha256:") + or authority.get("create_request_override") != "none" + or authority.get("capture_boundary") + != "run_start_before_gateway_and_measurement" + ): + limit_mismatches.append("authority") + if limit_mismatches: + raise ArchiveError( + "recorded effective sandbox limit drift: " + + ", ".join(sorted(set(limit_mismatches))) + ) + + +def run_checked( + args: list[str], + *, + cwd: Path, + timeout: float = 30.0, + env: dict[str, str] | None = None, +) -> subprocess.CompletedProcess[bytes]: + try: + completed = subprocess.run( + args, + cwd=cwd, + env=env, + stdin=subprocess.DEVNULL, + capture_output=True, + timeout=timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise ArchiveError(f"preflight command failed to execute: {args[0]}") from error + if completed.returncode != 0: + detail = completed.stderr.decode("utf-8", "replace")[:4096] + raise ArchiveError( + f"preflight command exited {completed.returncode}: {args[0]}: {detail}" + ) + return completed + + +def git_value(product_root: Path, *args: str) -> str: + return ( + run_checked(["git", *args], cwd=product_root) + .stdout.decode("utf-8", "strict") + .strip() + ) + + +def product_freeze_tag( + product_root: Path, + *, + disposition: str, + product_commit: str, + protocol_version: str = "v1.0", +) -> dict[str, Any]: + if protocol_version not in PROTOCOLS: + raise ArchiveError("unsupported EXP1 protocol version") + name = PROTOCOLS[protocol_version]["final_tag"] + if disposition != "final": + return { + "availability": "unavailable", + "reason": "pre-freeze smoke/exploratory archive", + "required_final_tag": name, + } + reference = f"refs/tags/{name}" + tag_object = git_value(product_root, "rev-parse", f"{reference}^{{tag}}") + object_type = git_value(product_root, "cat-file", "-t", tag_object) + peeled_commit = git_value(product_root, "rev-parse", f"{reference}^{{}}") + if ( + object_type != "tag" + or not _is_git_sha1(tag_object) + or not _is_git_sha1(peeled_commit) + or peeled_commit != product_commit + ): + raise ArchiveError( + f"final product {name} tag is absent, lightweight, " + "or does not peel to the measured product commit" + ) + return { + "availability": "available", + "name": name, + "reference": reference, + "object_type": object_type, + "tag_object": tag_object, + "peeled_commit": peeled_commit, + } + + +def _is_git_sha1(value: Any) -> bool: + return ( + isinstance(value, str) + and len(value) == 40 + and all(character in "0123456789abcdef" for character in value) + ) + + +def paper_git_provenance(paper_root: Path, *, disposition: str) -> dict[str, Any]: + git_root = Path(git_value(paper_root, "rev-parse", "--show-toplevel")).resolve( + strict=True + ) + frozen_scope = list(PAPER_FROZEN_SCOPE) + status = git_value( + paper_root, + "status", + "--porcelain=v1", + "--", + *frozen_scope, + *BENCHMARK_GIT_EXCLUSIONS, + ) + dirty = bool(status) + if disposition == "final" and dirty: + raise ArchiveError( + "final paper benchmark/protocol/analysis scope is not a clean frozen commit" + ) + return { + "root": os.fspath(git_root), + "paper_root": os.fspath(paper_root.resolve(strict=True)), + "frozen_scope": frozen_scope, + "generated_exclusions": BENCHMARK_GIT_EXCLUSIONS, + "branch": git_value(paper_root, "branch", "--show-current"), + "commit": git_value(paper_root, "rev-parse", "HEAD"), + "dirty": dirty, + "status_porcelain": status.splitlines(), + "freeze_state": ( + "clean_frozen_commit" if disposition == "final" else "pre_freeze_worktree" + ), + } + + +def validate_source( + run_path: Path, + *, + run_id: str, + disposition: str, + run_status: str, + expected_plan_hash: str, + protocol_version: str = "v1.0", +) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any], dict[str, Any]]: + expected = DISPOSITIONS[disposition] + manifest = envelope_data(run_path / "run-manifest.json") + plan = envelope_data(run_path / "expanded-plan.json") + report = envelope_data(run_path / "report.json") + intent = envelope_data(run_path / "intent-plan.json") + if manifest["run_id"] != run_id or report["run_id"] != run_id: + raise ArchiveError("run identity mismatch") + if manifest["name"] != expected["name"] or intent["name"] != expected["name"]: + raise ArchiveError("preset identity mismatch") + if run_status not in RUN_STATUSES: + raise ArchiveError("requested archive run status is invalid") + if manifest["state"] != run_status or report["state"] != run_status: + raise ArchiveError("run state does not match requested archive status") + if run_status == "completed": + if manifest["correctness"] != "pass" or report["correctness_verdict"] != "pass": + raise ArchiveError("run correctness is not pass") + else: + failure = manifest.get("failure") + if ( + manifest.get("correctness") != "fail" + or report.get("correctness_verdict") != "fail" + or not isinstance(failure, dict) + or not isinstance(failure.get("code"), str) + or not failure["code"] + or not isinstance(failure.get("message"), str) + or not failure["message"] + ): + raise ArchiveError( + "failed archive lacks explicit terminal failure evidence" + ) + if ( + manifest["plan_hash"] != expected_plan_hash + or plan["plan_hash"] != expected_plan_hash + ): + raise ArchiveError("plan hash mismatch") + estimates = plan["estimates"] + for key, expected_value in ( + ("cell_count", expected["cells"]), + ("trial_batch_count", expected["batches"]), + ("issued_operation_request_count", expected["requests"]), + ): + if estimates[key] != expected_value: + raise ArchiveError(f"plan estimate mismatch: {key}") + if plan["effective_environment"]["client_cohort"] != "product_cli": + raise ArchiveError("archive source is not product_cli") + validate_exp1_environment(manifest["environment"]) + validate_protocol_transport( + protocol_version=protocol_version, + environment=manifest["environment"], + manifest=manifest, + plan=plan, + completed=run_status == "completed", + ) + if ( + len(plan["cells"]) != expected["cells"] + or len(report["cells"]) != expected["cells"] + ): + raise ArchiveError("cell count mismatch") + for cell in plan["cells"]: + protocol = cell["protocol"] + if ( + protocol["warmups"] != expected["warmups"] + or protocol["measured_trials"] != expected["measured"] + or cell["operation"]["cell"].get("workspace_profile") != "paper-100m" + ): + raise ArchiveError("cell protocol or workspace profile drift") + if run_status == "completed": + for cell in report["cells"]: + counts = cell["counts"] + if ( + counts["warmup"] != expected["warmups"] + or counts["measured_attempted"] != expected["measured"] + or counts["successful"] != expected["measured"] + or any( + counts[field] != 0 + for field in ( + "product_failed", + "correctness_failed", + "infrastructure_failed", + "cleanup_invalid", + "missing_primary_latency", + ) + ) + ): + raise ArchiveError("report cell is not fully reportable") + if any( + check["failed"] or check["passed"] != expected["measured"] + for check in cell["checks"] + ): + raise ArchiveError("report correctness check failed") + if report["warnings"]: + raise ArchiveError("report contains warnings") + return manifest, plan, report, intent + + +def inspect_observations( + run_path: Path, + *, + expected_requests: int, + expected_batches: int, + allow_partial: bool = False, +) -> tuple[dict[str, Any], list[bytes]]: + counts: collections.Counter[str] = collections.Counter() + metric_availability: dict[str, collections.Counter[str]] = collections.defaultdict( + collections.Counter + ) + request_ids: list[str] = [] + resource_lines: list[bytes] = [] + sequences: list[int] = [] + with (run_path / "observations.ndjson").open("rb") as stream: + for raw_line in stream: + envelope = json.loads(raw_line) + data = envelope["data"] + sequences.append(data["sequence"]) + record = data["record"] + kind = record["record"] + payload = record["data"] + counts[kind] += 1 + if kind == "request": + request_ids.append(payload["request_id"]) + elif kind == "resource": + reading = payload["reading"] + metric_availability[reading["metric_id"]][ + reading["value"]["availability"] + ] += 1 + resource_lines.append(raw_line) + if sequences != list(range(1, len(sequences) + 1)): + raise ArchiveError("observation sequence is not contiguous") + complete = ( + counts["request"] == expected_requests + and len(set(request_ids)) == expected_requests + and counts["trial"] == expected_batches + and counts["operation"] == expected_batches + and bool(resource_lines) + ) + valid_partial = ( + allow_partial + and len(set(request_ids)) == len(request_ids) + and counts["request"] <= expected_requests + and counts["trial"] <= expected_batches + and counts["operation"] <= expected_batches + ) + if not complete and not valid_partial: + raise ArchiveError("observation counts or identities are incomplete") + return ( + { + "schema_version": 1, + "observation_counts": dict(sorted(counts.items())), + "resource_availability": { + metric: dict(sorted(availability.items())) + for metric, availability in sorted(metric_availability.items()) + }, + "request_id_count": len(request_ids), + "unique_request_id_count": len(set(request_ids)), + }, + resource_lines, + ) + + +def inspect_cli_evidence( + run_path: Path, *, allow_failures: bool = False +) -> dict[str, Any]: + metadata = [] + for path in sorted((run_path / "cli-subprocesses").glob("*.json")): + value = load_json(path) + schema_version = value.get("schema_version", 1) + if schema_version == 2: + _validate_cli_evidence_commit(run_path, path, value) + elif schema_version != 1: + raise ArchiveError("raw CLI metadata schema is unsupported") + metadata.append(value) + if not allow_failures: + if value["return_code"] != 0 or value["response_validation"] != "passed": + raise ArchiveError("raw CLI invocation did not pass") + if value["stderr_bytes"] != 0: + raise ArchiveError("raw CLI invocation emitted stderr") + for item in value["sanitized_argv"]: + if ( + "gateway-auth-token" in item + and item != "--gateway-auth-token=[REDACTED]" + ): + raise ArchiveError("raw CLI argv is not fully redacted") + if not metadata and not allow_failures: + raise ArchiveError("raw CLI metadata is absent") + request_ids = {item["request_id"] for item in metadata} + if len(request_ids) != len(metadata): + raise ArchiveError("raw CLI request IDs are not unique") + return { + "schema_version": 1, + "invocation_count": len(metadata), + "unique_request_id_count": len(request_ids), + "return_codes": dict( + sorted( + collections.Counter( + str(item["return_code"]) for item in metadata + ).items() + ) + ), + "response_validation": dict( + sorted( + collections.Counter( + item["response_validation"] for item in metadata + ).items() + ) + ), + "stderr_bytes": sum(item["stderr_bytes"] for item in metadata), + "auth_redaction": "passed", + "terminal_failure_evidence_allowed": allow_failures, + } + + +def _validate_cli_evidence_commit( + run_path: Path, + metadata_path: Path, + metadata: dict[str, Any], +) -> None: + invocation_id = metadata.get("invocation_id") + if ( + not isinstance(invocation_id, str) + or len(invocation_id) != 64 + or any(character not in "0123456789abcdef" for character in invocation_id) + or metadata_path.name != f"{invocation_id}.json" + or metadata.get("evidence_commit") != CLI_EVIDENCE_COMMIT_PROTOCOL + ): + raise ArchiveError("raw CLI durable commit marker is invalid") + evidence_root = run_path / "cli-subprocesses" + if ( + evidence_root.is_symlink() + or not evidence_root.is_dir() + or metadata_path.is_symlink() + or not metadata_path.is_file() + ): + raise ArchiveError("raw CLI durable commit marker is unsafe") + for stream in ("stdout", "stderr"): + relative = f"cli-subprocesses/{invocation_id}.{stream}" + if metadata.get(f"{stream}_path") != relative: + raise ArchiveError("raw CLI payload path is invalid") + packed_payload = decode_cli_evidence_payload(metadata, stream) + payload_path = run_path / relative + if payload_path.is_symlink() or not payload_path.is_file(): + raise ArchiveError("raw CLI committed payload is absent or unsafe") + if payload_path.parent.resolve(strict=True) != evidence_root.resolve(strict=True): + raise ArchiveError("raw CLI committed payload escaped evidence directory") + payload = payload_path.read_bytes() + if payload != packed_payload: + raise ArchiveError("raw CLI committed payload failed integrity validation") + + +def decode_cli_evidence_payload( + metadata: dict[str, Any], + stream: str, +) -> bytes: + if stream not in {"stdout", "stderr"}: + raise ArchiveError("raw CLI durable commit payload kind is invalid") + try: + packed_payload = base64.b64decode( + metadata[f"{stream}_base64"], validate=True + ) + except (KeyError, TypeError, ValueError, binascii.Error) as error: + raise ArchiveError("raw CLI durable commit payload is invalid") from error + expected_bytes = metadata.get(f"{stream}_bytes") + expected_sha256 = metadata.get(f"{stream}_sha256") + packed_sha256 = f"sha256:{hashlib.sha256(packed_payload).hexdigest()}" + if expected_bytes != len(packed_payload) or expected_sha256 != packed_sha256: + raise ArchiveError( + "raw CLI durable commit payload failed integrity validation" + ) + return packed_payload + + +def capture_cli_help(product_bin: Path, destination: Path) -> list[dict[str, Any]]: + destination.mkdir() + env = { + name: os.environ[name] + for name in ("PATH", "SystemRoot", "WINDIR") + if name in os.environ + } + records = [] + for role, (name, help_arg) in CLI_HELP.items(): + executable = (product_bin / name).resolve(strict=True) + if executable.parent != product_bin.resolve(strict=True): + raise ArchiveError(f"unsafe CLI help executable: {name}") + completed = run_checked( + [os.fspath(executable), help_arg], + cwd=product_bin.parent, + timeout=15.0, + env=env, + ) + if completed.stderr: + raise ArchiveError(f"CLI help emitted stderr: {name}") + (destination / f"{role}.stdout").write_bytes(completed.stdout) + (destination / f"{role}.stderr").write_bytes(completed.stderr) + record = { + "role": role, + "executable": os.fspath(executable), + "executable_sha256": f"sha256:{sha256_file(executable)}", + "argv": [os.fspath(executable), help_arg], + "return_code": completed.returncode, + "stdout_bytes": len(completed.stdout), + "stdout_sha256": f"sha256:{sha256_bytes(completed.stdout)}", + "stderr_bytes": len(completed.stderr), + "stderr_sha256": f"sha256:{sha256_bytes(completed.stderr)}", + } + write_json(destination / f"{role}.json", record) + records.append(record) + return records + + +def capture_cleanup( + *, + paper_root: Path, + product_root: Path, + run_id: str, + run_status: str, +) -> dict[str, Any]: + if os.name != "nt": + raise ArchiveError("EXP1 archive cleanup proof requires native Windows") + process_script = ( + "$names=@('sandbox-gateway.exe','sandbox-manager-cli.exe'," + "'sandbox-runtime-cli.exe','sandbox-observability-cli.exe');" + f"$run='{run_id}';" + "$p=@(Get-CimInstance Win32_Process | " + 'Where-Object {$_.Name -in $names -and $_.CommandLine -like "*$run*"} | ' + "Select-Object ProcessId,Name,CommandLine);" + "$p | ConvertTo-Json -Compress" + ) + process_result = run_checked( + ["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", process_script], + cwd=paper_root, + ) + process_text = process_result.stdout.decode("utf-8-sig", "strict").strip() + processes = [] if not process_text else json.loads(process_text) + if isinstance(processes, dict): + processes = [processes] + run_containers = ( + run_checked( + [ + "docker", + "ps", + "-a", + "--filter", + f"label=eos.benchmark.run_id={run_id}", + "--format", + "{{.ID}} {{.Names}} {{.Status}}", + ], + cwd=paper_root, + ) + .stdout.decode("utf-8", "strict") + .splitlines() + ) + gateway_containers = ( + run_checked( + [ + "docker", + "ps", + "-a", + "--filter", + "label=eos.benchmark.gateway_instance_id", + "--format", + "{{.ID}} {{.Names}} {{.Status}}", + ], + cwd=paper_root, + ) + .stdout.decode("utf-8", "strict") + .splitlines() + ) + run_volumes = ( + run_checked( + [ + "docker", + "volume", + "ls", + "--filter", + f"label=eos.benchmark.run_id={run_id}", + "--format", + "{{.Name}}", + ], + cwd=paper_root, + ) + .stdout.decode("utf-8", "strict") + .splitlines() + ) + gateway_volumes = ( + run_checked( + [ + "docker", + "volume", + "ls", + "--filter", + "label=eos.benchmark.gateway_instance_id", + "--format", + "{{.Name}}", + ], + cwd=paper_root, + ) + .stdout.decode("utf-8", "strict") + .splitlines() + ) + proof = { + "schema_version": 1, + "run_id": run_id, + "run_workspace_exists": ( + paper_root / ".benchmark-state/runs" / run_id + ).exists(), + "runtime_exists": (paper_root / ".benchmark-state/runtime" / run_id).exists(), + "matching_product_processes": processes, + "run_labeled_containers": run_containers, + "gateway_labeled_containers": gateway_containers, + "run_labeled_volumes": run_volumes, + "gateway_labeled_volumes": gateway_volumes, + "product_branch": git_value(product_root, "branch", "--show-current"), + "product_commit": git_value(product_root, "rev-parse", "HEAD"), + "product_status_porcelain": git_value(product_root, "status", "--porcelain"), + "product_checkout_policy": ( + "clean_exact_recorded_treatment" + if run_status == "completed" + else "post_run_drift_recorded_failed_ineligible" + ), + } + if ( + proof["run_workspace_exists"] + or proof["runtime_exists"] + or processes + or run_containers + or gateway_containers + or run_volumes + or gateway_volumes + or proof["product_branch"] != "main" + or (run_status == "completed" and proof["product_status_porcelain"]) + ): + raise ArchiveError("post-run cleanup proof failed") + return proof + + +def capture_environment( + *, + paper_root: Path, + product_root: Path, + product_bin: Path, + product_archive: Path, + image: str, + manifest: dict[str, Any], + command: list[str], + cli_help: list[dict[str, Any]], + cleanup: dict[str, Any], + run_status: str, + protocol_version: str = "v1.0", +) -> dict[str, Any]: + image_data = json.loads( + run_checked(["docker", "image", "inspect", image], cwd=paper_root).stdout + )[0] + docker_info = json.loads( + run_checked(["docker", "info", "--format", "{{json .}}"], cwd=paper_root).stdout + ) + binaries = { + role: { + "path": record["executable"], + "sha256": record["executable_sha256"], + "bytes": Path(record["executable"]).stat().st_size, + } + for role, record in ((item["role"], item) for item in cli_help) + } + daemon = product_bin.parent / "dist/sandbox-daemon-linux-amd64" + binaries["daemon"] = { + "path": os.fspath(daemon.resolve(strict=True)), + "sha256": f"sha256:{sha256_file(daemon)}", + "bytes": daemon.stat().st_size, + } + treatment = manifest["treatment"] + expected_hashes = { + "gateway": treatment["gateway_binary_hash"], + "manager": treatment["manager_cli_binary_hash"], + "runtime": treatment["runtime_cli_binary_hash"], + "observability": treatment["observability_cli_binary_hash"], + "daemon": treatment["daemon_binary_hash"], + } + for role, expected in expected_hashes.items(): + if binaries[role]["sha256"] != expected: + raise ArchiveError(f"product binary drift: {role}") + if ( + treatment.get("source_dirty") is not False + or treatment.get("source_diff_hash") is not None + ): + raise ArchiveError("recorded at-run product treatment was not clean") + if ( + run_status == "completed" + and cleanup["product_commit"] != treatment["source_commit"] + ): + raise ArchiveError("product source commit drift") + validate_cleanup_proof( + cleanup, + run_status=run_status, + recorded_product_commit=treatment["source_commit"], + ) + recorded_environment = manifest["environment"] + validate_exp1_environment(recorded_environment) + is_final = manifest["name"] == DISPOSITIONS["final"]["name"] + freeze_tag = product_freeze_tag( + product_root, + disposition=("final" if is_final else "exploratory"), + product_commit=treatment["source_commit"], + protocol_version=protocol_version, + ) + return { + "schema_version": 1, + "run_id": manifest["run_id"], + "disposition": manifest["name"], + "recorded_run_environment": recorded_environment, + "final_host": recorded_environment["host"], + "sandbox_limits": recorded_environment["sandbox_limits"], + "sanitized_campaign_command": command, + "product": { + "root": os.fspath(product_root), + "branch": cleanup["product_branch"], + "commit": treatment["source_commit"], + "dirty": treatment["source_dirty"], + "recorded_treatment": treatment, + "post_run_checkout": { + "branch": cleanup["product_branch"], + "commit": cleanup["product_commit"], + "status_porcelain": cleanup["product_status_porcelain"], + "dirty": bool(cleanup["product_status_porcelain"]), + "capture_boundary": product_checkout_capture_boundary(run_status), + }, + "freeze_tag": freeze_tag, + "bin_dir": os.fspath(product_bin), + "archive": { + "path": os.fspath(product_archive.resolve(strict=True)), + "bytes": product_archive.stat().st_size, + "sha256": f"sha256:{sha256_file(product_archive)}", + }, + "binaries": binaries, + }, + "docker": { + "server_version": docker_info["ServerVersion"], + "driver": docker_info["Driver"], + "cgroup_version": docker_info["CgroupVersion"], + "os_type": docker_info["OSType"], + "architecture": docker_info["Architecture"], + "warnings": docker_info["Warnings"], + }, + "image": { + "requested": image, + "id": image_data["Id"], + "repo_digests": image_data["RepoDigests"], + "os": image_data["Os"], + "architecture": image_data["Architecture"], + "size": image_data["Size"], + }, + "cleanup": cleanup, + } + + +def locate_fixture_manifest(paper_root: Path, plan: dict[str, Any]) -> Path: + profiles = plan["selected_workspace_profiles"] + if len(profiles) != 1 or profiles[0]["id"] != "paper-100m": + raise ArchiveError("unexpected workspace profile set") + candidates = sorted( + (paper_root / ".benchmark-state/fixtures/paper-100m").glob( + "*/fixture-manifest.json" + ) + ) + if len(candidates) != 1: + raise ArchiveError("paper-100m fixture manifest identity is ambiguous") + manifest = load_json(candidates[0]) + identity = manifest["identity"] + fixture = profiles[0]["fixture"] + if ( + identity["profile_id"] != profiles[0]["id"] + or identity["profile_version"] != profiles[0]["version"] + or identity["profile_generator_version"] != profiles[0]["generator_version"] + or identity["seed"] != plan["canonical_plan"]["seed"] + or manifest["actual_file_count"] != fixture["file_count"] + or manifest["actual_logical_bytes"] != fixture["logical_bytes"] + or identity["fixture"] != fixture + ): + raise ArchiveError("fixture manifest identity or realized size mismatch") + return candidates[0] + + +def archive_inventory(root: Path) -> tuple[list[dict[str, Any]], int, str]: + entries = [] + total_bytes = 0 + tree = hashlib.sha256() + for path in sorted( + (path for path in root.rglob("*") if path.is_file()), + key=lambda item: item.relative_to(root).as_posix(), + ): + relative = path.relative_to(root).as_posix() + if relative == "archive-manifest.json": + continue + size = path.stat().st_size + digest = sha256_file(path) + entries.append( + { + "path": relative, + "bytes": size, + "sha256": f"sha256:{digest}", + } + ) + total_bytes += size + tree.update(relative.encode("utf-8")) + tree.update(b"\0") + tree.update(bytes.fromhex(digest)) + tree.update(b"\n") + return entries, total_bytes, f"sha256:{tree.hexdigest()}" + + +def benchmark_source_inventory(paper_root: Path) -> dict[str, Any]: + benchmark = (paper_root / "benchmark").resolve(strict=True) + entries = [] + total_bytes = 0 + tree = hashlib.sha256() + for path in sorted( + ( + path + for path in benchmark.rglob("*") + if path.is_file() + and not path.is_symlink() + and not BENCHMARK_EXCLUDED_PARTS.intersection( + path.relative_to(benchmark).parts + ) + and path.suffix != ".pyc" + ), + key=lambda item: item.relative_to(benchmark).as_posix(), + ): + relative = path.relative_to(benchmark).as_posix() + size = path.stat().st_size + digest = sha256_file(path) + entries.append( + { + "path": relative, + "bytes": size, + "sha256": f"sha256:{digest}", + } + ) + total_bytes += size + tree.update(relative.encode("utf-8")) + tree.update(b"\0") + tree.update(bytes.fromhex(digest)) + tree.update(b"\n") + return { + "schema_version": 1, + "root": os.fspath(benchmark), + "file_count": len(entries), + "bytes": total_bytes, + "content_tree_sha256": f"sha256:{tree.hexdigest()}", + "excluded_parts": sorted(BENCHMARK_EXCLUDED_PARTS), + "files": entries, + } + + +def file_identity(path: Path, paper_root: Path) -> dict[str, Any]: + resolved = path.resolve(strict=True) + return { + "path": resolved.relative_to(paper_root).as_posix(), + "bytes": resolved.stat().st_size, + "sha256": f"sha256:{sha256_file(resolved)}", + } + + +def write_campaign_manifests( + archive_path: Path, + *, + paper_root: Path, + disposition: str, + run_status: str, + protocol_version: str = "v1.0", +) -> None: + if protocol_version not in PROTOCOLS: + raise ArchiveError("unsupported EXP1 protocol version") + source_manifest_path = archive_path / "benchmark-source-manifest.json" + campaign_manifest_path = archive_path / "campaign-manifest.json" + if source_manifest_path.exists() or campaign_manifest_path.exists(): + raise ArchiveError("campaign provenance manifest already exists") + raw = archive_path / "raw" + manifest = envelope_data(raw / "run-manifest.json") + plan = envelope_data(raw / "expanded-plan.json") + report = envelope_data(raw / "report.json") + if ( + run_status not in RUN_STATUSES + or manifest.get("state") != run_status + or report.get("state") != run_status + ): + raise ArchiveError("campaign manifest run status drift") + fixture = load_json(archive_path / "fixture-manifest.json") + environment = load_json(archive_path / "environment-preflight.txt") + benchmark_source = benchmark_source_inventory(paper_root) + write_json(source_manifest_path, benchmark_source) + raw_files, raw_bytes, raw_tree_hash = archive_inventory(raw) + paper_git = paper_git_provenance(paper_root, disposition=disposition) + protocol_paths = [paper_root / relative for relative in PAPER_PROTOCOL_PATHS] + analysis_paths = [paper_root / relative for relative in PAPER_ANALYSIS_PATHS] + campaign_manifest = { + "schema_version": 1, + "run_id": manifest["run_id"], + "disposition": disposition, + "run_status": run_status, + "eligibility": archive_eligibility(disposition, run_status), + "state": manifest["state"], + "correctness": report["correctness_verdict"], + "started_at": manifest["started_at"], + "ended_at": manifest["ended_at"], + "paper_git": { + **paper_git, + "note": ( + "The paper checkout is intentionally dirty during pre-freeze " + "execution; benchmark_source.content_tree_sha256 is the " + "authoritative run-time benchmark identity." + if disposition != "final" + else ( + "The paper benchmark, protocol, and analysis scope is a " + "clean frozen commit." + ) + ), + }, + "benchmark_source": { + "manifest": "benchmark-source-manifest.json", + "manifest_sha256": (f"sha256:{sha256_file(source_manifest_path)}"), + "file_count": benchmark_source["file_count"], + "bytes": benchmark_source["bytes"], + "content_tree_sha256": benchmark_source["content_tree_sha256"], + "capture_boundary": benchmark_source_capture_boundary(run_status), + }, + "protocol": { + "id": PROTOCOLS[protocol_version]["id"], + "version": ( + protocol_version + if protocol_version == "v1.1" or disposition == "final" + else "pre-freeze-exp1" + ), + "freeze_state": ("frozen" if disposition == "final" else "pre_freeze"), + "files": [file_identity(path, paper_root) for path in protocol_paths], + }, + "plan": { + "hash": plan["plan_hash"], + "cells": plan["estimates"]["cell_count"], + "trial_batches": plan["estimates"]["trial_batch_count"], + "issued_operation_requests": plan["estimates"][ + "issued_operation_request_count" + ], + "client_cohort": plan["effective_environment"]["client_cohort"], + "expanded_plan_sha256": ( + f"sha256:{sha256_file(raw / 'expanded-plan.json')}" + ), + }, + "fixture": { + "fixture_hash": fixture["fixture_hash"], + "tree_hash": fixture["tree_hash"], + "manifest_sha256": ( + f"sha256:{sha256_file(archive_path / 'fixture-manifest.json')}" + ), + }, + "product": environment["product"], + "docker": environment["docker"], + "image": environment["image"], + "final_host": environment["final_host"], + "sandbox_limits": environment["sandbox_limits"], + "definition_snapshot": manifest["definition_snapshot"], + "artifact_schemas": manifest["artifact_schemas"], + "analysis_and_archiving_code": { + "files": [file_identity(path, paper_root) for path in analysis_paths], + }, + "raw_corpus": { + "path": "raw", + "file_count": len(raw_files), + "bytes": raw_bytes, + "content_tree_sha256": raw_tree_hash, + "observations_sha256": ( + f"sha256:{sha256_file(raw / 'observations.ndjson')}" + ), + "events_sha256": f"sha256:{sha256_file(raw / 'events.ndjson')}", + "report_sha256": f"sha256:{sha256_file(raw / 'report.json')}", + }, + "cleanup": environment["cleanup"], + } + write_json(campaign_manifest_path, campaign_manifest) + + +def finalize_existing_archive(path: Path, *, paper_root: Path) -> dict[str, Any]: + prior = verify_archive(path) + archive_manifest = load_json(path / "archive-manifest.json") + disposition = archive_manifest["disposition"] + if disposition not in DISPOSITIONS: + raise ArchiveError("existing archive disposition is invalid") + run_status = archive_manifest.get("run_status", "completed") + if run_status not in RUN_STATUSES: + raise ArchiveError("existing archive run status is invalid") + write_campaign_manifests( + path, + paper_root=paper_root, + disposition=disposition, + run_status=run_status, + protocol_version=archive_manifest.get("protocol_version", "v1.0"), + ) + entries, total_bytes, tree_hash = archive_inventory(path) + archive_manifest.update( + { + "archive_file_count": len(entries), + "archive_bytes": total_bytes, + "content_tree_sha256": tree_hash, + "files": entries, + "supersedes_preliminary_content_tree_sha256": prior["content_tree_sha256"], + } + ) + write_json(path / "archive-manifest.json", archive_manifest) + result = verify_archive(path) + result["supersedes_preliminary_content_tree_sha256"] = prior["content_tree_sha256"] + return result + + +def verify_archive(path: Path) -> dict[str, Any]: + manifest = load_json(path / "archive-manifest.json") + entries, total_bytes, tree_hash = archive_inventory(path) + if ( + entries != manifest["files"] + or total_bytes != manifest["archive_bytes"] + or tree_hash != manifest["content_tree_sha256"] + ): + raise ArchiveError("archive inventory verification failed") + _validate_archive_provenance(path, manifest) + return { + "run_id": manifest["run_id"], + "archive_path": os.fspath(path.resolve(strict=True)), + "archive_file_count": len(entries), + "archive_bytes": total_bytes, + "content_tree_sha256": tree_hash, + "verified": True, + } + + +def _validate_archive_provenance(path: Path, archive_manifest: dict[str, Any]) -> None: + raw_manifest = envelope_data(path / "raw/run-manifest.json") + raw_report = envelope_data(path / "raw/report.json") + raw_environment = envelope_data(path / "raw/environment-metadata.json") + raw_plan_path = path / "raw/expanded-plan.json" + raw_plan = envelope_data(raw_plan_path) if raw_plan_path.is_file() else {} + copied_manifest = envelope_data(path / "run-manifest.json") + copied_report = envelope_data(path / "report.json") + preflight = load_json(path / "environment-preflight.txt") + if raw_manifest != copied_manifest: + raise ArchiveError("archived run-manifest copies disagree") + if raw_report != copied_report: + raise ArchiveError("archived report copies disagree") + if raw_manifest.get("run_id") != archive_manifest.get("run_id"): + raise ArchiveError("archive and run manifest identities disagree") + recorded_environment = raw_manifest.get("environment") + if raw_environment != recorded_environment: + raise ArchiveError("run-start environment artifact and run manifest disagree") + validate_exp1_environment(recorded_environment) + if ( + preflight.get("recorded_run_environment") != recorded_environment + or preflight.get("final_host") != recorded_environment["host"] + or preflight.get("sandbox_limits") != recorded_environment["sandbox_limits"] + ): + raise ArchiveError( + "archive preflight did not preserve run-start provenance exactly" + ) + campaign_path = path / "campaign-manifest.json" + if not campaign_path.is_file(): + raise ArchiveError("campaign provenance manifest is missing") + campaign = load_json(campaign_path) + if ( + campaign.get("final_host") != recorded_environment["host"] + or campaign.get("sandbox_limits") != recorded_environment["sandbox_limits"] + or campaign.get("product") != preflight.get("product") + ): + raise ArchiveError( + "campaign manifest did not preserve run-start/product provenance exactly" + ) + disposition = archive_manifest.get("disposition") + if disposition not in DISPOSITIONS: + raise ArchiveError("archive disposition is invalid") + run_status = archive_manifest.get("run_status", "completed") + if run_status not in RUN_STATUSES: + raise ArchiveError("archive run status is invalid") + expected_eligibility = archive_eligibility(disposition, run_status) + benchmark_source = campaign.get("benchmark_source") + if ( + archive_manifest.get("eligibility", expected_eligibility) + != expected_eligibility + or raw_manifest.get("state") != run_status + or raw_report.get("state") != run_status + or campaign.get("state") != run_status + or campaign.get("run_status", run_status) != run_status + or campaign.get("eligibility") != expected_eligibility + or raw_manifest.get("correctness") != raw_report.get("correctness_verdict") + or campaign.get("correctness") != raw_report.get("correctness_verdict") + or not isinstance(benchmark_source, dict) + or benchmark_source.get("capture_boundary") + != benchmark_source_capture_boundary(run_status) + ): + raise ArchiveError( + "archive terminal status or eligibility provenance is invalid" + ) + if run_status == "completed": + if raw_report.get("correctness_verdict") != "pass": + raise ArchiveError("completed archive correctness is not pass") + else: + failure = raw_manifest.get("failure") + if ( + raw_report.get("correctness_verdict") != "fail" + or not isinstance(failure, dict) + or not isinstance(failure.get("code"), str) + or not failure["code"] + or not isinstance(failure.get("message"), str) + or not failure["message"] + or not (path / "failures.md").is_file() + ): + raise ArchiveError("failed archive lacks terminal failure evidence") + protocol = campaign.get("protocol") + protocol_version = protocol_version_from_campaign( + campaign, disposition=disposition + ) + if ( + protocol.get("freeze_state") + != ("frozen" if disposition == "final" else "pre_freeze") + or ( + protocol_version == "v1.1" + and archive_manifest.get("protocol_version") != "v1.1" + ) + or ( + protocol_version == "v1.0" + and archive_manifest.get("protocol_version") not in (None, "v1.0") + ) + ): + raise ArchiveError("campaign protocol freeze provenance is invalid") + validate_protocol_transport( + protocol_version=protocol_version, + environment=recorded_environment, + manifest=raw_manifest, + plan=raw_plan, + completed=run_status == "completed", + ) + analysis = campaign.get("analysis_and_archiving_code") + analysis_paths = ( + { + item.get("path") + for item in analysis.get("files", []) + if isinstance(item, dict) + } + if isinstance(analysis, dict) + else set() + ) + if "experiments/analysis/scripts/generate_exp1_tables.py" not in analysis_paths: + raise ArchiveError("table generator identity is absent from the archive") + product = preflight.get("product") + paper_git = campaign.get("paper_git") + if not isinstance(product, dict) or not isinstance(paper_git, dict): + raise ArchiveError("freeze source provenance is incomplete") + treatment = raw_manifest.get("treatment") + cleanup = preflight.get("cleanup") + archived_cleanup = load_json(path / "cleanup/cleanup-proof.json") + post_run_checkout = product.get("post_run_checkout") + if ( + not isinstance(treatment, dict) + or treatment.get("source_dirty") is not False + or treatment.get("source_diff_hash") is not None + or product.get("commit") != treatment.get("source_commit") + or product.get("dirty") is not False + or product.get("recorded_treatment", treatment) != treatment + or not isinstance(cleanup, dict) + or campaign.get("cleanup") != cleanup + or archived_cleanup != cleanup + ): + raise ArchiveError("recorded at-run product or cleanup provenance is invalid") + validate_cleanup_proof( + cleanup, + run_status=run_status, + recorded_product_commit=treatment["source_commit"], + ) + if run_status == "failed": + if ( + not isinstance(post_run_checkout, dict) + or post_run_checkout.get("branch") != cleanup["product_branch"] + or post_run_checkout.get("commit") != cleanup["product_commit"] + or post_run_checkout.get("status_porcelain") + != cleanup["product_status_porcelain"] + or post_run_checkout.get("dirty") + != bool(cleanup["product_status_porcelain"]) + or post_run_checkout.get("capture_boundary") + != product_checkout_capture_boundary(run_status) + ): + raise ArchiveError("failed archive post-run product provenance is invalid") + elif post_run_checkout is not None and ( + not isinstance(post_run_checkout, dict) + or post_run_checkout.get("capture_boundary") + != product_checkout_capture_boundary(run_status) + ): + raise ArchiveError("completed archive post-run product provenance is invalid") + freeze_tag = product.get("freeze_tag") + if not isinstance(freeze_tag, dict): + raise ArchiveError("product freeze-tag provenance is missing") + if disposition == "final": + required_tag = PROTOCOLS[protocol_version]["final_tag"] + if ( + freeze_tag.get("availability") != "available" + or freeze_tag.get("name") != required_tag + or freeze_tag.get("reference") != f"refs/tags/{required_tag}" + or freeze_tag.get("object_type") != "tag" + or not _is_git_sha1(freeze_tag.get("tag_object")) + or freeze_tag.get("peeled_commit") != product.get("commit") + or paper_git.get("dirty") is not False + or paper_git.get("status_porcelain") != [] + or paper_git.get("freeze_state") != "clean_frozen_commit" + or paper_git.get("generated_exclusions") != BENCHMARK_GIT_EXCLUSIONS + or not _is_git_sha1(paper_git.get("commit")) + ): + raise ArchiveError("final source/tag freeze provenance is invalid") + elif ( + freeze_tag.get("availability") != "unavailable" + or freeze_tag.get("required_final_tag") + != PROTOCOLS[protocol_version]["final_tag"] + or paper_git.get("freeze_state") != "pre_freeze_worktree" + ): + raise ArchiveError("pre-freeze source/tag provenance is invalid") + + +def archive(args: argparse.Namespace) -> dict[str, Any]: + if args.protocol_version != "v1.1": + raise ArchiveError("new archives require explicit --protocol-version v1.1") + paper_root = args.paper_root.resolve(strict=True) + product_root = args.product_root.resolve(strict=True) + product_bin = args.product_bin_dir.resolve(strict=True) + product_archive = args.product_archive.resolve(strict=True) + archive_root = (paper_root / "experiments/runs").resolve() + run_path = (paper_root / ".benchmark-state/results" / args.run_id).resolve( + strict=True + ) + target = archive_root / args.run_id + staging = archive_root / f".{args.run_id}.staging" + if target.exists() or staging.exists(): + raise ArchiveError("archive target or staging path already exists") + if target.parent != archive_root or staging.parent != archive_root: + raise ArchiveError("archive path escaped the fixed archive root") + manifest, plan, report, intent = validate_source( + run_path, + run_id=args.run_id, + disposition=args.disposition, + run_status=args.run_status, + expected_plan_hash=args.expected_plan_hash, + protocol_version=args.protocol_version, + ) + observation_summary, resource_lines = inspect_observations( + run_path, + expected_requests=DISPOSITIONS[args.disposition]["requests"], + expected_batches=DISPOSITIONS[args.disposition]["batches"], + allow_partial=args.run_status == "failed", + ) + cli_summary = inspect_cli_evidence( + run_path, allow_failures=args.run_status == "failed" + ) + cleanup = capture_cleanup( + paper_root=paper_root, + product_root=product_root, + run_id=args.run_id, + run_status=args.run_status, + ) + archive_root.mkdir(parents=True, exist_ok=True) + staging.mkdir() + (staging / "raw").mkdir() + for source in sorted(run_path.iterdir(), key=lambda item: item.name): + destination = staging / "raw" / source.name + if source.is_dir(): + shutil.copytree(source, destination) + else: + shutil.copy2(source, destination) + for name in ("run-manifest.json", "expanded-plan.json", "report.json"): + shutil.copy2(run_path / name, staging / name) + fixture_manifest = locate_fixture_manifest(paper_root, plan) + shutil.copy2(fixture_manifest, staging / "fixture-manifest.json") + (staging / "intent-plan.yml").write_text( + yaml.safe_dump( + intent, + allow_unicode=True, + sort_keys=True, + default_flow_style=False, + ), + encoding="utf-8", + newline="\n", + ) + cli_help = capture_cli_help(product_bin, staging / "cli-help") + (staging / "resources").mkdir() + with (staging / "resources/resource-observations.ndjson").open("wb") as stream: + for line in resource_lines: + stream.write(line) + write_json(staging / "resources/resource-summary.json", observation_summary) + (staging / "logs").mkdir() + shutil.copy2(run_path / "events.ndjson", staging / "logs/events.ndjson") + write_json(staging / "logs/cli-summary.json", cli_summary) + (staging / "cleanup").mkdir() + write_json(staging / "cleanup/cleanup-proof.json", cleanup) + command = [ + os.fspath( + (paper_root / ".venv/Scripts/sandbox-benchmark.exe").resolve(strict=True) + ), + "run", + "--test-repository-root", + os.fspath(paper_root), + "--product-root", + os.fspath(product_root), + "--product-bin-dir", + os.fspath(product_bin), + "--plan", + DISPOSITIONS[args.disposition]["name"], + ] + environment = capture_environment( + paper_root=paper_root, + product_root=product_root, + product_bin=product_bin, + product_archive=product_archive, + image=args.image, + manifest=manifest, + command=command, + cli_help=cli_help, + cleanup=cleanup, + run_status=args.run_status, + protocol_version=args.protocol_version, + ) + write_json(staging / "environment-preflight.txt", environment) + unavailable = observation_summary["resource_availability"] + unavailable_lines = [ + f"- `{metric}`: {availability.get('unavailable', 0)} unavailable, " + f"{availability.get('available', 0)} available observations" + for metric, availability in unavailable.items() + if availability.get("unavailable", 0) + ] + if args.run_status == "failed": + failure_text = ( + "# Failures, exclusions, warnings, and unavailable fields\n\n" + "- Failures: terminal campaign failure; authoritative raw evidence is " + "preserved below `raw/`.\n" + "- Exclusions: this entire corpus is ineligible for pilot projection, " + "final tables, and manuscript claims.\n" + f"- Report warnings: {len(report['warnings'])}; preserved verbatim in " + "`raw/report.json`.\n" + "- Correctness: fail.\n" + "- Pilot/final eligibility: failed_ineligible.\n\n" + "## Recorded terminal failure\n\n" + "```json\n" + + json.dumps( + manifest["failure"], + indent=2, + sort_keys=True, + ensure_ascii=False, + ) + + "\n```\n\n" + "## Explicitly unavailable resource observations\n\n" + + ("\n".join(unavailable_lines) if unavailable_lines else "- None.") + + "\n\n" + "Unavailable observations remain unavailable; they are never encoded as zero.\n" + ) + else: + failure_text = ( + "# Failures, exclusions, warnings, and unavailable fields\n\n" + "- Failures: none.\n" + "- Exclusions: none.\n" + "- Report warnings: none.\n" + "- Correctness: pass.\n" + "- Pilot/final eligibility: " + + ( + "ineligible smoke qualification evidence." + if args.disposition == "smoke" + else ( + "ineligible exploratory pilot evidence." + if args.disposition == "exploratory" + else "eligible only after complete frozen-corpus verification." + ) + ) + + "\n\n" + "## Explicitly unavailable resource observations\n\n" + + ("\n".join(unavailable_lines) if unavailable_lines else "- None.") + + "\n\n" + "Unavailable observations remain unavailable; they are never encoded as zero.\n" + ) + (staging / "failures.md").write_text(failure_text, encoding="utf-8", newline="\n") + write_campaign_manifests( + staging, + paper_root=paper_root, + disposition=args.disposition, + run_status=args.run_status, + protocol_version=args.protocol_version, + ) + entries, total_bytes, tree_hash = archive_inventory(staging) + write_json( + staging / "archive-manifest.json", + { + "schema_version": 1, + "run_id": args.run_id, + "disposition": args.disposition, + "run_status": args.run_status, + "eligibility": archive_eligibility(args.disposition, args.run_status), + "protocol_version": args.protocol_version, + "source_path": os.fspath(run_path), + "plan_hash": args.expected_plan_hash, + "archive_file_count": len(entries), + "archive_bytes": total_bytes, + "content_tree_sha256": tree_hash, + "files": entries, + }, + ) + staging.replace(target) + return verify_archive(target) + + +def parser() -> argparse.ArgumentParser: + command = argparse.ArgumentParser() + command.add_argument("--run-id") + command.add_argument("--disposition", choices=sorted(DISPOSITIONS)) + command.add_argument( + "--protocol-version", + choices=("v1.1",), + help="required explicitly when creating a new archive", + ) + command.add_argument( + "--run-status", + choices=sorted(RUN_STATUSES), + default="completed", + help=( + "expected terminal status; failed must be requested explicitly and " + "is always archived as ineligible" + ), + ) + command.add_argument("--expected-plan-hash") + command.add_argument("--paper-root", type=Path) + command.add_argument("--product-root", type=Path) + command.add_argument("--product-bin-dir", type=Path) + command.add_argument("--product-archive", type=Path) + command.add_argument("--image") + command.add_argument("--verify", type=Path) + command.add_argument("--finalize-existing", type=Path) + return command + + +def main() -> int: + args = parser().parse_args() + try: + if args.verify is not None and args.finalize_existing is not None: + raise ArchiveError( + "--verify and --finalize-existing are mutually exclusive" + ) + if args.verify is not None: + result = verify_archive(args.verify.resolve(strict=True)) + elif args.finalize_existing is not None: + if args.paper_root is None: + raise ArchiveError("--paper-root is required with --finalize-existing") + result = finalize_existing_archive( + args.finalize_existing.resolve(strict=True), + paper_root=args.paper_root.resolve(strict=True), + ) + else: + required = ( + "run_id", + "disposition", + "expected_plan_hash", + "paper_root", + "product_root", + "product_bin_dir", + "product_archive", + "image", + "protocol_version", + ) + missing = [name for name in required if getattr(args, name) is None] + if missing: + raise ArchiveError( + f"archive arguments are missing: {', '.join(missing)}" + ) + result = archive(args) + except (ArchiveError, KeyError, OSError, ValueError, json.JSONDecodeError) as error: + print(f"archive_error: {error}", file=sys.stderr) + return 1 + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/scripts/capture_exp1_baseline.py b/ephemeral-sandbox-v1/experiments/scripts/capture_exp1_baseline.py new file mode 100644 index 0000000..f67de7e --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/capture_exp1_baseline.py @@ -0,0 +1,80 @@ +from __future__ import annotations + +import argparse +import hashlib +import json +import subprocess +from pathlib import Path + + +def _git(root: Path, *arguments: str) -> bytes: + completed = subprocess.run( + ["git", "-C", str(root), *arguments], + check=True, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + return completed.stdout + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def _repository(root: Path) -> dict[str, object]: + tracked = [ + item.decode("utf-8") + for item in _git(root, "ls-files", "-z", "--cached").split(b"\0") + if item + ] + return { + "root": str(root), + "commit": _git(root, "rev-parse", "HEAD").decode().strip(), + "branch": _git(root, "branch", "--show-current").decode().strip(), + "status_porcelain_v2": _git( + root, "status", "--porcelain=v2", "--branch", "--untracked-files=all" + ).decode(), + "tracked_files": { + relative: f"sha256:{_sha256(root / relative)}" + for relative in sorted(tracked) + }, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--paper-root", type=Path, required=True) + parser.add_argument("--product-root", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + arguments = parser.parse_args() + output = arguments.output.resolve() + if output.exists(): + raise SystemExit("baseline output already exists") + payload = { + "schema_version": 1, + "purpose": "EXP1 pre-edit tracked-file SHA-256 baseline", + "initial_observation": { + "paper_status": "clean at the recorded commit before this capture script was added", + "product_status": "clean at the recorded commit", + "new_campaign_files": [ + "experiments/scripts/capture_exp1_baseline.py", + "experiments/exp1-pre-edit-baseline.json", + ], + }, + "paper": _repository(arguments.paper_root.resolve(strict=True)), + "product": _repository(arguments.product_root.resolve(strict=True)), + } + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text( + json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/scripts/cli_environment_smoke.sh b/ephemeral-sandbox-v1/experiments/scripts/cli_environment_smoke.sh new file mode 100644 index 0000000..a97da96 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/cli_environment_smoke.sh @@ -0,0 +1,535 @@ +#!/usr/bin/env bash +set -euo pipefail + +started_seconds="$(date +%s)" +artifact_directory="${1:?usage: cli_environment_smoke.sh ARTIFACT_DIRECTORY}" +paper_root="${PAPER_ROOT:-/srv/eos-benchmark/paper}" +product_root="${PRODUCT_ROOT:-/srv/eos-benchmark/product}" +product_bin_dir="${PRODUCT_BIN_DIR:-${product_root}/target/release}" +python_bin="${PYTHON_BIN:-${paper_root}/.venv/bin/python}" +image_reference="${IMAGE_REFERENCE:-ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf}" +maximum_seconds="${MAX_CLI_SMOKE_SECONDS:-180}" + +gateway_binary="${product_bin_dir}/sandbox-gateway" +manager_cli="${product_bin_dir}/sandbox-manager-cli" +runtime_cli="${product_bin_dir}/sandbox-runtime-cli" +observability_cli="${product_bin_dir}/sandbox-observability-cli" +daemon_binary="${product_root}/dist/sandbox-daemon-linux-amd64" +gateway_template="${paper_root}/benchmark/defaults/gateway.yml" + +runtime_directory="${artifact_directory}/cli-runtime" +gateway_config="${runtime_directory}/effective-config.yml" +gateway_log="${artifact_directory}/gateway.log" +gateway_pid_file="${runtime_directory}/gateway.pid" +gateway_registry="${runtime_directory}/registry.json" +summary_file="${artifact_directory}/cli-env-smoke-summary.json" +gateway_instance_id="cli-env-smoke-$(date -u +%Y%m%dT%H%M%SZ)-$$" + +gateway_pid="" +sandbox_id="" +gateway_token="" +batch_sandbox_ids=() + +fail() { + printf 'FAIL\t%s\n' "$1" >&2 + exit 1 +} + +require_executable() { + [[ -f "$1" && ! -L "$1" && -x "$1" ]] || + fail "missing or unsafe executable: $1" +} + +stop_gateway() { + [[ -n "$gateway_pid" ]] || return 0 + if kill -0 "$gateway_pid" 2>/dev/null; then + kill -TERM -- "-${gateway_pid}" 2>/dev/null || + kill -TERM "$gateway_pid" 2>/dev/null || + true + for _ in $(seq 1 50); do + kill -0 "$gateway_pid" 2>/dev/null || break + sleep 0.1 + done + if kill -0 "$gateway_pid" 2>/dev/null; then + kill -KILL -- "-${gateway_pid}" 2>/dev/null || + kill -KILL "$gateway_pid" 2>/dev/null || + true + fi + wait "$gateway_pid" 2>/dev/null || true + fi + gateway_pid="" +} + +cleanup_gateway_volumes() { + local volume_name + local volume_gateway_id + local cleanup_stdout="${artifact_directory}/owned-volume-cleanup.stdout" + local cleanup_stderr="${artifact_directory}/owned-volume-cleanup.stderr" + : >"$cleanup_stdout" + : >"$cleanup_stderr" + while IFS= read -r volume_name; do + [[ -n "$volume_name" ]] || continue + volume_gateway_id="$( + docker volume inspect \ + --format '{{ index .Labels "eos.gateway_instance_id" }}' \ + "$volume_name" 2>>"$cleanup_stderr" + )" || return 1 + [[ "$volume_gateway_id" == "$gateway_instance_id" ]] || return 1 + docker volume rm -- "$volume_name" \ + >>"$cleanup_stdout" 2>>"$cleanup_stderr" || return 1 + done < <( + docker volume ls -q \ + --filter "label=eos.gateway_instance_id=${gateway_instance_id}" + ) +} + +require_no_owned_docker_resources() { + local owned_containers + local owned_volumes + owned_containers="$( + docker ps -aq \ + --filter "label=eos.gateway_instance_id=${gateway_instance_id}" + )" + owned_volumes="$( + docker volume ls -q \ + --filter "label=eos.gateway_instance_id=${gateway_instance_id}" + )" + [[ -z "$owned_containers" && -z "$owned_volumes" ]] +} + +cleanup_on_exit() { + local status="$?" + trap - EXIT INT TERM + set +e + if [[ -n "$sandbox_id" ]]; then + "$manager_cli" destroy_sandbox --sandbox-id "$sandbox_id" \ + >"${artifact_directory}/cleanup-destroy.json" \ + 2>"${artifact_directory}/cleanup-destroy.stderr" + if [[ "$?" -ne 0 ]]; then + status=1 + fi + fi + stop_gateway + if ! cleanup_gateway_volumes; then + status=1 + fi + if ! require_no_owned_docker_resources; then + status=1 + fi + unset SANDBOX_GATEWAY_AUTH_TOKEN gateway_token + exit "$status" +} + +for executable in \ + "$gateway_binary" \ + "$manager_cli" \ + "$runtime_cli" \ + "$observability_cli" \ + "$daemon_binary" +do + require_executable "$executable" +done +[[ -x "$python_bin" ]] || + fail "Python interpreter is missing or not executable: $python_bin" +for command_name in date docker find grep kill sed seq setsid sleep; do + command -v "$command_name" >/dev/null 2>&1 || + fail "missing command: $command_name" +done +[[ -f "$gateway_template" && ! -L "$gateway_template" ]] || + fail "gateway configuration template is missing or unsafe" +[[ "$image_reference" =~ @sha256:[0-9a-f]{64}$ ]] || + fail "sandbox image is not pinned by a full sha256 digest" + +mkdir -p -- "$artifact_directory" "$runtime_directory" +trap cleanup_on_exit EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +gateway_socket="$( + "$python_bin" - <<'PY' +import socket + +with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.bind(("127.0.0.1", 0)) + host, port = sock.getsockname() +print(f"{host}:{port}") +PY +)" +gateway_token="$( + "$python_bin" - <<'PY' +import secrets + +print(secrets.token_urlsafe(48)) +PY +)" +export SANDBOX_GATEWAY_SOCKET="$gateway_socket" +export SANDBOX_GATEWAY_AUTH_TOKEN="$gateway_token" +export EOS_SHARED_BASE_CACHE="${runtime_directory}/shared-base-cache" + +PAPER_ROOT="$paper_root" \ +GATEWAY_CONFIG="$gateway_config" \ +GATEWAY_INSTANCE_ID="$gateway_instance_id" \ +GATEWAY_PID_FILE="$gateway_pid_file" \ +GATEWAY_REGISTRY="$gateway_registry" \ +GATEWAY_SOCKET="$gateway_socket" \ +QUALIFICATION_ROOT="$artifact_directory" \ +DAEMON_BINARY="$daemon_binary" \ +"$python_bin" - <<'PY' +import os +from pathlib import Path + +import yaml + +paper_root = Path(os.environ["PAPER_ROOT"]) +template = paper_root / "benchmark" / "defaults" / "gateway.yml" +config = yaml.safe_load(template.read_text(encoding="utf-8")) +config["gateway"].update( + { + "bind_addr": os.environ["GATEWAY_SOCKET"], + "pid_path": os.environ["GATEWAY_PID_FILE"], + "max_concurrent_connections": 64, + } +) +config["manager"].update( + { + "registry_path": os.environ["GATEWAY_REGISTRY"], + "workspace_roots": [os.environ["QUALIFICATION_ROOT"]], + } +) +config["manager"]["docker"].update( + { + "daemon_binary_path": os.environ["DAEMON_BINARY"], + "daemon_config_yaml_path": os.environ["GATEWAY_CONFIG"], + "gateway_instance_id": os.environ["GATEWAY_INSTANCE_ID"], + } +) +Path(os.environ["GATEWAY_CONFIG"]).write_text( + yaml.safe_dump(config, sort_keys=True), + encoding="utf-8", +) +PY + +setsid "$gateway_binary" serve \ + --backend docker \ + --config-yaml "$gateway_config" >"$gateway_log" 2>&1 & +gateway_pid="$!" + +readiness_stdout="${artifact_directory}/gateway-readiness.json" +readiness_stderr="${artifact_directory}/gateway-readiness.stderr" +gateway_ready=0 +for _ in $(seq 1 120); do + if ! kill -0 "$gateway_pid" 2>/dev/null; then + fail "sandbox gateway exited during startup; inspect gateway.log" + fi + if "$manager_cli" list_docker_images \ + >"$readiness_stdout" 2>"$readiness_stderr"; then + gateway_ready=1 + break + fi + sleep 0.25 +done +[[ "$gateway_ready" -eq 1 ]] || + fail "sandbox gateway did not become CLI-ready within 30 seconds" +[[ ! -s "$readiness_stderr" ]] || + fail "list_docker_images emitted stderr" + +run_batch() { + local batch_index="$1" + local batch_label + local batch_prefix + local workspace_directory + local fixture_marker + local list_images_stdout + local list_images_stderr + local create_stdout + local create_stderr + local exec_stdout + local exec_stderr + local write_stdout + local write_stderr + local read_alpha_stdout + local read_alpha_stderr + local edit_stdout + local edit_stderr + local read_omega_stdout + local read_omega_stderr + local snapshot_stdout + local snapshot_stderr + local destroy_stdout + local destroy_stderr + local list_after_stdout + local list_after_stderr + local destroyed_sandbox_id + + printf -v batch_label '%02d' "$batch_index" + batch_prefix="${artifact_directory}/batch-${batch_label}" + workspace_directory="${artifact_directory}/cli-workspace-batch-${batch_label}" + fixture_marker="CLI_ENV_BATCH_${batch_label}" + mkdir -p -- "${workspace_directory}/src" + printf '%s fixture\n' "$fixture_marker" >"${workspace_directory}/README.txt" + printf 'initial fixture\n' >"${workspace_directory}/src/main.txt" + + list_images_stdout="${batch_prefix}-00-list-images.json" + list_images_stderr="${batch_prefix}-00-list-images.stderr" + "$manager_cli" list_docker_images \ + >"$list_images_stdout" 2>"$list_images_stderr" + [[ ! -s "$list_images_stderr" ]] || + fail "batch ${batch_label} list_docker_images emitted stderr" + + create_stdout="${batch_prefix}-01-create-sandbox.json" + create_stderr="${batch_prefix}-01-create-sandbox.stderr" + "$manager_cli" create_sandbox \ + --image "$image_reference" \ + --workspace-bind-root "$workspace_directory" \ + >"$create_stdout" 2>"$create_stderr" + [[ ! -s "$create_stderr" ]] || + fail "batch ${batch_label} create_sandbox emitted stderr" + + sandbox_id="$( + EXPECTED_WORKSPACE="$workspace_directory" \ + "$python_bin" - "$create_stdout" <<'PY' +import json +import os +import sys + +value = json.loads(open(sys.argv[1], encoding="utf-8").read()) +assert isinstance(value, dict) +assert isinstance(value.get("id"), str) and value["id"] +assert value.get("state") == "ready" +assert value.get("workspace_root") == os.environ["EXPECTED_WORKSPACE"] +print(value["id"]) +PY + )" || fail "batch ${batch_label} create_sandbox response failed strict validation" + + exec_stdout="${batch_prefix}-02-exec-command.json" + exec_stderr="${batch_prefix}-02-exec-command.stderr" + "$runtime_cli" \ + --sandbox-id "$sandbox_id" \ + --request-id "cli-env-smoke-${batch_label}-exec" \ + exec_command \ + --timeout-ms 30000 \ + 'printf "CLI_ENV_EXEC_OK\n"; test -f README.txt; cat README.txt' \ + >"$exec_stdout" 2>"$exec_stderr" + [[ ! -s "$exec_stderr" ]] || + fail "batch ${batch_label} exec_command emitted stderr" + + write_stdout="${batch_prefix}-03-file-write.json" + write_stderr="${batch_prefix}-03-file-write.stderr" + "$runtime_cli" \ + --sandbox-id "$sandbox_id" \ + --request-id "cli-env-smoke-${batch_label}-write" \ + file_write \ + --path cli-smoke.txt \ + --content 'CLI_ENV_FILE_ALPHA' \ + >"$write_stdout" 2>"$write_stderr" + [[ ! -s "$write_stderr" ]] || + fail "batch ${batch_label} file_write emitted stderr" + + read_alpha_stdout="${batch_prefix}-04-file-read-alpha.json" + read_alpha_stderr="${batch_prefix}-04-file-read-alpha.stderr" + "$runtime_cli" \ + --sandbox-id "$sandbox_id" \ + --request-id "cli-env-smoke-${batch_label}-read-alpha" \ + file_read \ + --path cli-smoke.txt \ + --limit 10 \ + >"$read_alpha_stdout" 2>"$read_alpha_stderr" + [[ ! -s "$read_alpha_stderr" ]] || + fail "batch ${batch_label} first file_read emitted stderr" + + edit_stdout="${batch_prefix}-05-file-edit.json" + edit_stderr="${batch_prefix}-05-file-edit.stderr" + "$runtime_cli" \ + --sandbox-id "$sandbox_id" \ + --request-id "cli-env-smoke-${batch_label}-edit" \ + file_edit \ + --path cli-smoke.txt \ + --edits '[{"old_string":"ALPHA","new_string":"OMEGA"}]' \ + >"$edit_stdout" 2>"$edit_stderr" + [[ ! -s "$edit_stderr" ]] || + fail "batch ${batch_label} file_edit emitted stderr" + + read_omega_stdout="${batch_prefix}-06-file-read-omega.json" + read_omega_stderr="${batch_prefix}-06-file-read-omega.stderr" + "$runtime_cli" \ + --sandbox-id "$sandbox_id" \ + --request-id "cli-env-smoke-${batch_label}-read-omega" \ + file_read \ + --path cli-smoke.txt \ + --limit 10 \ + >"$read_omega_stdout" 2>"$read_omega_stderr" + [[ ! -s "$read_omega_stderr" ]] || + fail "batch ${batch_label} second file_read emitted stderr" + + snapshot_stdout="${batch_prefix}-07-observability-snapshot.json" + snapshot_stderr="${batch_prefix}-07-observability-snapshot.stderr" + "$observability_cli" snapshot \ + --sandbox-id "$sandbox_id" \ + >"$snapshot_stdout" 2>"$snapshot_stderr" + [[ ! -s "$snapshot_stderr" ]] || + fail "batch ${batch_label} observability snapshot emitted stderr" + + SANDBOX_ID="$sandbox_id" \ + FIXTURE_MARKER="$fixture_marker" \ + "$python_bin" - \ + "$exec_stdout" \ + "$read_alpha_stdout" \ + "$read_omega_stdout" \ + "$snapshot_stdout" <<'PY' +import json +import os +import sys + + +def load(path: str) -> object: + return json.loads(open(path, encoding="utf-8").read()) + + +def strings(value: object) -> list[str]: + if isinstance(value, str): + return [value] + if isinstance(value, list): + return [text for item in value for text in strings(item)] + if isinstance(value, dict): + return [text for item in value.values() for text in strings(item)] + return [] + + +command, alpha, omega, snapshot = map(load, sys.argv[1:]) +assert isinstance(command, dict) +assert command.get("status") == "ok" +assert command.get("exit_code") == 0 +command_text = "\n".join(strings(command)) +assert "CLI_ENV_EXEC_OK" in command_text +assert os.environ["FIXTURE_MARKER"] in command_text +assert "CLI_ENV_FILE_ALPHA" in "\n".join(strings(alpha)) +omega_text = "\n".join(strings(omega)) +assert "CLI_ENV_FILE_OMEGA" in omega_text +assert "CLI_ENV_FILE_ALPHA" not in omega_text +assert os.environ["SANDBOX_ID"] in "\n".join(strings(snapshot)) +PY + + destroy_stdout="${batch_prefix}-08-destroy-sandbox.json" + destroy_stderr="${batch_prefix}-08-destroy-sandbox.stderr" + "$manager_cli" destroy_sandbox \ + --sandbox-id "$sandbox_id" \ + >"$destroy_stdout" 2>"$destroy_stderr" + [[ ! -s "$destroy_stderr" ]] || + fail "batch ${batch_label} destroy_sandbox emitted stderr" + + destroyed_sandbox_id="$sandbox_id" + sandbox_id="" + + list_after_stdout="${batch_prefix}-09-list-sandboxes-after.json" + list_after_stderr="${batch_prefix}-09-list-sandboxes-after.stderr" + "$manager_cli" list_sandboxes \ + >"$list_after_stdout" 2>"$list_after_stderr" + [[ ! -s "$list_after_stderr" ]] || + fail "batch ${batch_label} post-cleanup list_sandboxes emitted stderr" + + SANDBOX_ID="$destroyed_sandbox_id" \ + "$python_bin" - "$destroy_stdout" "$list_after_stdout" <<'PY' +import json +import os +import sys + +destroyed = json.loads(open(sys.argv[1], encoding="utf-8").read()) +listed = json.loads(open(sys.argv[2], encoding="utf-8").read()) +assert destroyed.get("id") == os.environ["SANDBOX_ID"] +assert os.environ["SANDBOX_ID"] not in json.dumps(listed, sort_keys=True) +PY + batch_sandbox_ids+=("$destroyed_sandbox_id") +} + +run_batch 1 +run_batch 2 + +stop_gateway +cleanup_gateway_volumes || + fail "failed to remove a Docker volume owned by the qualification gateway" +require_no_owned_docker_resources || + fail "the qualification gateway left a container or volume behind" + +nonempty_stderr="$( + find "$artifact_directory" -maxdepth 1 -type f -name '*.stderr' -size +0 -print +)" +[[ -z "$nonempty_stderr" ]] || + fail "one or more product CLI commands emitted stderr" +if grep -Eiq \ + '"level"[[:space:]]*:[[:space:]]*"(warn|error)"|(^|[^[:alpha:]])(WARN|ERROR|PANIC)([^[:alpha:]]|$)' \ + "$gateway_log"; then + fail "gateway log contains a warning, error, or panic" +fi +token_leak_files="$(grep -rlF -- "$gateway_token" "$artifact_directory" || true)" +if [[ -n "$token_leak_files" ]]; then + while IFS= read -r leaked_file; do + sed -i "s/${gateway_token}/[REDACTED]/g" "$leaked_file" + done <<<"$token_leak_files" + fail "gateway authentication token appeared in archived output and was redacted" +fi + +elapsed_seconds="$(( $(date +%s) - started_seconds ))" +[[ "$elapsed_seconds" -le "$maximum_seconds" ]] || + fail "CLI environment smoke took ${elapsed_seconds}s, above the ${maximum_seconds}s budget" + +BATCH_1_ID="${batch_sandbox_ids[0]}" \ +BATCH_2_ID="${batch_sandbox_ids[1]}" \ +GATEWAY_INSTANCE_ID="$gateway_instance_id" \ +IMAGE_REFERENCE="$image_reference" \ +ELAPSED_SECONDS="$elapsed_seconds" \ +"$python_bin" - "$summary_file" <<'PY' +import json +import os +import sys + +operation_sequence = [ + "list_docker_images", + "create_sandbox", + "exec_command", + "file_write", + "file_read", + "file_edit", + "file_read", + "snapshot", + "destroy_sandbox", + "list_sandboxes", +] +batch_ids = [os.environ["BATCH_1_ID"], os.environ["BATCH_2_ID"]] +summary = { + "schema_version": 1, + "client_cohort": "product_cli", + "state": "completed", + "correctness": "pass", + "completed_batches": 2, + "total_batches": 2, + "operation_count": 20, + "warning_count": 0, + "failure_count": 0, + "cleanup": "pass", + "elapsed_seconds": int(os.environ["ELAPSED_SECONDS"]), + "gateway_instance_id": os.environ["GATEWAY_INSTANCE_ID"], + "image_reference": os.environ["IMAGE_REFERENCE"], + "operation_sequence": operation_sequence, + "sandbox_ids": batch_ids, + "batches": [ + { + "batch_index": index, + "sandbox_id": sandbox_id, + "state": "completed", + "correctness": "pass", + "operation_count": len(operation_sequence), + "cleanup": "pass", + } + for index, sandbox_id in enumerate(batch_ids, start=1) + ], +} +with open(sys.argv[1], "w", encoding="utf-8") as handle: + json.dump(summary, handle, indent=2, sort_keys=True) + handle.write("\n") +PY + +unset SANDBOX_GATEWAY_AUTH_TOKEN gateway_token +trap - EXIT INT TERM +printf 'PASS\tCLI-only environment smoke completed\n' +printf 'INFO\tsummary=%s\n' "$summary_file" +printf 'INFO\telapsed_seconds=%s\n' "$elapsed_seconds" diff --git a/ephemeral-sandbox-v1/experiments/scripts/prepare_final_host_bundle.ps1 b/ephemeral-sandbox-v1/experiments/scripts/prepare_final_host_bundle.ps1 new file mode 100644 index 0000000..f5ca499 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/prepare_final_host_bundle.ps1 @@ -0,0 +1,141 @@ +param( + [Parameter(Mandatory = $true)] + [string]$OutputDirectory, + [string]$ProductRoot = "" +) + +$ErrorActionPreference = "Stop" +$paperRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..\..")).Path +if (-not $ProductRoot) { + $ProductRoot = Join-Path $paperRoot "..\..\ephemeral-sandbox" +} +$product = (Resolve-Path -LiteralPath $ProductRoot).Path +$output = [System.IO.Path]::GetFullPath($OutputDirectory) + +if (Test-Path -LiteralPath $output) { + if ((Get-ChildItem -Force -LiteralPath $output | Measure-Object).Count -ne 0) { + throw "OutputDirectory must not exist or must be empty: $output" + } +} else { + New-Item -ItemType Directory -Path $output | Out-Null +} + +$expectedCommit = "b22862550e0a7cb4fe61ce581831e9244cc492b5" +$expectedHashes = @{ + "target/release/sandbox-gateway" = "f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf" + "target/release/sandbox-catalog-export" = "c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba" + "target/release/sandbox-manager-cli" = "0be4f0c26f8f50b76b175d04cfeec61529a605bcda9ffcd6782a09096ba2983f" + "target/release/sandbox-runtime-cli" = "e9ac5f6c7a5f9c07a3de166b320e7d6065fa9480a7f18d6d59114337d15e28e7" + "target/release/sandbox-observability-cli" = "6b2dae2369344cbb3960a76f6ccdfa869a7aa9b7a7a255f8a634f4a52d5cfdb5" + "dist/sandbox-daemon-linux-amd64" = "a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22" +} + +$branch = (& git -c core.longpaths=true -C $product branch --show-current).Trim() +$commit = (& git -c core.longpaths=true -C $product rev-parse HEAD).Trim() +$productStatus = @(& git -c core.longpaths=true -C $product status --porcelain=v1) +if ($LASTEXITCODE -ne 0 -or $branch -ne "main" -or $commit -ne $expectedCommit -or $productStatus.Count -ne 0) { + throw "Product must be clean main at $expectedCommit" +} + +foreach ($relative in $expectedHashes.Keys) { + $path = Join-Path $product ($relative.Replace("/", "\")) + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + throw "Missing product artifact: $path" + } + $actual = (Get-FileHash -Algorithm SHA256 -LiteralPath $path).Hash.ToLowerInvariant() + if ($actual -ne $expectedHashes[$relative]) { + throw "Product artifact hash mismatch: $relative" + } +} + +$productBundle = Join-Path $output "product-main.bundle" +& git -c core.longpaths=true -C $product bundle create $productBundle main +if ($LASTEXITCODE -ne 0) { + throw "git bundle creation failed" +} + +$artifactArchive = Join-Path $output "product-artifacts.tar.gz" +& tar.exe -czf $artifactArchive -C $product ` + "target/release/sandbox-gateway" ` + "target/release/sandbox-catalog-export" ` + "target/release/sandbox-manager-cli" ` + "target/release/sandbox-runtime-cli" ` + "target/release/sandbox-observability-cli" ` + "dist/sandbox-daemon-linux-amd64" +if ($LASTEXITCODE -ne 0) { + throw "product artifact archive creation failed" +} + +$paperArchive = Join-Path $output "paper-snapshot.tar.gz" +& tar.exe -czf $paperArchive ` + "--exclude=.venv" ` + "--exclude=.benchmark-state" ` + "--exclude=__pycache__" ` + "--exclude=.pytest_cache" ` + "--exclude=experiments/runs" ` + -C $paperRoot "." +if ($LASTEXITCODE -ne 0) { + throw "paper snapshot archive creation failed" +} + +$paperRepository = (Resolve-Path -LiteralPath (Join-Path $paperRoot "..")).Path +$paperCommit = (& git -c core.longpaths=true -C $paperRepository rev-parse HEAD).Trim() +$paperStatus = @( + & git -c core.longpaths=true -C $paperRepository status --porcelain=v1 -- "ephemeral-sandbox-v1" +) +$archives = @{} +foreach ($path in @($productBundle, $artifactArchive, $paperArchive)) { + $archives[[System.IO.Path]::GetFileName($path)] = @{ + bytes = (Get-Item -LiteralPath $path).Length + sha256 = (Get-FileHash -Algorithm SHA256 -LiteralPath $path).Hash.ToLowerInvariant() + } +} +$manifest = @{ + schema_version = 1 + created_at_utc = [DateTime]::UtcNow.ToString("o") + selected_host = "eos-benchmark-ubuntu24" + selected_paths = @{ + product = "/srv/eos-benchmark/product" + paper = "/srv/eos-benchmark/paper" + } + product = @{ + branch = $branch + commit = $commit + dirty = $false + release_tag = "v0.1.4" + linux_release_archive_sha256 = "308563ad38bc7a9c5000acd54251db872e2e6a58bf70846d14760fef2b0d713c" + qualification_client_cohort = "product_cli" + artifact_sha256 = $expectedHashes + } + paper = @{ + repository_commit = $paperCommit + dirty = ($paperStatus.Count -ne 0) + status = $paperStatus + } + archives = $archives +} +$manifest | ConvertTo-Json -Depth 8 | + Set-Content -LiteralPath (Join-Path $output "bundle-manifest.json") -Encoding UTF8 + +@" +On eos-benchmark-ubuntu24, after copying this directory to an off-clock staging path: + +sudo install -d -o "`$USER" -g "`$USER" /srv/eos-benchmark +git clone ./product-main.bundle /srv/eos-benchmark/product +mkdir /srv/eos-benchmark/paper +tar -xzf ./product-artifacts.tar.gz -C /srv/eos-benchmark/product +tar -xzf ./paper-snapshot.tar.gz -C /srv/eos-benchmark/paper +cd /srv/eos-benchmark/paper +bash experiments/scripts/stage_final_host.sh 2>&1 | tee staging-final-host.txt + +After staging, quiesce or reboot the host if provisioning caused background activity. +Then run: + +cd /srv/eos-benchmark/paper +bash experiments/scripts/qualify_final_host.sh +"@ | Set-Content -LiteralPath (Join-Path $output "HANDOFF.txt") -Encoding UTF8 + +Get-ChildItem -File -LiteralPath $output | + Select-Object Name, Length, @{Name = "SHA256"; Expression = { + (Get-FileHash -Algorithm SHA256 -LiteralPath $_.FullName).Hash.ToLowerInvariant() + }} diff --git a/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_content_file.py b/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_content_file.py new file mode 100644 index 0000000..9c053de --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_content_file.py @@ -0,0 +1,206 @@ +from __future__ import annotations + +import argparse +import asyncio +import hashlib +import json +import os +import platform +import tempfile +import time +from pathlib import Path +from typing import Any + + +def _sha256_bytes(value: bytes) -> str: + return f"sha256:{hashlib.sha256(value).hexdigest()}" + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return f"sha256:{digest.hexdigest()}" + + +def _error_kind(stderr: bytes) -> str | None: + try: + value = json.loads(stderr) + except (UnicodeDecodeError, json.JSONDecodeError): + return None + error = value.get("error") + return error.get("kind") if isinstance(error, dict) else None + + +async def _probe( + executable: Path, payload_bytes: int, content_file: Path +) -> dict[str, Any]: + payload = b"x" * payload_bytes + content_file.write_bytes(payload) + arguments = [ + str(executable), + "--gateway-socket", + "127.0.0.1:1", + "--gateway-auth-token", + "exp1-diagnostic-token", + "--sandbox-id", + "exp1-diagnostic", + "--request-id", + f"exp1-content-file-{payload_bytes}", + "file_write", + "--path", + "diagnostic.bin", + "--content-file", + str(content_file), + ] + started_ns = time.monotonic_ns() + process: asyncio.subprocess.Process | None = None + try: + process = await asyncio.create_subprocess_exec( + *arguments, + cwd=executable.parent.parent, + env={ + name: os.environ[name] + for name in ("PATH", "SystemRoot", "WINDIR", "TEMP", "TMP") + if name in os.environ + }, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=10) + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "payload_sha256": _sha256_bytes(payload), + "payload_file_prepared_before_clock": True, + "process_started": True, + "pid_was_assigned": process.pid is not None, + "return_code": process.returncode, + "stdout_bytes": len(stdout), + "stderr_bytes": len(stderr), + "stdout_sha256": _sha256_bytes(stdout), + "stderr_sha256": _sha256_bytes(stderr), + "stderr_error_kind": _error_kind(stderr), + "elapsed_ns": ended_ns - started_ns, + "expected_gateway_connection": "deliberately unavailable", + } + except OSError as error: + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "payload_sha256": _sha256_bytes(payload), + "payload_file_prepared_before_clock": True, + "process_started": False, + "return_code": None, + "error_type": type(error).__name__, + "errno": error.errno, + "winerror": getattr(error, "winerror", None), + "error": str(error), + "elapsed_ns": ended_ns - started_ns, + "expected_gateway_connection": "not attempted", + } + except TimeoutError: + if process is not None and process.returncode is None: + process.kill() + await process.wait() + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "payload_sha256": _sha256_bytes(payload), + "payload_file_prepared_before_clock": True, + "process_started": process is not None, + "return_code": None if process is None else process.returncode, + "error_type": "TimeoutError", + "error": "diagnostic child did not terminate within 10 seconds", + "elapsed_ns": ended_ns - started_ns, + } + + +async def _run(executable: Path) -> dict[str, Any]: + with tempfile.TemporaryDirectory(prefix="exp1-content-file-") as temp_dir: + temp_root = Path(temp_dir) + probes = [ + await _probe(executable, 4096, temp_root / "payload-4096.txt"), + await _probe(executable, 262144, temp_root / "payload-262144.txt"), + ] + accepted = all( + probe["process_started"] is True + and probe["return_code"] == 1 + and probe["stderr_error_kind"] == "connection_error" + for probe in probes + ) + return { + "schema_version": 1, + "kind": "native_windows_asyncio_cli_content_file_probe", + "timestamp_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "host": { + "computer_name": platform.node(), + "operating_system": platform.system(), + "release": platform.release(), + "machine": platform.machine(), + }, + "python": { + "version": platform.python_version(), + "implementation": platform.python_implementation(), + "subprocess_api": "asyncio.create_subprocess_exec", + }, + "runtime_cli": { + "path": str(executable), + "sha256": _sha256(executable), + }, + "sanitized_argument_shape": [ + str(executable), + "--gateway-socket", + "127.0.0.1:1", + "--gateway-auth-token", + "[REDACTED]", + "--sandbox-id", + "exp1-diagnostic", + "--request-id", + "PAYLOAD-SPECIFIC-ID", + "file_write", + "--path", + "diagnostic.bin", + "--content-file", + "PREPARED-PAYLOAD-FILE", + ], + "timing_boundary": ( + "payload file preparation precedes the clock; process creation, CLI file read, " + "request construction, and gateway connection attempt follow the clock" + ), + "probes": probes, + "acceptance": { + "all_payload_processes_started": all( + probe["process_started"] is True for probe in probes + ), + "required_payload_process_started": probes[1]["process_started"] is True, + "required_payload_reached_gateway_transport": ( + probes[1]["return_code"] == 1 + and probes[1]["stderr_error_kind"] == "connection_error" + ), + "passed": accepted, + }, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--runtime-cli", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + arguments = parser.parse_args() + executable = arguments.runtime_cli.resolve(strict=True) + output = arguments.output.resolve() + if output.exists(): + raise SystemExit("diagnostic output already exists") + result = asyncio.run(_run(executable)) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + return 0 if result["acceptance"]["passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_payload.py b/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_payload.py new file mode 100644 index 0000000..17f4b19 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/probe_exp1_windows_cli_payload.py @@ -0,0 +1,173 @@ +from __future__ import annotations + +import argparse +import asyncio +import hashlib +import json +import os +import platform +import time +from pathlib import Path +from typing import Any + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return f"sha256:{digest.hexdigest()}" + + +async def _probe(executable: Path, payload_bytes: int) -> dict[str, Any]: + payload = "x" * payload_bytes + arguments = [ + str(executable), + "--gateway-socket", + "127.0.0.1:1", + "--gateway-auth-token", + "exp1-diagnostic-token", + "--sandbox-id", + "exp1-diagnostic", + "--request-id", + f"exp1-python-subprocess-{payload_bytes}", + "file_write", + "--path", + "diagnostic.bin", + "--content", + payload, + ] + started_ns = time.monotonic_ns() + process: asyncio.subprocess.Process | None = None + try: + process = await asyncio.create_subprocess_exec( + *arguments, + cwd=executable.parent.parent, + env={ + name: os.environ[name] + for name in ("PATH", "SystemRoot", "WINDIR", "TEMP", "TMP") + if name in os.environ + }, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=10) + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "process_started": True, + "pid_was_assigned": process.pid is not None, + "return_code": process.returncode, + "stdout_bytes": len(stdout), + "stderr_bytes": len(stderr), + "stdout_sha256": f"sha256:{hashlib.sha256(stdout).hexdigest()}", + "stderr_sha256": f"sha256:{hashlib.sha256(stderr).hexdigest()}", + "elapsed_ns": ended_ns - started_ns, + "expected_gateway_connection": "deliberately unavailable", + } + except OSError as error: + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "process_started": False, + "return_code": None, + "error_type": type(error).__name__, + "errno": error.errno, + "winerror": getattr(error, "winerror", None), + "error": str(error), + "elapsed_ns": ended_ns - started_ns, + "expected_gateway_connection": "not attempted", + } + except TimeoutError: + if process is not None and process.returncode is None: + process.kill() + await process.wait() + ended_ns = time.monotonic_ns() + return { + "payload_bytes": payload_bytes, + "process_started": process is not None, + "return_code": None if process is None else process.returncode, + "error_type": "TimeoutError", + "error": "diagnostic child did not terminate within 10 seconds", + "elapsed_ns": ended_ns - started_ns, + } + + +async def _run(executable: Path) -> dict[str, Any]: + probes = [ + await _probe(executable, 4096), + await _probe(executable, 262144), + ] + return { + "schema_version": 1, + "kind": "native_windows_asyncio_cli_payload_probe", + "timestamp_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "host": { + "computer_name": platform.node(), + "operating_system": platform.system(), + "release": platform.release(), + "machine": platform.machine(), + }, + "python": { + "version": platform.python_version(), + "implementation": platform.python_implementation(), + "subprocess_api": "asyncio.create_subprocess_exec", + }, + "runtime_cli": { + "path": str(executable), + "sha256": _sha256(executable), + }, + "sanitized_argument_shape": [ + str(executable), + "--gateway-socket", + "127.0.0.1:1", + "--gateway-auth-token", + "[REDACTED]", + "--sandbox-id", + "exp1-diagnostic", + "--request-id", + "PAYLOAD-SPECIFIC-ID", + "file_write", + "--path", + "diagnostic.bin", + "--content", + "PAYLOAD", + ], + "probes": probes, + "acceptance": { + "small_payload_process_started": probes[0]["process_started"] is True, + "required_payload_process_started": probes[1]["process_started"] is True, + "required_payload_failed_before_gateway": ( + probes[1]["process_started"] is False + and probes[1].get("winerror") == 206 + ), + }, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--runtime-cli", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + arguments = parser.parse_args() + executable = arguments.runtime_cli.resolve(strict=True) + output = arguments.output.resolve() + if output.exists(): + raise SystemExit("diagnostic output already exists") + result = asyncio.run(_run(executable)) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + accepted = result["acceptance"] + return ( + 0 + if accepted["small_payload_process_started"] + and accepted["required_payload_failed_before_gateway"] + else 1 + ) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/scripts/project_exp1_final_runtime.py b/ephemeral-sandbox-v1/experiments/scripts/project_exp1_final_runtime.py new file mode 100644 index 0000000..ae5228c --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/project_exp1_final_runtime.py @@ -0,0 +1,1239 @@ +#!/usr/bin/env python3 +"""Project EXP1 final duration from immutable structural run evidence.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import sys +from dataclasses import dataclass +from datetime import datetime +from fractions import Fraction +from pathlib import Path +from typing import Any, Iterable + +FINAL_CELLS = 19 +FINAL_BATCHES = 1_938 +FINAL_REQUESTS = 5_610 +FINAL_WARMUPS = 2 +FINAL_MEASURED = 100 +LIMIT_NS = 1_400_000_000_000 +EXPECTED = { + "smoke": { + "name": "paper-env-smoke", + "disposition": "smoke", + "eligibility": "qualification_only", + "batches": 19, + "requests": 55, + "warmups": 0, + "measured": 1, + }, + "pilot": { + "name": "paper-pilot", + "disposition": "exploratory", + "eligibility": "exploratory_ineligible", + "batches": 133, + "requests": 385, + "warmups": 2, + "measured": 5, + }, +} +PHASES = ("setup", "operation", "verify", "teardown") +STABLE_HOST_FIELDS = ( + "computer_name", + "operating_system", + "architecture", + "os_version", + "os_build_number", + "cpu_model", + "logical_processors", + "processor_logical_processors", + "total_memory_bytes", + "filesystem", + "volume_root", + "docker_engine_version", + "monotonic_clock", +) +PROTOCOLS = { + "v1.0": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.0", + "environment_identity": "isolated_loopback_per_execution_block", + }, + "v1.1": { + "id": "ephemeral-sandbox-v1-practical-performance-v1.1", + "environment_identity": ("isolated_windows_named_pipe_per_execution_block"), + }, +} +V11_GATEWAY_TRANSPORT = { + "transport": "windows_named_pipe", + "scope": "local_only", + "rotation": "per_execution_block", +} +SAFE_NPIPE_ENDPOINT = re.compile(r"npipe://\./pipe/[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z") +SHA256_IDENTITY = re.compile(r"sha256:[0-9a-f]{64}\Z") +V11_PREFREEZE_MUTABLE_STATUS_PATHS = frozenset( + { + "progress.md", + "experiments/experiment_log.md", + "paper_state.json", + "plan/progress.md", + } +) +V11_PROTOCOL_FILE_PATHS = frozenset( + { + "progress.md", + "plan/task-packets/exp1-cli-performance-campaign.md", + "experiment_inventory.md", + "experiments/exp1-v1.1-protocol-amendment.md", + "experiments/environment_setup.md", + "experiments/expected_tables.md", + "experiments/experiment_log.md", + "benchmark/PAPER_ARTIFACT.md", + "paper_state.json", + "plan/progress.md", + } +) + + +class ProjectionError(RuntimeError): + """An input or structural runtime invariant failed closed.""" + + +@dataclass(frozen=True) +class TrialSpan: + trial_id: str + warmup: bool + start_ns: int + end_ns: int + phase_ns: dict[str, int] + + @property + def active_ns(self) -> int: + return self.end_ns - self.start_ns + + +@dataclass(frozen=True) +class CellProfile: + semantic_key: str + semantic_sha256: str + family_id: str + operation_id: str + leading_ns: int + trailing_ns: int + trials: tuple[TrialSpan, ...] + gaps_ns: tuple[int, ...] + + @property + def warmups(self) -> tuple[TrialSpan, ...]: + return tuple(trial for trial in self.trials if trial.warmup) + + @property + def measured(self) -> tuple[TrialSpan, ...]: + return tuple(trial for trial in self.trials if not trial.warmup) + + +@dataclass(frozen=True) +class RunProfile: + role: str + identity: dict[str, Any] + plan: dict[str, Any] + elapsed_ns: int + run_residual_ns: int + family_residual_ns: dict[str, int] + cells: dict[str, CellProfile] + + +def load_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def envelope_data(path: Path, schema_name: str | None = None) -> dict[str, Any]: + value = load_json(path) + if schema_name is not None and value.get("schema_name") != schema_name: + raise ProjectionError(f"artifact schema is invalid: {path}") + data = value.get("data") + if not isinstance(data, dict): + raise ProjectionError(f"artifact is not an envelope: {path}") + return data + + +def expanded_plan_data(path: Path) -> dict[str, Any]: + value = load_json(path) + if isinstance(value, dict) and isinstance(value.get("data"), dict): + if value.get("schema_name") != "eos_benchmark_expanded_plan": + raise ProjectionError("final plan envelope schema is invalid") + value = value["data"] + if not isinstance(value, dict): + raise ProjectionError("final expanded plan is not an object") + return value + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return f"sha256:{digest.hexdigest()}" + + +def canonical_json(value: Any) -> str: + return json.dumps( + value, + sort_keys=True, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ) + + +def canonical_sha256(value: Any) -> str: + return f"sha256:{hashlib.sha256(canonical_json(value).encode()).hexdigest()}" + + +def elapsed_ns(manifest: dict[str, Any]) -> int: + try: + started = datetime.fromisoformat(manifest["started_at"].replace("Z", "+00:00")) + ended = datetime.fromisoformat(manifest["ended_at"].replace("Z", "+00:00")) + except (AttributeError, KeyError, ValueError) as error: + raise ProjectionError("run timestamps are invalid") from error + if started.tzinfo is None or ended.tzinfo is None or ended <= started: + raise ProjectionError("run timestamps are not positive offset-aware values") + delta = ended - started + return ( + delta.days * 86_400 + delta.seconds + ) * 1_000_000_000 + delta.microseconds * 1_000 + + +def archive_inventory(root: Path) -> tuple[list[dict[str, Any]], int, str]: + entries: list[dict[str, Any]] = [] + total_bytes = 0 + tree = hashlib.sha256() + for path in sorted( + (item for item in root.rglob("*") if item.is_file()), + key=lambda item: item.relative_to(root).as_posix(), + ): + if path.is_symlink(): + raise ProjectionError(f"archive contains a symlink: {path}") + relative = path.relative_to(root).as_posix() + if relative == "archive-manifest.json": + continue + size = path.stat().st_size + digest = sha256_file(path) + entries.append({"path": relative, "bytes": size, "sha256": digest}) + total_bytes += size + tree.update(relative.encode()) + tree.update(b"\0") + tree.update(bytes.fromhex(digest.removeprefix("sha256:"))) + tree.update(b"\n") + return entries, total_bytes, f"sha256:{tree.hexdigest()}" + + +def verify_archive_inventory(root: Path, manifest: dict[str, Any]) -> None: + entries, total_bytes, tree_hash = archive_inventory(root) + if ( + entries != manifest.get("files") + or total_bytes != manifest.get("archive_bytes") + or tree_hash != manifest.get("content_tree_sha256") + or len(entries) != manifest.get("archive_file_count") + ): + raise ProjectionError("archive inventory verification failed") + + +def _required_dict(value: Any, label: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise ProjectionError(f"{label} is not an object") + return value + + +def _validate_protocol_transport( + campaign: dict[str, Any], + manifest: dict[str, Any], + plan: dict[str, Any], + protocol_version: str, +) -> None: + protocol = _required_dict(campaign.get("protocol"), "protocol") + expected_version = ( + protocol_version if protocol_version == "v1.1" else "pre-freeze-exp1" + ) + if protocol.get("version") != expected_version or protocol.get("id") not in ( + PROTOCOLS[protocol_version]["id"], + None, + ): + raise ProjectionError("archive protocol version is invalid") + environment = _required_dict(manifest.get("environment"), "run environment") + if protocol_version == "v1.0": + if environment.get("gateway_endpoint_identity") not in ( + None, + PROTOCOLS["v1.0"]["environment_identity"], + ): + raise ProjectionError("legacy v1.0 gateway identity drift") + return + if ( + protocol.get("id") != PROTOCOLS["v1.1"]["id"] + or environment.get("gateway_endpoint_identity") + != PROTOCOLS["v1.1"]["environment_identity"] + or environment.get("gateway_transport") != V11_GATEWAY_TRANSPORT + ): + raise ProjectionError("v1.1 named-pipe environment identity is invalid") + policy = _required_dict(manifest.get("gateway_policy"), "gateway policy") + if ( + policy.get("protocol_version") != PROTOCOLS["v1.1"]["id"] + or any(policy.get(key) != value for key, value in V11_GATEWAY_TRANSPORT.items()) + or policy.get("mode") != "isolated" + or policy.get("isolated_runtime_per_execution_block") is not True + or policy.get("loopback_only") is not False + ): + raise ProjectionError("v1.1 named-pipe gateway policy is invalid") + planned = plan.get("execution_blocks") + launched = manifest.get("gateway_execution_blocks") + if ( + not isinstance(planned, list) + or not isinstance(launched, list) + or len(planned) != len(launched) + ): + raise ProjectionError("v1.1 execution-block endpoint count is invalid") + endpoints: set[str] = set() + for expected, observed in zip(planned, launched): + endpoint = observed.get("endpoint_uri") if isinstance(observed, dict) else None + if ( + not isinstance(expected, dict) + or not isinstance(observed, dict) + or observed.get("block_id") != expected.get("block_id") + or observed.get("family_id") != expected.get("family_id") + or any( + observed.get(key) != value + for key, value in V11_GATEWAY_TRANSPORT.items() + ) + or not isinstance(observed.get("gateway_instance_id"), str) + or not observed["gateway_instance_id"] + or not isinstance(endpoint, str) + or SAFE_NPIPE_ENDPOINT.fullmatch(endpoint) is None + or endpoint in endpoints + ): + raise ProjectionError("v1.1 execution-block endpoint evidence is unsafe") + endpoints.add(endpoint) + + +def _stable_host(campaign: dict[str, Any]) -> dict[str, Any]: + host = _required_dict(campaign.get("final_host"), "final host") + try: + return {field: host[field] for field in STABLE_HOST_FIELDS} + except KeyError as error: + raise ProjectionError("final-host identity is incomplete") from error + + +def _cleanup_identity(campaign: dict[str, Any]) -> dict[str, Any]: + cleanup = _required_dict(campaign.get("cleanup"), "cleanup") + if ( + cleanup.get("run_workspace_exists") is not False + or cleanup.get("runtime_exists") is not False + or cleanup.get("matching_product_processes") != [] + or cleanup.get("run_labeled_containers") != [] + or cleanup.get("gateway_labeled_containers") != [] + or cleanup.get("run_labeled_volumes") != [] + or cleanup.get("gateway_labeled_volumes") != [] + or cleanup.get("product_branch") != "main" + or cleanup.get("product_status_porcelain") != "" + ): + raise ProjectionError("source archive cleanup proof did not pass") + return { + key: cleanup.get(key) + for key in ( + "product_branch", + "product_commit", + "product_status_porcelain", + "product_checkout_policy", + ) + } + + +def _identity(campaign: dict[str, Any], manifest: dict[str, Any]) -> dict[str, Any]: + benchmark_source = _required_dict( + campaign.get("benchmark_source"), "benchmark source" + ) + fixture = _required_dict(campaign.get("fixture"), "fixture") + product = _required_dict(campaign.get("product"), "product") + image = _required_dict(campaign.get("image"), "image") + protocol = _required_dict(campaign.get("protocol"), "protocol") + paper_git = _required_dict(campaign.get("paper_git"), "paper Git") + return { + "benchmark_source": { + key: benchmark_source.get(key) + for key in ("content_tree_sha256", "manifest_sha256") + }, + "fixture": { + key: fixture.get(key) + for key in ("fixture_hash", "tree_hash", "manifest_sha256") + }, + "product": { + "branch": product.get("branch"), + "commit": product.get("commit"), + "dirty": product.get("dirty"), + "archive": product.get("archive"), + "binaries": product.get("binaries"), + "recorded_treatment": product.get("recorded_treatment"), + }, + "image": { + key: image.get(key) + for key in ("id", "requested", "repo_digests", "architecture", "os") + }, + "docker": campaign.get("docker"), + "host": _stable_host(campaign), + "sandbox_limits": campaign.get("sandbox_limits"), + "definition_snapshot": campaign.get("definition_snapshot"), + "artifact_schemas": campaign.get("artifact_schemas"), + "protocol_files": protocol.get("files"), + "freeze_state": { + "protocol": protocol.get("freeze_state"), + "paper_git": paper_git.get("freeze_state"), + }, + "analysis_and_archiving_code": campaign.get("analysis_and_archiving_code"), + "treatment": manifest.get("treatment"), + "lifecycle": manifest.get("fixed_lifecycle_policy"), + "gateway": manifest.get("gateway_policy"), + } + + +def _validated_protocol_files( + value: Any, + *, + protocol_version: str, +) -> list[dict[str, Any]]: + if not isinstance(value, list): + raise ProjectionError("cross-run protocol-file identity is invalid") + seen: set[str] = set() + for entry in value: + if not isinstance(entry, dict): + raise ProjectionError("cross-run protocol-file identity is invalid") + path = entry.get("path") + size = entry.get("bytes") + digest = entry.get("sha256") + if ( + not isinstance(path, str) + or not path + or path in seen + or not isinstance(size, int) + or isinstance(size, bool) + or size < 0 + or not isinstance(digest, str) + or SHA256_IDENTITY.fullmatch(digest) is None + ): + raise ProjectionError("cross-run protocol-file identity is invalid") + seen.add(path) + if protocol_version == "v1.1" and seen != V11_PROTOCOL_FILE_PATHS: + raise ProjectionError("v1.1 protocol-file identity set is invalid") + return value + + +def _validate_prefreeze_campaign( + campaign: dict[str, Any], protocol_version: str +) -> None: + protocol = _required_dict(campaign.get("protocol"), "protocol") + paper_git = _required_dict(campaign.get("paper_git"), "paper Git") + if ( + protocol.get("freeze_state") != "pre_freeze" + or paper_git.get("freeze_state") != "pre_freeze_worktree" + ): + raise ProjectionError("projection archive is not in the pre-freeze state") + _validated_protocol_files( + protocol.get("files"), + protocol_version=protocol_version, + ) + + +def _cross_run_provenance(value: Any, *, protocol_version: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise ProjectionError("cross-run provenance identity is invalid") + normalized = json.loads(canonical_json(value)) + freeze_state = normalized.get("freeze_state") + if freeze_state != { + "paper_git": "pre_freeze_worktree", + "protocol": "pre_freeze", + }: + raise ProjectionError("cross-run freeze-state identity is invalid") + protocol_files = _validated_protocol_files( + normalized.get("protocol_files"), + protocol_version=protocol_version, + ) + for entry in protocol_files: + if entry["path"] in V11_PREFREEZE_MUTABLE_STATUS_PATHS: + entry["bytes"] = "" + entry["sha256"] = "" + return normalized + + +def _read_journal( + path: Path, + *, + schema_name: str, + schema_version: int, +) -> list[dict[str, Any]]: + records: list[dict[str, Any]] = [] + with path.open("r", encoding="utf-8") as stream: + for expected_sequence, line in enumerate(stream, 1): + try: + value = json.loads(line) + if ( + value.get("schema_name") != schema_name + or value.get("schema_version") != schema_version + ): + raise ProjectionError(f"journal schema is invalid: {path}") + data = value["data"] + if data["sequence"] != expected_sequence: + raise ProjectionError(f"journal sequence is invalid: {path}") + if "monotonic_offset_ns" in data: + offset = data["monotonic_offset_ns"] + if not isinstance(offset, int) or offset < 0: + raise ProjectionError( + f"journal monotonic offsets are invalid: {path}" + ) + records.append(data) + except (KeyError, TypeError, json.JSONDecodeError) as error: + raise ProjectionError(f"journal record is invalid: {path}") from error + return records + + +def _cell_semantic(cell: dict[str, Any]) -> str: + required = { + "family_id": cell.get("family_id"), + "operation_id": cell.get("operation_id"), + "operation_semantic_revision": cell.get("operation_semantic_revision"), + "factor_schema_revision": cell.get("factor_schema_revision"), + "comparison_key": cell.get("comparison_key"), + "operation": cell.get("operation"), + "protocol": { + key: cell.get("protocol", {}).get(key) + for key in ("destructive", "timeout_ms", "cleanup") + }, + } + return canonical_json(required) + + +def _plan_semantics(plan: dict[str, Any]) -> dict[str, Any]: + cells = plan.get("cells") + blocks = plan.get("execution_blocks") + if not isinstance(cells, list) or not isinstance(blocks, list): + raise ProjectionError("expanded plan cells or execution blocks are invalid") + by_id: dict[str, str] = {} + semantics: dict[str, dict[str, Any]] = {} + for cell in cells: + cell_id = cell.get("cell_id") + semantic = _cell_semantic(cell) + if not isinstance(cell_id, str) or cell_id in by_id or semantic in semantics: + raise ProjectionError("expanded plan cell identity is ambiguous") + by_id[cell_id] = semantic + semantics[semantic] = cell + block_semantics = [] + for block in blocks: + try: + cell_keys = [by_id[cell_id] for cell_id in block["cell_ids"]] + except (KeyError, TypeError) as error: + raise ProjectionError( + "execution block references an unknown cell" + ) from error + block_semantics.append( + { + "family_id": block.get("family_id"), + "restart_reason": block.get("restart_reason"), + "cells": cell_keys, + } + ) + canonical = _required_dict(plan.get("canonical_plan"), "canonical plan") + stable_canonical = { + key: canonical.get(key) + for key in ( + "configuration_base", + "seed", + "environment", + "operations", + ) + } + protocol = _required_dict(canonical.get("protocol"), "canonical protocol") + stable_canonical["protocol"] = { + key: protocol.get(key) + for key in ("order", "resource_interval_ms", "timeout_ms") + } + effective = dict( + _required_dict(plan.get("effective_environment"), "effective environment") + ) + effective.pop("free_space_bytes", None) + return { + "canonical": stable_canonical, + "effective_environment": effective, + "cells": set(semantics), + "blocks": block_semantics, + "cell_by_id": by_id, + "cell_data": semantics, + } + + +def _validate_plan( + plan: dict[str, Any], + *, + role: str, +) -> dict[str, Any]: + expected = EXPECTED.get(role) + if role == "final": + expected = { + "name": "paper-good-pass", + "batches": FINAL_BATCHES, + "requests": FINAL_REQUESTS, + "warmups": FINAL_WARMUPS, + "measured": FINAL_MEASURED, + } + assert expected is not None + estimates = _required_dict(plan.get("estimates"), "plan estimates") + canonical = _required_dict(plan.get("canonical_plan"), "canonical plan") + if ( + plan.get("runnable") is not True + or plan.get("validation") != [] + or canonical.get("name") != expected["name"] + or plan.get("effective_environment", {}).get("client_cohort") != "product_cli" + or estimates.get("cell_count") != FINAL_CELLS + or estimates.get("trial_batch_count") != expected["batches"] + or estimates.get("issued_operation_request_count") != expected["requests"] + ): + raise ProjectionError(f"{role} expanded plan violates the fixed protocol") + if any( + cell.get("protocol", {}).get("warmups") != expected["warmups"] + or cell.get("protocol", {}).get("measured_trials") != expected["measured"] + for cell in plan.get("cells", []) + ): + raise ProjectionError(f"{role} cell repetition policy is invalid") + return _plan_semantics(plan) + + +def _state_times( + events: Iterable[dict[str, Any]], + *, + kind: str, + identity_field: str, + required_states: tuple[str, ...], +) -> tuple[dict[str, dict[str, int]], dict[str, dict[str, Any]]]: + values: dict[str, dict[str, int]] = {} + metadata: dict[str, dict[str, Any]] = {} + for event in events: + payload = event.get("data", {}) + if payload.get("kind") != kind: + continue + identity = payload.get(identity_field) + state = payload.get("state") + if not isinstance(identity, str) or state not in required_states: + raise ProjectionError(f"{kind} event is invalid") + states = values.setdefault(identity, {}) + if state in states: + raise ProjectionError(f"{kind} transition is duplicated") + states[state] = event["monotonic_offset_ns"] + metadata.setdefault(identity, payload) + if any( + metadata[identity].get(key) != payload.get(key) + for key in metadata[identity] + if key not in {"state"} + ): + raise ProjectionError(f"{kind} identity metadata drifted") + if not values or any( + set(states) != set(required_states) for states in values.values() + ): + raise ProjectionError(f"{kind} transitions are incomplete") + if any( + any( + states[left] > states[right] + for left, right in zip(required_states, required_states[1:]) + ) + for states in values.values() + ): + raise ProjectionError(f"{kind} transitions are unordered") + return values, metadata + + +def _validate_phases(events: list[dict[str, Any]], trials: set[str]) -> None: + values: dict[tuple[str, str], dict[str, int]] = {} + for event in events: + payload = event.get("data", {}) + if payload.get("kind") != "trial_phase": + continue + trial_id = payload.get("trial_id") + phase = payload.get("phase") + state = payload.get("state") + if ( + trial_id not in trials + or phase not in PHASES + or state + not in { + "running", + "completed", + } + ): + raise ProjectionError("trial phase event is invalid") + states = values.setdefault((trial_id, phase), {}) + if state in states: + raise ProjectionError("trial phase transition is duplicated") + states[state] = event["monotonic_offset_ns"] + expected = {(trial_id, phase) for trial_id in trials for phase in PHASES} + if set(values) != expected or any( + set(states) != {"running", "completed"} + or states["running"] > states["completed"] + for states in values.values() + ): + raise ProjectionError("trial phase transitions are incomplete or unordered") + + +def _validate_observations( + observations: list[dict[str, Any]], + trial_states: dict[str, dict[str, int]], + trial_metadata: dict[str, dict[str, Any]], + cell_by_id: dict[str, str], + role: str, +) -> dict[str, dict[str, Any]]: + by_kind: dict[str, list[dict[str, Any]]] = {} + for envelope in observations: + record = _required_dict(envelope.get("record"), "observation record") + kind = record.get("record") + data = record.get("data") + if not isinstance(kind, str) or not isinstance(data, dict): + raise ProjectionError("observation record is invalid") + by_kind.setdefault(kind, []).append(data) + trials = by_kind.get("trial", []) + if len(trials) != EXPECTED[role]["batches"]: + raise ProjectionError("trial observation count is invalid") + by_trial: dict[str, dict[str, Any]] = {} + for trial in trials: + trial_id = trial.get("trial_id") + if trial_id not in trial_states or trial_id in by_trial: + raise ProjectionError("trial observation identity is invalid") + event_meta = trial_metadata[trial_id] + warmup = event_meta.get("warmup") + if ( + trial.get("cell_id") != event_meta.get("cell_id") + or trial.get("cell_id") not in cell_by_id + or trial.get("warmup") is not warmup + or trial.get("kind") != ("warmup" if warmup else "measured") + or trial.get("status") != "success" + or trial.get("product_succeeded") is not True + or trial.get("checks_passed") is not True + or trial.get("cleanup_baseline_restored") is not True + or trial.get("infrastructure_failed") is not False + or trial.get("reportable") is not (not warmup) + ): + raise ProjectionError("trial observation did not pass") + phase_ns: dict[str, int] = {} + for phase in PHASES: + value = trial.get(f"{phase}_ns") + if not isinstance(value, int) or value < 0: + raise ProjectionError("trial phase duration is invalid") + phase_ns[phase] = value + active_ns = ( + trial_states[trial_id]["completed"] - trial_states[trial_id]["preparing"] + ) + if sum(phase_ns.values()) > active_ns: + raise ProjectionError("trial phase sum exceeds its active span") + by_trial[trial_id] = {**trial, "_phase_ns": phase_ns} + if set(by_trial) != set(trial_states): + raise ProjectionError("trial events and observations do not match") + + requests = by_kind.get("request", []) + request_ids: set[str] = set() + request_counts: dict[str, int] = {} + for request in requests: + request_id = request.get("request_id") + trial_id = request.get("trial_id") + if ( + not isinstance(request_id, str) + or request_id in request_ids + or trial_id not in by_trial + or request.get("status") != "success" + or request.get("cell_id") != by_trial[trial_id]["cell_id"] + or request.get("warmup") is not by_trial[trial_id]["warmup"] + ): + raise ProjectionError("request observation is invalid") + request_ids.add(request_id) + request_counts[trial_id] = request_counts.get(trial_id, 0) + 1 + if len(requests) != EXPECTED[role]["requests"] or any( + request_counts.get(trial_id, 0) != trial["request_count"] + for trial_id, trial in by_trial.items() + ): + raise ProjectionError("request observations do not match trial counts") + operations = by_kind.get("operation", []) + if len(operations) != len(by_trial) or { + item.get("trial_id") for item in operations + } != set(by_trial): + raise ProjectionError("operation observations do not match trials") + if any(item.get("passed") is not True for item in by_kind.get("check", [])): + raise ProjectionError("a correctness observation failed") + return by_trial + + +def load_run_profile( + root: Path, + role: str, + *, + verify_inventory: bool = True, + protocol_version: str = "v1.0", +) -> RunProfile: + if protocol_version not in PROTOCOLS: + raise ProjectionError("unsupported EXP1 protocol version") + expected = EXPECTED[role] + root = root.resolve(strict=True) + archive_manifest = _required_dict( + load_json(root / "archive-manifest.json"), "archive manifest" + ) + if verify_inventory: + verify_archive_inventory(root, archive_manifest) + campaign = _required_dict( + load_json(root / "campaign-manifest.json"), "campaign manifest" + ) + _validate_prefreeze_campaign(campaign, protocol_version) + manifest = envelope_data( + root / "raw/run-manifest.json", "eos_benchmark_run_manifest" + ) + plan = envelope_data(root / "raw/expanded-plan.json", "eos_benchmark_expanded_plan") + if ( + archive_manifest.get("run_id") != manifest.get("run_id") + or campaign.get("run_id") != manifest.get("run_id") + or archive_manifest.get("disposition") != expected["disposition"] + or campaign.get("disposition") != expected["disposition"] + or archive_manifest.get("run_status") != "completed" + or campaign.get("run_status") != "completed" + or campaign.get("state") != "completed" + or campaign.get("correctness") != "pass" + or campaign.get("eligibility") != expected["eligibility"] + or manifest.get("state") != "completed" + or manifest.get("correctness") != "pass" + or manifest.get("failure") is not None + or manifest.get("name") != expected["name"] + or manifest.get("plan_hash") != plan.get("plan_hash") + or archive_manifest.get("plan_hash") != plan.get("plan_hash") + ): + raise ProjectionError(f"{role} archive terminal provenance is invalid") + if archive_manifest.get("protocol_version") not in ( + None if protocol_version == "v1.0" else protocol_version, + protocol_version, + ): + raise ProjectionError(f"{role} archive protocol identity is invalid") + _validate_protocol_transport(campaign, manifest, plan, protocol_version) + cleanup = _cleanup_identity(campaign) + if cleanup["product_commit"] != campaign.get("product", {}).get("commit"): + raise ProjectionError("post-run product identity drifted") + raw_corpus = _required_dict(campaign.get("raw_corpus"), "raw corpus") + if ( + raw_corpus.get("events_sha256") != sha256_file(root / "raw/events.ndjson") + or raw_corpus.get("observations_sha256") + != sha256_file(root / "raw/observations.ndjson") + or campaign.get("plan", {}).get("expanded_plan_sha256") + != sha256_file(root / "raw/expanded-plan.json") + ): + raise ProjectionError("raw corpus identity is invalid") + semantics = _validate_plan(plan, role=role) + events = _read_journal( + root / "raw/events.ndjson", + schema_name="eos_benchmark_event", + schema_version=1, + ) + observations = _read_journal( + root / "raw/observations.ndjson", + schema_name="eos_benchmark_observation", + schema_version=5, + ) + family_states, _ = _state_times( + events, + kind="family_state", + identity_field="family", + required_states=("preparing", "running", "completed"), + ) + cell_states, _ = _state_times( + events, + kind="cell_state", + identity_field="cell_id", + required_states=("preparing", "running", "completed"), + ) + trial_states, trial_metadata = _state_times( + events, + kind="trial_state", + identity_field="trial_id", + required_states=("preparing", "completed"), + ) + _validate_phases(events, set(trial_states)) + trial_observations = _validate_observations( + observations, + trial_states, + trial_metadata, + semantics["cell_by_id"], + role, + ) + if set(cell_states) != set(semantics["cell_by_id"]): + raise ProjectionError("cell events do not match the expanded plan") + family_blocks = { + block["family_id"]: block["cell_ids"] for block in plan["execution_blocks"] + } + if set(family_states) != set(family_blocks): + raise ProjectionError("family events do not match execution blocks") + elapsed = elapsed_ns(manifest) + family_durations = { + family: states["completed"] - states["preparing"] + for family, states in family_states.items() + } + run_residual = elapsed - sum(family_durations.values()) + if run_residual < 0: + raise ProjectionError("run residual duration is negative") + family_residual: dict[str, int] = {} + for family, cell_ids in family_blocks.items(): + prior_completed = -1 + cell_total = 0 + for cell_id in cell_ids: + states = cell_states[cell_id] + if ( + states["preparing"] < family_states[family]["preparing"] + or states["completed"] > family_states[family]["completed"] + or states["preparing"] < prior_completed + ): + raise ProjectionError("cell intervals are not sequentially nested") + prior_completed = states["completed"] + cell_total += states["completed"] - states["preparing"] + residual = family_durations[family] - cell_total + if residual < 0: + raise ProjectionError("family residual duration is negative") + family_residual[family] = residual + trials_by_cell: dict[str, list[tuple[TrialSpan, dict[str, Any]]]] = { + cell_id: [] for cell_id in cell_states + } + for trial_id, states in trial_states.items(): + metadata = trial_metadata[trial_id] + cell_id = metadata["cell_id"] + if cell_id not in trials_by_cell: + raise ProjectionError("trial references an unknown cell") + observation = trial_observations[trial_id] + trials_by_cell[cell_id].append( + ( + TrialSpan( + trial_id=trial_id, + warmup=metadata["warmup"], + start_ns=states["preparing"], + end_ns=states["completed"], + phase_ns=observation["_phase_ns"], + ), + observation, + ) + ) + cells: dict[str, CellProfile] = {} + for cell_id, trial_pairs in trials_by_cell.items(): + trial_pairs.sort(key=lambda item: item[0].start_ns) + trials = tuple(item[0] for item in trial_pairs) + expected_warmups = expected["warmups"] + expected_measured = expected["measured"] + if ( + len(trials) != expected_warmups + expected_measured + or [trial.warmup for trial in trials] + != [True] * expected_warmups + [False] * expected_measured + ): + raise ProjectionError("cell trial order or count is invalid") + states = cell_states[cell_id] + if ( + trials[0].start_ns < states["preparing"] + or trials[-1].end_ns > states["completed"] + ): + raise ProjectionError("trial intervals escaped their cell") + gaps = tuple( + right.start_ns - left.end_ns for left, right in zip(trials, trials[1:]) + ) + if any(gap < 0 for gap in gaps): + raise ProjectionError("trial intervals overlap") + semantic = semantics["cell_by_id"][cell_id] + cell = semantics["cell_data"][semantic] + profile = CellProfile( + semantic_key=semantic, + semantic_sha256=canonical_sha256(json.loads(semantic)), + family_id=cell["family_id"], + operation_id=cell["operation_id"], + leading_ns=trials[0].start_ns - states["preparing"], + trailing_ns=states["completed"] - trials[-1].end_ns, + trials=trials, + gaps_ns=gaps, + ) + cells[semantic] = profile + return RunProfile( + role=role, + identity={ + "run_id": manifest["run_id"], + "protocol_version": protocol_version, + "plan_hash": plan["plan_hash"], + "archive_manifest_sha256": sha256_file(root / "archive-manifest.json"), + "archive_content_tree_sha256": archive_manifest["content_tree_sha256"], + "campaign_manifest_sha256": sha256_file(root / "campaign-manifest.json"), + "raw_events_sha256": raw_corpus["events_sha256"], + "raw_observations_sha256": raw_corpus["observations_sha256"], + "provenance": _identity(campaign, manifest), + }, + plan=semantics, + elapsed_ns=elapsed, + run_residual_ns=run_residual, + family_residual_ns=family_residual, + cells=cells, + ) + + +def validate_cross_run( + smoke: RunProfile, + pilot: RunProfile, + final_plan: dict[str, Any], +) -> dict[str, Any]: + final_semantics = _validate_plan(final_plan, role="final") + protocol_version = smoke.identity.get("protocol_version") + if ( + protocol_version not in PROTOCOLS + or pilot.identity.get("protocol_version") != protocol_version + ): + raise ProjectionError("smoke and pilot protocol versions drifted") + if _cross_run_provenance( + smoke.identity["provenance"], + protocol_version=protocol_version, + ) != _cross_run_provenance( + pilot.identity["provenance"], + protocol_version=protocol_version, + ): + raise ProjectionError("smoke and pilot provenance identities drifted") + for candidate in (pilot.plan, final_semantics): + if ( + smoke.plan["canonical"] != candidate["canonical"] + or smoke.plan["effective_environment"] != candidate["effective_environment"] + or smoke.plan["cells"] != candidate["cells"] + or smoke.plan["blocks"] != candidate["blocks"] + ): + raise ProjectionError("smoke, pilot, and final plan semantics drifted") + return final_semantics + + +def ceil_fraction(value: Fraction) -> int: + return -(-value.numerator // value.denominator) + + +def seconds_string(nanoseconds: int) -> str: + whole, fraction = divmod(nanoseconds, 1_000_000_000) + return f"{whole}.{fraction:09d}" + + +def fraction_value(value: Fraction) -> dict[str, int]: + return { + "numerator_ns": value.numerator, + "denominator": value.denominator, + "ceil_ns": ceil_fraction(value), + } + + +def project_structural( + smoke: RunProfile, + pilot: RunProfile, + final_plan: dict[str, Any], + *, + final_plan_sha256: str, + script_sha256: str, + protocol_version: str = "v1.0", +) -> dict[str, Any]: + validate_cross_run(smoke, pilot, final_plan) + family_ids = sorted(pilot.family_residual_ns) + cell_keys = sorted(pilot.cells) + if set(smoke.family_residual_ns) != set(family_ids) or set(smoke.cells) != set( + cell_keys + ): + raise ProjectionError("smoke and pilot structural identities drifted") + run_fixed = max(smoke.run_residual_ns, pilot.run_residual_ns) + family_fixed = { + family: max( + smoke.family_residual_ns[family], + pilot.family_residual_ns[family], + ) + for family in family_ids + } + central = Fraction(run_fixed + sum(family_fixed.values())) + envelope = run_fixed + sum(family_fixed.values()) + cell_results: list[dict[str, Any]] = [] + for key in cell_keys: + smoke_cell = smoke.cells[key] + pilot_cell = pilot.cells[key] + leading = max(smoke_cell.leading_ns, pilot_cell.leading_ns) + trailing = max(smoke_cell.trailing_ns, pilot_cell.trailing_ns) + fixed = leading + trailing + warmups = pilot_cell.warmups + measured = pilot_cell.measured + smoke_cold = smoke_cell.measured + if ( + len(warmups) != 2 + or len(measured) != 5 + or len(smoke_cold) != 1 + or len(pilot_cell.gaps_ns) != 6 + ): + raise ProjectionError("source cell does not have the required trial shape") + warmup_gaps = pilot_cell.gaps_ns[:2] + measured_gaps = pilot_cell.gaps_ns[2:] + if len(measured_gaps) != 4: + raise ProjectionError("pilot measured transitions are incomplete") + central_cell = ( + Fraction(fixed) + + sum((Fraction(trial.active_ns) for trial in warmups), Fraction()) + + sum((Fraction(gap) for gap in warmup_gaps), Fraction()) + + Fraction( + FINAL_MEASURED * sum(trial.active_ns for trial in measured), + len(measured), + ) + + Fraction( + (FINAL_MEASURED - 1) * sum(measured_gaps), + len(measured_gaps), + ) + ) + warmup_active_max = max( + [trial.active_ns for trial in warmups] + + [trial.active_ns for trial in smoke_cold] + ) + measured_active_max = max(trial.active_ns for trial in measured) + gap_max = max(pilot_cell.gaps_ns) + envelope_cell = ( + fixed + + FINAL_WARMUPS * warmup_active_max + + FINAL_MEASURED * measured_active_max + + (FINAL_WARMUPS + FINAL_MEASURED - 1) * gap_max + ) + central += central_cell + envelope += envelope_cell + cell_results.append( + { + "semantic_sha256": pilot_cell.semantic_sha256, + "family_id": pilot_cell.family_id, + "operation_id": pilot_cell.operation_id, + "fixed": { + "leading_ns": leading, + "trailing_ns": trailing, + "total_ns": fixed, + }, + "source_samples": { + "smoke_cold_active_ns": [trial.active_ns for trial in smoke_cold], + "pilot_warmup_active_ns": [trial.active_ns for trial in warmups], + "pilot_measured_active_ns": [trial.active_ns for trial in measured], + "pilot_transition_gap_ns": list(pilot_cell.gaps_ns), + }, + "central_projected": fraction_value(central_cell), + "envelope_projected_ns": envelope_cell, + } + ) + central_ns = ceil_fraction(central) + runtime_pass = ( + pilot.elapsed_ns <= LIMIT_NS and central_ns <= LIMIT_NS and envelope <= LIMIT_NS + ) + return { + "schema_version": 2, + "model_revision": 1, + "protocol_version": protocol_version, + "purpose": "EXP1 Gate 3 runtime projection; not manuscript evidence", + "analysis_script_sha256": script_sha256, + "final_plan": { + "sha256": final_plan_sha256, + "plan_hash": final_plan["plan_hash"], + "cells": FINAL_CELLS, + "warmups_per_cell": FINAL_WARMUPS, + "measured_trials_per_cell": FINAL_MEASURED, + "trial_batches": FINAL_BATCHES, + "issued_operation_requests": FINAL_REQUESTS, + }, + "limit_ns": LIMIT_NS, + "inputs": { + "smoke": smoke.identity, + "pilot": pilot.identity, + }, + "decomposition": { + "run_fixed_ns": run_fixed, + "family_fixed_ns": family_fixed, + "cells": cell_results, + }, + "models": { + "pilot_elapsed_lower_bound_ns": pilot.elapsed_ns, + "central_structural": fraction_value(central), + "observed_envelope_ns": envelope, + }, + "display_seconds": { + "pilot_elapsed": seconds_string(pilot.elapsed_ns), + "central_structural": seconds_string(central_ns), + "observed_envelope": seconds_string(envelope), + "limit": seconds_string(LIMIT_NS), + }, + "pass_conditions": { + "pilot_elapsed_within_limit": pilot.elapsed_ns <= LIMIT_NS, + "central_structural_within_limit": central_ns <= LIMIT_NS, + "observed_envelope_within_limit": envelope <= LIMIT_NS, + }, + "gate_3_runtime_pass": runtime_pass, + "decision": ( + "pass_runtime_projection" + if runtime_pass + else "block_freeze_and_final_runtime_projection_exceeds_1400_seconds" + ), + "scientific_use": ( + "Exploratory capacity decision only. No smoke, pilot, or projected " + "value is eligible for manuscript tables." + ), + } + + +def render_json(value: dict[str, Any]) -> str: + return ( + json.dumps( + value, + indent=2, + sort_keys=True, + ensure_ascii=False, + allow_nan=False, + ) + + "\n" + ) + + +def parser() -> argparse.ArgumentParser: + command = argparse.ArgumentParser() + command.add_argument("--smoke-archive", type=Path, required=True) + command.add_argument("--pilot-archive", type=Path, required=True) + command.add_argument("--final-plan", type=Path, required=True) + command.add_argument("--output", type=Path, required=True) + command.add_argument("--protocol-version", choices=sorted(PROTOCOLS), required=True) + return command + + +def main() -> int: + args = parser().parse_args() + try: + smoke_path = args.smoke_archive.resolve(strict=True) + pilot_path = args.pilot_archive.resolve(strict=True) + final_plan_path = args.final_plan.resolve(strict=True) + output = args.output.resolve() + if output.exists() or output.is_symlink(): + raise ProjectionError("projection output already exists") + output.parent.mkdir(parents=True, exist_ok=True) + final_plan = expanded_plan_data(final_plan_path) + result = project_structural( + load_run_profile( + smoke_path, "smoke", protocol_version=args.protocol_version + ), + load_run_profile( + pilot_path, "pilot", protocol_version=args.protocol_version + ), + final_plan, + final_plan_sha256=sha256_file(final_plan_path), + script_sha256=sha256_file(Path(__file__).resolve(strict=True)), + protocol_version=args.protocol_version, + ) + rendered = render_json(result) + output.write_text(rendered, encoding="utf-8", newline="\n") + except ( + KeyError, + OSError, + TypeError, + ValueError, + json.JSONDecodeError, + ProjectionError, + ) as error: + print(f"projection_error: {error}", file=sys.stderr) + return 1 + print(rendered, end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ephemeral-sandbox-v1/experiments/scripts/qualify_final_host.sh b/ephemeral-sandbox-v1/experiments/scripts/qualify_final_host.sh new file mode 100644 index 0000000..df2b975 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/qualify_final_host.sh @@ -0,0 +1,143 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_directory="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +paper_root="$(cd -- "${script_directory}/../.." && pwd -P)" +product_root="/srv/eos-benchmark/product" +product_bin_dir="${product_root}/target/release" +image_reference="ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" +qualification_id="qualification-$(date -u +%Y%m%dT%H%M%SZ)" +qualification_directory="${paper_root}/experiments/runs/${qualification_id}" +preflight_log="${qualification_directory}/environment-preflight.txt" +smoke_stdout="${qualification_directory}/cli-env-smoke-driver.stdout" +smoke_stderr="${qualification_directory}/cli-env-smoke-driver.stderr" +smoke_summary="${qualification_directory}/cli-env-smoke-summary.json" +smoke_elapsed="${qualification_directory}/cli-env-smoke-elapsed-seconds.txt" + +fail() { + printf 'FAIL\t%s\n' "$1" >&2 + exit 1 +} + +snapshot_owned_state() { + local output="$1" + { + find "${paper_root}/.benchmark-state/runtime" -mindepth 1 -maxdepth 1 -printf 'runtime\t%f\n' 2>/dev/null || true + find "${paper_root}/.benchmark-state/runs" -mindepth 1 -maxdepth 1 -printf 'run\t%f\n' 2>/dev/null || true + docker ps -aq --filter label=eos.gateway_instance_id 2>/dev/null | + sed 's/^/container\t/' + docker volume ls -q --filter label=eos.gateway_instance_id 2>/dev/null | + sed 's/^/volume\t/' + ps -eo pid=,args= | + awk -v binary="${product_bin_dir}/sandbox-gateway" \ + 'index($0, binary) && index($0, " serve ") {print "process\t" $0}' + } | sort >"$output" +} + +[[ "$paper_root" == "/srv/eos-benchmark/paper" ]] || + fail "qualification must run from /srv/eos-benchmark/paper" +mkdir -p -- "$qualification_directory" +before_state="$(mktemp)" +after_state="$(mktemp)" +evidence_finalized=0 + +finalize_on_exit() { + local status="$?" + trap - EXIT + set +e + if [[ "$evidence_finalized" -eq 0 && -f "$before_state" ]]; then + snapshot_owned_state "$after_state" + diff -u -- "$before_state" "$after_state" \ + >"${qualification_directory}/leak-diff.txt" + fi + rm -f -- "$before_state" "$after_state" + exit "$status" +} + +trap finalize_on_exit EXIT +snapshot_owned_state "$before_state" +if [[ -s "$before_state" ]]; then + cp -- "$before_state" "${qualification_directory}/preexisting-owned-state.txt" + fail "pre-existing benchmark-owned process, runtime, container, or volume state is not allowed" +fi + +export PRODUCT_ROOT="$product_root" +export PRODUCT_BIN_DIR="$product_bin_dir" +export IMAGE_REFERENCE="$image_reference" +export PAPER_ROOT="$paper_root" +export PYTHON_BIN="${paper_root}/.venv/bin/python" + +bash "${script_directory}/verify_environment.sh" 2>&1 | tee "$preflight_log" + +smoke_started="$(date +%s)" +if ! bash "${script_directory}/cli_environment_smoke.sh" \ + "$qualification_directory" >"$smoke_stdout" 2>"$smoke_stderr"; then + fail "CLI-only environment smoke failed; inspect archived stdout and stderr" +fi +smoke_seconds="$(( $(date +%s) - smoke_started ))" +printf '%s\n' "$smoke_seconds" >"$smoke_elapsed" +[[ "$smoke_seconds" -le 180 ]] || + fail "CLI-only environment smoke exceeded the 180-second acceptance budget" +[[ ! -s "$smoke_stderr" ]] || + fail "CLI-only environment smoke emitted stderr output" + +sandbox_ids="$( + "${paper_root}/.venv/bin/python" - "$smoke_summary" <<'PY' +import json +import os +import sys + +value = json.loads(open(sys.argv[1], encoding="utf-8").read()) +assert value["client_cohort"] == "product_cli" +assert value["state"] == "completed" +assert value["correctness"] == "pass" +assert value["completed_batches"] == 2 +assert value["total_batches"] == 2 +assert value["operation_count"] == 20 +assert value["warning_count"] == 0 +assert value["failure_count"] == 0 +assert value["cleanup"] == "pass" +assert value["elapsed_seconds"] <= 180 +assert value["image_reference"] == os.environ["IMAGE_REFERENCE"] +assert value["gateway_instance_id"].startswith("cli-env-smoke-") +operation_sequence = [ + "list_docker_images", + "create_sandbox", + "exec_command", + "file_write", + "file_read", + "file_edit", + "file_read", + "snapshot", + "destroy_sandbox", + "list_sandboxes", +] +assert value["operation_sequence"] == operation_sequence +assert len(value["batches"]) == 2 +assert len(value["sandbox_ids"]) == 2 +assert len(set(value["sandbox_ids"])) == 2 +for index, batch in enumerate(value["batches"], start=1): + assert batch["batch_index"] == index + assert batch["sandbox_id"] == value["sandbox_ids"][index - 1] + assert batch["state"] == "completed" + assert batch["correctness"] == "pass" + assert batch["operation_count"] == len(operation_sequence) + assert batch["cleanup"] == "pass" +print(",".join(value["sandbox_ids"])) +PY +)" || fail "CLI-only environment smoke summary failed strict acceptance" + +snapshot_owned_state "$after_state" +if ! diff -u -- "$before_state" "$after_state" >"${qualification_directory}/leak-diff.txt"; then + fail "benchmark-owned process, runtime, container, or volume baseline changed" +fi +[[ ! -s "$after_state" ]] || + fail "CLI-only environment smoke left benchmark-owned state behind" +evidence_finalized=1 + +printf 'PASS\tfinal-host qualification completed\n' +printf 'INFO\tqualification_id=%s\n' "$qualification_id" +printf 'INFO\tclient_cohort=product_cli\n' +printf 'INFO\tsandbox_ids=%s\n' "$sandbox_ids" +printf 'INFO\tsmoke_elapsed_seconds=%s\n' "$smoke_seconds" +printf 'INFO\tartifact_directory=%s\n' "$qualification_directory" diff --git a/ephemeral-sandbox-v1/experiments/scripts/qualify_windows_docker_environment.ps1 b/ephemeral-sandbox-v1/experiments/scripts/qualify_windows_docker_environment.ps1 new file mode 100644 index 0000000..04a6c1a --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/qualify_windows_docker_environment.ps1 @@ -0,0 +1,749 @@ +[CmdletBinding()] +param( + [string]$PaperRoot = "", + [string]$ProductRoot = "", + [string]$PackageRoot = "", + [string]$ArtifactDirectory = "", + [string]$ExpectedComputerName = "DESKTOP-OLP1ADS", + [string]$ImageReference = "ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf", + [int]$MaximumSeconds = 180 +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = "Stop" + +$expectedProductCommit = "b22862550e0a7cb4fe61ce581831e9244cc492b5" +$expectedDockerVersion = "29.0.1" +$expectedReleaseArchiveSha256 = "9f2327578c186897578f0d502893d894aed52be27306f43f75afa3205eba9fdb" +$expectedHashes = [ordered]@{ + "bin\sandbox-gateway.exe" = "3a96bedcfa9857bd3881155d758ec2d969f6265456ec3b2878eb6dbb26dc9368" + "bin\sandbox-manager-cli.exe" = "b43ec520edc2f436adc8aa7e8b2b50680bb9021883fe23d79a85b17afd2e10fe" + "bin\sandbox-runtime-cli.exe" = "df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4" + "bin\sandbox-observability-cli.exe" = "0e0471e52750805570876a6244868764c44e166ec653627b9ebd490176e2fcbe" + "config\windows-amd64.yml" = "0f0efd15e5111851054e0f7c1ce0f3eaebb3b3047c1b9e2322544036f5daf5db" + "dist\sandbox-daemon-linux-amd64" = "2da4395cd835e5325bc3e55b9c2f3b67565ea7c698fce5e086167ec4a2092a39" +} + +if (-not $PaperRoot) { + $PaperRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot "..\..")).Path +} +$PaperRoot = (Resolve-Path -LiteralPath $PaperRoot).Path +if (-not $ProductRoot) { + $ProductRoot = Join-Path $PaperRoot "..\..\ephemeral-sandbox" +} +$ProductRoot = (Resolve-Path -LiteralPath $ProductRoot).Path +if (-not $PackageRoot) { + $PackageRoot = Join-Path $ProductRoot "target\windows-v0.1.4" +} +$PackageRoot = (Resolve-Path -LiteralPath $PackageRoot).Path +if (-not $ArtifactDirectory) { + $qualificationId = "qualification-windows-docker-$([DateTime]::UtcNow.ToString('yyyyMMddTHHmmssZ'))" + $ArtifactDirectory = Join-Path $PaperRoot "experiments\runs\$qualificationId" +} +$ArtifactDirectory = [System.IO.Path]::GetFullPath($ArtifactDirectory) +if (Test-Path -LiteralPath $ArtifactDirectory) { + if ((Get-ChildItem -Force -LiteralPath $ArtifactDirectory | Measure-Object).Count -ne 0) { + throw "ArtifactDirectory must not exist or must be empty: $ArtifactDirectory" + } +} else { + New-Item -ItemType Directory -Path $ArtifactDirectory | Out-Null +} + +$startedAt = [DateTime]::UtcNow +$preflightLog = Join-Path $ArtifactDirectory "environment-preflight.txt" +$gatewayStdout = Join-Path $ArtifactDirectory "gateway.stdout.log" +$gatewayStderr = Join-Path $ArtifactDirectory "gateway.stderr.log" +$gatewayPidFile = Join-Path $ArtifactDirectory "gateway.pid" +$gatewayConfig = Join-Path $ArtifactDirectory "effective-windows-docker.yml" +$gatewayRegistry = Join-Path $ArtifactDirectory "gateway-registry.json" +$summaryPath = Join-Path $ArtifactDirectory "windows-docker-cli-env-summary.json" +$beforeStatePath = Join-Path $ArtifactDirectory "owned-state-before.txt" +$afterStatePath = Join-Path $ArtifactDirectory "owned-state-after.txt" +$gatewayInstanceId = "cli-env-windows-$([DateTime]::UtcNow.ToString('yyyyMMddTHHmmssZ'))-$PID" +$authToken = [guid]::NewGuid().ToString("N") +$gatewayProcess = $null +$activeSandboxId = $null +$batchSandboxIds = [System.Collections.Generic.List[string]]::new() +$cleanupErrors = [System.Collections.Generic.List[string]]::new() +$primaryError = $null +$originalSharedBaseCache = $env:EOS_SHARED_BASE_CACHE + +function Write-Record([string]$Kind, [string]$Message) { + $line = "$Kind`t$Message" + Write-Host $line + Add-Content -LiteralPath $preflightLog -Value $line -Encoding utf8 +} + +function Pass([string]$Message) { + Write-Record "PASS" $Message +} + +function Info([string]$Message) { + Write-Record "INFO" $Message +} + +function Fail([string]$Message) { + Write-Record "FAIL" $Message + throw $Message +} + +function Quote-ProcessArgument([string]$Value) { + if ($Value -match '[\s"]') { + return '"' + ($Value -replace '"', '\"') + '"' + } + return $Value +} + +function Assert-X64Pe([string]$Path) { + $bytes = [System.IO.File]::ReadAllBytes($Path) + if ($bytes.Length -lt 128 -or $bytes[0] -ne 0x4d -or $bytes[1] -ne 0x5a) { + Fail "not a valid PE executable: $Path" + } + $peOffset = [BitConverter]::ToInt32($bytes, 0x3c) + if ($peOffset -lt 0 -or $peOffset + 6 -gt $bytes.Length) { + Fail "invalid PE header offset: $Path" + } + if ( + $bytes[$peOffset] -ne 0x50 -or + $bytes[$peOffset + 1] -ne 0x45 -or + $bytes[$peOffset + 2] -ne 0 -or + $bytes[$peOffset + 3] -ne 0 + ) { + Fail "missing PE signature: $Path" + } + $machine = [BitConverter]::ToUInt16($bytes, $peOffset + 4) + if ($machine -ne 0x8664) { + Fail "PE executable is not x64: $Path" + } +} + +function Assert-X64Elf([string]$Path) { + $bytes = [System.IO.File]::ReadAllBytes($Path) + if ( + $bytes.Length -lt 20 -or + $bytes[0] -ne 0x7f -or + $bytes[1] -ne 0x45 -or + $bytes[2] -ne 0x4c -or + $bytes[3] -ne 0x46 -or + $bytes[4] -ne 2 + ) { + Fail "daemon is not ELF64: $Path" + } + $machine = [BitConverter]::ToUInt16($bytes, 18) + if ($machine -ne 0x3e) { + Fail "daemon ELF is not x86-64: $Path" + } +} + +function Get-FreeTcpSocket { + $listener = [System.Net.Sockets.TcpListener]::new( + [System.Net.IPAddress]::Loopback, + 0 + ) + $listener.Start() + try { + $port = ([System.Net.IPEndPoint]$listener.LocalEndpoint).Port + } finally { + $listener.Stop() + } + return "127.0.0.1:$port" +} + +function Get-OwnedState { + $records = [System.Collections.Generic.List[string]]::new() + @(& docker ps -aq --filter "label=eos.gateway_instance_id") | + Where-Object { $_ } | + ForEach-Object { $records.Add("container`t$_") } + @(& docker volume ls -q --filter "label=eos.gateway_instance_id") | + Where-Object { $_ } | + ForEach-Object { $records.Add("volume`t$_") } + Get-Process -Name "sandbox-gateway" -ErrorAction SilentlyContinue | + ForEach-Object { $records.Add("process`t$($_.Id)") } + return @($records | Sort-Object) +} + +function Invoke-ProductCli( + [string]$RecordName, + [string]$Executable, + [string[]]$Arguments +) { + $stdoutPath = Join-Path $ArtifactDirectory "$RecordName.json" + $stderrPath = Join-Path $ArtifactDirectory "$RecordName.stderr" + $output = @(& $Executable @Arguments 2> $stderrPath) + $exitCode = $LASTEXITCODE + ($output -join [Environment]::NewLine) | + Set-Content -LiteralPath $stdoutPath -Encoding utf8 + if ($exitCode -ne 0) { + Fail "$RecordName exited with code $exitCode" + } + if ((Get-Item -LiteralPath $stderrPath).Length -ne 0) { + Fail "$RecordName emitted stderr" + } + $raw = Get-Content -Raw -LiteralPath $stdoutPath + try { + return $raw | ConvertFrom-Json + } catch { + Fail "$RecordName did not produce valid JSON" + } +} + +function Invoke-ManagerCli([string]$RecordName, [string[]]$Arguments) { + $prefix = @( + "--gateway-socket", $gatewaySocket, + "--gateway-auth-token", $authToken + ) + return Invoke-ProductCli $RecordName $managerCli ($prefix + $Arguments) +} + +function Invoke-RuntimeCli( + [string]$RecordName, + [string]$SandboxId, + [string[]]$Arguments +) { + $prefix = @( + "--gateway-socket", $gatewaySocket, + "--gateway-auth-token", $authToken, + "--sandbox-id", $SandboxId + ) + return Invoke-ProductCli $RecordName $runtimeCli ($prefix + $Arguments) +} + +function Invoke-ObservabilityCli( + [string]$RecordName, + [string]$SandboxId +) { + $arguments = @( + "--gateway-socket", $gatewaySocket, + "--gateway-auth-token", $authToken, + "snapshot", + "--sandbox-id", $SandboxId + ) + return Invoke-ProductCli $RecordName $observabilityCli $arguments +} + +function Stop-QualificationGateway { + if ($null -eq $gatewayProcess) { + return + } + $process = Get-Process -Id $gatewayProcess.Id -ErrorAction SilentlyContinue + if ($process) { + Stop-Process -Id $process.Id -ErrorAction SilentlyContinue + try { + Wait-Process -Id $process.Id -Timeout 5 -ErrorAction Stop + } catch { + Stop-Process -Id $process.Id -Force -ErrorAction SilentlyContinue + } + } + $script:gatewayProcess = $null +} + +function Remove-QualificationVolumes { + $volumes = @( + & docker volume ls -q ` + --filter "label=eos.gateway_instance_id=$gatewayInstanceId" + ) | Where-Object { $_ } + foreach ($volume in $volumes) { + $record = @(& docker volume inspect $volume | ConvertFrom-Json)[0] + $actualGatewayId = $record.Labels."eos.gateway_instance_id" + if ($actualGatewayId -ne $gatewayInstanceId) { + throw "refusing cleanup of volume with mismatched owner: $volume" + } + & docker volume rm -- $volume | + Add-Content -LiteralPath (Join-Path $ArtifactDirectory "owned-volume-cleanup.stdout") -Encoding utf8 + if ($LASTEXITCODE -ne 0) { + throw "failed to remove qualification-owned volume: $volume" + } + } +} + +function Cleanup-Qualification { + if ($activeSandboxId) { + try { + $cleanupStdout = Join-Path $ArtifactDirectory "cleanup-destroy.json" + $cleanupStderr = Join-Path $ArtifactDirectory "cleanup-destroy.stderr" + $arguments = @( + "--gateway-socket", $gatewaySocket, + "--gateway-auth-token", $authToken, + "destroy_sandbox", + "--sandbox-id", $activeSandboxId + ) + $output = @(& $managerCli @arguments 2> $cleanupStderr) + ($output -join [Environment]::NewLine) | + Set-Content -LiteralPath $cleanupStdout -Encoding utf8 + if ($LASTEXITCODE -ne 0) { + $cleanupErrors.Add("product CLI failed to destroy active sandbox $activeSandboxId") + } + } catch { + $cleanupErrors.Add("cleanup destroy exception: $($_.Exception.Message)") + } + $script:activeSandboxId = $null + } + Stop-QualificationGateway + try { + Remove-QualificationVolumes + } catch { + $cleanupErrors.Add($_.Exception.Message) + } + $ownedContainers = @(@( + & docker ps -aq ` + --filter "label=eos.gateway_instance_id=$gatewayInstanceId" + ) | Where-Object { $_ }) + $ownedVolumes = @(@( + & docker volume ls -q ` + --filter "label=eos.gateway_instance_id=$gatewayInstanceId" + ) | Where-Object { $_ }) + if ($ownedContainers.Count -ne 0) { + $cleanupErrors.Add("qualification gateway left $($ownedContainers.Count) container(s)") + } + if ($ownedVolumes.Count -ne 0) { + $cleanupErrors.Add("qualification gateway left $($ownedVolumes.Count) volume(s)") + } + if ($null -eq $originalSharedBaseCache) { + Remove-Item Env:EOS_SHARED_BASE_CACHE -ErrorAction SilentlyContinue + } else { + $env:EOS_SHARED_BASE_CACHE = $originalSharedBaseCache + } +} + +$gateway = Join-Path $PackageRoot "bin\sandbox-gateway.exe" +$managerCli = Join-Path $PackageRoot "bin\sandbox-manager-cli.exe" +$runtimeCli = Join-Path $PackageRoot "bin\sandbox-runtime-cli.exe" +$observabilityCli = Join-Path $PackageRoot "bin\sandbox-observability-cli.exe" +$daemon = Join-Path $PackageRoot "dist\sandbox-daemon-linux-amd64" +$templateConfig = Join-Path $PackageRoot "config\windows-amd64.yml" +$releaseArchive = Join-Path ( + Split-Path -Parent $ProductRoot +) "final-host-staging\v0.1.4-windows-release-input\ephemeral-sandbox-windows-amd64.zip" +$gatewaySocket = Get-FreeTcpSocket + +try { + $beforeState = @(Get-OwnedState) + $beforeState | Set-Content -LiteralPath $beforeStatePath -Encoding utf8 + + if ([Environment]::OSVersion.Platform -ne [PlatformID]::Win32NT) { + Fail "qualification host is not Windows" + } + if (-not [Environment]::Is64BitOperatingSystem) { + Fail "Windows host is not 64-bit" + } + if ($env:COMPUTERNAME -ne $ExpectedComputerName) { + Fail "computer name is not $ExpectedComputerName" + } + $os = Get-CimInstance Win32_OperatingSystem + $buildNumber = [int]$os.BuildNumber + if ($buildNumber -lt 22000) { + Fail "Windows build is older than the Windows 11 baseline" + } + Pass "native Windows x64 host $ExpectedComputerName uses build $buildNumber" + + $computer = Get-CimInstance Win32_ComputerSystem + if ([int]$computer.NumberOfLogicalProcessors -lt 8) { + Fail "host exposes fewer than 8 logical processors" + } + if ([int64]$computer.TotalPhysicalMemory -lt 15GB) { + Fail "host exposes less than 15 GiB physical memory" + } + Pass "host capacity is $($computer.NumberOfLogicalProcessors) logical CPUs and $($computer.TotalPhysicalMemory) bytes memory" + + foreach ($root in @($PaperRoot, $ProductRoot, $PackageRoot)) { + $drive = [System.IO.Path]::GetPathRoot($root).TrimEnd("\").TrimEnd(":") + $volume = Get-Volume -DriveLetter $drive + if ($volume.FileSystem -ne "NTFS") { + Fail "host path is not on NTFS: $root" + } + if ([int64]$volume.Size -lt 100GB) { + Fail "host filesystem is smaller than 100 GiB: $root" + } + if ([int64]$volume.SizeRemaining -lt 20GB) { + Fail "host filesystem has less than 20 GiB free: $root" + } + } + Pass "paper, product, and package roots are on NTFS with required capacity" + + $branch = (& git -C $ProductRoot branch --show-current).Trim() + $commit = (& git -C $ProductRoot rev-parse HEAD).Trim() + $status = @(& git -C $ProductRoot status --porcelain=v1) + if ($branch -ne "main" -or $commit -ne $expectedProductCommit -or $status.Count -ne 0) { + Fail "product checkout is not clean main at $expectedProductCommit" + } + Pass "product checkout is clean main at $commit" + + if (-not (Test-Path -LiteralPath $releaseArchive -PathType Leaf)) { + Fail "official Windows release archive is missing" + } + $releaseArchiveHash = (Get-FileHash -Algorithm SHA256 -LiteralPath $releaseArchive).Hash.ToLowerInvariant() + if ($releaseArchiveHash -ne $expectedReleaseArchiveSha256) { + Fail "official Windows release archive hash mismatch" + } + Pass "official v0.1.4 Windows archive hash matches" + + foreach ($relativePath in $expectedHashes.Keys) { + $path = Join-Path $PackageRoot $relativePath + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Fail "release payload is missing: $relativePath" + } + $actualHash = (Get-FileHash -Algorithm SHA256 -LiteralPath $path).Hash.ToLowerInvariant() + if ($actualHash -ne $expectedHashes[$relativePath]) { + Fail "release payload hash mismatch: $relativePath" + } + } + foreach ($path in @($gateway, $managerCli, $runtimeCli, $observabilityCli)) { + Assert-X64Pe $path + } + Assert-X64Elf $daemon + Pass "native x64 gateway/CLIs and Linux x86-64 daemon match the v0.1.4 package" + + $managerHelp = @(& $managerCli help 2>&1) -join "`n" + $runtimeHelp = @(& $runtimeCli --sandbox-id preflight-only help 2>&1) -join "`n" + $observabilityHelp = @(& $observabilityCli help 2>&1) -join "`n" + foreach ($operation in @("list_docker_images", "create_sandbox", "destroy_sandbox")) { + if ($managerHelp -notmatch "(?m)\b$operation\b") { + Fail "manager CLI does not expose $operation" + } + } + foreach ($operation in @("exec_command", "file_read", "file_write", "file_edit")) { + if ($runtimeHelp -notmatch "(?m)\b$operation\b") { + Fail "runtime CLI does not expose $operation" + } + } + if ($observabilityHelp -notmatch "(?m)\bsnapshot\b") { + Fail "observability CLI does not expose snapshot" + } + Pass "native product CLI catalogs expose every smoke operation" + + $dockerRecord = (& docker info --format "{{.ServerVersion}}|{{.OSType}}|{{.Architecture}}|{{.Driver}}|{{.CgroupVersion}}|{{.OperatingSystem}}").Trim() + if ($LASTEXITCODE -ne 0) { + Fail "Docker Desktop engine is unreachable" + } + $dockerFields = $dockerRecord.Split("|") + if ( + $dockerFields.Count -ne 6 -or + $dockerFields[0] -ne $expectedDockerVersion -or + $dockerFields[1] -ne "linux" -or + $dockerFields[2] -notin @("x86_64", "amd64") -or + $dockerFields[3] -ne "overlayfs" -or + $dockerFields[4] -ne "2" -or + $dockerFields[5] -notmatch "Docker Desktop" + ) { + Fail "Docker Desktop engine does not match the selected contract: $dockerRecord" + } + Pass "Docker Desktop $expectedDockerVersion provides Linux AMD64, overlayfs, and cgroup v2" + + if ($ImageReference -notmatch '@sha256:[0-9a-f]{64}$') { + Fail "sandbox image is not pinned by a full digest" + } + $image = @(& docker image inspect $ImageReference | ConvertFrom-Json)[0] + if ($image.Os -ne "linux" -or $image.Architecture -notin @("amd64", "x86_64")) { + Fail "pinned image is not Linux AMD64" + } + if ($image.RepoDigests -notcontains "ubuntu@$($ImageReference.Split('@')[1])") { + Fail "local image RepoDigests do not contain the pinned manifest" + } + Pass "exact pinned Ubuntu sandbox image is present locally" + + $quotedArtifactDirectory = "'" + ($ArtifactDirectory -replace "'", "''") + "'" + $quotedRegistry = "'" + ($gatewayRegistry -replace "'", "''") + "'" + $quotedDaemon = "'" + ($daemon -replace "'", "''") + "'" + $quotedDaemonConfig = "'" + ($templateConfig -replace "'", "''") + "'" + $config = Get-Content -Raw -LiteralPath $templateConfig + $managerHeader = @" +manager: + registry_path: $quotedRegistry + workspace_roots: + - $quotedArtifactDirectory +"@ + $config = [regex]::Replace($config, "(?m)^manager:\r?\n", "$managerHeader`r`n") + $config = [regex]::Replace( + $config, + "(?m)^ daemon_binary_path:.*$", + " daemon_binary_path: $quotedDaemon" + ) + $config = [regex]::Replace( + $config, + "(?m)^ daemon_config_yaml_path:.*$", + " daemon_config_yaml_path: $quotedDaemonConfig" + ) + $config = [regex]::Replace( + $config, + "(?m)^ gateway_instance_id:.*$", + " gateway_instance_id: $gatewayInstanceId" + ) + [System.IO.File]::WriteAllText( + $gatewayConfig, + $config, + [System.Text.UTF8Encoding]::new($false) + ) + + $env:EOS_SHARED_BASE_CACHE = Join-Path $ArtifactDirectory "shared-base-cache" + $gatewayArguments = @( + "serve", + "--backend", "docker", + "--config-yaml", $gatewayConfig, + "--gateway-socket", $gatewaySocket, + "--auth-token", $authToken, + "--pid-file", $gatewayPidFile + ) + $argumentLine = ($gatewayArguments | ForEach-Object { Quote-ProcessArgument $_ }) -join " " + $gatewayProcess = Start-Process ` + -FilePath $gateway ` + -ArgumentList $argumentLine ` + -WorkingDirectory $PackageRoot ` + -RedirectStandardOutput $gatewayStdout ` + -RedirectStandardError $gatewayStderr ` + -WindowStyle Hidden ` + -PassThru + + $ready = $false + for ($attempt = 0; $attempt -lt 120; $attempt++) { + if ($gatewayProcess.HasExited) { + Fail "native Windows gateway exited during startup" + } + $readinessStderr = Join-Path $ArtifactDirectory "gateway-readiness.stderr" + $savedErrorActionPreference = $ErrorActionPreference + $ErrorActionPreference = "Continue" + try { + $readinessOutput = @( + & $managerCli ` + --gateway-socket $gatewaySocket ` + --gateway-auth-token $authToken ` + list_docker_images 2> $readinessStderr + ) + $readinessExitCode = $LASTEXITCODE + } finally { + $ErrorActionPreference = $savedErrorActionPreference + } + if ($readinessExitCode -eq 0) { + ($readinessOutput -join [Environment]::NewLine) | + Set-Content -LiteralPath (Join-Path $ArtifactDirectory "gateway-readiness.json") -Encoding utf8 + $ready = $true + break + } + Start-Sleep -Milliseconds 250 + } + if (-not $ready) { + Fail "native Windows gateway did not become CLI-ready within 30 seconds" + } + if ((Get-Item -LiteralPath (Join-Path $ArtifactDirectory "gateway-readiness.stderr")).Length -ne 0) { + Fail "final gateway readiness check emitted stderr" + } + Pass "native Windows gateway became ready for native product CLIs" + + for ($batch = 1; $batch -le 2; $batch++) { + $batchLabel = "{0:D2}" -f $batch + $workspace = Join-Path $ArtifactDirectory "cli-workspace-batch-$batchLabel" + New-Item -ItemType Directory -Path (Join-Path $workspace "src") -Force | Out-Null + Set-Content -LiteralPath (Join-Path $workspace "README.txt") -Value "CLI_ENV_WINDOWS_BATCH_$batchLabel fixture" -Encoding utf8 + Set-Content -LiteralPath (Join-Path $workspace "src\main.txt") -Value "initial fixture" -Encoding utf8 + + $null = Invoke-ManagerCli "batch-$batchLabel-00-list-images" @("list_docker_images") + $created = Invoke-ManagerCli "batch-$batchLabel-01-create-sandbox" @( + "create_sandbox", + "--image", $ImageReference, + "--workspace-bind-root", $workspace + ) + if (-not $created.id) { + Fail "batch $batchLabel create_sandbox response has no sandbox ID" + } + $activeSandboxId = [string]$created.id + $returnedWorkspace = [string]$created.workspace_root + if ($returnedWorkspace.StartsWith("\\?\")) { + $returnedWorkspace = $returnedWorkspace.Substring(4) + } + if ( + $created.state -ne "ready" -or + -not [string]::Equals( + [System.IO.Path]::GetFullPath($returnedWorkspace), + [System.IO.Path]::GetFullPath($workspace), + [System.StringComparison]::OrdinalIgnoreCase + ) + ) { + Fail "batch $batchLabel create_sandbox response failed strict validation" + } + + $executed = Invoke-RuntimeCli "batch-$batchLabel-02-exec-command" $activeSandboxId @( + "--request-id", "cli-env-windows-$batchLabel-exec", + "exec_command", + "--timeout-ms", "30000", + 'printf "CLI_ENV_EXEC_OK\n"; test -f README.txt; cat README.txt' + ) + if ( + $executed.status -ne "ok" -or + [int]$executed.exit_code -ne 0 -or + ($executed | ConvertTo-Json -Depth 100) -notmatch "CLI_ENV_EXEC_OK" -or + ($executed | ConvertTo-Json -Depth 100) -notmatch "CLI_ENV_WINDOWS_BATCH_$batchLabel" + ) { + Fail "batch $batchLabel exec_command response failed strict validation" + } + + $null = Invoke-RuntimeCli "batch-$batchLabel-03-file-write" $activeSandboxId @( + "--request-id", "cli-env-windows-$batchLabel-write", + "file_write", + "--path", "cli-smoke.txt", + "--content", "CLI_ENV_FILE_ALPHA" + ) + $readAlpha = Invoke-RuntimeCli "batch-$batchLabel-04-file-read-alpha" $activeSandboxId @( + "--request-id", "cli-env-windows-$batchLabel-read-alpha", + "file_read", + "--path", "cli-smoke.txt", + "--limit", "10" + ) + if (($readAlpha | ConvertTo-Json -Depth 100) -notmatch "CLI_ENV_FILE_ALPHA") { + Fail "batch $batchLabel first file_read failed correctness" + } + $null = Invoke-RuntimeCli "batch-$batchLabel-05-file-edit" $activeSandboxId @( + "--request-id", "cli-env-windows-$batchLabel-edit", + "file_edit", + "--path", "cli-smoke.txt", + "--edits", '[{\"old_string\":\"ALPHA\",\"new_string\":\"OMEGA\"}]' + ) + $readOmega = Invoke-RuntimeCli "batch-$batchLabel-06-file-read-omega" $activeSandboxId @( + "--request-id", "cli-env-windows-$batchLabel-read-omega", + "file_read", + "--path", "cli-smoke.txt", + "--limit", "10" + ) + $readOmegaText = $readOmega | ConvertTo-Json -Depth 100 + if ($readOmegaText -notmatch "CLI_ENV_FILE_OMEGA" -or $readOmegaText -match "CLI_ENV_FILE_ALPHA") { + Fail "batch $batchLabel second file_read failed correctness" + } + + $snapshot = Invoke-ObservabilityCli "batch-$batchLabel-07-observability-snapshot" $activeSandboxId + if (($snapshot | ConvertTo-Json -Depth 100) -notmatch [regex]::Escape($activeSandboxId)) { + Fail "batch $batchLabel snapshot does not identify the sandbox" + } + $destroyed = Invoke-ManagerCli "batch-$batchLabel-08-destroy-sandbox" @( + "destroy_sandbox", + "--sandbox-id", $activeSandboxId + ) + if ($destroyed.id -ne $activeSandboxId) { + Fail "batch $batchLabel destroy_sandbox response failed strict validation" + } + $destroyedSandboxId = $activeSandboxId + $activeSandboxId = $null + $listed = Invoke-ManagerCli "batch-$batchLabel-09-list-sandboxes-after" @("list_sandboxes") + if (($listed | ConvertTo-Json -Depth 100) -match [regex]::Escape($destroyedSandboxId)) { + Fail "batch $batchLabel sandbox remains in manager listing after destroy" + } + $batchSandboxIds.Add($destroyedSandboxId) + Pass "product-CLI batch $batchLabel completed and destroyed sandbox $destroyedSandboxId" + } +} catch { + $primaryError = $_ +} finally { + Cleanup-Qualification +} + +if ($primaryError) { + Fail "qualification failed: $($primaryError.Exception.Message)" +} +if ($cleanupErrors.Count -ne 0) { + Fail "qualification cleanup failed: $($cleanupErrors -join '; ')" +} + +$afterState = @(Get-OwnedState) +$afterState | Set-Content -LiteralPath $afterStatePath -Encoding utf8 +$beforeStateText = @(Get-Content -LiteralPath $beforeStatePath -ErrorAction SilentlyContinue) +$afterStateText = @(Get-Content -LiteralPath $afterStatePath -ErrorAction SilentlyContinue) +$stateDifference = Compare-Object -ReferenceObject $beforeStateText -DifferenceObject $afterStateText +$stateDifference | + Out-String | + Set-Content -LiteralPath (Join-Path $ArtifactDirectory "owned-state-diff.txt") -Encoding utf8 +if ($stateDifference) { + Fail "global EOS-owned Docker/process baseline changed" +} + +$nonemptyCliStderr = @( + Get-ChildItem -File -LiteralPath $ArtifactDirectory -Filter "batch-*.stderr" | + Where-Object { $_.Length -ne 0 } +) +if ($nonemptyCliStderr.Count -ne 0) { + Fail "one or more native product CLI stderr files are nonempty" +} + +$gatewayLogText = @( + Get-Content -Raw -LiteralPath $gatewayStdout -ErrorAction SilentlyContinue + Get-Content -Raw -LiteralPath $gatewayStderr -ErrorAction SilentlyContinue +) -join "`n" +if ($gatewayLogText -match '(?im)"level"\s*:\s*"(warn|error)"|(^|[^a-z])(WARN|ERROR|PANIC)([^a-z]|$)') { + Fail "native gateway log contains a warning, error, or panic" +} + +$tokenLeakFiles = [System.Collections.Generic.List[string]]::new() +Get-ChildItem -File -LiteralPath $ArtifactDirectory | + Where-Object { $_.Extension -in @(".txt", ".json", ".log", ".stderr", ".stdout", ".yml") } | + ForEach-Object { + $content = Get-Content -Raw -LiteralPath $_.FullName -ErrorAction SilentlyContinue + if ($content -and $content.Contains($authToken)) { + $redacted = $content.Replace($authToken, "[REDACTED]") + Set-Content -LiteralPath $_.FullName -Value $redacted -Encoding utf8 + $tokenLeakFiles.Add($_.FullName) + } + } +if ($tokenLeakFiles.Count -ne 0) { + Fail "gateway authentication token appeared in archived output and was redacted" +} + +$elapsedSeconds = [int][Math]::Ceiling(([DateTime]::UtcNow - $startedAt).TotalSeconds) +if ($elapsedSeconds -gt $MaximumSeconds) { + Fail "qualification exceeded the $MaximumSeconds-second budget" +} + +$operationSequence = @( + "list_docker_images", + "create_sandbox", + "exec_command", + "file_write", + "file_read", + "file_edit", + "file_read", + "snapshot", + "destroy_sandbox", + "list_sandboxes" +) +$summary = [ordered]@{ + schema_version = 1 + qualification_target = "windows_docker_desktop" + client_cohort = "product_cli" + state = "completed" + correctness = "pass" + completed_batches = 2 + total_batches = 2 + operation_count = 20 + operation_sequence = $operationSequence + warning_count = 0 + failure_count = 0 + cleanup = "pass" + elapsed_seconds = $elapsedSeconds + computer_name = $ExpectedComputerName + windows_build = $buildNumber + docker_version = $expectedDockerVersion + docker_engine = "linux/amd64" + docker_storage_driver = "overlayfs" + docker_cgroup_version = "2" + image_reference = $ImageReference + product_commit = $expectedProductCommit + release_archive_sha256 = $expectedReleaseArchiveSha256 + gateway_instance_id = $gatewayInstanceId + sandbox_ids = @($batchSandboxIds) + batches = @( + for ($index = 0; $index -lt $batchSandboxIds.Count; $index++) { + [ordered]@{ + batch_index = $index + 1 + sandbox_id = $batchSandboxIds[$index] + state = "completed" + correctness = "pass" + operation_count = 10 + cleanup = "pass" + } + } + ) + artifact_sha256 = $expectedHashes +} +$summary | + ConvertTo-Json -Depth 8 | + Set-Content -LiteralPath $summaryPath -Encoding utf8 + +Pass "Windows Docker Desktop environment qualification completed" +Info "summary=$summaryPath" +Info "artifact_directory=$ArtifactDirectory" +Info "elapsed_seconds=$elapsedSeconds" diff --git a/ephemeral-sandbox-v1/experiments/scripts/stage_final_host.sh b/ephemeral-sandbox-v1/experiments/scripts/stage_final_host.sh new file mode 100644 index 0000000..4304052 --- /dev/null +++ b/ephemeral-sandbox-v1/experiments/scripts/stage_final_host.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +set -euo pipefail + +paper_root="/srv/eos-benchmark/paper" +product_root="/srv/eos-benchmark/product" +product_bin_dir="${product_root}/target/release" +expected_hostname="eos-benchmark-ubuntu24" +expected_product_commit="b22862550e0a7cb4fe61ce581831e9244cc492b5" +expected_docker_version="29.0.1" +image_reference="ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" +expected_catalog_sha256="c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba" +expected_gateway_sha256="f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf" +expected_daemon_sha256="a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22" +expected_manager_cli_sha256="0be4f0c26f8f50b76b175d04cfeec61529a605bcda9ffcd6782a09096ba2983f" +expected_runtime_cli_sha256="e9ac5f6c7a5f9c07a3de166b320e7d6065fa9480a7f18d6d59114337d15e28e7" +expected_observability_cli_sha256="6b2dae2369344cbb3960a76f6ccdfa869a7aa9b7a7a255f8a634f4a52d5cfdb5" + +fail() { + printf 'FAIL\t%s\n' "$1" >&2 + exit 1 +} + +[[ "$(id -u)" -ne 0 ]] || fail "run staging as the benchmark user, not root" +[[ "$(hostname -s)" == "$expected_hostname" ]] || + fail "logical hostname is not ${expected_hostname}" +[[ "$(pwd -P)" == "$paper_root" ]] || fail "run from ${paper_root}" +[[ -d "$product_root/.git" ]] || fail "product checkout is missing" +[[ "$(git -C "$product_root" branch --show-current)" == "main" ]] || + fail "product checkout is not on main" +[[ -z "$(git -C "$product_root" status --porcelain=v1)" ]] || + fail "product checkout is dirty" +[[ "$(git -C "$product_root" rev-parse HEAD)" == "$expected_product_commit" ]] || + fail "product checkout is not at the selected commit" +[[ "$(docker version --format '{{.Server.Version}}' 2>/dev/null)" == "$expected_docker_version" ]] || + fail "Docker server is not ${expected_docker_version}" +command -v python3.13 >/dev/null 2>&1 || fail "python3.13 is unavailable" +[[ "$(python3.13 --version 2>&1)" == Python\ 3.13.* ]] || + fail "python3.13 does not report CPython 3.13" + +catalog_binary="${product_bin_dir}/sandbox-catalog-export" +gateway_binary="${product_bin_dir}/sandbox-gateway" +manager_cli_binary="${product_bin_dir}/sandbox-manager-cli" +runtime_cli_binary="${product_bin_dir}/sandbox-runtime-cli" +observability_cli_binary="${product_bin_dir}/sandbox-observability-cli" +daemon_binary="${product_root}/dist/sandbox-daemon-linux-amd64" +for artifact in \ + "$catalog_binary" \ + "$gateway_binary" \ + "$manager_cli_binary" \ + "$runtime_cli_binary" \ + "$observability_cli_binary" \ + "$daemon_binary" +do + [[ -f "$artifact" && ! -L "$artifact" && -s "$artifact" ]] || + fail "staged artifact is missing or unsafe: ${artifact}" + chmod 0755 -- "$artifact" +done +[[ "$(sha256sum "$catalog_binary" | awk '{print $1}')" == "$expected_catalog_sha256" ]] || + fail "catalog exporter hash mismatch" +[[ "$(sha256sum "$gateway_binary" | awk '{print $1}')" == "$expected_gateway_sha256" ]] || + fail "gateway hash mismatch" +[[ "$(sha256sum "$manager_cli_binary" | awk '{print $1}')" == "$expected_manager_cli_sha256" ]] || + fail "manager CLI hash mismatch" +[[ "$(sha256sum "$runtime_cli_binary" | awk '{print $1}')" == "$expected_runtime_cli_sha256" ]] || + fail "runtime CLI hash mismatch" +[[ "$(sha256sum "$observability_cli_binary" | awk '{print $1}')" == "$expected_observability_cli_sha256" ]] || + fail "observability CLI hash mismatch" +[[ "$(sha256sum "$daemon_binary" | awk '{print $1}')" == "$expected_daemon_sha256" ]] || + fail "daemon hash mismatch" + +docker pull "$image_reference" +[[ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image_reference")" == "linux/amd64" ]] || + fail "staged image platform is not linux/amd64" + +if [[ ! -d .venv ]]; then + python3.13 -m venv .venv +fi +[[ "$(.venv/bin/python --version 2>&1)" == Python\ 3.13.* ]] || + fail "existing paper virtual environment is not CPython 3.13" +.venv/bin/python -m pip install --require-hashes \ + -r experiments/final-host-requirements.lock +.venv/bin/python -m pip install --no-deps --no-build-isolation -e ./benchmark + +printf 'PASS\toff-clock staging completed\n' +printf 'INFO\tproduct_commit=%s\n' "$expected_product_commit" +printf 'INFO\timage_reference=%s\n' "$image_reference" +printf 'INFO\tclient_cohort=product_cli\n' +printf 'INFO\tpython=%s\n' "$(.venv/bin/python --version 2>&1)" +printf 'INFO\tpip=%s\n' "$(.venv/bin/python -m pip --version)" +printf 'INFO\tnext=bash experiments/scripts/qualify_final_host.sh\n' diff --git a/ephemeral-sandbox-v1/experiments/scripts/verify_environment.sh b/ephemeral-sandbox-v1/experiments/scripts/verify_environment.sh index 6cb47d3..da424da 100644 --- a/ephemeral-sandbox-v1/experiments/scripts/verify_environment.sh +++ b/ephemeral-sandbox-v1/experiments/scripts/verify_environment.sh @@ -8,10 +8,24 @@ paper_root="$(cd -- "${script_directory}/../.." && pwd -P)" : "${PRODUCT_ROOT:?Set PRODUCT_ROOT to the absolute ephemeral-sandbox checkout}" : "${PRODUCT_BIN_DIR:?Set PRODUCT_BIN_DIR to its prebuilt executable directory}" -image_reference="${IMAGE_REFERENCE:-ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf}" -preset="${BENCHMARK_PRESET:-paper-env-smoke}" +expected_hostname="eos-benchmark-ubuntu24" +expected_paper_root="/srv/eos-benchmark/paper" +expected_product_root="/srv/eos-benchmark/product" +expected_product_bin_dir="${expected_product_root}/target/release" +expected_product_commit="b22862550e0a7cb4fe61ce581831e9244cc492b5" +expected_docker_version="29.0.1" +expected_image_reference="ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf" +expected_catalog_sha256="c841597bab53612a2f424088264a0fce383b54ded480050d99fbed1c529ac8ba" +expected_gateway_sha256="f1f8420bfa6ea6370d90fbf8428c432fe6f1031b0cb7cc7d32ac543dc8be2faf" +expected_daemon_sha256="a55d4775b992c02d603ca294746fb314e99d59774732ab7b8e7bf24ef010fb22" +expected_manager_cli_sha256="0be4f0c26f8f50b76b175d04cfeec61529a605bcda9ffcd6782a09096ba2983f" +expected_runtime_cli_sha256="e9ac5f6c7a5f9c07a3de166b320e7d6065fa9480a7f18d6d59114337d15e28e7" +expected_observability_cli_sha256="6b2dae2369344cbb3960a76f6ccdfa869a7aa9b7a7a255f8a634f4a52d5cfdb5" +image_reference="${IMAGE_REFERENCE:-$expected_image_reference}" maximum_seconds="${MAX_PREFLIGHT_SECONDS:-60}" minimum_memory_bytes=$((15 * 1024 * 1024 * 1024)) +minimum_storage_bytes=$((100 * 1024 * 1024 * 1024)) +minimum_available_storage_bytes=$((20 * 1024 * 1024 * 1024)) pass() { printf 'PASS\t%s\n' "$1" @@ -38,7 +52,7 @@ require_executable() { [[ -x "$path" ]] || fail "file is not executable: $path" } -for command_name in awk date docker findmnt git head mktemp nproc od sha256sum tr uname; do +for command_name in awk date df docker file findmnt git grep head hostname lsblk mktemp nproc od realpath sed sha256sum systemd-detect-virt tr uname; do require_command "$command_name" done @@ -48,6 +62,19 @@ pass "host operating system is Linux" [[ "$(uname -m)" == "x86_64" ]] || fail "host architecture is not x86_64" pass "host architecture is x86_64" +[[ "$(hostname -s)" == "$expected_hostname" ]] || + fail "logical hostname is not ${expected_hostname}" +pass "logical hostname is ${expected_hostname}" + +if grep -Eiq '(microsoft|wsl)' /proc/sys/kernel/osrelease /proc/version; then + fail "WSL is not an eligible final host" +fi +if systemd-detect-virt --container >/dev/null 2>&1; then + fail "a container is not an eligible final host" +fi +virtualization="$(systemd-detect-virt 2>/dev/null || true)" +pass "host is neither WSL nor a container; virtualization=${virtualization:-none}" + [[ -r /etc/os-release ]] || fail "/etc/os-release is unavailable" # shellcheck disable=SC1091 . /etc/os-release @@ -55,6 +82,16 @@ pass "host architecture is x86_64" fail "host is not Ubuntu 24.04" pass "host distribution is Ubuntu 24.04" +resolved_product_root="$(realpath -e -- "$PRODUCT_ROOT")" +resolved_product_bin_dir="$(realpath -e -- "$PRODUCT_BIN_DIR")" +[[ "$paper_root" == "$expected_paper_root" ]] || + fail "paper root is not ${expected_paper_root}" +[[ "$resolved_product_root" == "$expected_product_root" ]] || + fail "product root is not ${expected_product_root}" +[[ "$resolved_product_bin_dir" == "$expected_product_bin_dir" ]] || + fail "product binary directory is not ${expected_product_bin_dir}" +pass "paper, product, and binary paths match the selected environment" + cpu_count="$(nproc)" [[ "$cpu_count" -ge 8 ]] || fail "host exposes fewer than 8 logical CPUs" pass "logical CPU count is ${cpu_count}" @@ -65,57 +102,162 @@ memory_bytes="$(awk '/^MemTotal:/ {print $2 * 1024}' /proc/meminfo | awk '{print pass "usable memory bytes are ${memory_bytes}" paper_filesystem="$(findmnt -n -o FSTYPE --target "$paper_root")" -product_filesystem="$(findmnt -n -o FSTYPE --target "$PRODUCT_ROOT")" +product_filesystem="$(findmnt -n -o FSTYPE --target "$resolved_product_root")" [[ "$paper_filesystem" == "ext4" ]] || fail "paper root is not on ext4" [[ "$product_filesystem" == "ext4" ]] || fail "product root is not on ext4" pass "paper and product roots are on ext4" -[[ -r /sys/fs/cgroup/cgroup.controllers ]] || fail "cgroup v2 controllers are unavailable" -pass "cgroup v2 is available" +for storage_path in "$paper_root" "$resolved_product_root"; do + storage_size="$( + df -B1 --output=size -- "$storage_path" | + awk 'NR == 2 {gsub(/[[:space:]]/, "", $1); print $1}' + )" + [[ "$storage_size" =~ ^[0-9]+$ && "$storage_size" -ge "$minimum_storage_bytes" ]] || + fail "filesystem containing ${storage_path} is smaller than 100 GiB" + storage_source="$(findmnt -n -o SOURCE --target "$storage_path")" + backing_record="$(lsblk -s -n -o NAME,TYPE,ROTA,TRAN -- "$storage_source" 2>/dev/null)" || + fail "cannot inspect storage backing for ${storage_path}" + if ! awk ' + $2 == "disk" && $3 == "0" && ($4 == "nvme" || $1 ~ /^nvme/) { found = 1 } + END { exit found ? 0 : 1 } + ' <<<"$backing_record"; then + fail "storage containing ${storage_path} is not provably NVMe-backed" + fi + backing_evidence="$( + awk ' + BEGIN { separator = "" } + { + printf "%s%s:%s:rota=%s:transport=%s", + separator, $1, $2, $3, ($4 == "" ? "unknown" : $4) + separator = "," + } + ' <<<"$backing_record" + )" + pass "storage path=${storage_path} size_bytes=${storage_size} source=${storage_source} backing=${backing_evidence}" +done +pass "paper and product filesystems are at least 100 GiB and NVMe-backed" + +[[ -r /sys/fs/cgroup/cgroup.controllers ]] || + fail "cgroup v2 controllers are unavailable" +cgroup_controllers="$(< /sys/fs/cgroup/cgroup.controllers)" +for controller in cpu memory pids; do + grep -qw "$controller" <<<"$cgroup_controllers" || + fail "cgroup v2 ${controller} controller is unavailable" +done +pass "cgroup v2 cpu, memory, and pids controllers are available" -docker_version="$(docker version --format '{{.Server.Version}}' 2>/dev/null)" || +docker_record="$( + docker info --format \ + '{{.ServerVersion}}|{{.OSType}}|{{.Architecture}}|{{.Driver}}|{{.CgroupVersion}}' \ + 2>/dev/null +)" || fail "Docker server is unreachable" -[[ -n "$docker_version" ]] || fail "Docker server version is empty" -pass "Docker server version is ${docker_version}" +IFS='|' read -r docker_version docker_os docker_architecture docker_driver docker_cgroup_version \ + <<<"$docker_record" +[[ "$docker_version" == "$expected_docker_version" ]] || + fail "Docker server version is not ${expected_docker_version}" +[[ "$docker_os" == "linux" ]] || fail "Docker server operating system is not Linux" +[[ "$docker_architecture" == "x86_64" || "$docker_architecture" == "amd64" ]] || + fail "Docker server architecture is not AMD64" +[[ "$docker_driver" == "overlayfs" ]] || fail "Docker storage driver is not overlayfs" +[[ "$docker_cgroup_version" == "2" ]] || fail "Docker does not use cgroup v2" +pass "Docker ${docker_version} is Linux AMD64 with overlayfs and cgroup v2" [[ "$image_reference" =~ @sha256:[0-9a-f]{64}$ ]] || fail "IMAGE_REFERENCE is not pinned by a full sha256 digest" +[[ "$image_reference" == "$expected_image_reference" ]] || + fail "IMAGE_REFERENCE differs from the protocol-pinned image" image_platform="$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image_reference" 2>/dev/null)" || fail "pinned image is not present locally; stage it before measurement" [[ "$image_platform" == "linux/amd64" ]] || fail "pinned image platform is not linux/amd64" image_id="$(docker image inspect --format '{{.Id}}' "$image_reference")" -pass "pinned image is present as ${image_id}" - -[[ -d "$PRODUCT_ROOT/.git" ]] || fail "PRODUCT_ROOT is not a Git checkout" -product_branch="$(git -C "$PRODUCT_ROOT" branch --show-current)" +image_digest="${image_reference##*@}" +image_repo_digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$image_reference")" +grep -Eq "(^|/)ubuntu@${image_digest}$" <<<"$image_repo_digests" || + fail "local image RepoDigests do not contain the protocol-pinned manifest" +pass "pinned image is present as ${image_id} with manifest ${image_digest}" + +[[ -d "$resolved_product_root/.git" ]] || fail "PRODUCT_ROOT is not a Git checkout" +product_branch="$(git -C "$resolved_product_root" branch --show-current)" [[ "$product_branch" == "main" ]] || fail "product checkout is not on main" -product_status="$(git -C "$PRODUCT_ROOT" status --porcelain=v1)" +product_status="$(git -C "$resolved_product_root" status --porcelain=v1)" [[ -z "$product_status" ]] || fail "product checkout is dirty" -product_commit="$(git -C "$PRODUCT_ROOT" rev-parse HEAD)" -pass "product checkout is clean main at ${product_commit}" - -catalog_binary="${PRODUCT_BIN_DIR}/sandbox-catalog-export" -gateway_binary="${PRODUCT_BIN_DIR}/sandbox-gateway" -daemon_binary="${PRODUCT_ROOT}/dist/sandbox-daemon-linux-amd64" -git_amd64="${PRODUCT_ROOT}/dist/git/linux-amd64.tar" -git_arm64="${PRODUCT_ROOT}/dist/git/linux-arm64.tar" +product_commit="$(git -C "$resolved_product_root" rev-parse HEAD)" +[[ "$product_commit" == "$expected_product_commit" ]] || + fail "product commit is not ${expected_product_commit}" +pass "product checkout is clean main at the selected commit ${product_commit}" + +catalog_binary="${resolved_product_bin_dir}/sandbox-catalog-export" +gateway_binary="${resolved_product_bin_dir}/sandbox-gateway" +manager_cli_binary="${resolved_product_bin_dir}/sandbox-manager-cli" +runtime_cli_binary="${resolved_product_bin_dir}/sandbox-runtime-cli" +observability_cli_binary="${resolved_product_bin_dir}/sandbox-observability-cli" +daemon_binary="${resolved_product_root}/dist/sandbox-daemon-linux-amd64" require_executable "$catalog_binary" require_executable "$gateway_binary" +require_executable "$manager_cli_binary" +require_executable "$runtime_cli_binary" +require_executable "$observability_cli_binary" require_executable "$daemon_binary" -require_regular_file "$git_amd64" -require_regular_file "$git_arm64" -daemon_magic="$(head -c 4 "$daemon_binary" | od -An -t x1 | tr -d ' \n')" -[[ "$daemon_magic" == "7f454c46" ]] || fail "Linux daemon is not an ELF executable" -pass "all prebuilt product artifacts are present and safe" +for binary_path in \ + "$catalog_binary" \ + "$gateway_binary" \ + "$manager_cli_binary" \ + "$runtime_cli_binary" \ + "$observability_cli_binary" \ + "$daemon_binary" +do + binary_record="$(file -b -- "$binary_path")" + grep -q 'ELF 64-bit' <<<"$binary_record" || + fail "prebuilt artifact is not ELF64: ${binary_path}" + grep -q 'x86-64' <<<"$binary_record" || + fail "prebuilt artifact is not x86-64: ${binary_path}" +done +catalog_sha256="$(sha256sum "$catalog_binary" | awk '{print $1}')" +gateway_sha256="$(sha256sum "$gateway_binary" | awk '{print $1}')" +manager_cli_sha256="$(sha256sum "$manager_cli_binary" | awk '{print $1}')" +runtime_cli_sha256="$(sha256sum "$runtime_cli_binary" | awk '{print $1}')" +observability_cli_sha256="$(sha256sum "$observability_cli_binary" | awk '{print $1}')" +daemon_sha256="$(sha256sum "$daemon_binary" | awk '{print $1}')" +[[ "$catalog_sha256" == "$expected_catalog_sha256" ]] || + fail "catalog exporter hash differs from the staged bundle" +[[ "$gateway_sha256" == "$expected_gateway_sha256" ]] || + fail "gateway hash differs from the staged bundle" +[[ "$manager_cli_sha256" == "$expected_manager_cli_sha256" ]] || + fail "manager CLI hash differs from the staged v0.1.4 bundle" +[[ "$runtime_cli_sha256" == "$expected_runtime_cli_sha256" ]] || + fail "runtime CLI hash differs from the staged v0.1.4 bundle" +[[ "$observability_cli_sha256" == "$expected_observability_cli_sha256" ]] || + fail "observability CLI hash differs from the staged v0.1.4 bundle" +[[ "$daemon_sha256" == "$expected_daemon_sha256" ]] || + fail "daemon hash differs from the staged bundle" +pass "prebuilt Linux AMD64 artifact hashes match the selected bundle" + +manager_help="$("$manager_cli_binary" help 2>&1)" || + fail "manager CLI help failed" +runtime_help="$("$runtime_cli_binary" --sandbox-id preflight-only help 2>&1)" || + fail "runtime CLI help failed" +observability_help="$("$observability_cli_binary" help 2>&1)" || + fail "observability CLI help failed" +for operation in list_docker_images create_sandbox destroy_sandbox; do + grep -qw "$operation" <<<"$manager_help" || + fail "manager CLI does not expose ${operation}" +done +for operation in exec_command file_read file_write file_edit; do + grep -qw "$operation" <<<"$runtime_help" || + fail "runtime CLI does not expose ${operation}" +done +grep -qw snapshot <<<"$observability_help" || + fail "observability CLI does not expose snapshot" +pass "product CLI catalogs expose the required environment-smoke operations" benchmark_python="${paper_root}/.venv/bin/python" -benchmark_cli="${paper_root}/.venv/bin/sandbox-benchmark" -require_executable "$benchmark_python" -require_executable "$benchmark_cli" +[[ -x "$benchmark_python" ]] || + fail "paper virtual-environment Python is missing or not executable" python_version="$("$benchmark_python" --version 2>&1)" case "$python_version" in "Python 3.13."*) @@ -153,39 +295,43 @@ PY )" pass "paper workspace profile is valid as ${profile_record}" -plan_json="$(mktemp)" -trap 'rm -f -- "$plan_json"' EXIT - -"$benchmark_cli" validate \ - --test-repository-root "$paper_root" \ - --product-root "$PRODUCT_ROOT" \ - --product-bin-dir "$PRODUCT_BIN_DIR" \ - --plan "$preset" >"$plan_json" - -plan_record="$( - "$benchmark_python" - "$plan_json" <<'PY' -import json -import sys - -value = json.loads(open(sys.argv[1], encoding="utf-8").read()) -assert value["runnable"] is True -print( - f'plan_hash={value["plan_hash"]} ' - f'cells={value["estimates"]["cell_count"]} ' - f'trials={value["estimates"]["trial_batch_count"]}' -) -PY +available_free_space_bytes="$( + df -B1 --output=avail -- "$paper_root" | + awk 'NR == 2 {gsub(/[[:space:]]/, "", $1); print $1}' )" -pass "benchmark preset validates: ${plan_record}" - +[[ "$available_free_space_bytes" =~ ^[0-9]+$ && + "$available_free_space_bytes" -ge "$minimum_available_storage_bytes" ]] || + fail "paper filesystem has less than 20 GiB available" +pass "paper filesystem has at least 20 GiB available" + +cpu_model="$(awk -F: '/^model name/ {sub(/^[[:space:]]+/, "", $2); print $2; exit}' /proc/cpuinfo)" +paper_source_commit="$( + git -C "$paper_root" rev-parse HEAD 2>/dev/null || + printf 'snapshot-without-git-metadata' +)" +printf 'INFO\thostname=%s\n' "$expected_hostname" printf 'INFO\tkernel=%s\n' "$(uname -r)" +printf 'INFO\tvirtualization=%s\n' "${virtualization:-none}" +printf 'INFO\tcpu_model=%s\n' "${cpu_model:-unknown}" +printf 'INFO\tlogical_cpus=%s\n' "$cpu_count" +printf 'INFO\tmemory_bytes=%s\n' "$memory_bytes" +printf 'INFO\tpaper_filesystem=%s\n' "$paper_filesystem" +printf 'INFO\tproduct_filesystem=%s\n' "$product_filesystem" +printf 'INFO\tpaper_source_commit=%s\n' "$paper_source_commit" printf 'INFO\tproduct_commit=%s\n' "$product_commit" +printf 'INFO\tdocker_version=%s\n' "$docker_version" +printf 'INFO\tdocker_storage_driver=%s\n' "$docker_driver" +printf 'INFO\tdocker_cgroup_version=%s\n' "$docker_cgroup_version" printf 'INFO\timage_reference=%s\n' "$image_reference" printf 'INFO\timage_id=%s\n' "$image_id" -printf 'INFO\tcatalog_sha256=%s\n' "$(sha256sum "$catalog_binary" | awk '{print $1}')" -printf 'INFO\tgateway_sha256=%s\n' "$(sha256sum "$gateway_binary" | awk '{print $1}')" -printf 'INFO\tdaemon_sha256=%s\n' "$(sha256sum "$daemon_binary" | awk '{print $1}')" +printf 'INFO\tcatalog_sha256=%s\n' "$catalog_sha256" +printf 'INFO\tgateway_sha256=%s\n' "$gateway_sha256" +printf 'INFO\tmanager_cli_sha256=%s\n' "$manager_cli_sha256" +printf 'INFO\truntime_cli_sha256=%s\n' "$runtime_cli_sha256" +printf 'INFO\tobservability_cli_sha256=%s\n' "$observability_cli_sha256" +printf 'INFO\tdaemon_sha256=%s\n' "$daemon_sha256" printf 'INFO\tworkspace_fixture_hash=%s\n' "$profile_record" +printf 'INFO\tavailable_free_space_bytes=%s\n' "$available_free_space_bytes" elapsed_seconds="$(( $(date +%s) - started_seconds ))" [[ "$elapsed_seconds" -le "$maximum_seconds" ]] || diff --git a/ephemeral-sandbox-v1/figures/concept-figure-review.md b/ephemeral-sandbox-v1/figures/concept-figure-review.md new file mode 100644 index 0000000..6badacd --- /dev/null +++ b/ephemeral-sandbox-v1/figures/concept-figure-review.md @@ -0,0 +1,168 @@ +# Concept-figure provenance and drafting-stage review + +Review date: 2026-07-30. + +## Provenance boundary + +The four PNGs were supplied by the author from +`C:\Users\yifan\Downloads` and moved into `figures/concept/` with filenames +matching the approved prompt/spec sources. The external generator, model, +seed, source session, and label-placement maps were not supplied. The prompt +files under `figures/source/` are the available regeneration specifications; +they do not prove that the supplied PNGs were generated exactly from those +prompts. + +The author directed that the four files be used unchanged in the drafting-stage +manuscript. Visual redesign, resolution normalization, topology repair, and +style-family harmonization are deferred to PW7. These assets are explanatory +concept diagrams and are not experimental evidence. + +## File inventory + +| Asset | Dimensions and metadata | Bytes | SHA-256 | +|---|---:|---:|---| +| `concept/fig_system_architecture.png` | 1536 x 1024, 96 x 96 dpi | 1,356,038 | `bef02bc0fc10e475dce7c75647e1efcf964e9f40ca46985055ae7442a2f7467a` | +| `concept/fig_publication_sequence.png` | 1672 x 941, 96 x 96 dpi | 1,367,977 | `08bb7881f356946553d1f7e34704f4b7d6791e8f939897db21f1226111efec82` | +| `concept/fig_lifecycle_state_machine.png` | 3200 x 1800, 300 x 300 dpi | 212,531 | `dd0c2f719c788072d45e52519e2b15850457caa97eaa0e5743ec9fce4f37464f` | +| `concept/fig_reconciliation_decision.png` | 1024 x 1536, 96 x 96 dpi | 1,389,136 | `b0445f2d0c93439270d649d3c51e2d002722b8b445cdd5e84c561b52d46feb2d` | + +## Drafting-stage QA + +### System architecture + +- Strengths: names the shared LayerStack, two private sessions, capture and + reconciliation, the active-manifest visibility boundary, post-commit work, + and sessionless file paths. +- Deferred disparities: the two private-session publication paths do not + converge as cleanly as the prompt requires; one orange route visually + approaches the resolved-changeset area; the green return path does not + clearly originate at the promoted layer; the shared-state band overlaps the + publication column; resolution and aspect ratio differ from the prompt. +- Current gate: accepted unchanged for drafting; topology and resolution review + required at PW7. + +### Publication sequence + +- Strengths: separates private execution, precommit planning, accepted + publication, and post-commit services; the active-manifest boundary and + cleanup non-rollback note are prominent. +- Deferred disparities: the supplied composition uses horizontal phase groups + and per-step owner tags rather than the requested five vertical swimlanes; + some requested caller/session labels are absent or reorganized; icon density + and raster resolution differ from the shared specification. +- Current gate: accepted unchanged for drafting; layout, label inventory, and + resolution review required at PW7. + +### Lifecycle state machine + +- Strengths: distinguishes precommit and post-commit regions, explicit retry, + no-op, discard, published, destroying, closed, and finalize-failed outcomes. + The file satisfies the requested 3200 x 1800 and 300 dpi delivery target. +- Lifecycle qualifier: the diagram is a normal-path abstraction. It omits + holder-exit recovery artifacts, shutdown convergence, and lease-aware + remount outcomes. The shared `Destroying` node does not encode whether its + predecessor was publication, no-op, or discard; therefore + `Published but not closed` must be read only for the committed-publication + path. +- PW3 revalidation: pass as a qualified normal-path abstraction. Section 6 + separately describes the omitted holder-exit recovery artifact, shutdown + convergence, and lease-aware remount outcomes, and the caption prevents the + shared `Destroying` node from extending `Published but not closed` to the + no-op or discard paths. Submission-final topology review remains PW7. + +### Reconciliation decision flow + +- Strengths: exposes base validation, protected-drop handling, source and + ignored routes, fingerprint checks, narrow text-merge eligibility, + whole-candidate rejection, no-op, and accepted publication. +- Deferred disparities: the portrait 1024 x 1536 raster differs from the + requested wide high-resolution layout; lower rejection routing is visually + close to the active-manifest replacement terminal; the no-op and commit + terminals require careful caption interpretation; style differs from the + more icon-heavy architecture and sequence figures. +- Current gate: accepted unchanged for drafting; terminal routing, layout, + resolution, and family-style review required at PW7. + +## Gate summary + +| Gate | Drafting-stage result | Final requirement | +|---|---|---| +| File existence and hashes | Pass | Recheck after any PW7 replacement | +| Prompt/spec source present | Pass | Preserve with final assets | +| Concept-only evidence boundary | Pass | Keep captions non-empirical | +| Label/topology review | Completed with recorded disparities | Repair or explicitly accept at PW7 | +| Final-width and raster review | Exceptions recorded | Reassess at PW7 | +| Style-family consistency | Deferred by author | Resolve or explicitly waive at PW7 | +| Lifecycle/prose consistency | Pass with an explicit normal-path qualifier | Repeat after any PW7 replacement | +| Manuscript placement and captions | Pass in the final PW3 render | Repeat after any PW7 replacement | + +## PW7 final-phase queue + +No PNG is to be changed during PW3. The final paper-polish phase must revisit: + +1. the architecture figure's route convergence, promoted-layer return edge, + band overlap, aspect ratio, and raster resolution; +2. the publication figure's requested swimlane layout, missing or reorganized + labels, icon density, and raster resolution; +3. the lifecycle figure's omitted exceptional branches and shared + `Destroying`-node ambiguity; +4. the reconciliation figure's portrait layout, terminal-edge separation, + no-op/commit interpretation, and raster resolution; and +5. cross-figure typography, palette, stroke, icon, spacing, final-size, + grayscale, and family-style consistency. + +PW7 may repair these items or explicitly accept an item with a recorded +submission-stage waiver. The current author waiver permits drafting-stage use +only. + +## PW7 submission-stage disposition + +Review date: 2026-08-02. The final PDF was rebuilt after PW4--PW7 integration, +then rendered in color and grayscale at the manuscript's final page size. +Architecture, publication sequence, reconciliation, and lifecycle figures were +each inspected in both renderings. All primary labels, arrows, terminal states, +and captions remain readable; no clipping, overlap, broken glyph, or caption +mismatch was found. The existing raster files were retained unchanged, so the +hashes in the inventory above remain the final asset hashes. + +The following explicit waivers close the deferred items without changing a +source-defined or evidence-bearing assertion: + +- Architecture: the visual convergence and band-spacing differences are + cosmetic. The caption and Section 4 specify the publication boundary, and + no route is used as proof of a result. +- Publication sequence: its phased horizontal composition is readable and + preserves private/precommit/accepted/post-commit separation. It is accepted + instead of the earlier swimlane preference. +- Lifecycle: pale dotted post-commit annotations have lower grayscale contrast, + but the main states and directed edges remain readable; the caption carries + the normal-path and ``published but not closed'' qualifiers. +- Reconciliation: the portrait layout is legible in the final PDF; captions and + prose, rather than color alone, state the whole-candidate reject/no-op rules. +- Family style: icon density and raster dimensions differ, but the common + outline, arrow, and state-color language stays interpretable in grayscale. + +No regeneration was warranted: the observed differences are explanatory-layout +choices, while replacement would risk text/topology drift without improving an +evidence claim. These figures remain concept-method illustrations rather than +experimental evidence. + +## PW3 compiled-PDF review + +- The successful recorded build preserves the four asset hashes above. +- The 14-page US-letter PDF places the architecture on page 3, publication + sequence on page 6, reconciliation flow on page 8 immediately before + Section 6, and lifecycle state machine on page 10 immediately before + Section 7. +- Figure and table boundaries have no clipping, overlap, missing glyphs, + broken labels, or unresolved references. The two Section 7 tables remain + after their introducing prose. +- All four figures are readable at their compiled color size for drafting. + The grayscale render preserves the core topology and primary labels, but + pale dashed annotations and category cues lose contrast, especially the + lifecycle figure's pink post-commit/failure notes. That contrast weakness, + together with final typography and style-family consistency, remains PW7 + debt rather than a PW3 image edit. +- The final PDF-wide color and grayscale contact sheets show consistent + margins, page numbering, and section transitions. The sparse final + References page is expected while `references.bib` remains comment-only. diff --git a/ephemeral-sandbox-v1/figures/concept/fig_lifecycle_state_machine.png b/ephemeral-sandbox-v1/figures/concept/fig_lifecycle_state_machine.png new file mode 100644 index 0000000..a6df36c Binary files /dev/null and b/ephemeral-sandbox-v1/figures/concept/fig_lifecycle_state_machine.png differ diff --git a/ephemeral-sandbox-v1/figures/concept/fig_publication_sequence.png b/ephemeral-sandbox-v1/figures/concept/fig_publication_sequence.png new file mode 100644 index 0000000..b461480 Binary files /dev/null and b/ephemeral-sandbox-v1/figures/concept/fig_publication_sequence.png differ diff --git a/ephemeral-sandbox-v1/figures/concept/fig_reconciliation_decision.png b/ephemeral-sandbox-v1/figures/concept/fig_reconciliation_decision.png new file mode 100644 index 0000000..52114c8 Binary files /dev/null and b/ephemeral-sandbox-v1/figures/concept/fig_reconciliation_decision.png differ diff --git a/ephemeral-sandbox-v1/figures/concept/fig_system_architecture.png b/ephemeral-sandbox-v1/figures/concept/fig_system_architecture.png new file mode 100644 index 0000000..580359d Binary files /dev/null and b/ephemeral-sandbox-v1/figures/concept/fig_system_architecture.png differ diff --git a/ephemeral-sandbox-v1/figures/figure_plan.md b/ephemeral-sandbox-v1/figures/figure_plan.md index cdd9b73..0d900b0 100644 --- a/ephemeral-sandbox-v1/figures/figure_plan.md +++ b/ephemeral-sandbox-v1/figures/figure_plan.md @@ -1,6 +1,36 @@ -# PW0 figure and table plan - -No figure or table asset is generated in PW0. Concept-method assets are illustrative only and must be checked against final terminology and tagged source before inclusion. Every evidence-result asset is blocked until its named frozen source or experimental data exists. +# Figure and table plan + +No figure or table asset was generated in PW0. PW2.5 now provides four +self-contained concept-figure prompts under `figures/source/` and four +author-supplied PNG review drafts under `figures/concept/`. Their hashes, +provenance boundary, and drafting-stage QA are recorded in +[`concept-figure-review.md`](concept-figure-review.md). The author accepted the +supplied files unchanged for the drafting manuscript and deferred visual, +resolution, topology, and style-family repair to PW7. Concept-method assets are +illustrative only and must still be checked against final terminology and tagged +source before submission. Every evidence-result asset remains blocked until its +named frozen source or experimental data exists. + +## PW2.5 prompt package + +All four prompts use the shared visual language in +[`source/STYLE_GUIDE.md`](source/STYLE_GUIDE.md), while repeating the relevant +style instructions so each prompt can be sent independently to an image +generator. + +| Figure ID | Diagram | Separate prompt | Figure-specific composition | Status | +|---|---|---|---|---| +| `fig:system-architecture` | LayerStack and workspace-session architecture | [`fig_system_architecture_prompt.md`](source/fig_system_architecture_prompt.md) | Three semantic bands with parallel private sessions and one publication boundary | Final color/grayscale review passed; topology/layout waiver recorded. | +| `fig:publication-sequence` | Workspace-to-publication sequence | [`fig_publication_sequence_prompt.md`](source/fig_publication_sequence_prompt.md) | Five vertical swimlanes with a highlighted active-manifest commit line | Final color/grayscale review passed; non-swimlane composition waived. | +| `fig:lifecycle-state-machine` | Workspace-session lifecycle state machine | [`fig_lifecycle_state_machine_prompt.md`](source/fig_lifecycle_state_machine_prompt.md) | Precommit/post-commit state regions with retry and cleanup-failure branches | Final color/grayscale review passed; qualified normal-path abstraction retained. | +| `fig:reconciliation-decision` | Current-head reconciliation decision flow | [`fig_reconciliation_decision_prompt.md`](source/fig_reconciliation_decision_prompt.md) | Source/ignored branches, narrow merge gates, and shared whole-candidate terminals | Final color/grayscale review passed; portrait layout and style-family variation waived. | + +Review and finalization order: + +1. integrated all four review drafts unchanged during PW3; +2. revalidated lifecycle terminology against completed Section 6; +3. inspected compiled placement, captions, final-size color, and grayscale; +4. repair or explicitly accept all recorded visual disparities during PW7. ## Planned assets diff --git a/ephemeral-sandbox-v1/figures/source/STYLE_GUIDE.md b/ephemeral-sandbox-v1/figures/source/STYLE_GUIDE.md new file mode 100644 index 0000000..f3eee26 --- /dev/null +++ b/ephemeral-sandbox-v1/figures/source/STYLE_GUIDE.md @@ -0,0 +1,124 @@ +# Ephemeral Sandbox concept-figure style guide + +## Preferred style + +Use a flat, vector-inspired academic systems-diagram style that combines +**Classic Academic** precision with **Modern Minimal** spacing. The diagrams +should look like polished architecture figures, not product marketing art. + +Why this style fits the paper: + +- precise boundaries matter more than visual spectacle; +- the figures must remain readable at the manuscript's approximately + 6.5-inch text width; +- flat geometry is easier to verify against source and terminology; +- restrained color avoids implying performance, security, or maturity that + the paper has not measured; +- the same visual grammar can cover architecture, sequence, state, and + decision diagrams. + +Avoid isometric servers, 3D stacks, glossy gradients, shadows, glassmorphism, +cartoon robots, brains, clouds, shields, locks, speed lines, dashboards, fake +terminal text, and decorative infrastructure icons. + +## Canvas and output + +- Full-width figure for a 10 pt, single-column `article` manuscript. +- Preferred landscape canvas: 16:9, at least 3200 x 1800 pixels. +- Decision flow may use 4:3, at least 2800 x 2100 pixels. +- Raster output: PNG, 300 DPI or higher. +- Preserve generous outer margins and at least 3% spacing between groups. +- Do not place the paper caption or a large title inside the artwork. +- Keep text equivalent to at least 8 pt when the figure is rendered at + 6.5 inches wide. + +## Typography and geometry + +- Neutral sans serif: Inter, Helvetica, Arial, or a close equivalent. +- Sentence case, not all caps. +- Short labels, normally one to four words. +- Charcoal text `#222222`. +- White background `#FFFFFF`; optional group bands `#F8FAFC`. +- Thin neutral borders `#6B7280`, consistent 1.5--2 px visual weight. +- Rounded rectangles with a small, consistent corner radius. +- Use alignment, whitespace, and bands before using icons. +- If icons are necessary, use only simple monochrome geometric glyphs. + +## Semantic palette + +Use color together with labels, border styles, or shapes so the figure remains +interpretable in grayscale. + +| Meaning | Color | Usage | +|---|---|---| +| Shared/durable state | Navy `#0072B2` | Active manifest, LayerStack, promoted layer | +| Private/session state | Sky blue `#56B4E9` | Private overlay, upper/work, session | +| Planning/reconciliation | Orange `#E69F00` | Capture plan, fingerprint, merge | +| Accepted publication | Green `#009E73` | Resolved changeset, commit path | +| Rejection/failure | Vermilion `#D55E00` | Dashed rejection and failure paths | +| Post-commit/non-atomic work | Purple `#CC79A7` | Audit attribution and cleanup boundary | +| Neutral/control | Gray `#8C8C8C` | Control arrows, inactive structure | + +Do not use green and red as the only distinction. Pair them with the words +“accepted” and “rejected,” and use solid versus dashed lines. + +## Arrow grammar + +- Solid navy/blue arrow: data or filesystem-view flow. +- Solid gray arrow: control flow or state transition. +- Solid green arrow: accepted publication path. +- Dashed vermilion arrow: rejection, retry, or failure. +- Dotted gray arrow: optional or semantically separate path. +- Purple dotted or dashed arrow: post-commit attribution/cleanup. +- Avoid crossing arrows; route connections orthogonally when possible. + +## Terminology rules + +Use exact canonical terms: + +- LayerStack +- active manifest +- active head +- lease +- workspace session +- private overlay +- implicit session +- explicit session +- capture +- candidate changeset +- current-head reconciliation +- rejection +- atomic data publication + +Never label the mechanism: + +- transaction; +- serializable snapshot; +- VM; +- secure sandbox; +- semantic merge; +- Git commit; +- universal network isolation; +- cross-platform; +- faster, scalable, efficient, or higher throughput. + +## Generated-text fallback + +Exact text is part of the scientific content. If the image generator cannot +render every required label faithfully: + +1. generate the same composition with numbered nodes and no prose; +2. return a label-placement map; +3. add exact labels later with a deterministic SVG, PDF, or LaTeX overlay; +4. never accept misspelled or substituted terminology. + +## Family consistency + +All four figures should reuse: + +- the same palette and semantic meanings; +- the same border radius and line weight; +- the same typography; +- the same arrow grammar; +- the same visual representation of the active manifest, LayerStack, private + overlay, rejection, and post-commit boundary. diff --git a/ephemeral-sandbox-v1/figures/source/fig_lifecycle_state_machine_prompt.md b/ephemeral-sandbox-v1/figures/source/fig_lifecycle_state_machine_prompt.md new file mode 100644 index 0000000..b8bf422 --- /dev/null +++ b/ephemeral-sandbox-v1/figures/source/fig_lifecycle_state_machine_prompt.md @@ -0,0 +1,169 @@ +# Figure prompt: Workspace-session lifecycle state machine + +## Status + +**Provisional until PW3 completes Section 6.** The image generator may produce +a review draft, but the asset must not be treated as manuscript-final until +the lifecycle/recovery prose and D9 boundary are revalidated. + +## Figure contract + +```yaml +figure_id: fig:lifecycle-state-machine +filename: figures/concept/fig_lifecycle_state_machine.png +figure_class: concept-method +role: method-detail +message: "Precommit failure, rejection, no-op, publication, discard, close, and post-commit cleanup failure are distinct lifecycle outcomes." +core_conclusion: "Data publication and session closure are separate transitions; cleanup failure cannot undo an accepted active-manifest transition." +evidence_hierarchy: + hero_evidence: "The split between the precommit state region and the post-commit published region." + supporting_evidence: "Retry, discard, no-op, closed, and published-but-not-closed paths." +layout: "Left-to-right state machine with vertically separated precommit and post-commit regions." +backend: hybrid +source: "C4, D1, D6, D8, D9, Section 3 definitions, and forthcoming PW3 Section 6." +backup: "Deterministic labels and transitions over a generated state-layout base." +caption_takeaway: "A session can be retained before commit or fail to close after commit; only the latter already changed the active head." +evidence_status: illustrative-only +reviewer_risk: "The current Section 6 is still a placeholder; exact transition names and implicit-session cleanup behavior require PW3 verification." +``` + +## Copy-paste prompt + +```text +Create a review-draft lifecycle state-machine diagram for an arXiv systems +paper about Ephemeral Sandbox. The diagram must distinguish precommit session +handling from post-commit cleanup. Its central message is: data publication +and session closure are separate transitions, so a cleanup failure after +publication does not roll back visible data. + +This figure is PROVISIONAL until the paper’s Lifecycle and Recovery section is +completed. Do not embellish or infer missing states. + +STYLE +Use a flat 2D vector-inspired Classic Academic × Modern Minimal style. White +background, rounded state nodes, thin gray borders, neutral sans-serif labels, +large whitespace, no gradients, no shadows, no 3D, no isometric servers, no +icons except simple state markers, no logos, and no watermarks. Landscape +16:9, at least 3200 x 1800 pixels, 300 DPI or higher. It must be readable at +approximately 6.5 inches wide. + +PALETTE +- Active/private session states: sky blue #56B4E9. +- Planning/finalization states: orange #E69F00. +- Published/accepted states: green #009E73. +- Rejection/precommit failure: vermilion #D55E00, dashed arrows. +- Post-commit cleanup states: purple #CC79A7. +- Closed/neutral terminal state: gray #8C8C8C. +- Text: charcoal #222222. +Use labels and line styles in addition to color. + +REGIONS +Create two clearly labeled horizontal regions: +1. upper region: “Before data commit” +2. lower or right-side region: “After data commit” +Separate them with a strong green boundary labeled exactly: +“Active manifest replaced” +Do not label the boundary a transaction. + +CORE STATE MACHINE +Use these exact state nodes: +- “Active session” +- “Finalizing” +- “Capture” +- “Reconcile” +- “No-op” +- “Published” +- “Destroying” +- “Closed” +- “Finalize failed” + +MAIN TRANSITIONS +Draw solid gray arrows: +“Active session” -- “publish requested” --> “Finalizing” +“Finalizing” -- “capture filesystem delta” --> “Capture” +“Capture” -- “candidate changeset” --> “Reconcile” +“Reconcile” -- “all changes resolved” --> the boundary +The boundary “Active manifest replaced” leads to “Published”. +“Published” --> “Destroying” --> “Closed”. + +PRECOMMIT BRANCHES +From “Capture” and “Reconcile”, draw dashed vermilion arrows labeled +“precommit failure or rejection” back to “Active session”. +Add a small note beside the return path: +“Explicit session retained for retry or destroy”. +From “Reconcile”, draw a neutral branch labeled “empty changeset” to “No-op”. +From “No-op”, draw a solid gray arrow to “Destroying”. +From “Active session”, draw a separate neutral path labeled +“destroy without publication” to “Destroying”, with a small tag “discard”. +Keep all of these paths before the commit boundary. + +POST-COMMIT BRANCHES +From “Published”, add a dotted purple side annotation: +“Best-effort attribution”. +It must not gate the transition to “Destroying”. +From “Destroying”, draw: +- solid gray arrow labeled “cleanup succeeds” to “Closed”; +- dashed purple arrow labeled “cleanup fails” to “Finalize failed”. +Place a callout next to “Finalize failed”: +“Published but not closed”. +Draw no rollback arrow from “Finalize failed” to “Active session” or to the +old manifest. + +IMPLICIT-SESSION CALLOUT +Outside the main state machine, add one small neutral callout: +“Implicit session: enters finalization after command ledger drains”. +Connect it with a dotted gray arrow to “Finalizing”. +Do not imply every tool call creates an implicit session. + +CONSTRAINTS +Do not add process checkpointing, automatic semantic verification, automatic +task decomposition, transaction rollback, security guarantees, timeouts, +retry counts, performance metrics, or cross-platform claims. +Do not invent recovery transitions beyond the labels above. + +OUTPUT +Return a high-resolution review PNG suitable for: +figures/concept/fig_lifecycle_state_machine.png +Also return a label-placement map and explicitly mark the result +“PROVISIONAL — verify after PW3”. +If exact text is unreliable, create a numbered state layout and preserve all +transition geometry for a deterministic label overlay. +``` + +## Exact label inventory + +- Before data commit +- After data commit +- Active manifest replaced +- Active session +- Finalizing +- Capture +- Reconcile +- No-op +- Published +- Destroying +- Closed +- Finalize failed +- publish requested +- capture filesystem delta +- candidate changeset +- all changes resolved +- precommit failure or rejection +- Explicit session retained for retry or destroy +- empty changeset +- destroy without publication +- discard +- Best-effort attribution +- cleanup succeeds +- cleanup fails +- Published but not closed +- Implicit session: enters finalization after command ledger drains + +## Post-generation acceptance + +- Precommit return paths never cross the commit boundary. +- Only “Active manifest replaced” leads into “Published”. +- Cleanup failure ends at “Finalize failed” with no data rollback. +- No-op and discard are visually distinct from successful publication. +- The asset is visibly marked provisional in its delivery metadata, not as a + large watermark inside the artwork. diff --git a/ephemeral-sandbox-v1/figures/source/fig_publication_sequence_prompt.md b/ephemeral-sandbox-v1/figures/source/fig_publication_sequence_prompt.md new file mode 100644 index 0000000..ee84f8e --- /dev/null +++ b/ephemeral-sandbox-v1/figures/source/fig_publication_sequence_prompt.md @@ -0,0 +1,210 @@ +# Figure prompt: Workspace-to-publication sequence + +## Figure contract + +```yaml +figure_id: fig:publication-sequence +filename: figures/concept/fig_publication_sequence.png +figure_class: concept-method +role: method-detail +message: "A workspace delta remains private through capture and reconciliation; accepted data becomes public at active-manifest replacement, before best-effort attribution and cleanup." +core_conclusion: "The sequence has a precise data-commit boundary and separate precommit rejection and post-commit phases." +evidence_hierarchy: + hero_evidence: "The highlighted active-manifest replacement line." + supporting_evidence: "Lease/mount/execute/capture flow, whole-candidate rejection, and downstream audit/cleanup." +layout: "Five-lane sequence diagram with time flowing downward." +backend: hybrid +source: "C1/C2, D1--D8, Sections 4--5, and plan/terminology.md" +backup: "Deterministic labels and arrows over a generated background." +caption_takeaway: "Execution and capture stay private; only the accepted manifest transition exposes data, while attribution and cleanup occur afterward." +evidence_status: illustrative-only +reviewer_risk: "Do not make command response equal publication completion, include audit in the commit, or imply every file operation creates a session." +``` + +## Copy-paste prompt + +```text +Create a publication-quality sequence diagram for an arXiv systems paper. The +figure must show how an Ephemeral Sandbox workspace moves from a leased private +execution view to one durable public head. The most important visual fact is +that public data visibility begins at active-manifest replacement; audit +attribution and cleanup occur afterward and are not part of atomic data +publication. + +STYLE +Use a flat 2D vector-inspired Classic Academic × Modern Minimal style. White +background, no gradients, no shadows, no 3D, no isometric objects, no logos, +no watermarks, no cartoon agents, and no fake numeric data. Use thin neutral +lines, rounded message boxes, generous whitespace, and neutral sans-serif +text. Landscape 16:9 or 16:10, at least 3200 pixels wide, 300 DPI or higher. +The diagram must remain readable at approximately 6.5 inches wide. + +PALETTE AND LINES +- Shared/durable LayerStack: navy #0072B2. +- Private workspace activity: sky blue #56B4E9. +- Planning/reconciliation: orange #E69F00. +- Accepted commit path: green #009E73. +- Rejection/failure: vermilion #D55E00 with dashed arrows. +- Post-commit work: purple #CC79A7 with dotted arrows. +- Neutral control: gray #8C8C8C. +- Text: charcoal #222222. +Do not rely on color alone. + +SWIMLANES +Create five vertical swimlanes with these exact headers, left to right: +1. “Caller” +2. “Session service” +3. “Workspace / OverlayFS” +4. “LayerStack” +5. “Post-commit services” +Use subtle alternating lane backgrounds and a downward time direction. +Do not put a large title inside the figure. + +SEQUENCE +Show the following numbered conceptual phases without fake timings: + +1. Caller → Session service: + message label “Create or admit session”. + +2. Session service → LayerStack: + message label “Acquire lease”. + LayerStack returns “Manifest + ordered layers”. + +3. Session service → Workspace / OverlayFS: + message label “Mount leased lowers”. + Inside the workspace lane show a compact blue state box: + “Private upper / work”. + +4. Caller → Session service: + message label “Execute command(s)”. + Session service → Workspace / OverlayFS: + message label “Holder + runner”. + Add a small note in the session lane: + “Implicit: auto-finalize after ledger drains” + and a second note: + “Explicit: multiple calls until publish or destroy”. + Do not equate returning an initial command response with publication. + +5. Workspace / OverlayFS → Session service: + message label “Command ledger drained”. + +6. Session service → Workspace / OverlayFS: + message label “Capture upper tree”. + Workspace returns an orange object labeled “Candidate changeset”. + Add a small note: “filesystem delta, not process state”. + +7. Session service → LayerStack: + message label “Plan against leased base”. + Add a compact orange note in the LayerStack lane: + “route + protect + fingerprint”. + +8. Inside LayerStack, show a narrow orange critical region labeled: + “Writer lock: reread active manifest”. + Then show “Current-head reconciliation”. + From reconciliation draw two branches: + - dashed vermilion branch left, label “Rejected”, ending at + “Active manifest unchanged”; + - solid green branch downward, label “All changes resolved”. + Add a tiny no-op branch labeled “No-op” that also leaves the current head + selected. + +9. On the accepted branch, inside the LayerStack lane show these exact boxes + in order: + “Stage changes” + → “Sync staging tree” + → “Promote layer” + → “Write layer digest” + → “Recheck manifest” + → “Replace active manifest”. + +COMMIT BOUNDARY +Across all lanes, draw a strong horizontal green line immediately after +“Replace active manifest”. Label the line exactly: +“Atomic data publication — public visibility begins” +This is the hero element of the figure. +Everything above it is private or precommit. Everything below it is +post-commit. Do not place audit or cleanup above this line. + +POST-COMMIT PHASE +Below the green line: +- LayerStack → Post-commit services: + dotted purple message “Best-effort audit attribution”. +- Session service → Workspace / OverlayFS: + purple/gray message “Destroy workspace”. +- Session service → LayerStack: + purple/gray message “Release lease”. +Group these steps in a pale region labeled exactly: +“Outside atomic data publication”. + +FAILURE SEMANTICS +Use dashed vermilion only for precommit rejection or failure. +Use a separate purple dashed annotation below the commit line: +“Cleanup failure does not roll back visible data”. +Do not draw a rollback arrow from post-commit cleanup to the active manifest. + +CONSTRAINTS +Use only the requested terminology. Do not add performance numbers, duration +bars, throughput arrows, security shields, Git commits, database transactions, +semantic merge claims, universal network isolation, or cross-platform icons. +Do not imply that sessionless file read/write/edit follows this exact session +sequence; add a small neutral footnote: +“Sequence shown for workspace-session publication.” + +OUTPUT +Return a high-resolution PNG suitable for: +figures/concept/fig_publication_sequence.png +Also return a label-placement map. If exact text is unreliable, preserve the +five-lane layout using numbered message markers and provide a deterministic +overlay plan instead of hallucinating text. +``` + +## Exact label inventory + +- Caller +- Session service +- Workspace / OverlayFS +- LayerStack +- Post-commit services +- Create or admit session +- Acquire lease +- Manifest + ordered layers +- Mount leased lowers +- Private upper / work +- Execute command(s) +- Holder + runner +- Implicit: auto-finalize after ledger drains +- Explicit: multiple calls until publish or destroy +- Command ledger drained +- Capture upper tree +- Candidate changeset +- filesystem delta, not process state +- Plan against leased base +- route + protect + fingerprint +- Writer lock: reread active manifest +- Current-head reconciliation +- Rejected +- Active manifest unchanged +- All changes resolved +- No-op +- Stage changes +- Sync staging tree +- Promote layer +- Write layer digest +- Recheck manifest +- Replace active manifest +- Atomic data publication — public visibility begins +- Best-effort audit attribution +- Destroy workspace +- Release lease +- Outside atomic data publication +- Cleanup failure does not roll back visible data +- Sequence shown for workspace-session publication. + +## Post-generation acceptance + +- Time and arrows read unambiguously from top to bottom. +- The green commit boundary follows manifest replacement exactly. +- Rejection ends with the active manifest unchanged. +- Audit, destruction, and lease release are below the commit boundary. +- No post-commit rollback arrow exists. +- The caption can explain the sequence without relying on color alone. diff --git a/ephemeral-sandbox-v1/figures/source/fig_reconciliation_decision_prompt.md b/ephemeral-sandbox-v1/figures/source/fig_reconciliation_decision_prompt.md new file mode 100644 index 0000000..1058086 --- /dev/null +++ b/ephemeral-sandbox-v1/figures/source/fig_reconciliation_decision_prompt.md @@ -0,0 +1,207 @@ +# Figure prompt: Current-head reconciliation decision flow + +## Figure contract + +```yaml +figure_id: fig:reconciliation-decision +filename: figures/concept/fig_reconciliation_decision.png +figure_class: concept-method +role: method-detail +message: "Publication resolves every planned change through explicit route, fingerprint, structure, and merge gates, or rejects the whole candidate." +core_conclusion: "Only eligible exact-file text divergence is merged; ignored-route behavior and whole-candidate rejection remain explicit." +evidence_hierarchy: + hero_evidence: "Source-route fingerprint divergence leading to a narrowly eligible merge gate." + supporting_evidence: "Protected-path/base checks, ignored-route branch, opaque-directory checks, and whole-candidate commit/reject outcomes." +layout: "Top-down decision flow with source and ignored branches that rejoin at whole-candidate resolution." +backend: hybrid +source: "C2, D4--D6, Section 5, and plan/terminology.md" +backup: "Deterministic labels over a generated flowchart layout." +caption_takeaway: "A candidate is committed only after every change resolves; required binary, oversized, structural, or conflicting merges reject rather than partially publish." +evidence_status: illustrative-only +reviewer_risk: "Do not imply all binary/oversized writes reject: the restriction applies only when divergence requires a merge." +``` + +## Copy-paste prompt + +```text +Create a publication-quality decision-flow diagram for the Current-head +reconciliation method in an arXiv systems paper about Ephemeral Sandbox. The +reader should understand that every planned change must resolve, only a narrow +class of concurrent exact-file text divergence can merge, and one unresolved +change rejects the whole candidate changeset. + +STYLE +Use a flat 2D vector-inspired Classic Academic × Modern Minimal flowchart. +White background, no gradients, no shadows, no 3D, no isometric objects, no +logos, no watermarks, no fake code, and no fake numeric results. Use rounded +rectangles for actions, diamonds for decisions, thin neutral borders, short +exact labels, and orthogonal arrows with minimal crossings. Use a neutral +sans-serif font. Preferred canvas 4:3 at least 2800 x 2100 pixels, or a roomy +16:9 landscape if needed, 300 DPI or higher. Text must remain readable at +approximately 6.5 inches wide. + +PALETTE +- Input/shared state: navy #0072B2. +- Planning and decisions: orange #E69F00. +- Accepted/resolved path: green #009E73. +- Rejection: vermilion #D55E00 with dashed arrows. +- Ignored-route branch: neutral gray #8C8C8C. +- Notes/boundaries: purple #CC79A7. +- Text: charcoal #222222. +Use words and line styles in addition to color. + +TOP INPUT +At the top center, draw a navy rounded rectangle: +“Candidate changeset” +Then an orange action: +“Validate leased base” +Then a decision diamond: +“Base identity valid?” +The “no” branch goes by dashed vermilion arrow to: +“Reject whole candidate” +with a small result tag: +“Active manifest unchanged”. + +PROTECTION AND PLANNING +The “yes” branch continues to: +“Protected path or drop?” +For an explicit-session figure, the “yes” branch goes to +“Reject whole candidate”. +Add a compact purple footnote: +“Unsupported-special-file drop policy differs by entry point”. +The “no” branch goes to: +“Plan route”. +Beside planning, add a small orange callout: +“Opaque directory: protected descendant, mixed route, or expansion bound +violation → reject”. + +ROUTE SPLIT +From “Plan route”, split into two clearly labeled branches: + +LEFT NEUTRAL BRANCH — “Ignored route” +Action box: +“Carry wholesale change” +Small note: +“No source fingerprint merge”. + +RIGHT ORANGE BRANCH — “Source route” +Action box: +“Fingerprint leased base” +Then show a critical-region label: +“Writer lock: reread active manifest”. +Then a decision: +“Current fingerprint matches base?” +The “yes” branch goes to: +“Accept planned change”. +Add a small compatibility note: +“Compatible directory create may pass”. + +SOURCE DIVERGENCE +The “no” branch from fingerprint comparison goes to: +“Structural descendant changed?” +The “yes” branch goes to “Reject whole candidate”. +The “no” branch continues to: +“Exact regular-file write?” +The “no” branch goes to “Reject whole candidate”. +The “yes” branch continues to: +“Merge inputs eligible?” +Place a concise note next to this diamond: +“base + current + command: +UTF-8, no NUL, each ≤ 8 MiB”. +The “no” branch goes to “Reject whole candidate”. +The “yes” branch continues to: +“Three-way line merge clean?” +The “no” branch goes to “Reject whole candidate”. +The “yes” branch goes to: +“Resolved merged write”. + +IMPORTANT QUALIFIER +Add a clearly visible purple note near the merge path: +“Binary or oversized writes are not categorically rejected. +They reject only when concurrent divergence requires this merge.” +Add another short note: +“Clean merge ≠ semantic correctness”. + +WHOLE-CANDIDATE JOIN +Join these green/neutral success nodes: +- “Carry wholesale change” +- “Accept planned change” +- “Resolved merged write” +into one action: +“Record resolved change”. +Then draw a decision: +“Every planned change resolved?” +The “no” branch goes to “Reject whole candidate”. +The “yes” branch goes to: +“Resolved changeset”. +From there: +- if empty, branch to “No-op; current head remains”; +- if nonempty, branch to “Stage and commit whole changeset”. +End the accepted branch at: +“Active manifest replacement”. + +VISUAL EMPHASIS +Make “Reject whole candidate” one shared terminal node reached by all dashed +vermilion rejection paths. Do not draw partial-commit arrows. +Make “Active manifest replacement” the accepted green terminal node. +The decision tree must communicate all-or-none data handling, not a full +database transaction. + +CONSTRAINTS +Do not add performance claims, runtime graphs, security imagery, Git merge +terminology, arbitrary binary merge, semantic verification, automatic tests, +partial publication, or rollback after active-manifest replacement. +Do not simplify the qualifier into “binary files reject” or “files above 8 MiB +reject”. + +OUTPUT +Return a high-resolution PNG suitable for: +figures/concept/fig_reconciliation_decision.png +Also return a label-placement map. If exact text cannot be guaranteed, produce +a numbered-node base with the same branch topology for a deterministic text +overlay; do not paraphrase the technical labels. +``` + +## Exact label inventory + +- Candidate changeset +- Validate leased base +- Base identity valid? +- Reject whole candidate +- Active manifest unchanged +- Protected path or drop? +- Unsupported-special-file drop policy differs by entry point +- Plan route +- Opaque directory: protected descendant, mixed route, or expansion bound violation → reject +- Ignored route +- Carry wholesale change +- No source fingerprint merge +- Source route +- Fingerprint leased base +- Writer lock: reread active manifest +- Current fingerprint matches base? +- Accept planned change +- Compatible directory create may pass +- Structural descendant changed? +- Exact regular-file write? +- Merge inputs eligible? +- base + current + command: UTF-8, no NUL, each ≤ 8 MiB +- Three-way line merge clean? +- Resolved merged write +- Binary or oversized writes are not categorically rejected. They reject only when concurrent divergence requires this merge. +- Clean merge ≠ semantic correctness +- Record resolved change +- Every planned change resolved? +- Resolved changeset +- No-op; current head remains +- Stage and commit whole changeset +- Active manifest replacement + +## Post-generation acceptance + +- Source and ignored routes are distinct. +- All rejection paths converge on one whole-candidate rejection terminal. +- Merge is available only after exact-file and eligibility gates. +- The binary/oversized qualifier is readable and unambiguous. +- No partial publication path exists. +- Accepted visibility ends at active-manifest replacement. diff --git a/ephemeral-sandbox-v1/figures/source/fig_system_architecture_prompt.md b/ephemeral-sandbox-v1/figures/source/fig_system_architecture_prompt.md new file mode 100644 index 0000000..9c1dadf --- /dev/null +++ b/ephemeral-sandbox-v1/figures/source/fig_system_architecture_prompt.md @@ -0,0 +1,191 @@ +# Figure prompt: LayerStack and workspace-session architecture + +## Figure contract + +```yaml +figure_id: fig:system-architecture +filename: figures/concept/fig_system_architecture.png +figure_class: concept-method +role: overview +message: "One active LayerStack supplies leased read-only views to multiple private workspace sessions, and only accepted active-manifest replacement advances the shared head." +core_conclusion: "Shared history and session-private execution are separated until atomic data publication." +evidence_hierarchy: + hero_evidence: "Shared LayerStack feeding two private overlays that converge through one publication boundary." + supporting_evidence: "Holder/runner execution, lease pinning, rejection, and post-commit audit/cleanup." +layout: "Three horizontal semantic bands in a left-to-right architecture." +backend: hybrid +source: "C1/C2, D1--D8, Sections 3--5, and plan/terminology.md" +backup: "Deterministic SVG or LaTeX label overlay if generated text is unreliable." +caption_takeaway: "Sessions share leased lower history but keep writable state private; accepted data becomes public only when the active manifest is replaced." +evidence_status: illustrative-only +reviewer_risk: "Do not depict filesystem-enforced immutability, universal network isolation, a security boundary, or audit/cleanup inside atomic publication." +``` + +## Copy-paste prompt + +```text +Create a publication-quality concept-method architecture diagram for an arXiv +systems paper titled “Ephemeral Sandbox.” The reader should understand one +idea immediately: one shared LayerStack provides leased read-only views to +multiple private workspace sessions, and only accepted active-manifest +replacement advances the shared active head. + +STYLE +Use a flat 2D vector-inspired academic systems-diagram style: Classic Academic +precision combined with Modern Minimal spacing. White background, no gradients, +no shadows, no 3D, no isometric perspective, no clip art, no logos, no +watermarks, no fake terminal text. Use clean rounded rectangles, thin neutral +borders, generous whitespace, and a neutral sans-serif font. The final figure +will be printed approximately 6.5 inches wide, so all labels must remain +readable at that size. Landscape 16:9 canvas, at least 3200 x 1800 pixels, +300 DPI or higher. + +SEMANTIC PALETTE +- Shared/durable state: navy #0072B2. +- Private/session state: sky blue #56B4E9. +- Capture and reconciliation: orange #E69F00. +- Accepted publication: green #009E73. +- Rejection/failure: vermilion #D55E00. +- Post-commit/non-atomic work: purple #CC79A7. +- Neutral control structure: gray #8C8C8C. +- Text: charcoal #222222. +Do not rely on color alone; pair color with exact labels and line styles. + +LAYOUT +Create three clearly separated horizontal bands with a left-to-right reading +order. Do not add a large title inside the artwork. + +TOP BAND — SHARED DURABLE STATE +Label the pale band “Shared durable state.” +At the left, draw a small box labeled exactly “Active manifest.” +It points to a horizontal ordered stack labeled exactly “LayerStack.” +Inside the stack show three flat layer slabs labeled, from highest precedence +to lowest: “Newest layer”, “Earlier layer”, and “Base”. +Add a small directional annotation “newest first”. +From the active manifest and layer stack, draw two navy lease connections +downward. Each connection has a small tag labeled exactly “Lease”. +The leases must visually pin a logical view; do not depict them as global locks. + +MIDDLE BAND — PRIVATE SESSION STATE +Label the pale blue band “Private session state.” +Place two parallel session groups to show multiplicity: +“Workspace session A” and “Workspace session B”. +Inside each group, show: +1. a read-only lower stack labeled “Leased lowers”; +2. a distinct writable block labeled “Upper / work”; +3. the combined mounted view labeled “Private overlay”; +4. a small process box labeled “Namespace holder”; +5. one or two small process nodes labeled “Runner”. +Use solid blue arrows from each leased lower stack into its private overlay. +Use a separate blue writable connection from “Upper / work” into the same +overlay. Connect “Namespace holder” to “Runner” with a neutral control arrow. +Add one short annotation beneath the two sessions: “Shared lowers; unique +writable state”. +Do not draw virtual machines, security shields, locks, or claims of complete +isolation. + +RIGHT-SIDE PUBLICATION PATH +From each private overlay, allow an orange path to converge into one box +labeled exactly “Capture”. +Then draw boxes in this exact order: +“Candidate changeset” → “Current-head reconciliation” → “Resolved changeset”. +Use orange for capture and reconciliation, then green for the resolved +changeset. +From reconciliation, draw one dashed vermilion branch labeled “Rejected” that +ends at a box labeled “Active manifest unchanged”. +From “Resolved changeset”, draw a solid green path through: +“Stage + sync” → “Promoted layer” → “Active manifest replacement”. +Make “Active manifest replacement” the strongest visual boundary, using a +green outlined box and a thin vertical marker labeled exactly +“Public data visibility”. +Draw a green arrow from the promoted layer into the top-band LayerStack, +showing that the accepted layer is prepended to shared history. + +POST-COMMIT BOUNDARY +After “Active manifest replacement”, draw a visually separate pale region +labeled exactly “Outside atomic data publication”. +Inside it place two small boxes: “Audit attribution” and “Session cleanup”. +Connect them with purple dotted arrows from the manifest replacement. +They must appear downstream and outside the green publication boundary. + +SESSIONLESS FILE-OPERATION CALLOUT +Add a small neutral inset in the bottom-left corner labeled exactly +“Sessionless file operations”. +Inside it show: +“read” → “Active LayerStack” +“write / edit” → “Direct head amendment” +Use dotted gray arrows and keep this inset visually separate from the workspace +sessions. This prevents the diagram from implying that every tool call creates +a session. + +ARROW GRAMMAR +- Solid navy/blue: filesystem-view or private-state flow. +- Solid gray: process/control relationship. +- Solid green: accepted publication. +- Dashed vermilion: rejection. +- Dotted purple: post-commit work outside atomic publication. +- Dotted gray: separate sessionless paths. +Avoid crossing arrows. + +EXACTNESS AND CONSTRAINTS +Use only the labels requested above. Do not add numbers, axes, performance +curves, speed symbols, security claims, semantic correctness claims, Git +terminology, cross-platform claims, or decorative infrastructure. +Do not label the system a transaction, VM, secure sandbox, or serializable +snapshot. + +OUTPUT +Return a high-resolution PNG suitable for: +figures/concept/fig_system_architecture.png +Also return a short label-placement map. If exact text cannot be rendered +faithfully, generate a numbered text-light base composition and preserve the +same layout for a deterministic label overlay; never invent or misspell labels. +``` + +## Exact label inventory + +- Shared durable state +- Active manifest +- LayerStack +- Newest layer +- Earlier layer +- Base +- newest first +- Lease +- Private session state +- Workspace session A +- Workspace session B +- Leased lowers +- Upper / work +- Private overlay +- Namespace holder +- Runner +- Shared lowers; unique writable state +- Capture +- Candidate changeset +- Current-head reconciliation +- Resolved changeset +- Rejected +- Active manifest unchanged +- Stage + sync +- Promoted layer +- Active manifest replacement +- Public data visibility +- Outside atomic data publication +- Audit attribution +- Session cleanup +- Sessionless file operations +- read +- Active LayerStack +- write / edit +- Direct head amendment + +## Post-generation acceptance + +- Shared and private bands are visually distinct. +- Both sessions share lower history but have separate upper/work state. +- Rejection leaves the active manifest unchanged. +- Public visibility is marked only at active-manifest replacement. +- Audit and cleanup are visibly outside atomic data publication. +- Sessionless file operations are not routed through an implicit session. +- No text is misspelled or smaller than final-paper readability permits. diff --git a/ephemeral-sandbox-v1/lanes/paper-writing.md b/ephemeral-sandbox-v1/lanes/paper-writing.md index efad934..ee987de 100644 --- a/ephemeral-sandbox-v1/lanes/paper-writing.md +++ b/ephemeral-sandbox-v1/lanes/paper-writing.md @@ -1,6 +1,6 @@ # Lane 1: paper writing -Status: **ready to start the evidence-bearing systems core** +Status: **PW0--PW3 complete** This lane turns frozen or clearly provisional evidence into manuscript text. It can advance before performance and multi-agent results exist, but it must not convert hypotheses, test source, public product descriptions, or exploratory runs into demonstrated results. @@ -61,10 +61,10 @@ The paper lane has eight phases. PW0–PW3 can start from baseline source eviden | Phase | Manuscript work | Status | Entry gate | |---|---|---|---| -| PW0 | Scaffold and terminology | Ready | Story and skeleton available | -| PW1 | Sections 2–3: goals/non-goals and system model | Ready | Baseline source inventory | -| PW2 | Sections 4–5: workspace execution and publication | Ready | C1/C2 design evidence | -| PW3 | Sections 6–7: lifecycle, implementation, and CLI | Ready | C3/C4 design evidence | +| PW0 | Scaffold and terminology | Complete | Story and skeleton available | +| PW1 | Sections 2–3: goals/non-goals and system model | Complete | Baseline source inventory | +| PW2 | Sections 4–5: workspace execution and publication | Complete | C1/C2 design evidence | +| PW3 | Sections 6–7: lifecycle, implementation, and CLI | Complete | C3/C4 design evidence | | PW4 | Section 8 methodology; Section 9 related work and source limitations | Partially ready | Protocol lock for Evaluation Methodology | | PW5 | Section 8 results and measured failure analysis | Blocked | Evidence lock | | PW6 | Sections 1 and 10; final framing synthesis | Blocked | PW5 complete | @@ -320,17 +320,18 @@ The paper lane must not copy a number from a console, chat, exploratory notebook ## Completion checklist -- [ ] Create `main.tex`, section files, and reproducible build command. -- [ ] Draft System Model and Invariants. -- [ ] Draft Goals, Non-goals, and Threat-model Boundary. -- [ ] Draft LayerStack and Workspace-session Design. -- [ ] Draft Capture and Publication. -- [ ] Draft Lifecycle and Recovery. -- [ ] Draft Operational Interface. -- [ ] Draft Implementation. +- [x] Create `main.tex`, section files, and reproducible build command. +- [x] Draft System Model and Invariants. +- [x] Draft Goals, Non-goals, and Threat-model Boundary. +- [x] Draft LayerStack and Workspace-session Design. +- [x] Draft Capture and Publication. +- [x] Draft Lifecycle and Recovery. +- [x] Draft Operational Interface. +- [x] Draft Implementation. - [ ] Draft locked Evaluation Methodology. - [ ] Draft source-proven Limitations. -- [ ] Create and verify concept figure specifications. +- [x] Create and verify concept figure specifications; retain final visual + repair and submission acceptance for PW7. - [ ] Verify scholarly citations and sentence-level support. - [ ] Replace baseline source links with `paper-v1-freeze`. - [ ] Write Results from frozen artifacts. diff --git a/ephemeral-sandbox-v1/literature/paper_inventory.md b/ephemeral-sandbox-v1/literature/paper_inventory.md new file mode 100644 index 0000000..2b1ca13 --- /dev/null +++ b/ephemeral-sandbox-v1/literature/paper_inventory.md @@ -0,0 +1,33 @@ +# Related-work inventory + +This inventory is the submission-stage review set for the manuscript's narrow +runtime-publication claim. `citation_lock.json` records a terminal primary +metadata check for every identifier that can be represented by the paper's +DOI/arXiv verifier. The remaining union-mount record was checked at the +official USENIX proceedings page; its identifier is a proceedings URL rather +than a DOI or arXiv record. + +| Cluster | Primary record | Verification | Manuscript role | +| --- | --- | --- | --- | +| Isolated SWE execution | `swe_minisandbox_2026` | arXiv lock | Contrast: isolated execution and low-overhead environments are not introduced here. | +| Hybrid sandbox session | `agentbay_2025` | arXiv lock | Contrast: human/agent session control is distinct from durable code publication. | +| Checkpoint/rollback | `deltabox_2026` | arXiv lock | Contrast: process-state rollback exceeds this paper's filesystem-only claim. | +| Reversible execution traces | `shepherd_2026` | arXiv lock | Contrast: a trace/replay substrate is not a publication protocol. | +| Asynchronous SWE delegation | `caid_2026` | arXiv lock | Closest orchestration comparison: Git worktrees and manager policies. | +| Multi-agent concurrency control | `coagent_2026` | arXiv lock | Closest shared-state contrast: speculative in-place effects plus repair. | +| Change-intent control | `claimplane_2026` | arXiv lock | Closest admission-control comparison: authority and scope before writes. | +| Private-workspace awareness | `palantir_2012` | Crossref lock | Historical contrast: detect and communicate conflict, rather than enforce a runtime transition. | +| Collaboration-risk detection | `crystal_2013` | Crossref lock | Historical contrast: diagnose textual/build/test risks before merge. | +| Semantic merge | `threewaymerge_2018` | Crossref lock | Limitation boundary: a bounded text merge is not semantic merge correctness. | +| Union filesystem mechanism | Pendry and McKusick, *Union Mounts in 4.4BSD-Lite* | Official USENIX record | Background only: upper/lower views, copy-up, and whiteouts predate this system. | +| Optimistic validation | `occ_1981` | Crossref lock | Background only: tentative work plus validation is a general pattern, not a serializability claim. | +| Snapshot-isolation terminology | `snapshot_isolation_1995` | Crossref lock | Guardrail: do not call the system serializable snapshot isolation. | +| Collaborative coding evaluation | `cooperbench_2026` | arXiv lock | Candidate workload/motivation; not evidence for this system. | +| Historical conflict dataset | `agenticflict_2026` | Crossref lock | Narrow motivation only; no universal conflict-rate inference. | +| Role-separated teamwork | `teambench_2026` | arXiv lock | Evaluation context; role separation is not workspace publication. | +| Repository issue resolution | `swebench_2024` | arXiv lock | Workload context; it is not a concurrent-session benchmark. | +| Long-horizon research replication | `paperbench_2025` | arXiv lock | Workload context; it does not isolate integration effects. | + +No product blog, tool documentation, or unreviewed summary is used as scholarly +evidence. Current product interfaces may motivate engineering choices, but +they are not citations for empirical performance or correctness claims. diff --git a/ephemeral-sandbox-v1/literature/positioning.md b/ephemeral-sandbox-v1/literature/positioning.md new file mode 100644 index 0000000..fa8317e --- /dev/null +++ b/ephemeral-sandbox-v1/literature/positioning.md @@ -0,0 +1,38 @@ +# Submission-stage positioning + +## Claim that remains + +Ephemeral Sandbox is presented as a source-defined runtime publication protocol +for coding-agent tool calls and explicit workspace sessions. A private view is +derived from leased LayerStack history; on close or publication, the system +captures a delta, validates it against the active head, and either publishes a +new durable layer or returns a structured rejection. The contribution is this +bounded composition and operational contract, not a new filesystem primitive, +an agent planner, a universal coordination algorithm, or a semantic merge +theory. + +## Distinctions required in the manuscript + +- From CAID: the manager/worktree orchestration policy is external here; the + paper defines runtime capture and publication semantics. +- From CoAgent and Claim Plane: the paper neither keeps all effects live and + repairable nor performs pre-write task-intent admission. It isolates a + session's workspace effects and validates at publication. +- From DeltaBox and Shepherd: there is no claim of complete process-state + checkpointing, rollback, replay, or reversible execution trace. +- From Palantir and Crystal: awareness or early diagnosis is distinct from + enforcing a durable publish-or-reject transition. +- From semantic merge work: the bounded text merge is not proof that a clean + merge is semantically correct. +- From union mounts and optimistic concurrency control: copy-up, validation, + and restart/abort patterns are established mechanisms. The manuscript does + not call its lease behavior serializable snapshot isolation. + +## Prohibited novelty and evidence language + +Do not say that the system invents isolation, copy-on-write workspaces, +immutable histories, validation before commit, three-way merge, optimistic +concurrency, or multi-agent coordination. Do not infer throughput, coding-task +success, resource capacity, security, or cross-platform behavior from the +single frozen local treatment. Cite preprints as preprints and attach their +reported results only to their stated version, workload, and setting. diff --git a/ephemeral-sandbox-v1/literature/related_work_matrix.md b/ephemeral-sandbox-v1/literature/related_work_matrix.md new file mode 100644 index 0000000..80275e6 --- /dev/null +++ b/ephemeral-sandbox-v1/literature/related_work_matrix.md @@ -0,0 +1,20 @@ +# Related-work matrix + +| Work family | Private executable state | Durable-state rule | What it does not establish for this paper | +| --- | --- | --- | --- | +| SWE-MiniSandbox and AgentBay | Isolated execution environments or sessions | Environment/session management | Validation of a captured filesystem delta against the current project head. | +| DeltaBox and Shepherd | Checkpoints, rollback, forks, or replayable traces | Reversible agent/environment state | A filesystem-only publication contract or semantic merge guarantee. | +| CAID | Isolated Git worktrees managed by an orchestrator | Commit/merge plus executable verification | An orchestrator-independent runtime protocol for tool-call sessions. | +| CoAgent | Shared mutable state with order-filtered reads and speculative effects | Repair and reordering at quiescence | Private copy-on-write execution followed by publish-or-reject. | +| Claim Plane | Intent-bound execution scope and authority | Admission, fencing, and scope promotion | The operating-system/runtime mechanics of private overlays and capture. | +| Palantir and Crystal | Developer-private branches/workspaces | Awareness and speculative conflict diagnosis | Atomic data publication, structured rejection, or runtime session lifecycle. | +| Verified Three-Way Program Merge | Program variants | Semantic conflict-freedom reasoning | The bounded line-oriented merge in this implementation. | +| Union mounts | Writable upper over shared lower namespace | Copy-up and whiteouts | Leased version history, active-head validation, and multiwriter reconciliation. | +| Optimistic concurrency control | Tentative work | Validation and abort/restart | Filesystem serializability, semantic correctness, or fairness. | + +The matrix is intentionally asymmetric. It identifies established pieces from +which Ephemeral Sandbox composes its design, then confines the paper claim to +the source-defined composition: leased LayerStack history, a private executable +view, captured-delta validation, bounded reconciliation, and one controlled +durable transition. The paper does not claim ownership of the constituent +mechanisms. diff --git a/ephemeral-sandbox-v1/main.pdf b/ephemeral-sandbox-v1/main.pdf index 5baaf95..2482f57 100644 Binary files a/ephemeral-sandbox-v1/main.pdf and b/ephemeral-sandbox-v1/main.pdf differ diff --git a/ephemeral-sandbox-v1/main.tex b/ephemeral-sandbox-v1/main.tex index 1620a06..34c975c 100644 --- a/ephemeral-sandbox-v1/main.tex +++ b/ephemeral-sandbox-v1/main.tex @@ -3,40 +3,35 @@ \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage[margin=1in]{geometry} +\usepackage{graphicx} +\graphicspath{{sections/}} +\usepackage{float} \usepackage[hidelinks]{hyperref} -\title{Ephemeral Sandbox: A LayerStack Workspace OS for Parallel Coding Agents\\ - \large\textit{Provisional working title}} -\author{\texttt{[AUTHOR INFORMATION NEEDED]}\\ - \texttt{[AFFILIATION INFORMATION NEEDED]}} -\date{Draft manuscript scaffold} +\title{Ephemeral Sandbox: Private LayerStack Workspaces with Controlled Publication for Coding Agents} +\author{Anonymous authors} +\date{} \begin{document} \maketitle \begin{abstract} -\noindent\textbf{Provisional design-first draft---not submission text.} -Agent systems can fan out intelligence, but conventional operating-system -workspaces provide no agent-level protocol for fanning concurrent writes back -into one durable project. Coding-agent teams and swarms inspect changing code, -perform multi-file edits, run tests, and start services; sharing one mutable -workspace permits live-state interference, while copying workspaces alone -defers integration. We present Ephemeral Sandbox, a workspace OS organized -around LayerStack: one durable project history, many private executable -sessions, and controlled publication. A sessionless -\texttt{exec\_command} tool call receives an implicit isolated session over a -leased LayerStack snapshot. When its command ledger drains, the runtime -captures the private overlay, reconciles the complete delta against the current -head, and either publishes one new layer or returns a structured rejection. -Eligible text divergence can use a bounded three-way merge; structural, -binary, oversized, or conflicting changes reject. Explicit sessions let -multiple command and file tool calls share private state before publication or -discard. Source and correctness tests establish these mechanisms, but not -formal security, semantic merge correctness, process-state rollback, resource -coordination, or improved multi-agent throughput. Whether Ephemeral Sandbox -raises the workload-dependent useful-work concurrency ceiling remains an -empirical question. +Coding agents need private executable state while their resulting changes still +need to join one durable project. A shared mutable workspace permits +live-state interference, while copied workspaces defer integration to an +external process. We present Ephemeral Sandbox, a runtime publication protocol +organized around LayerStack: leased project history, private executable +workspace sessions, capture of a filesystem delta, current-head validation, +bounded reconciliation, and a controlled publish-or-reject transition. A +sessionless command receives an implicit isolated session; explicit sessions +allow several command and file operations to share private state before a +publication attempt or discard. The source-defined mechanism separates atomic +data publication from attribution and cleanup. A frozen local campaign reports +startup, public-CLI, and selected resource observations for one pinned +implementation and fixture. The paper does not claim semantic merge +correctness, process-state rollback, security, cross-platform behavior, +comparative advantage, or improved team-level throughput. \end{abstract} \input{sections/01-introduction} @@ -47,6 +42,7 @@ \input{sections/06-lifecycle-recovery} \input{sections/07-implementation-interface} \input{sections/08-evaluation} +\input{sections/generated_results_tables} \input{sections/09-limitations-related-work} \input{sections/10-conclusion} diff --git a/ephemeral-sandbox-v1/numeric_evidence.json b/ephemeral-sandbox-v1/numeric_evidence.json new file mode 100644 index 0000000..0b5b04b --- /dev/null +++ b/ephemeral-sandbox-v1/numeric_evidence.json @@ -0,0 +1,2606 @@ +{ + "schema_version": "ai-research-writing/numeric-evidence-v2", + "entries": [ + { + "aggregate": "identity", + "note": "table1.cpu.logical_processors; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/logical_processors\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.cpu.logical_processors" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 48.0 + }, + { + "aggregate": "identity", + "note": "table1.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/recorded_run_environment/host/total_memory_bytes\"}; units=bytes->bytes; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.memory_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 137438953472.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.memory_bytes; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/memory_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.memory_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 512.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.pids_limit; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/pids_limit\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.pids_limit" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 256.0 + }, + { + "aggregate": "identity", + "note": "table1.sandbox_limits.vcpus; upstream=environment-preflight.txt#{\"pointer\":\"/sandbox_limits/vcpus\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.sandbox_limits.vcpus" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1.0 + }, + { + "aggregate": "identity", + "note": "table1.seed; upstream=fixture-manifest.json#{\"pointer\":\"/identity/seed\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.seed" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 20260712.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.measured; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/measured_trials\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.measured" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.trials.warmups; upstream=expanded-plan.json#{\"invariant\":\"all equal\",\"pointer\":\"/data/cells/*/protocol/warmups\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.trials.warmups" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 2.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.file_count; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/file_count\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.file_count" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 4000.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.logical_mib; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/logical_bytes\"}; units=bytes->MiB; aggregation=identity {\"divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.logical_mib" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table1.workspace_fixture.maximum_depth; upstream=fixture-manifest.json#{\"pointer\":\"/identity/fixture/maximum_depth\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table1.workspace_fixture.maximum_depth" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1659.8107 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1749.7388 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1794.9019450000008 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_sandbox.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:d479308c8b3bd02e2d89919bafcd824d9b289bab41d2d40ba1cb4ca3968bf6b2\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_sandbox.none.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.6185790825907366 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 32.89655 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 35.6847 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 36.096376 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 30.359842997206 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 109.11905 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 120.107005 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 131.58766000000003 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.create_workspace.none.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.create_workspace.none.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 45.3204753420739 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table2.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table2.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 27.23455 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.937635 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 32.66597900000007 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8f45e06d13b98d71a0cc703f215771e8f949417e18593feedf485b08be865cbf\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 36.44471318802348 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 45.44085 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 50.973254999999995 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 56.30326700000011 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.fixture_read.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:45ca26b8ebec93c4c0f5d17e4e4477a5ab95d63fa24d1dfd0254a40e740e35fb\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.fixture_read.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 108.88291336149418 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 26.71865 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.29143 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.996465000000022 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 37.28339526491812 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 45.9976 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 53.281615 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 60.09194300000015 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.exec_command.noop.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.exec_command.noop.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 107.61432697082469 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 48.93455 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 53.328845 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 63.09657600000165 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:315abad68138d27b23e3e71b10e3d681d495613af62276fe5ec901852dfabb55\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 20.171524365907672 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 187.35685 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 208.008075 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 294.3917890000002 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.262144.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 26.357148435723182 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 38.6474 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 42.61549 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 46.87779200000013 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8de2d3ce51e15f97f3c760fe8ffd328a24eb1033c7de7ce88afccca8bf5b565e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 25.608367032181523 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 134.61035 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 141.75804 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 146.72477300000017 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_edit.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ccfc5b51f0bd7c3b4ea211621c4c95af8a40200ebe886fc2fb7513b0e500726e\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_edit.4096.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 37.216098524026926 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 20.7811 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 24.366515 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 25.48393800000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:1d46f2fd658c9ef9840dedba040ab4ac59474ea716140a617837580625ac65b0\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 47.68014601293585 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 37.78825 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 57.79372999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 60.15640100000002 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.262144.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 125.54290904967944 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 11.4248 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 15.772725 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 19.96439500000001 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:22e51aa86c53171ed08483202c70fd8275700d0232908737242749a59d59cc16\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 84.9999254477021 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 26.575 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 30.591984999999998 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 39.73821200000005 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_read.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:08051500fba8d2e8a3bf66bc2650b93d4bd03042bb85b316916c988c9f38c185\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_read.4096.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 185.30259988470985 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 45.06585 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 49.94722 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 53.543718 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:4b9f88a3ef05291334cd2aa7c5a5248d5c380a6bab530ae7966f132bb3b6fd75\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 22.02875636155663 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 125.9685 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 137.255055 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 147.37442700000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.262144.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.262144.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 39.354743449708884 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 38.8126 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 41.38427999999999 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 51.56190400000005 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c1.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:8bf5249d7df59689f7b23a7a3764aeda8d82b31170b332dde620f2e81fe11ee6\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c1.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 25.551107882015895 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p50_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p50 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.5,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p50_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 108.75565 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p95_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p95 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.95,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p95_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 111.979635 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.p99_ms; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=linear_quantile_p99 {\"interpolation\":\"linear\",\"position\":\"(n-1)*q\",\"probability\":0.99,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.p99_ms" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 116.06709000000002 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.samples; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"available_n\",\"metric_id\":\"batch_makespan_ns\",\"trial_scope\":\"reportable_measured\"}; units=count->count; aggregation=identity {}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.samples" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 100.0 + }, + { + "aggregate": "identity", + "note": "table3.file_write.4096.c5.throughput_ops_s; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:b87af3269b33724949335fca092c5b6adb183e4894bff8806b801ac84db783ce\",\"field\":\"raw_points[].value\",\"metric_id\":\"throughput_ops_s\",\"trial_scope\":\"reportable_measured\"}; units=operations_per_second->operations_per_second; aggregation=arithmetic_mean {\"denominator\":100}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table3.file_write.4096.c5.throughput_ops_s" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 45.9991049224615 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 25.5 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0108203125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 30.4712 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 28.171875 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:2f8ff3901a35439793e55958d898b63fb9010bc2f1c016a59d688b9b9157f2fd\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c1.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.00390625 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 34.125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0270703125 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 79.50166 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 43.52734375 + }, + { + "aggregate": "identity", + "note": "table4.create_workspace.none.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:46dca2a11cd160fb27148faef398b025082c5a87cbdf187c58b26045fac29c21\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.create_workspace.none.c5.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.01953125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 13.875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.001171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 35.00335 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 20.03125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c1.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:49abffc1163b5d9c8d96c8c091f2a31d46f8303444af6d2b9a78ea9a4e92b8f5\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c1.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 13.125 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0026171875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 63.07729 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 21.5546875 + }, + { + "aggregate": "identity", + "note": "table4.exec_command.noop.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:328ad82e72b0d80ac2466f6d7b8610fb125d828ea73a4efc98c7bb317beeab6d\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.exec_command.noop.c5.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 62.6015625 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1.2505859375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 148.73234 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 66.71484375 + }, + { + "aggregate": "identity", + "note": "table4.file_edit.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:ae045e86ae9a922cccd9f5820ba8a29db3155f5c9d8eb81129b6a3de48f7daab\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_edit.262144.c5.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1.2578125 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 14.47265625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 38.43103 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 14.9765625 + }, + { + "aggregate": "identity", + "note": "table4.file_read.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:36ce0418b24fc3acb183f7cc8c35d1a66f76dcab3aa1e6050c02a6b99f5f0603\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_read.262144.c5.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.daemon_rss_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"daemon_rss_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.daemon_rss_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 51.04296875 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_read_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_read_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_read_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 0.0 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_block_write_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_block_write_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_block_write_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1.250625 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_cpu_time_ns; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_cpu_time_ns\",\"trial_scope\":\"reportable_measured\"}; units=nanoseconds->milliseconds; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1000000}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_cpu_time_ns" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 95.11521 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.sandbox_memory_peak_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"sandbox_memory_peak_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=maximum {\"denominator\":null,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.sandbox_memory_peak_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 54.26953125 + }, + { + "aggregate": "identity", + "note": "table4.file_write.262144.c5.upperdir_bytes; upstream=report.json#{\"artifact\":\"report.json\",\"cell_id\":\"sha256:bc22c4d681e3996f98d860c08a3bdc58b7cdb1152be031dcd9f213db570e7aaa\",\"field\":\"raw_points[].value\",\"metric_id\":\"upperdir_bytes\",\"trial_scope\":\"reportable_measured\"}; units=bytes->MiB; aggregation=arithmetic_mean {\"denominator\":100,\"unit_divisor\":1048576}; eligibility=frozen_final_candidate", + "representations": [ + "raw" + ], + "selector": { + "column": "value", + "kind": "csv", + "where": { + "evidence_id": "table4.file_write.262144.c5.upperdir_bytes" + } + }, + "source": "experiments/analysis/final-v11-019fb86c-tables-a/numeric-provenance.csv", + "tolerance": 0.005, + "value": 1.2578125 + } + ] +} diff --git a/ephemeral-sandbox-v1/paper_skeleton.md b/ephemeral-sandbox-v1/paper_skeleton.md index eaf02f8..e6115bd 100644 --- a/ephemeral-sandbox-v1/paper_skeleton.md +++ b/ephemeral-sandbox-v1/paper_skeleton.md @@ -1,467 +1,108 @@ -# Ephemeral Sandbox v1: paper and work skeleton +# Ephemeral Sandbox v1: final paper skeleton -Status: working manuscript skeleton. Design claims are provisionally grounded in source baseline [`b22862550e0a7cb4fe61ce581831e9244cc492b5`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5). No experiment may become a paper result until rerun against the annotated `paper-v1-freeze` tag with complete provenance. +**Scope status:** reconciled on 2026-08-02. Design/interface claims refer to +product commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`; quantitative claims +refer only to final EXP1 v1.1 output selectors. -Current readiness, active lanes, synchronization gates, and immediate actions are tracked in [`progress.md`](./progress.md). +## Core contract -## Core paper contract +- **Thesis:** Ephemeral Sandbox makes a private execution-to-shared-history + publication boundary explicit through leased LayerStack workspaces, capture, + current-head reconciliation, and active-manifest publication; EXP1 v1.1 + describes bounded public-CLI observations for one treatment. +- **System contribution:** a source-defined runtime protocol and interface, + not a claim to invent union mounts, leases, optimistic validation, or merge. +- **Completed evaluation question:** what startup, public operation, and + preregistered resource observations occur in the fixed EXP1 v1.1 treatment? +- **Claims to avoid:** competitive results, security/correctness evaluation, + broad reliability, useful-work, agent productivity, and extrapolation beyond + the 19-cell campaign. -**Recommended title:** *Ephemeral Sandbox: A LayerStack Workspace OS for Parallel Coding Agents* - -**Research question:** How can a sandbox runtime raise the practical concurrency ceiling for coding agents by giving each agent a private, executable workspace over shared project history, then reconciling accepted changes through well-defined publication semantics? - -**Thesis:** Agent products can fan out intelligence, but conventional operating-system workspaces provide no agent-level protocol for fanning concurrent writes back into one durable project. Ephemeral Sandbox makes the workspace session the boundary between private execution and shared LayerStack history, with capture, current-head reconciliation, and atomic data publication. Source and correctness tests can establish the mechanism; whether it raises useful-work concurrency requires measurement. - -**Method phrase:** **One durable project truth, many private executable sessions, controlled publication.** - -**Evidence boundary:** The paper does not claim a universal concurrency threshold, generic speedup, formal sandbox security, universal egress denial, process-state rollback, semantic merge correctness, or a complete agent coordination/resource plane. - -## Manuscript skeleton - -### 0. Title and abstract - -**Purpose:** State the artifact, systems problem, mechanism, and evidence boundary in 150–200 words. - -**Include:** - -- OS/runtime abstraction mismatch for autonomous agent teams and swarms; -- private LayerStack workspace sessions; -- capture, merge/reject, and atomic data publication; -- explicit statement that concurrency-ceiling improvement is measured, not assumed. - -**Evidence:** [`paper_story.md`](./paper_story.md), C1–C5 in [`claim_evidence_map.md`](./claim_evidence_map.md). - -**Gate:** Rewrite last, after frozen evaluation results and limitations are complete. +## Section plan ### 1. Introduction -**Purpose:** Make the situation, issue, and bounded opportunity unmistakable. - -**Paragraph plan:** - -1. Agent systems can fan out research, review, planning, and separated modules, but fine-grained interdependent writers remain difficult to integrate. -2. Multi-agent coding is immature for model and coordination reasons; runtime-created interference should not become an artificial ceiling. -3. Conventional OS units—users, processes, files, descriptors, locks, and ports—do not encode agent task, project base, private session, resource ownership, or publish/reject state. -4. Structured teams and exploratory swarms encounter different dependency, conflict, staleness, resource, and integration limits. -5. Define useful-work rate and the workload-dependent concurrency ceiling. -6. Introduce the LayerStack workspace-session insight and state the limitations. -7. List contributions, each paired with its evidence type. - -**Evidence:** Motivation claims M0–M6; source contributions C1–C4; evaluation contribution C5. - -**Gate:** Do not say “raises the ceiling” in past tense until C5 has frozen results. - -### 2. Problem statement and system model - -**Purpose:** Define precisely what is shared, what is private, and what success means. - -**Entities:** - -- durable project head and ordered LayerStack history; -- leased base manifest/layers; -- implicit or explicit workspace session; -- private filesystem overlay and namespace-scoped execution; -- captured candidate changeset; -- publish, reject, retry, discard, and published-but-not-closed outcomes; -- management, runtime, and observability clients; -- coding worker, structured team, exploratory swarm, and orchestrator. - -**State split:** - -- **Private execution state:** uncommitted files, commands, processes, and experiments within a session. -- **Shared durable state:** accepted LayerStack history and active manifest. -- **Public operational state exposed by v1:** request/scope, lifecycle results, bounded traces/events/resources/topology/LayerStack views. -- **Coordination state not established by v1:** task intent, agent ownership, handoffs, port/service leases, resource budgets, and integration scheduling. - -**Useful-work definition:** For workload \(W\), runtime \(R\), agent configuration \(A\), and worker count \(n\), measure verification-passing, durably accepted contribution units per wall time or cost. The concurrency ceiling is the workload-specific region in which marginal workers no longer improve that rate. - -**Failure taxonomy:** shared-file visibility, multi-file inconsistency, stale bases, Git/worktree state, service/port collisions, compute/I/O saturation, lifecycle residue, audit gaps, lossy filesystem observation, and semantic integration. - -**Gate:** Separate source-proven runtime properties from workload hypotheses. - -### 3. Design goals and non-goals - -**Goals:** - -1. Stable private execution over changing shared history. -2. Faithful filesystem change capture. -3. Conflict-aware all-or-none data publication. -4. Atomic visibility of an accepted LayerStack head transition. -5. Explicit lifecycle and partial-failure outcomes. -6. Role-separated operational contracts for orchestration. -7. Measurable isolation, publication, scaling, and useful-work behavior. - -**Non-goals:** - -- formal security proof or complete noninterference; -- universal network-egress denial; -- process checkpoint/rollback; -- semantic merge correctness; -- general task decomposition, communication, or scheduling; -- current v1 port leases, service discovery, intent negotiation, handoffs, or resource budgets; -- Windows OverlayFS/reflink or generic cross-platform durability. - -### 4. Design overview - -**Purpose:** Explain the whole mechanism before implementation detail. - -**Pipeline:** - -1. Select and lease a LayerStack base. -2. Project shared lower layers with a unique private upper/work pair. -3. Execute tool calls in the session's namespace-scoped workspace. -4. Capture the complete private filesystem delta. -5. Reconcile the delta against the current durable head. -6. Merge eligible text or reject the unresolved data changeset as a whole. -7. Stage, sync, promote, and atomically replace the active manifest. -8. Attribute best-effort, close, retry, or report partial cleanup failure. - -**Key distinction:** Data publication is atomic at the active-manifest visibility boundary; audit, accounting, and cleanup are later fallible phases. - -**Evidence:** System stages in [`paper_story.md`](./paper_story.md); source map in [`project_inventory.md`](./project_inventory.md). - -### 5. LayerStack and workspace-session design - -#### 5.1 Shared version history - -- layer/manifest representation; -- content digests and append-oriented history; -- active head and base selection; -- exact immutability wording. - -#### 5.2 Leases, compaction, and snapshot stability - -- manifest/layer lease acquisition; -- leased logical snapshot; -- squash/GC constraints; -- remount/substitution and restart questions. - -#### 5.3 Private workspace projection - -- shared lower layers; -- unique upper/work directories; -- OverlayFS whiteouts and opaque directories; -- Linux/platform boundary. - -#### 5.4 Tool-call execution - -- implicit session for sessionless `exec_command`; -- explicit session for multiple command/file calls; -- user, mount, PID, and optional network namespaces; -- shared-network default for implicit command sessions; -- sessionless file-operation qualifier. - -#### 5.5 Capture - -- writes, deletions, symlinks, empty and opaque directories; -- ordinary `.wh.*` names versus kernel whiteouts; -- protected drops and unsupported entry behavior. - -**Evidence:** C1, D1–D4 in [`claim_evidence_map.md`](./claim_evidence_map.md). - -### 6. Conflict-aware publication - -#### 6.1 Planning and validation - -- verify base and changeset; -- route/protect paths; -- fingerprint base/current/candidate states. - -#### 6.2 Reconciliation - -- unchanged-current fast path; -- eligible bounded three-way text merge; -- binary, oversized, invalid-UTF-8, structural, delete/modify, and conflicting-edit rejection; -- no semantic compatibility claim. - -#### 6.3 Durable commit - -- stage layer; -- sync and promote; -- construct manifest; -- atomic active-manifest replacement; -- no partial data changeset visibility. - -#### 6.4 Partial failures - -- precommit failure and retry; -- reject and retry; -- no-op publication; -- commit followed by audit failure; -- published-but-not-closed cleanup failure. - -**Evidence:** C2, C4, D5–D9. - -### 7. Operational interface - -**Purpose:** Show how an orchestrator invokes and observes the runtime without conflating the CLI contract with runtime correctness. - -#### 7.1 Management client - -- host/fleet lifecycle, selection, compaction, and export; -- 8 baseline operations, regenerated at freeze. - -#### 7.2 Runtime client - -- sandbox-scoped command, file, attribution, and session lifecycle; -- 10 baseline operations, regenerated at freeze. - -#### 7.3 Read-only observability client - -- snapshot, trace, events, resources, daemon, topology, cgroup, and LayerStack; -- 8 baseline operations, regenerated at freeze. - -#### 7.4 Shared protocol - -- sandbox scope; -- request IDs; -- connection and token discovery; -- newline-framed JSON; -- stdout/stderr and exit-status contracts; -- catalog-derived help. - -#### 7.5 Boundary - -- smaller role-specific surfaces are an interface/least-exposure property; -- they are not alone an authorization or security guarantee; -- v1 is not a full task, intent, resource-lease, service-discovery, or handoff plane. - -**Evidence:** [`cli_contract_matrix.md`](./cli_contract_matrix.md), C3, D10–D13. - -### 8. Implementation - -**Purpose:** Record source organization and engineering choices without turning modules into contributions. - -**Cover:** - -- Rust crates and process boundaries; -- LayerStack storage and publish locking; -- workspace and OverlayFS lifecycle; -- namespace holder/runner pattern; -- operation services and command ledger; -- catalog/projection/help generation; -- observability and audit storage; -- source-derived operational complexity and the distinction between logical and physically allocated storage; -- platform assumptions and dependencies. - -**Gate:** Every source citation must be changed from the baseline commit to `paper-v1-freeze`. - -#### 8.1 Operational cost model - -Define \(L\) layers, \(S\) live sessions, \(U\) upperdir entries, \(C\) captured changes, \(F\) validated paths, \(B_p\) published bytes, \(Q\) concurrent publishers, and \(R\) layer references retained across leases. Summarize: - -- \(O(L)\) lease/session setup metadata and \(O(R)\), commonly \(O(SL)\), live lease-manifest metadata; -- metadata-only capture over the private upperdir, including per-directory sorting; -- repeated layer/path/fingerprint work during planning and resolution; -- byte-proportional hashing/copy/sync plus manifest work inside the serialized publication path; -- the current line-merge algorithm's edit-distance-dependent time and retained-trace space; -- squash/GC costs and lease-induced history retention. - -These are source-derived drivers, not latency or storage results. The authoritative working analysis is [`complexity_and_evolution.md`](./complexity_and_evolution.md). - -### 9. Evaluation methodology - -#### RQ1: Does each session observe a stable private executable workspace? - -**Protocol:** concurrent leased sessions, controlled bases, cross-session file/process/network probes, compaction during leases, cleanup and restart. - -**Measures:** visibility matrix, namespace identities, escaped-state failures, exact base/manifest/lease IDs, cleanup outcomes. - -#### RQ2: Does publication implement the documented merge/reject and visibility semantics? - -**Protocol:** disjoint stale writes, same-path merge, binary/structural conflicts, delete/modify, protected paths, concurrent publishers, storage and cleanup fault injection. - -**Measures:** accept/reject class, changed paths, unrelated-path leakage, pre/post manifest, layer digest, retry outcome, partial failures. - -#### RQ3: How do latency, storage, and resources scale? - -**Protocol:** 1/2/4/8/... workers on fixed no-op, command, capture, publish, and conflict workloads with warmups and repeated measured runs. Independently vary layer depth, live-lease count and age, upperdir entry structure, changed-path count, payload bytes, merge line/edit-distance shape, and storage backend. - -**Measures:** session start, execution, capture, publish, squash/GC, and end-to-end latency distributions; CPU, RSS/PSS, I/O, logical/allocated/shared/exclusive storage, writer wait/hold time, queueing, failures, and residual history after lease release. - -#### RQ4: Does the runtime move useful-work concurrency under controlled coding workloads? - -**Baselines:** shared mutable directory, Git worktrees, Ephemeral sessions. - -**Workload families:** - -- **Structured team:** explicit roles, task dependencies, handoffs, shared interfaces/services, and an integration lane. -- **Exploratory swarm:** redundant or competing proposals, hot-path overlap, rapidly changing heads, and result selection. - -**Controls:** same model, prompts, budgets, repositories, tool versions, test harness, initial commits, worker counts, seeds/repeats, and integration acceptance. - -**Primary measure:** verification-passing, durably accepted contribution units per wall time and resource cost. - -**Secondary measures:** conflicts, retries, stale bases, duplicate work, integration/selection latency, clean textual publishes that fail tests, resource saturation, and human/lead repair if present. - -#### RQ5: How complete is operational attribution and recovery? - -**Protocol:** correlated accept, reject, retry, audit failure, cleanup failure, cancellation, and daemon restart. - -**Measures:** request/base/path/result linkage, missing or `unknown` attribution, recovery time, residual sessions/processes/resources. - -**Provenance required for every number:** source tag/commit, benchmark commit, dirty status, OS/kernel/filesystem, hardware, cgroup/runtime settings, toolchain, binary/image digests, workload commit, exact command/environment, seeds/repeats, raw events/logs, exclusions, and analysis commit. - -### 10. Results - -Do not draft numerical prose until frozen data exists. Reserve the following result structure: - -1. isolation correctness matrix; -2. publication and injected-failure matrix; -3. latency/resource scaling curves; -4. useful-work curves by baseline and workload family; -5. conflict, retry, integration, and selection decomposition; -6. audit/recovery completeness; -7. negative results and the observed limiting regime. - -Every result paragraph must state workload, runtime, model/agent configuration, worker range, metric, uncertainty, and exact evidence artifact. - -### 11. Related work - -Organize by: - -1. agent sandboxes and isolated execution; -2. concurrent coding-agent coordination and integration; -3. reversible and versioned agent state; -4. private-workspace awareness and semantic merge; -5. union filesystems, copy-on-write, leases, and optimistic concurrency; -6. multi-agent coding benchmarks and product motivation. - -**Position:** Ephemeral's bounded niche is a model- and orchestrator-independent runtime publication protocol: private executable sessions over leased shared history, followed by source-defined capture, current-head reconciliation, and atomic data publication. - -**Evidence:** [`references/related_work.md`](./references/related_work.md). - -### 12. Limitations and discussion - -State prominently: - -- workspace isolation is necessary for one failure class, not sufficient for collaboration; -- clean text reconciliation is not semantic correctness; -- shared networking remains possible and no universal egress guarantee is made; -- filesystem capture is not process rollback; -- audit is best-effort and not transactionally coupled to data publication; -- resource ownership/admission and general agent coordination remain open; -- publication is serialized and v1 keeps reconciliation plus byte hashing/copy/sync inside the writer critical section; -- per-lease manifest metadata scales with retained layer references, and long-lived leases can retain obsolete history; -- the 8 MiB text-merge byte limit does not eliminate the current diff trace's pathological line/edit-distance memory envelope; -- logical copy-on-write behavior is not a claim about physical extent sharing or cheap publication; -- the platform and crash claims are limited to the tested Linux environment; -- any concurrency result is workload- and runtime-specific. - -Discuss the larger ambition separately from demonstrated results: the runtime should not force agent teams or swarms into low concurrency before task dependencies, agent quality, verification, or physical resources become the actual limit. - -#### 12.1 Future evolution - -Present LayerStack 2.0 only as a candidate response to the byte-copy/storage term: a qualified same-filesystem storage domain plus capability-gated reflink publication/copy-up, with a correctness-preserving copy fallback. Do not call reflink \(O(1)\), do not import preregistered thresholds as results, and disclose the pinned stock Windows Docker Desktop/WSL 2 feasibility failure (`FICLONE` `errno=95`). Also identify shorter writer critical sections, shared/compact lease metadata, indexed layered lookups, and independently bounded merge CPU/RSS as future targets. Evidence and wording boundaries are in [`complexity_and_evolution.md`](./complexity_and_evolution.md). - -### 13. Conclusion - -Return to the evidence: - -- what the tagged source implements; -- what correctness and fault tests establish; -- what the measured workloads show; -- which limits remain. - -Closing direction: multi-agent coding should be limited by the work and the agents, not prematurely by a shared mutable workspace and an undefined merge-back procedure. - -## Figure and table skeleton - -| ID | Type | Message | Evidence class | Status | -|---|---|---|---|---| -| Figure 1 | Teaser/problem diagram | Teams and swarms can fan out work but collide at workspace, resource, and integration planes; LayerStack sessions introduce a controlled publication boundary. | Concept-method | Planned | -| Figure 2 | Architecture | One durable LayerStack history, many leased private session views, capture, reconciliation, and one accepted head transition. | Concept-method | Planned | -| Figure 3 | State machine | Create → execute → capture → publish/reject → retry/discard/close, including published-but-not-closed. | Concept-method grounded in source | Planned | -| Table 1 | Design comparison | Shared directory vs worktree vs Ephemeral session across visibility, base identity, services/resources, merge-back, and attribution. | Mechanism comparison | Planned | -| Table 2 | CLI contract | Final management/runtime/observability operations and scopes from the tagged projections. | Source evidence | Baseline matrix exists; regenerate | -| Table 3 | Publication matrix | Exact merge/reject behavior by path/change class. | Source plus correctness tests | Planned | -| Table 4 | Operational cost model | Source-derived stage, scaling variables, critical section, live/peak space, and required measurement. | Source analysis, not a measured result | Drafted in `complexity_and_evolution.md` | -| Table 5 | Experimental setup | Hardware, software, workloads, agents, worker counts, controls, seeds, and metrics. | Experimental provenance | Blocked on freeze | -| Table 6 | Correctness/fault results | Isolation, publication, restart, audit, and cleanup outcomes. | Experimental result | Not run | -| Figure 4 | Useful-work scaling | Accepted verified progress versus workers for each baseline and workload family. | Experimental result | Not run | -| Figure 5 | Cost decomposition | Conflict, retry, integration, verification, and resource costs across concurrency. | Experimental result | Not run | - -Generated concept figures communicate the mechanism only; they cannot serve as experimental evidence. Numerical figures and tables must be generated deterministically from preserved raw data and analysis code. - -## Work packages - -### WP1. Freeze and provenance - -**Tasks:** resolve paper-specific source fixes, create annotated `paper-v1-freeze`, record commit/tag object/dirty status/toolchain/lockfile/binary digests, and relink all source claims. - -**Done when:** every source claim names the freeze commit and the checkout/build provenance is archived. - -### WP2. Interface freeze - -**Tasks:** regenerate the three CLI catalogs/help snapshots, update operation counts and scopes, archive raw outputs, and document website drift. - -**Done when:** [`cli_contract_matrix.md`](./cli_contract_matrix.md) matches the tagged binaries and contract tests pass separately from runtime tests. - -### WP3. Runtime correctness and fault testing - -**Tasks:** run the isolation, lease/compaction, capture, publication, concurrent-publisher, injected-storage-fault, audit, cleanup, cancellation, and restart matrices on the paper Linux platform. +State the workspace/publication gap; introduce the bounded protocol; present +the source-grounded contributions and completed RQ3 evidence; state the narrow +evaluation boundary on the first page. Do not promise answers to RQ1/RQ2/RQ4/ +RQ5 or describe source/test inspection as an empirical result. -**Done when:** commands, logs, configurations, and exact outcomes are archived and mapped to C1, C2, and C4. +### 2. Goals and non-goals -### WP4. Scaling and resource evaluation +Retain stable private execution, controlled data publication, lifecycle +outcomes, and role-separated operations as implementation goals. State the +security, semantic merge, process rollback, egress, coordination, and +cross-platform boundaries. -**Tasks:** run controlled session/start/capture/publish/conflict microbenchmarks across worker counts and collect latency, CPU, memory, I/O, storage, queueing, and failures. +### 3. System model and invariants -**Done when:** repeatable raw runs, uncertainty analysis, and deterministic plots exist. +Define LayerStack history, lease, private upper/work pair, implicit command +session, explicit session, candidate changeset, reconciliation, publication, +and lifecycle outcomes. Qualify that data visibility, attribution, and cleanup +are different phases. -### WP5. Multi-agent workflow evaluation +### 4. Workspace execution -**Tasks:** finalize structured-team and exploratory-swarm workloads; run shared-directory, Git-worktree, and Ephemeral baselines with matched models, budgets, prompts, tools, worker counts, and seeds. +Describe source-defined OverlayFS projection and holder/runner execution. +Distinguish sessionless `exec_command` from sessionless file paths and shared +from isolated networking. State Linux/OverlayFS support accurately. -**Done when:** useful-work, conflict, retry, integration, selection, verification, and resource results have complete provenance. +### 5. Capture and publication -### WP6. Analysis and claim update +Describe protected drops, routing, current-head validation, narrow merge +eligibility, whole-candidate rejection, staging, and manifest replacement. +Do not call the protocol semantic merge correctness or a complete transaction. -**Tasks:** compute metrics, uncertainty, negative results, cost decomposition, and the observed workload-specific ceiling; update C5 and every quantitative manuscript sentence. +### 6. Lifecycle and recovery -**Done when:** every number is recomputable from raw data and the claim map labels its strength correctly. +Cover retryable precommit failure, discard, no-op, committed publication, +published-but-not-closed, holder exit artifacts, and conservative cleanup. +State restart/lease/substitution behavior as unmeasured. -### WP7. Manuscript, figures, and citations +### 7. Implementation and operational interface -**Tasks:** draft evidence-bearing core sections first; create source-grounded system figures and deterministic result plots; verify scholarly metadata and sentence-level support; then rewrite Introduction, Abstract, title, and Conclusion. +Use the measured-source 8/10/8 matrix. Explain all-client request IDs, endpoint +URI resolution, local named-pipe default on Windows, and TCP compatibility. +Keep the operational cost model source-derived rather than measured. -**Done when:** all citations and claims pass review and no placeholder is presented as a result. +### 8. Evaluation: Methodology and bounded results -### WP8. Skeptical review and release +Open with the completed EQ1 and state why broader RQs are not evaluated. +Describe the Windows/Docker Desktop environment, pinned image, `product_cli`, +named-pipe treatment, 19 cells, fixture, concurrency/payloads, warmups, +trials, randomized blocks, no-retry rule, correctness inclusion gate, cleanup, +and resource sampling. Explain v1.0 qualitatively, v1.1 remediation, and the +Table-1 reader compatibility erratum. Include only deterministic tables +generated from the frozen registry and selector-bound numeric prose. -**Tasks:** run source/evidence, systems novelty, experimental methodology, statistics, limitations, and reproducibility reviews; build the final paper and archive submission artifacts. +### 9. Limitations and related work -**Done when:** high-severity objections are resolved or recorded as explicit blockers and the final package builds from the frozen inputs. +Open with evaluation and system limitations, then position execution isolation, +reversible state, coding-agent coordination, conflict work, and foundational +filesystem/concurrency work from the verified literature matrix. Related Work +must not make numerical comparisons. -## Dependency order +### 10. Conclusion -1. Fix source blockers and freeze v1. -2. Freeze the operational contract. -3. Run runtime correctness/fault tests. -4. Run scaling and workflow experiments. -5. Analyze results and update claims. -6. Draft the evidence-bearing system/evaluation/results sections. -7. Produce figures/tables and verify citations. -8. Rewrite title, abstract, introduction, limitations, and conclusion. -9. Conduct skeptical review, build, and package. +Recap the source-grounded protocol and one-treatment descriptive evidence, +then restate the unmeasured correctness, baseline, generalization, recovery, +and coordination questions. -## Current blockers +## Final result assets -1. `paper-v1-freeze` does not exist. -2. No paper-ready isolation, publication/fault, scaling/resource, or multi-agent workflow dataset exists. -3. Attribution is best-effort after data commit rather than atomically coupled to publication. -4. Explicit/implicit protected-drop behavior needs maintainer confirmation. -5. Lease/substitution and daemon-restart behavior needs fault testing. -6. Claim Plane requires deeper sentence-level novelty comparison. -7. Final structured-team and exploratory-swarm workloads, acceptance units, and integration policy must be fixed before measurement. +| Asset | Source | Disposition | +|---|---|---| +| Environment/provenance table | frozen Table 1 | deterministic LaTeX conversion | +| Startup table | frozen Table 2 | deterministic LaTeX conversion | +| Public CLI table | frozen Table 3 | deterministic LaTeX conversion | +| Resource table | frozen Table 4 | deterministic LaTeX conversion; call its workspace column `upperdir` delta | +| Numerical figure | none | omitted; tables retain exact values without adding a decorative plot | +| Four concept figures | prompt sources plus reviewed PNGs | repair, regenerate, waive, or remove each issue after final-width color/grayscale QA | -## Document map +## Claims moved out of the core -- Current readiness, workflow lanes, gates, and next actions: [`progress.md`](./progress.md) -- Paper-writing execution contract: [`lanes/paper-writing.md`](./lanes/paper-writing.md) -- Experiment protocol, freeze, run, and evidence-handoff contract: [`lanes/experiments.md`](./lanes/experiments.md) -- Story, title, abstracts, introduction, contributions, and claim boundaries: [`paper_story.md`](./paper_story.md) -- Source, tests, documentation, and evidence inventory: [`project_inventory.md`](./project_inventory.md) -- Source-derived operational interface: [`cli_contract_matrix.md`](./cli_contract_matrix.md) -- Claim strength and evidence requirements: [`claim_evidence_map.md`](./claim_evidence_map.md) -- Related-work metadata and positioning: [`references/related_work.md`](./references/related_work.md) +Former RQ1 isolation, RQ2 publication/fault behavior, RQ4 competitive +useful-work scaling, and RQ5 attribution/restart recovery are limitations and +future evaluation. The source-derived cost-model variables (layer depth, +leases, upperdir shape, publication bytes/queueing, merge shape, and retained +history) are hypotheses for future measurement, not EXP1 conclusions. diff --git a/ephemeral-sandbox-v1/paper_state.json b/ephemeral-sandbox-v1/paper_state.json index f4d726f..7d2e1a6 100644 --- a/ephemeral-sandbox-v1/paper_state.json +++ b/ephemeral-sandbox-v1/paper_state.json @@ -1,45 +1,68 @@ { "schema_version": "ai-research-writing/paper-state-v1", "mode": "full-paper", - "stage": "drafting", + "stage": "final-review", "target_venue": "arXiv cs.OS", "main_tex": "main.tex", "bibliography": "references.bib", "required_artifacts": [ + "ARTIFACTS.md", "BUILD.md", + "REPRODUCIBILITY.md", + "REVIEWER_GUIDE.md", + "SUBMISSION.md", + "submission_readiness.md", "plan/terminology.md", + "plan/source_revalidation.md", + "plan/reverse_outline_pw4_pw7.md", + "plan/reviewer_report.md", + "plan/pw4-pw7-package-manifest.md", "figures/figure_plan.md", + "figures/concept-figure-review.md", + "literature/paper_inventory.md", + "literature/related_work_matrix.md", + "literature/positioning.md", "complexity_and_evolution.md", "experiment_inventory.md", "experiments/environment_setup.md", + "experiments/exp1-v1.1-protocol-amendment.md", "experiments/expected_tables.md", - "experiments/experiment_log.md" + "experiments/experiment_log.md", + "experiments/analysis/exp1-final-handoff.md", + "experiments/analysis/exp1-gate0-7-final-report.md", + "experiments/analysis/final-v11-019fb86c-tables-a/numeric-evidence.json", + "numeric_evidence.json", + "sections/generated_results_tables.tex", + "sections/results_numeric_bindings.md", + "citation_requests.json", + "citation_lock.json" ], "blockers": [ - "The annotated paper-v1-freeze source tag is missing; baseline source links remain provisional.", - "experiment_inventory.md now contains a focused RQ3 practical-performance draft, but it is not protocol-locked and does not replace the broader RQ1-RQ5 evaluation charter.", - "Final isolation, publication/fault, performance, resource, and useful-work measurements are absent.", - "Author names and affiliations are unresolved.", - "Publication attribution remains best-effort after data commit rather than transactionally coupled to publication.", - "Explicit versus implicit protected-drop behavior still requires maintainer confirmation.", - "Lease/substitution behavior across daemon restart still requires fault testing.", - "The source-derived cost model has not been validated over layer depth, live leases, upperdir structure, changed paths/bytes, writer queueing, merge shape, or retained history.", - "The current text-merge byte gate does not independently bound retained diff-trace memory; adversarial CPU/RSS evidence or a stricter implementation bound is required.", - "LayerStack 2.0 remains future work; its integrated A/B/C results are absent and its pinned stock Windows Docker Desktop/WSL 2 reflink feasibility cell failed.", - "Claim Plane requires a complete sentence-level novelty audit." + "Paper owner must supply confirmed author names, affiliations, intended category/venue, and any required disclosure text before external submission." ], "build": { "status": "passed", "command": "latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex", "pdf": "main.pdf", - "external_inputs": [], - "input_sha256": "414eef7f33ae33849df5ac2305606499d01871d43d1aa00a692ef1a9b5feaf16", - "pdf_sha256": "ba4963d3d5f6352e1829946290265671599432e9984e301d5626de7316435327", + "external_inputs": [ + { + "name": "Eligible EXP1 archive content tree", + "source": "experiments/runs/019fb86c-096e-7589-a0a4-a6d6ef5d7f8b", + "archive_sha256": "606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986" + }, + { + "name": "Frozen EXP1 Table-A output tree", + "source": "experiments/analysis/final-v11-019fb86c-tables-a", + "archive_sha256": "27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04" + } + ], + "input_sha256": "6f1675a8021bf6acc982d9b7dc9f6c4d5755832ca937334c6212eb15b867b6a7", + "pdf_sha256": "6ffb429c6acb0f27af6dc493cd3fcc43a1710a22cba17fb669a11129ad06e8b2", + "attestation": "executed", "exit_code": 0, - "log": "plan\\pw1-build.log", - "log_sha256": "719696c0aa0efb9fce4796efaab3cce5b27dd17563c4b60d7f4951b539aa7f30", + "log": "plan\\pw4-pw7-build-attestation.log", + "log_sha256": "cffe435373442861725c7ca42e572c0a33ea029f02d1031fa38ba731745157c1", "tool_version": "Initial Win CP for (console input, console output, system): (CP437, CP65001, CP1252)", - "attestation": "executed", - "ran_at": "2026-07-30T00:48:19.855325+00:00" + "ran_at": "2026-08-02T05:28:00.336951+00:00" } } diff --git a/ephemeral-sandbox-v1/paper_story.md b/ephemeral-sandbox-v1/paper_story.md index b72cd8d..5a6f466 100644 --- a/ephemeral-sandbox-v1/paper_story.md +++ b/ephemeral-sandbox-v1/paper_story.md @@ -1,170 +1,105 @@ # Paper story: Ephemeral Sandbox v1 -Status: design story grounded in source baseline [`b22862550e0a7cb4fe61ce581831e9244cc492b5`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5), audited 2026-07-30. The final paper must use the annotated `paper-v1-freeze` commit. No performance or multi-agent workflow result is yet paper-ready. +**Status:** final scope reconciled for PW4--PW7 on 2026-08-02. +**Source snapshot:** product commit [`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8), tag `paper-v1.1-freeze`. +**Measurement:** sole eligible EXP1 final `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b`. -The section-by-section execution plan, evidence dependencies, figure/table plan, and work packages are in [`paper_skeleton.md`](./paper_skeleton.md). +## Title -## Working titles +**Ephemeral Sandbox: Private LayerStack Workspaces with Controlled Publication +for Coding Agents** -1. **Ephemeral Sandbox: A LayerStack Workspace OS for Parallel Coding Agents** -2. **Ephemeral Sandbox: LayerStack Sessions for Parallel Coding Agents** -3. **Ephemeral Sandbox: Private Sessions over LayerStack** -4. **Ephemeral Sandbox: LayerStack Isolation and Controlled Publication** -5. **Ephemeral Sandbox: Raising Coding-Agent Concurrency with a LayerStack OS** — use only if the final evaluation demonstrates this result. - -None claims measured speedup, a universal concurrency threshold, security, or semantic merge correctness. - -## Recommended title - -**Ephemeral Sandbox: A LayerStack Workspace OS for Parallel Coding Agents** - -Three independent reviewers audited the options from source-accuracy, concurrency-evidence, and related-work perspectives. The source reviewer preferred “Private LayerStack Workspace Sessions and Controlled Publication for Parallel Coding Agents”; the other two independently proposed the near-identical “Private LayerStack Workspaces with Controlled Publication for Parallel Coding Agents.” All three rejected an unconditional “raising the ceiling” title before measurement and warned that “OS” can invite broader kernel/security expectations. - -The recommendation synthesizes that consensus with the desired systems identity without forcing every mechanism into the title. It names the artifact, LayerStack substrate, workspace-OS role, and parallel-agent domain in ten words. The thesis and abstract carry the private-session and controlled-publication details. Options 2–4 are progressively more mechanism-specific; option 5 becomes eligible only if frozen experiments show higher workload-specific useful-work concurrency against named baselines. +The title describes a protocol and does not imply a general-purpose OS, +security guarantee, performance win, or improved agent productivity. ## One-sentence thesis -Ephemeral Sandbox is a LayerStack workspace OS that executes each sessionless command tool call in an isolated workspace session, optionally groups multiple tool calls in an explicit session, and returns accepted filesystem changes to shared history only through capture, current-head reconciliation, and atomic data publication; source and tests establish the mechanism, while whether it raises the useful-work concurrency ceiling remains unmeasured. - -## Problem gap - -Conventional operating systems expose collaboration primitives for human-operated applications and cooperative processes: users, files, descriptors, locks, process identifiers, ports, permissions, and mutable namespaces. They do not expose a first-class coding-agent role, delegated task, workspace-session boundary, base revision, resource ownership record, or publish/reject event. A human developer can often reconstruct these relationships from a terminal, editor, Git history, and team conversation. An autonomous multi-agent runtime must establish them explicitly and repeatedly at machine speed. - -This abstraction mismatch matters because concurrent coding-agent tool calls do more than write independent patches. They inspect a changing project, perform multi-file edits, run builds and tests, start services, consume shared resources, and return results that may have been produced against different project states. In one native mutable workspace, intermediate edits and execution side effects can leak across agents. Separate copies or Git worktrees reduce direct file interference but leave service ownership, resource admission, stale-base detection, integration ordering, verification, and attribution to external coordination. - -Two multi-agent regimes expose different limits. In an **agent team**, workers have relatively stable roles, delegated tasks, and an orchestrator or lead. Its concurrency ceiling can be dominated by dependency-aware scheduling, ownership and handoff delays, shared-service conflicts, and a serial integration/verification lane. In an **agent swarm**, many workers explore or implement concurrently with weaker central coordination. Its ceiling can be dominated by duplicated work, hot-path collisions, rapidly stale bases, resource saturation, publication contention, and the cost of attributing and selecting among competing results. These are workload models for evaluation, not claims that every product implements one regime cleanly or that either has a fixed agent-count threshold. - -The product survey in [“What agent teams prove—and what their runtimes leave open”](https://agent-infra-foundation.org/blog/2026/07/the-concurrency-ceiling-of-coding-agents/) supports a narrower, useful observation: current agent-team interfaces make parallel research, review, planning, and separated modules legible, while their own guidance remains cautious about interdependent concurrent writes. Task lists, messages, role canvases, dynamic plans, worktrees, and remote machines organize workers, but do not by themselves define private copy-on-write visibility, resource ownership, current-head reconciliation, atomic data publication, or an integration policy. These product documents establish engineering motivation, not scholarly evidence that a particular runtime raises coding concurrency. - -Existing evidence shows coordination and textual-integration failures in overlap-prone workloads, but it does not identify a universal agent-count ceiling or prove that the runtime substrate is the dominant cause. CooperBench v2 is especially instructive: agents operate in separate containers yet still suffer coordination and joint-integration failures. Isolation is therefore necessary for one class of interference, not sufficient for collaboration. - -For workload \(W\), runtime \(R\), agent configuration \(A\), and worker count \(n\), define useful-work rate \(U(W,R,A,n)\) as verification-passing, durably accepted contribution units per unit wall time or cost. A practical concurrency ceiling is a workload- and runtime-specific region where increasing \(n\) no longer improves \(U\) because conflict, retry, integration, verification, or resource costs dominate marginal accepted progress. There is no universal threshold and no presumption that \(U\) is monotonic. - -The systems gap studied here is narrower: a coding-agent runtime needs a private workspace-session boundary for tool execution and a defined path for merging captured session changes back into recorded project history. Ephemeral Sandbox can address direct workspace interference, namespace-scoped execution boundaries, stale/overlapping publication, and typed lifecycle outcomes. It cannot fix model quality, task decomposition, architectural disagreement, semantic merge correctness, general agent communication/scheduling, or verification design. - -Problem-framing sources and their bounded uses are audited in [`references/related_work.md`](./references/related_work.md) and [`claim_evidence_map.md`](./claim_evidence_map.md). The Agent Infra Foundation's [concurrency-ceiling article](https://agent-infra-foundation.org/blog/2026/07/the-concurrency-ceiling-of-coding-agents/) is motivation, not empirical proof. - -## Introduction opening (working draft) - -Agent systems have learned to fan out intelligence; their runtimes still lack a principled way to fan concurrent writes back into one durable project. Current agent teams are effective at research, review, planning, and work that can be divided into separated modules. Fine-grained coding is harder. Workers edit related files, run commands against changing dependencies, start services, and produce locally successful results that must eventually coexist at one project head. - -Multi-agent coding remains an immature systems setting. Agents can miscommunicate, choose poor decompositions, duplicate work, make incompatible architectural assumptions, and produce changes that merge textually but fail together. A workspace runtime cannot solve those problems. They are also not a reason to accept avoidable interference from a substrate that has no agent-level session or publication abstraction. The runtime should remove the conflicts it creates itself, so the measured limit reflects the coding workload, agents, and coordination method rather than a shared mutable checkout and an improvised merge procedure. - -Conventional operating systems were organized around human-operated applications and cooperative processes, not autonomous teams of coding agents. Their first-class units are users, processes, files, descriptors, locks, and ports. They do not represent an agent's delegated task, the project revision it observed, the lifetime of its private execution state, the resources it owns, or the decision that makes its result durable. Human developers reconstruct this context through editors, terminals, Git, and conversation. At multi-agent speed, leaving it implicit turns concurrent coding into a collection of filesystem, process, service, and integration races. - -Structured agent teams and exploratory swarms encounter this limit differently. Teams can stall at dependency handoffs, shared services, and a serial integration lane; swarms can saturate the same code paths and resources with duplicated or rapidly stale work. In both cases, adding workers can stop improving verification-passing, durably accepted progress when conflicts, retries, integration, verification, or resource costs dominate. This workload-dependent point is the concurrency ceiling—not a universal agent count and not a claim that more agents always help. - -Ephemeral Sandbox treats this ceiling as a runtime-systems problem. Its central abstraction is a private workspace session over a leased LayerStack base: one durable project history, many private executable views, and one controlled boundary for capture, current-head reconciliation, and atomic data publication. The mechanism isolates uncertain work without treating isolation as collaboration itself. It does not make agents agree on architecture, decompose tasks correctly, or validate semantic compatibility. The paper therefore asks whether explicit workspace sessions and controlled publication can move the measured useful-work ceiling under stated coding workloads, rather than assuming that result. - -## Technical challenges - -1. **Stable execution over changing history.** A session must retain an executable snapshot while other publications advance the head and compaction changes the physical layer representation. The baseline uses a manifest/layer lease and lease-aware squash/GC ([LayerStack](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack)). -2. **Faithful capture at the filesystem boundary.** The runtime must recover writes, deletes, symlinks, empty directories, and opaque directories from OverlayFS metadata without confusing ordinary `.wh.*` names for kernel whiteouts, and it must make unsupported entries explicit ([capture](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs), [tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/tests/unit/overlay_capture.rs)). -3. **Current-head reconciliation without partial visibility.** A delta produced against a leased base must be compared with the active head, merged only under defined rules, rejected as a whole when unresolved, and made visible through one durable manifest transition ([publish resolver](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs), [commit path](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs)). -4. **Lifecycle and failure attribution across non-atomic phases.** Capture, data commit, best-effort audit, and session destruction have distinct failure points. The interface must distinguish retryable precommit failure from published-but-not-closed cleanup failure ([publish service](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs), [tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs)). - -## Method insight - -**Make the workspace session the boundary between a tool call and shared LayerStack history.** Execute a sessionless command tool call in a private session—or group related tool calls in one explicit session—then cross back into shared history only by capturing and validating the session's complete filesystem delta against the current head. - -In compact form: **one durable project truth, many private executable sessions, controlled publication.** This makes coding concurrency an explicit runtime concern rather than an improvised consequence of task prompts, worktree conventions, and lead-agent merge repair. - -This is deliberately not described as a full transaction: the resolved data changeset is atomic at the active-manifest visibility boundary, while audit attribution, accounting, and post-publication cleanup are later, fallible phases. - -## System stages - -1. **Workspace projection.** The initial project is copied into a base layer. Session creation acquires an exact LayerStack manifest/layer lease and mounts those shared lowers with a fresh private upper/work pair ([workspace creation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/lifecycle/create.rs), [overlay mount](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs)). -2. **Tool-call execution.** A sessionless `exec_command` call creates one implicit workspace session with `publish_then_destroy`; an explicit session can host multiple command and file calls until the caller publishes or destroys it ([command path](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/command/service/exec_command.rs), [finalization model](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs)). A holder owns user, mount, PID, and optionally network namespaces; runners join the stored handles without moving the daemon itself into the session ([namespace holder](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-process/src/holder/namespace.rs), [setns](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-process/src/runner/setns/namespaces.rs)). -3. **Capture.** The runtime walks the private upperdir and translates kernel OverlayFS metadata into typed layer changes and protected drops ([capture](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs)). -4. **Publication.** Planning validates the base and routes paths; the exclusive commit phase rereads the active head, resolves fingerprints/eligible text merges, rejects the whole data changeset on unresolved conflict, or stages/syncs/promotes one layer and atomically replaces the active manifest ([plan](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/plan.rs), [resolve](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs), [publish](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs)). Audit attribution is best-effort after commit. -5. **Merge-back lifecycle and observability.** When an implicit command session drains, it captures, publishes or rejects, and destroys; an explicit session supports deliberate publish/retry or discard. Three catalog-derived clients separate management, sandbox runtime, and read-only observability operations ([CLI matrix](./cli_contract_matrix.md)). - -## Proposed contributions and evidence status - -1. **LayerStack-backed workspace-session protocol.** A sessionless command tool call receives an implicit private workspace over an exact LayerStack lease, while an explicit session can group multiple command/file calls before publication or discard. **Evidence:** implemented at the baseline; session lifecycle, capture, and squash/lease behavior have correctness tests; final-tag isolation and restart tests still need to run. See C1 in [`claim_evidence_map.md`](./claim_evidence_map.md). -2. **Conflict-aware atomic data publication.** A whole-changeset protocol with base validation, path fingerprints, structural rejection, bounded eligible text merge, staged layer promotion, and atomic active-manifest replacement. **Evidence:** implemented and covered by LayerStack and explicit-session publication tests; paper-platform fault testing still needed. This excludes semantic correctness, audit, accounting, and cleanup. See C2. -3. **Role-separated operational contract.** Separate management, sandbox-runtime, and read-only observability clients with catalog-derived help, typed scope, request IDs, connection/token discovery, and JSON/exit contracts. **Evidence:** implemented and contract-tested at 8/10/8 baseline operations; must be regenerated from the final tag. See C3. -4. **Explicit lifecycle and partial-failure semantics.** Persistent and implicit workspaces distinguish retryable precommit failure, discard, successful publication, and published-but-not-closed cleanup failure. **Evidence:** implemented and correctness-tested; protected-drop policy and daemon-restart recovery remain open. See C4. -5. **A concurrency-ceiling evaluation methodology.** Measure accepted verified progress, conflict/retry/integration costs, and runtime resources across worker counts and workspace strategies. **Evidence:** external presets and exploratory demos exist, but no final dataset. This is a planned contribution, not yet a result. Drop or demote it if the freeze evaluation is not completed. See C5. - -## Abstract option A: design-first - -Agent systems can fan out intelligence, but conventional operating-system workspaces provide no agent-level protocol for fanning concurrent writes back into one durable project. Coding-agent teams and swarms inspect changing code, perform multi-file edits, run tests, and start services; sharing one mutable workspace permits live-state interference, while copying workspaces alone defers integration. We present Ephemeral Sandbox, a workspace OS organized around LayerStack: one durable project history, many private executable sessions, and controlled publication. A sessionless `exec_command` tool call receives an implicit isolated session over a leased LayerStack snapshot. When its command ledger drains, the runtime captures the private overlay, reconciles the complete delta against the current head, and either publishes one new layer or returns a structured rejection. Eligible text divergence can use a bounded three-way merge; structural, binary, oversized, or conflicting changes reject. Explicit sessions let multiple command and file tool calls share private state before publication or discard. Source and correctness tests establish these mechanisms, but not formal security, semantic merge correctness, process-state rollback, resource coordination, or improved multi-agent throughput. Whether Ephemeral Sandbox raises the workload-dependent useful-work concurrency ceiling remains an empirical question. - -## Abstract option B: evaluation-ready template - -Conventional operating-system workspaces provide process- and file-level concurrency, but no agent-level unit connecting a delegated task, private execution state, resource ownership, and a controlled publication decision. Coding-agent teams and swarms can therefore lose useful work through mutable-state interference, stale results, resource collisions, and deferred integration. We present Ephemeral Sandbox, a LayerStack workspace OS in which each sessionless command tool call executes inside an implicit isolated workspace session, while explicit sessions can group multiple command and file calls. Merge-back captures the private overlay, validates its complete delta against the current LayerStack head, and either performs bounded eligible text reconciliation, publishes one layer through an atomic manifest transition, or rejects the data changeset. We evaluate isolation and publication using [MEASURED RESULT NEEDED] adversarial cases, then compare shared-directory, Git-worktree, and Ephemeral-session baselines across structured-team and exploratory-swarm workloads with [MEASURED RESULT NEEDED] workers. We report verification-passing durably accepted work, conflicts, retries, integration latency, resource cost, and attribution completeness. Ephemeral Sandbox achieves [MEASURED RESULT NEEDED] under [EXACT CONDITIONS NEEDED], while [LIMITING RESULT NEEDED] identifies when coordination, verification, conflicts, or resources dominate marginal progress. The result is workload-specific, not a universal speedup, security guarantee, or solution to semantic coordination. +Ephemeral Sandbox implements private executable workspace sessions over leased +LayerStack history and a controlled capture-to-publication boundary; EXP1 v1.1 +describes public-CLI startup, operation, and selected resource observations in +one explicitly bounded Windows/Docker Desktop treatment. + +## Task boundary + +- **Inputs:** a selected sandbox, a leased LayerStack view, session-scoped or + sessionless runtime operations, and a captured private filesystem delta. +- **Output:** either a source-defined current-head publication outcome or a + structured non-publication/lifecycle outcome. +- **Measured setting:** native Windows build 26200, Docker Desktop 29.0.1, + pinned Ubuntu 24.04 image, `paper-100m`, native `product_cli`, local named + pipes, concurrency 1/5, two warmups and 100 measured trials per cell. +- **Out of scope:** isolation correctness, publication/fault correctness, + competitive baselines, deeper scaling, useful-work, attribution completeness, + restart recovery, security, and multi-agent quality. + +## Gap and method insight + +Private copies alone defer the question of how a private execution view relates +to a changing shared project head. Ephemeral's system-level contribution is to +make that boundary explicit: **one recorded project history, private executable +sessions, controlled publication.** A lease fixes the logical source view; +capture produces a filesystem candidate; current-head reconciliation either +resolves the complete data changeset under source-defined rules or rejects it; +only an accepted layer becomes visible through active-manifest replacement. + +This is a composition and lifecycle/operational-contract claim. Union mounts, +copy-on-write views, leases, optimistic validation, and text merges are not +claimed as individual inventions. The Related Work section must position the +composition against DeltaBox, Shepherd, AgentBay, CAID, CoAgent, Claim Plane, +private-workspace conflict work, and foundational filesystem/concurrency work. + +## Contributions and evidence + +1. **Workspace-session protocol.** Source inspection establishes leased + LayerStack projection, private upper/work directories, sessionless command + finalization, and explicit-session lifecycle. This is not a measured + isolation or security claim. +2. **Controlled publication protocol.** Source inspection establishes capture, + current-head validation, bounded text merge eligibility, whole-candidate + rejection, and the active-manifest data-visibility boundary. This is not a + semantic-correctness, fault-tolerance, or full-transaction claim. +3. **Operational contract.** The measured source exposes 8 management, 10 + runtime, and 8 observability operations; all three accept an optional + validated request ID and use endpoint URI resolution with a legacy + `--gateway-socket` alias. This is contract evidence, not correctness proof. +4. **Focused RQ3 evidence.** The immutable 19-cell EXP1 v1.1 archive and + deterministic tables describe end-to-end public-CLI timings and seven + preregistered resource rows in one fixed treatment. No comparison is made. + +## Completed evaluation question + +Under the disclosed EXP1 v1.1 treatment, what startup, public-CLI operation, +and preregistered resource observations are recorded at concurrency 1 and 5? +Results answer this question descriptively. They do not establish why values +change, whether individual latency improves, or what happens in another host, +workload, product revision, or agent workflow. ## Claims to make -- v1 implements private writable Linux overlay views over leased recorded history, with namespace-scoped command/file execution. -- a sessionless `exec_command` call creates an implicit isolated workspace session and uses `publish_then_destroy` after its command ledger drains. -- an explicit workspace session can carry private state across multiple command and file tool calls before explicit publication or discard. -- leases retain a captured logical snapshot and constrain compaction/GC. -- capture translates kernel OverlayFS state into typed filesystem changes and protected drops. -- publication applies defined current-head validation, bounded text merge, structured reject, and all-or-none data-changeset behavior. -- successful data publication promotes a layer/digest and changes active-manifest visibility atomically in the implemented commit path. -- the baseline source contract contains 8 management, 10 runtime, and 8 observability operations, with the exact scopes in [`cli_contract_matrix.md`](./cli_contract_matrix.md). -- explicit lifecycle states distinguish precommit retry from post-commit cleanup failure. - -## Claims to be careful about - -- **Immutable history:** call layers append-oriented/content-digested and treated as immutable after promotion; do not claim filesystem-enforced immutability against external mutation. -- **Isolation:** specify private upperdir and namespace profile. Default/shared networking is not egress isolation. -- **Workspace OS:** define it as the runtime substrate for workspace-session execution, LayerStack history, publication, lifecycle, and observability; do not imply a general-purpose OS, kernel, or security monitor. -- **Per-tool-call sessions:** this is exact for sessionless `exec_command`. Sessionless `file_read` projects the active LayerStack, while sessionless `file_write` and `file_edit` amend the current head directly; do not claim that every runtime operation creates a workspace session. -- **Atomic:** scope it to one resolved data changeset and the active-manifest visibility boundary. -- **Attributable:** request correlation and a best-effort audit log exist, but audit is not transactionally coupled and may be `unknown`. -- **Conflict-aware:** enumerate fingerprints, structural checks, protected paths, and bounded line merge; do not imply semantic conflict detection. -- **Durable:** bind it to the tested Linux/filesystem/fault model; the source alone is not a cross-platform crash proof. -- **Concurrency ceiling:** define workload, runtime, agents, metric, cost basis, and uncertainty for every result. -- **Observability:** source-prove which views exist; do not imply that agents necessarily use them effectively. -- **Complexity:** present a stage-specific operational cost model, not one context-free runtime bound. Separate source-derived \(L,S,U,C,F,B,Q,R\) terms from measured latency, RSS, queueing, and physical allocation. -- **Bounded merge:** the 8 MiB byte gate bounds file inputs, not the current Myers trace's practical line/edit-distance memory envelope. -- **LayerStack 2.0:** describe it only as a candidate future response to v1 byte-copy/storage amplification; capability detection, extent-dependent cost, semantic preservation, and copy fallback are mandatory qualifiers. +- The measured source implements the bounded workspace/session/publication and + role-separated-interface mechanisms stated in the claim map. +- The final archive completed 1,900/1,900 reportable measured trials in its + bounded campaign, without generalizing that outcome to product reliability. +- Tables 2--4 report selector-bound end-to-end public-CLI and resource values + for the disclosed setting. +- The v1.0 endpoint-exhaustion failure motivated a preregistered local named- + pipe treatment; it is qualitative failure analysis, never a competing row. ## Claims to avoid -- a universal concurrency ceiling or a single-digit threshold; -- generic speedup, cheapness, scalability, productivity, or monotonic benefit from more agents; -- “solves” parallel-agent coordination; -- formal sandbox security, complete noninterference, or universal egress denial; -- process-state checkpoint/rollback; -- semantic merge correctness or serializable snapshot isolation; -- unsupported Windows overlay, reflink, locking, or durability claims; -- \(O(1)\), universally available, or already implemented reflink publication; -- atomic coupling of data, line attribution, accounting, and cleanup; -- current v1 support for intent negotiation, service/port ownership, scheduling, handoffs, or resource budgets unless added and source-proven before freeze. - -## Related-work position - -- **CAID:** orchestrates asynchronous engineers in Git worktrees and performs harness-level merge/verification; Ephemeral supplies a lower-level model/orchestrator-independent capture/publication runtime. -- **CoAgent:** speculates on shared mutable state and repairs/reorders effects; Ephemeral isolates writes and validates before durable visibility. -- **Claim Plane:** admits typed change intents and authority before work; Ephemeral's niche begins with private execution and ends with current-head publication. This is the highest novelty risk. -- **SWE-MiniSandbox:** optimizes isolated task environments; it does not reconcile results into shared history. -- **DeltaBox:** couples layered filesystem state with process checkpoint/rollback for tree search; Ephemeral does not roll back process state and instead focuses on multiwriter durable publication. -- **Shepherd:** records reversible agent/environment traces for meta-agent replay and intervention; Ephemeral records workspace history and publication outcomes, not full reversible execution. -- **Union mounts and OCC:** establish the primitives. Novelty, if any, lies in the composition and source-defined lifecycle/operational protocol. -- **Palantir, Crystal, and Verified Three-Way Program Merge:** show that private workspaces defer integration issues and that textual cleanliness is not semantic correctness. - -Full metadata, citation safety, and overlap risks are in [`references/related_work.md`](./references/related_work.md). - -## Complexity and evolution position - -The manuscript should include a compact source-derived cost table in Implementation and test its variables in Evaluation. The strongest current hypotheses are that history depth multiplies session/validation metadata, upperdir structure controls capture, published bytes and reconciliation work extend the serialized writer path, and long-lived leases trade stable views for retained history. These are mechanisms to measure, not performance conclusions. The derivation and experiment matrix are in [`complexity_and_evolution.md`](./complexity_and_evolution.md). - -LayerStack 2.0 belongs in Limitations/Future Work, not the abstract or v1 contributions. Its single-storage-domain/reflink protocol targets publication and allocation amplification while requiring unchanged blame, squash, remount, active execution, crash, memory, image, and privilege behavior. The only completed Windows evidence is negative and narrow: direct `FICLONE` failed with `errno=95` in the pinned stock Docker Desktop/WSL 2 cell, so the integrated candidate was not run and a copy fallback remains necessary. - -## Reviewer risks and blockers - -1. **Freeze/provenance:** `paper-v1-freeze` is absent. All current source citations are provisional. -2. **Evaluation gap:** no paper-ready isolation, fault, scaling, resource, or multi-agent workflow results exist. -3. **Novelty:** Claim Plane, CAID, Shepherd, CoAgent, DeltaBox, union mounts, and OCC substantially overlap individual ideas. The paper must make a composition/protocol claim and audit Claim Plane fully. -4. **Attribution mismatch:** the product aim says attributable integration, but source audit is best-effort after data commit. Either state this plainly, strengthen and test the implementation before freeze, or narrow the contribution. -5. **Failure model:** restart ordering for in-memory leases/substitutions and explicit/implicit protected-drop asymmetry need resolution. -6. **Atomicity wording:** reviewers will object if data, audit, and cleanup are described as one transaction. -7. **Semantic boundary:** clean text merge can still fail build/tests; evaluation should measure this rather than hide it. -8. **Platform scope:** core overlay/namespace behavior is Linux-only; Windows/reflink claims are unsupported. -9. **Cost-model validation:** layer-depth, live-lease, upperdir-shape, changed-path/byte, writer-queue, merge-shape, and retained-history predictions have not been validated on the freeze. -10. **Merge resource risk:** the current byte gate does not independently bound the retained diff trace; add adversarial CPU/RSS evidence or change the implementation before freeze. -11. **Benchmark validity:** compare matched agent/model/budget/workloads, preserve raw provenance, report uncertainty, and avoid treating deterministic demos or ignored benches as final evidence. +- speed, scalability, superiority, reliability, security, production readiness, + useful-work productivity, or universal concurrency claims; +- source inspection or test presence as measured correctness evidence; +- correctness of publication, isolation, recovery, or semantic integration; +- a general Windows/Linux equivalence, physical-storage advantage, or reflink + result. + +## Required limitations and reviewer risks + +The paper must foreground the missing baseline, narrow environment, one final +run, unmeasured correctness/recovery, post-commit attribution, protected-drop +asymmetry, daemon-restart lease gap, unbounded-in-practice merge trace risk, +unavailable storage metrics, and analysis compatibility erratum. The highest +novelty risk is Claim Plane; the story must distinguish pre-write intent and +authority from Ephemeral's execution-to-publication boundary without claiming +that either system subsumes the other. diff --git a/ephemeral-sandbox-v1/plan/progress.md b/ephemeral-sandbox-v1/plan/progress.md index fef0f26..8b8bbc5 100644 --- a/ephemeral-sandbox-v1/plan/progress.md +++ b/ephemeral-sandbox-v1/plan/progress.md @@ -23,6 +23,42 @@ Root [`progress.md`](../progress.md) is the authoritative project milestone trac - **2026-07-30 -- blockers:** Final native Ubuntu verification, live smoke, file-operation base-workspace policy, explicit sandbox-create timing, pilot duration, source/benchmark freeze, and protocol approval remain open. +- **2026-07-30 -- environment-qualification continuation:** Exhausted bounded + local discovery for `eos-benchmark-ubuntu24`; no DNS, SSH, hosts-file, + Docker-context, Hyper-V, or eligible Multipass route exists. The only local + Multipass instance is Ubuntu 22.04 and remains ineligible. +- **2026-07-30 -- final-host automation:** Added a reproducible Windows bundle + builder, hash-locked Linux CPython 3.13 dependency input/lock, off-clock host + staging script, strict network-free preflight, and an archival smoke + qualification wrapper with before/after leak comparison. +- **2026-07-30 -- transfer readiness:** Generated and integrity-checked the + final-host transfer bundle at + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\ephemeral-sandbox-v1-20260730-ready-v2`. + The strict verifier also failed closed on WSL at the exact-hostname gate. No + benchmark run or measurement was performed. +- **2026-07-30 -- CLI-only correction:** Replaced the direct-client environment + smoke with two independent product-CLI-controlled lifecycles. The final WSL + diagnostic completed 2/2 batches and 20 validated CLI calls with correctness + pass, empty CLI stderr, no gateway warnings/errors, and zero Docker resources + remaining for its unique gateway instance. This is automation evidence only; + the native-host gate remains unpassed. +- **2026-07-30 -- handoff boundary:** Corrected the external action: SSH and new + provisioning are optional. Completion requires running the supplied handoff + on an existing eligible native host, or provisioning one only if none exists, + and returning an all-pass qualification artifact set. +- **2026-07-30 -- host correction:** User clarification established native + Windows plus Docker Desktop as the final host. Ubuntu 24.04 is the sandbox + image. The earlier native-Ubuntu/SSH/CPython/ext4 handoff is superseded. +- **2026-07-30 -- Windows package and qualifier:** Checksum-verified and staged + the official `v0.1.4` Windows AMD64 package under ignored product `target/`. + Added a PowerShell-only strict host/package/Docker/image qualifier using the + native gateway and manager/runtime/observability CLIs. +- **2026-07-30 -- environment completion:** Accepted + `qualification-windows-docker-20260730-final-6`: native Windows/Docker + Desktop preflight pass, 2/2 independent batches, 20 validated product-CLI + calls, correctness pass, empty CLI stderr, zero warnings/failures, unchanged + EOS baseline, and zero resources remaining for the qualifier gateway. The + environment-only task is complete. ## PW0: manuscript scaffold and vocabulary @@ -62,3 +98,270 @@ Root [`progress.md`](../progress.md) is the authoritative project milestone trac - **2026-07-30 -- started:** Loaded the complete `ai-research-writing-skill` instructions, full-paper workflow, artifact contract, section-writing method, task-management rules, and reference map. Created `plan/task-packets/pw1-foundations.md` with the bounded scope, pre-draft theses, paragraph roles, evidence IDs, rejection checks, validation plan, and acceptance criteria. - **2026-07-30 -- input audit:** Read all 21 required PW1 paper artifacts completely. Confirmed the paper checkout is clean on `agent/pw0-build-pw1-prompt`; confirmed the read-only source checkout is clean on `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`. No source, experiment, later-section, bibliography, or figure artifact was changed. - **2026-07-30 -- evidence boundary:** PW1 uses M0--M6, C1--C4, and D1--D9. C5 and measured useful-work or concurrency-ceiling language remain pending evaluation; source-derived wording remains provisional until `paper-v1-freeze`. +- **2026-07-30 -- artifacts produced:** Drafted Sections 2 and 3 and completed the paragraph-level reverse outline in `plan/task-packets/pw1-foundations.md`. Updated only the authorized task, progress, machine-state, and build records; no claim-map or terminology correction was needed. +- **2026-07-30 -- recorded build:** The skill recorder executed `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex` with exit code 0 using Latexmk 4.88, pdfTeX 1.40.29, and BibTeX 0.99e from TeX Live 2026. It recorded input SHA-256 `8a9d0f97487ccf937efd82eb6a5726b2c7c8b5e3e31be1434945826a00731708`, PDF SHA-256 `ba4963d3d5f6352e1829946290265671599432e9984e301d5626de7316435327`, and build-log SHA-256 `719696c0aa0efb9fce4796efaab3cce5b27dd17563c4b60d7f4951b539aa7f30`. +- **2026-07-30 -- verification:** The full-paper quality gate, citation-key check, JSON/mode/stage check, exact section-count/order check, Section 2/3 heading/label check, 115-target relative-link check, prohibited-claim review, PDF/log hash check, and Git whitespace check passed. The source checkout remains clean on `main` at the audited baseline. The build log has no errors, undefined citations/references, missing files, or overfull boxes; five pre-existing underfull boxes remain in the Section 7 cost table. +- **2026-07-30 -- outcome:** PW1 is complete. Remaining scientific risks are the missing source freeze and frozen evaluation, best-effort post-commit attribution, unresolved explicit/implicit protected-drop policy, and untested lease/substitution behavior across daemon restart. + +## PW2: workspace execution and capture/publication + +- **2026-07-30 -- started:** Reloaded the complete `ai-research-writing-skill` instructions and the required workflow, artifact, section-writing, and task-management references. Created `plan/task-packets/pw2-workspace-publication.md` before manuscript drafting with the bounded scope, section theses, paragraph roles, evidence IDs, rejection checks, validation plan, and acceptance criteria. +- **2026-07-30 -- input audit:** Read the authoritative PW2 lane contract, C1/C2 and D1--D8 evidence, canonical terminology, story/skeleton/inventory/interface records, Sections 2--5, and the relevant read-only baseline source for workspace creation, execution, capture, reconciliation, commit, attribution, and cleanup. No standalone `lanes/prompts/pw2.md` exists, so the task packet records the lane charter as authoritative. +- **2026-07-30 -- worktree boundary:** Preserved the authorized PW1 record updates and unrelated pre-existing benchmark/experiment modifications. Recorded protected-file hashes before drafting; no source, benchmark, experiment, bibliography, figure, or non-target manuscript edit is authorized. +- **2026-07-30 -- artifacts produced:** Drafted Sections 4 and 5 from C1/C2 and D1--D8 and completed their paragraph-level reverse outlines. The drafts distinguish implicit commands, explicit sessions, session and sessionless file paths, source and ignored publication routes, merge eligibility, the active-manifest visibility point, and post-commit attribution/cleanup boundaries. +- **2026-07-30 -- recorded build:** The skill recorder executed `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex` with exit code 0 using Latexmk 4.88, pdfTeX 1.40.29, and BibTeX 0.99e from TeX Live 2026. It recorded input SHA-256 `36877891dbce1e066589d8a295e436654ccb097970620003cdaf45871f74311b`, PDF SHA-256 `801ac91c302ae3ea7d5827d34dd4da09278e8f537409e3426fd4ff30c8ed36e7`, and build-log SHA-256 `9fc4261ab327472e004f4f62466bf2266218d1b17a66cc5b852ee5dbe7b23265`. +- **2026-07-30 -- verification:** The full-paper quality gate, citation-key check, JSON/mode/stage check, exact section-count/order check, Section 4/5 heading/label check, 115-target relative-link check, prohibited-claim review, attested input/PDF/log hash check, protected-file hash review, and scoped and whole-worktree Git whitespace checks passed. The source checkout remains clean on `main` at the audited baseline. The build log has no errors, undefined citations/references, missing files, or overfull boxes; five pre-existing underfull boxes remain in the Section 7 cost table. +- **2026-07-30 -- outcome:** PW2 is complete. Remaining scientific risks are the missing source freeze and frozen fault/resource evidence, best-effort post-commit attribution, nonuniform protected-drop entry-point policy, unvalidated worst-case text-diff resource behavior, and lifecycle/recovery outcomes deferred to PW3. + +## PW2.5: concept-figure prompt package + +- **2026-07-30 -- started:** Loaded the figure workflow, figure-specification contract, venue/style presets, canonical terminology, current figure plan, and Sections 3--6. Created `plan/task-packets/pw2-5-figure-prompts.md` before authoring prompt sources. +- **2026-07-30 -- scope boundary:** This package creates prompts/specifications only. It does not generate images, edit manuscript inputs, refresh the PDF build, or treat a concept diagram as experimental evidence. Pre-existing paper, benchmark, and experiment changes remain untouched. +- **2026-07-30 -- artifacts produced:** Created one shared Classic Academic × Modern Minimal style guide and four separate, self-contained prompt/spec files for the system architecture, publication sequence, lifecycle state machine, and reconciliation decision flow. Updated `figures/figure_plan.md` with direct links, distinct compositions, generation order, and asset status. +- **2026-07-30 -- verification and outcome:** Confirmed all four prompts independently include style, palette, output, exact labels, and acceptance criteria. The 121-target relative-link check, full-paper quality gate, and scoped whitespace check pass. Prompt preparation is complete; image generation and visual QA remain pending, and the lifecycle asset is provisional until PW3. + +## PW2.5: concept-figure generation and review + +- **2026-07-30 -- started:** Loaded the image-generation and paper-figure workflows, created `plan/task-packets/pw2-5-figure-generation.md`, and delegated architecture, publication-sequence, and reconciliation assets to non-overlapping subagents. The lifecycle asset will use the next available agent slot. +- **2026-07-30 -- review contract:** Every PNG must pass visual inspection for exact labels, final-width readability, semantic palette, shared/private and commit/post-commit boundaries, and unsupported-claim avoidance. Failed assets receive targeted regeneration or editing rather than automatic acceptance. +- **2026-07-30 -- supplied assets:** The author supplied four PNGs, which were + moved to the expected `figures/concept/` paths. Generator/model/seed metadata + and label-placement maps were not supplied; prompt/spec sources remain the + available regeneration contract. +- **2026-07-30 -- drafting-stage review:** Recorded hashes, dimensions, + strengths, topology/layout/resolution/style disparities, and the + concept-only evidence boundary in `figures/concept-figure-review.md`. + The author accepted all four files unchanged for the drafting manuscript and + deferred visual repair and normalization to PW7. +- **2026-07-30 -- outcome:** Draft asset generation and review are complete; + submission-final visual acceptance remains open. Lifecycle/prose consistency + is carried into PW3. + +## PW3: lifecycle, recovery, implementation, and interface + +- **2026-07-30 -- started:** Loaded the full-paper, section-writing, figure, + task-management, and build contracts; created + `plan/task-packets/pw3-lifecycle-interface.md`; and passed the early + full-paper quality gate. +- **2026-07-30 -- evidence audit:** Confirmed the read-only source checkout is + clean on `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`. + Audited explicit and implicit finalization, guarded destruction, holder-exit + reconciliation, bounded recovery artifacts, shutdown, remount, role-separated + client projections, request construction, output/help behavior, and the + existing cost model. +- **2026-07-30 -- figure boundary:** PW3 will integrate the four author-supplied + PNGs without modifying their bytes. Recorded visual disparities are PW7 debt, + not hidden as passed final-quality gates. +- **2026-07-30 -- artifacts produced:** Replaced the Section 6 placeholder with + the explicit/implicit lifecycle, partial-failure, holder-exit, recovery, + shutdown, remount, and lease-aware compaction account. Completed Section 7's + role-separated operational contract while preserving the source-derived cost + model. Added four evidence-bounded figure inclusions and one baseline CLI + contract table. +- **2026-07-30 -- figure revalidation:** Passed the lifecycle image as a + qualified normal-path abstraction. Recorded its omitted holder-exit, + shutdown, and remount branches and shared cleanup-node ambiguity, plus all + architecture, publication, reconciliation, resolution, grayscale, and + style-family disparities, as PW7-only repair work. Rechecked all four hashes; + the PNG bytes remain unchanged. +- **2026-07-30 -- recorded build:** The final recorder run executed + `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex` with exit + code 0 using Latexmk 4.88, pdfTeX 1.40.29, and BibTeX 0.99e from TeX Live + 2026. It recorded input SHA-256 + `e31aa83e7db9c45cdf0c3c1f731fe66670fc6641dc20d755704a7673a552e18a`, + PDF SHA-256 + `bf893a5ac17e396233232b18d552e77ddddfc2bcba50786d457fd58db71604eb`, + and build-log SHA-256 + `39a836a6226fc9b8007dd23fda5deb9e2cd8bf3c453faf17539c9b356871e59f`. +- **2026-07-30 -- verification:** The full-paper quality gate, citation-key + check, structure/label/figure/hash/link checks, source-baseline check, and + whitespace checks pass. The final log has no errors, emergency stops, + undefined citations/references, missing files, or overfull boxes; seven + underfull boxes remain in the two narrow Section 7 tables. +- **2026-07-30 -- PDF inspection and outcome:** Rendered all 14 pages in color + and grayscale. Figures appear on pages 3, 6, 8, and 10; both Section 7 + tables follow their introductions; no clipping, overlap, broken glyph, or + page-number defect was found. PW3 is complete. Submission-final figure + repair/waiver remains PW7, and scientific/source-freeze/evaluation blockers + remain open. + +## EXP1: CLI-only focused performance campaign specification + +- **2026-07-30 -- tracker reconciliation:** Updated the authoritative root + tracker to record the accepted native Windows/Docker Desktop environment and + remove the stale native-Linux-host experiment gate. Gates 1 and 2 remain + complete; no performance measurement was run. +- **2026-07-30 -- pre-measurement decisions:** Fixed the focused RQ3 primary + timing boundary as end-to-end native product-CLI subprocess latency, selected + `paper-100m` for every operation family, and retained sandbox-create-to-ready + as a distinct manager-CLI metric. These decisions were recorded before + viewing performance results. +- **2026-07-30 -- next-agent contract:** Created + `plan/task-packets/exp1-cli-performance-campaign.md` with exact environment, + image, executable, hash, cohort, timing, workspace, implementation, test, + pilot, freeze, final-run, analysis, stop, artifact, and handoff requirements. + The final focused matrix is 19 cells after the required sandbox-create cell + is implemented. +- **2026-07-30 -- outcome:** Documentation/specification work is complete. + The immediate executable task is the reviewed `product_cli` benchmark cohort. + `paper-env-smoke`, `paper-pilot`, and `paper-good-pass` remain prohibited + until the implementation and validation gates in the EXP1 packet pass. +- **2026-07-31 -- implementation and qualification:** Implemented and reviewed + the native Windows `product_cli` cohort, fixed all 19 cells to `paper-100m`, + added manager-CLI sandbox-create timing, passed the full backend suite, and + completed clean CLI-only smoke and five-sample pilot runs. The conservative + final projection was 1,307.100411100 seconds against the author-approved + 1,400-second Gate-3 limit. +- **2026-07-31 -- freeze:** Froze protocol v1.0 at paper commit + `eb10c26d1bfd632772baf1bc331c985d0231f52d`, product commit + `0392b299ecaf3a75c8b6d04ed94d5a15593ca6a3`, and annotated product tag + object `0b4aaec5f13b0e52772b2adb7ca2807ee2223e6d`. No object was pushed. +- **2026-07-31 -- final stop condition:** The sole eligible final run + `019fb6e5-c00b-7b02-8a3c-d76bd1346eb4` stopped after 853 of 1,938 batches. + The product file-read request succeeded, but the mandatory post-response + snapshot CLI failed to connect with WSAEADDRINUSE 10048. Windows TCP/IP + Event 4227 at the same instant confirms local endpoint-reuse pressure from + high-rate connection churn. +- **2026-07-31 -- preservation and outcome:** Explicit cleanup completed, and + the 620,311,242-byte failed corpus was archived and independently verified at + `experiments/runs/019fb6e5-c00b-7b02-8a3c-d76bd1346eb4`, content-tree + `sha256:7efa643b12aba09f0ba5ecfbed5b5692a166a5c12931490402d3992d92f3ae6a`. + It is permanently `failed_ineligible`; Gates 5--7 fail and no partial, + pilot, or smoke number may enter the paper. A future attempt requires a + protocol amendment, new freeze, and explicit author authorization. +- **2026-07-31 -- v1.1 read-only forensics:** Independent archive and + primary-source audits identified 7,993 client-to-gateway TCP attempts. Of + 3,836 observability attempts, 3,626 were mandatory and only 210 periodic; + reducing resource cadence cannot address the mandatory failed boundary. + Frozen client and gateway code each implement one request per fresh TCP + connection. +- **2026-07-31 -- remediation proposal:** Recorded + `experiments/analysis/exp1-v1.1-remediation-decision.{md,json}`. The + recommended EXP1-preserving option is an active-store-only IPv4 dynamic-port + expansion, followed by a 20,000-connection same-rate ineligible qualifier, + fresh smoke/pilot, no-more-than-1,400-second projection, v1.1 + source/environment freeze, and exactly one newly authorized final. No + network, source, Git, Docker, or live-run mutation occurred. Explicit author + authorization and an elevated host action remain the exact blocker. +- **2026-07-31 -- v1.1 authorization and treatment amendment:** The author + directed the permanent CLI issue to be fixed and the goal resumed. EXP1 + v1.1 therefore replaces CLI-to-gateway loopback TCP with direct Windows + named-pipe IPC while retaining one fresh native CLI subprocess per measured + request. The earlier host-wide dynamic-port-range proposal is superseded and + was never applied. The v1.0 archive remains immutable and ineligible. +- **2026-07-31 -- resumed implementation state:** Typed TCP/npipe/Unix endpoint + discovery, the Windows named-pipe client, explicit `--gateway-endpoint` + CLI/MCP wiring, Windows local-IPC defaults, and compatibility aliases are + implemented. Real client named-pipe round-trip and concurrency-5 tests pass. + Gateway listener/lifecycle implementation and full product validation remain + in progress; no live qualifier, smoke, pilot, freeze, or final has started. +- **2026-07-31 -- local-IPC product gate passed:** Product direct `main` commit + `56c676d588fbb704bf3da8f67d22be910453644d` and fresh package + `target/windows-exp1-56c676d5` now carry the no-fallback named-pipe + treatment. Changed-crate tests, warnings-denied Clippy, and native + concurrency-5 packaged CLI round trips pass. The next blocking gate is the + ineligible 25,000-invocation qualifier with zero new TCP/IP 4227/4231 + events and bounded gateway handle/RSS growth; no live campaign stage has + started. +- **2026-07-31 -- final prequalification product candidate:** Compatibility + follow-up commit `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` preserves + legacy DNS TCP endpoints while retaining strict local named-pipe defaults + and rejects ambiguous Unix endpoint syntax. A fresh clean package at + `target/windows-exp1-5c48dae1` supersedes the `56c676d5` candidate for all + new work. Its ZIP SHA-256 is + `11e83246b2f509da9708a0237bb6ab600d042e1cb390c81fc41dc834d897c506`; + packaged concurrency-5 native manager-CLI named-pipe verification passed. +- **2026-07-31 -- v1.1 policy preregistration:** Added + `experiments/exp1-v1.1-protocol-amendment.md` before live qualification. + It locks exactly 25,000 manager-CLI calls in 5,000 concurrency-5 batches, + strict output/no-retry/no-TCP rules, event-log and owned-TCP checks, + periodic gateway sampling, fixed handle/memory growth caps, fail-closed + cleanup, exact provenance, and the unchanged postqualification + smoke/pilot/1,400-second/freeze/one-final sequence. The historical elevated + IPv4-range proposal is explicitly superseded and was never applied. +- **2026-07-31 -- v1.1 IPC qualifier passed:** Qualification + `718cf58dace44dba83bed54601854bc9` completed exactly 25,000/25,000 + manager-CLI `list_sandboxes` calls in 5,000 concurrency-5 batches. All + invocation records passed; request IDs were unique; the isolated gateway + owned no TCP endpoint; the System/Tcpip cursor interval contained no new + 4227/4231 event; peak/final handle growth was zero; peak/final private and + RSS growth were 389,120 and 475,136 bytes; and stop/PID-file cleanup passed. + The evidence is qualification-only. A checksum-verified retained ZIP at + `experiments/diagnostics/exp1-v11-ipc-qualification-718cf58dace44dba83bed54601854bc9.zip` + has SHA-256 + `2c4f87dc5bb123157f76e6be58b769bafef8943aba36ee8e9202601b50e62a02`. + Fresh v1.1 smoke is the next gate. +- **2026-07-31 -- v1.1 smoke passed:** Fresh run + `019fb83a-54bc-79db-b6ac-6189fb28f5f2` completed all 19 cells, 19 trial + batches, and 55 issued requests through four local Windows named-pipe + execution blocks. Correctness, CLI validation, warning, cleanup, and + protected-process gates passed. The immutable archive contains 1,073 files + and 29,778,180 bytes with content-tree + `sha256:c8e0e872d42c0df2ce2c19c4b030a29b615a7d250c95097dac9bff66fa4405e4`. + It is qualification-only. A strict preflight and one fresh five-sample + v1.1 pilot are now authorized; freeze and final remain prohibited. +- **2026-07-31 -- v1.1 pilot and Gate 3 passed:** Fresh exploratory run + `019fb84e-aef1-7fdc-9a56-1adbe712f30d` completed all 19 cells, 133 trial + batches, 38 warmups, 95 measured trials, and 385 issued requests with zero + product, correctness, infrastructure, cleanup, or primary-latency failures. + Its immutable 177,813,974-byte archive has content-tree + `sha256:e951342a73d94b2f21aec76d1926bd2b1fc196303fabde5ced01d6c3ab5a4da9`. + Two nine-file exploratory table generations are byte-identical at + `sha256:f3e2e0c4d6f39622f23251c2661eb270689cd980cd00730c53ae8a600067858e`. + These values remain ineligible for the manuscript. +- **2026-07-31 -- projection correction and decision:** A fail-closed + cross-run check initially treated mandatory evidence/status tracker updates + as scientific drift. Independent recursive comparison proved all scientific + identities identical. The corrected checker masks only byte/hash identity + for four exact pre-freeze status paths, enforces pre-freeze state and the + exact ten-file v1.1 set, and rejects all other drift. Full benchmark and + analysis tests pass 377 with five expected Windows symlink skips. Repeated + projection outputs are byte-identical at + `sha256:a7b4eda8cd1f15e59bca2e6495cb8b8c36914619e4c812143ac361bdb1803822`; + the conservative observed envelope is 1303.732241600 seconds against the + fixed 1400-second limit. Gate 3 passes. The scoped clean v1.1 freeze is the + next gate; the sole eligible final remains prohibited until freeze and + strict final preflight pass. +- **2026-07-31 -- v1.1 freeze and final preflight passed:** The measurement + source froze at paper commit + `1680b599129532f72e706b6acb12ef62c63759e2`, product clean direct `main` + `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, and annotated local + `paper-v1.1-freeze` tag object + `834c84534359f37653fb25ac45304091e82c37a6`. Freeze record + `exp1-v11-freeze-record-1680b59-5c48dae1.json` hashes to + `5b8ca3962f479f1776be0298acbbe7620b683a334c1964889b031122a0ffdc32`. + A fresh strict final validation passed 19 cells, 1,938 batches, 5,610 + requests, `product_cli`, `paper-100m`, four named-pipe blocks, and zero + findings/warnings. Gate 4 passed and exactly one final was authorized. +- **2026-07-31 -- sole v1.1 final and Gate 5 passed:** Run + `019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` completed all 19 cells, 1,938 + batches, 38 warmups, 1,900 successful/reportable measured trials, 5,610 + issued requests, and 4,800 correctness checks with zero classified failure, + report warning, or stderr byte. Its immutable archive contains 82,051 files + and 3,139,214,747 bytes at content tree + `sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`; + independent verification passed. Cleanup removed all owned runtime/workspace + state, and protected PID 62980 remained alive and untouched. +- **2026-07-31 -- post-freeze Table-1 compatibility erratum:** The frozen + generator failed closed because it recognized synthetic `os_edition` and + `os_build` keys but not the archive's canonical verified `os_caption` and + `os_build_number`. The archive was not mutated and the final was not rerun. + Local correction commit `538f6c98233863957082620329203348ddaa781c` + changes only this reader compatibility, preserves missing-field failure, + and has generator SHA-256 + `ff93953a6b8b94f10bc35138356a3039f6709a28ff0860d05c0da25e2064727b`. + The frozen generator identity + `7fd9c21d99ceb4b9fc3b962977ee9c0d270411ec2c6b76cc88960387a2fcbeb7` + remains disclosed. Replay on the frozen pilot proved numeric evidence, + numeric provenance, and Tables 2--4 byte-identical; only intended Table 1 + host presentation changed. +- **2026-07-31 -- Gates 6--7 and EXP1 complete:** Corrected final generation + into two fresh directories produced nine files/231,047 bytes each, with all + paths and hashes byte-identical at output tree + `sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. + Numeric-evidence v2 contains 153 unique selector-bound values; archive + source hashes and all output-manifest entries reverified. The claim map, + inventory, handoff, final Gate report, root/plan trackers, paper state, and + append-only log now define supported and unsafe wording. EXP1 has no + remaining execution or analysis blocker. Author review before LaTeX import + is the next external editorial action; broader paper work remains. diff --git a/ephemeral-sandbox-v1/plan/pw4-pw7-package-manifest.md b/ephemeral-sandbox-v1/plan/pw4-pw7-package-manifest.md new file mode 100644 index 0000000..51426fb --- /dev/null +++ b/ephemeral-sandbox-v1/plan/pw4-pw7-package-manifest.md @@ -0,0 +1,33 @@ +# PW4--PW7 package manifest + +## Included review artifacts + +- Manuscript: `main.tex`, `sections/`, `references.bib`, `main.pdf`. +- Evidence: `ARTIFACTS.md`, `numeric_evidence.json`, frozen archive and Table-A + paths named in `ARTIFACTS.md`, and `sections/results_numeric_bindings.md`. +- Reproduction: `scripts/generate_latex_results.py`, + `scripts/generate_bibliography.py`, `REPRODUCIBILITY.md`, and + `build_check.md`. +- Literature: `citation_requests.json`, `citation_lock.json`, + `citation_verification.md`, `literature/`, and `references/related_work.md`. +- Review and handoff: `REVIEWER_GUIDE.md`, `SUBMISSION.md`, + `submission_readiness.md`, `plan/reviewer_report.md`, and figure review + records. + +## Excluded material + +- Temporary PDF page renders under `plan/pw4-pw7-render/` are QA intermediates + only and are not submission artifacts. +- LaTeX auxiliary files, build caches, editor files, and prior draft prompts + are not release artifacts. +- Qualification, smoke, pilot, exploratory, and superseded experiment outputs + are not result evidence. +- The task packet and user-owned progress edits are planning context and are + not automatically staged for a paper commit. + +## Gate disposition + +**FAIL/GATED.** The declared review artifacts are complete and verified, but +author/affiliation/venue/disclosure metadata remains an external owner input. +Neither a pull-request merge nor external submission is performed by this +package. diff --git a/ephemeral-sandbox-v1/plan/reverse_outline_pw4_pw7.md b/ephemeral-sandbox-v1/plan/reverse_outline_pw4_pw7.md new file mode 100644 index 0000000..c90f9cf --- /dev/null +++ b/ephemeral-sandbox-v1/plan/reverse_outline_pw4_pw7.md @@ -0,0 +1,34 @@ +# PW4--PW7 reverse outline + +## Section 8: Evaluation + +1. State the only completed evaluation question and exclude broader correctness, + baseline, useful-work, and recovery questions. +2. Define the disclosed host, Docker Desktop, image, product/benchmark, fixture, + CLI transport, matrix, scheduling, trial, inclusion, cleanup, and sampling + conditions needed to reproduce interpretation. +3. Explain v1.0 failure and preregistered v1.1 local-IPC remediation without + reporting ineligible v1.0 or qualification values. +4. Disclose the Table-1 compatibility erratum and why it is numeric-neutral. +5. Present deterministic environment/startup/operation/resource tables. +6. Interpret only descriptive changes and campaign completeness, then restate + unavailable metrics and setting boundary. + +## Section 9: Limitations and Related Work + +1. Start with the evaluation boundary so source-defined mechanisms are not + mistaken for correctness or security evaluation. +2. Preserve attribution, protected-drop, restart/lease, diff-memory, physical + storage, and reflink/LayerStack 2.0 limitations. +3. Position execution-isolation systems, reversible state systems, coding-agent + control/integration systems, and foundational private-workspace/concurrency + work using verified citations. +4. State the composition/protocol niche and the Claim Plane novelty risk without + firstness, superiority, or empirical comparison language. + +## Section 10: Conclusion + +1. Recap the source-grounded execution-to-publication protocol. +2. State the one-treatment RQ3 evidence rather than a broad performance result. +3. Close on the explicit unanswered correctness, baseline, recovery, and + coordination questions. diff --git a/ephemeral-sandbox-v1/plan/reviewer_report.md b/ephemeral-sandbox-v1/plan/reviewer_report.md new file mode 100644 index 0000000..16b5f4b --- /dev/null +++ b/ephemeral-sandbox-v1/plan/reviewer_report.md @@ -0,0 +1,32 @@ +# PW8 skeptical reviewer report + +**Review outcome:** evidence-bounded manuscript is suitable for owner review; +external submission remains blocked by author and venue metadata. + +## Strengths + +- The contribution is stated as a runtime publication composition, not as a new + filesystem, merge, or coordination primitive. +- The only numerical displays are generated from the frozen Table-A output and + pass selector-level numeric verification. +- The evaluation names what it cannot establish: no baseline, workload suite, + useful-work result, security result, fault campaign, or cross-platform claim. +- Closest coordination, sandbox, reversibility, conflict-awareness, and merge + work is cited with terminal primary metadata records. + +## Findings and disposition + +| Severity | Finding | Disposition | +| --- | --- | --- | +| Release blocker | The manuscript uses `Anonymous authors`; no owner-approved author list, affiliations, venue/category, or disclosure text exists. | Remains an external owner action and is recorded in `SUBMISSION.md` and `paper_state.json`. | +| Major claim risk | A reader could mistake the local treatment for a comparative or team-productivity result. | Resolved by the abstract, Section 8 scope/threats text, table captions, claim map, and reviewer guide. | +| Major novelty risk | The implementation combines established isolated-workspace, optimistic-validation, and merge mechanisms. | Resolved by explicit positioning against CAID, CoAgent, Claim Plane, Palantir, Crystal, semantic merge, union mounts, and OCC. | +| Minor presentation issue | Sample-count cells used prose rather than the archived numeric display. | Resolved before the final build: the generated tables now render the frozen `100` sample value, verified by `check_numeric_evidence.py`. | +| Minor visual risk | Existing concept rasters differ in layout and palette density. | Resolved by final color/grayscale inspection and explicit non-evidence waivers in `figures/concept-figure-review.md`. | + +## Recommendation + +Do not broaden the claim set during authoring. After the owner supplies the +metadata listed above, rerun the documented projection, verification, build, +visual QA, and submission-readiness checks before deciding whether to mark the +paper submission-ready. diff --git a/ephemeral-sandbox-v1/plan/source_revalidation.md b/ephemeral-sandbox-v1/plan/source_revalidation.md new file mode 100644 index 0000000..8de4c23 --- /dev/null +++ b/ephemeral-sandbox-v1/plan/source_revalidation.md @@ -0,0 +1,41 @@ +# P2 measured-source revalidation + +**Audit date:** 2026-08-02 +**Historical baseline:** `b22862550e0a7cb4fe61ce581831e9244cc492b5` +**Measured source:** `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` +**Tag:** `paper-v1.1-freeze`, annotated object +`834c84534359f37653fb25ac45304091e82c37a6` + +## Revalidation outcome + +The manuscript retains only source-grounded behavior that exists at the +measured revision. Workspace/session, lease, capture, current-head resolution, +active-manifest publication, and lifecycle claims remain implementation claims. +They are not promoted to empirical correctness claims merely because source or +tests exist. `cli_contract_matrix.md` was regenerated from measured projection +sources and replaces the baseline matrix. + +## Material baseline-to-measured differences + +| Area | Measured-source change | Paper decision | +|---|---|---| +| Gateway transport | Typed `tcp`, `npipe`, and `unix` endpoint support; Windows defaults to local named pipe. | State that EXP1 used local named pipes per execution block and that TCP compatibility remains. | +| CLI configuration | Canonical `--gateway-endpoint URI`; `--gateway-socket` is a compatibility alias. | Use endpoint terminology in prose and contract matrix. | +| Request identity | Optional validated `--request-id` is accepted by management, runtime, and observability CLIs. | Replace the old runtime-only assertion. | +| Resource sampling | Workspace sampling adds available upperdir-byte observation. | Table 4 labels its field as `upperdir` delta and preserves unavailable allocated-block fields. | +| Workspace base | Measured revision can hash/reuse the shared base instead of always rebuilding it. | Treat as experiment implementation/provenance, not a general efficiency result. | +| Holder/supervisor and Docker executor | Completion polling/executor implementation changed. | Preserve lifecycle limits; make no new restart or reliability claim. | + +## Source anchors + +- CLI projections: `crates/sandbox-cli/src/projection/{manager,runtime,observability}.rs`. +- CLI behavior: `crates/sandbox-cli/src/{manager,runtime,observability,input,output}.rs`. +- Endpoint/client: `crates/sandbox-operations/client/src/{config,endpoint,client}.rs`. +- Gateway: `crates/sandbox-config/src/configs/gateway.rs` and + `crates/sandbox-gateway/src/gateway/{config,listener,lifecycle}.rs`. +- Workspace/session/publication: the existing LayerStack, workspace, + namespace-process, and runtime-operation source anchors now resolve at the + measured commit through the updated inventory and final manuscript snapshot. + +No historical freeze record, run archive, generated output tree, or v1.0 +evidence was edited in this revalidation. diff --git a/ephemeral-sandbox-v1/plan/task-packets/exp0-environment-qualification.md b/ephemeral-sandbox-v1/plan/task-packets/exp0-environment-qualification.md new file mode 100644 index 0000000..e666d4f --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/exp0-environment-qualification.md @@ -0,0 +1,68 @@ +# EXP0 environment qualification + +**Status:** Complete +**Date:** 2026-07-30 +**Scope:** Environment correctness only + +## Objective + +Qualify the native Windows workstation and Docker Desktop Linux engine using +only the released product CLIs for sandbox operations. + +## Corrected environment boundary + +- Host: native Windows x64, `DESKTOP-OLP1ADS`, build 26200. +- Container engine: Docker Desktop 29.0.1, Linux AMD64, `overlayfs`, cgroup v2. +- Sandbox guest: pinned Ubuntu 24.04 Linux AMD64 image. +- Product: clean `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`, release `v0.1.4`. +- Control path: native Windows gateway, manager CLI, runtime CLI, and + observability CLI. +- Daemon: released Linux x86-64 artifact running inside Docker containers. +- Host storage: NTFS. +- Python: not required for environment qualification. + +The earlier native-Ubuntu host contract was incorrect because it treated the +Ubuntu sandbox image as the host. It is superseded. + +## Work completed + +1. Downloaded and checksum-verified the official `v0.1.4` Windows AMD64 + release. +2. Staged the complete package under ignored + `ephemeral-sandbox\target\windows-v0.1.4`; product Git remained clean. +3. Added a PowerShell-only strict qualifier for the Windows host, package, + Docker Desktop engine, image, and CLI contracts. +4. Ran two independent product-CLI-controlled sandbox lifecycles. +5. Validated 20 CLI operations, response correctness, file correctness, + observability, product-CLI destruction, warning/failure absence, token + secrecy, and cleanup. +6. Archived the accepted artifact set. + +## Accepted evidence + +- Artifact directory: + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6` +- Summary: + `windows-docker-cli-env-summary.json` +- Archive: + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6.zip` +- Archive SHA-256: + `eea981665b031846677046d4c211e71ad144f8a32507c09058923241d4d0f7f9` + +## Acceptance result + +- Host/package/Docker/image preflight: pass. +- Client cohort: `product_cli`. +- Batches: 2/2. +- Validated operations: 20. +- Correctness: pass. +- Warnings/failures: 0/0. +- Cleanup: pass. +- Qualifier-owned containers/volumes after completion: 0/0. +- Elapsed time: 10 seconds, environment evidence only. + +## Verdict + +**GO.** The selected Windows plus Docker Desktop environment is qualified. +Nothing else is required for this environment-only task. diff --git a/ephemeral-sandbox-v1/plan/task-packets/exp0-focused-performance-protocol.md b/ephemeral-sandbox-v1/plan/task-packets/exp0-focused-performance-protocol.md index 8e7a870..5e64800 100644 --- a/ephemeral-sandbox-v1/plan/task-packets/exp0-focused-performance-protocol.md +++ b/ephemeral-sandbox-v1/plan/task-packets/exp0-focused-performance-protocol.md @@ -72,7 +72,7 @@ and an append-only experiment log. than an explicitly selected `paper-100m` base. - Initial sandbox creation/mount timing is not yet a first-class preserved operation metric. -- The 20-minute good-pass budget is a pilot acceptance target, not a verified - runtime. +- The 1,400-second good-pass budget is a pilot acceptance target, not a + verified runtime. - Depth 100 is stress-shaped and must not be described as a typical code repository. diff --git a/ephemeral-sandbox-v1/plan/task-packets/exp1-cli-performance-campaign.md b/ephemeral-sandbox-v1/plan/task-packets/exp1-cli-performance-campaign.md new file mode 100644 index 0000000..9b634e0 --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/exp1-cli-performance-campaign.md @@ -0,0 +1,564 @@ +# EXP1 CLI-only focused performance campaign + +**Status:** v1.0 preserved failed/ineligible; authorized v1.1 campaign resumed + +**Date prepared:** 2026-07-30 + +**Scope:** Complete Gates 0 and 3--7 of the focused RQ3 practical-performance +protocol + +**Authoritative protocol:** [`../../experiment_inventory.md`](../../experiment_inventory.md) + +**Run log:** [`../../experiments/experiment_log.md`](../../experiments/experiment_log.md) + +**Active amendment:** +[`../../experiments/exp1-v1.1-protocol-amendment.md`](../../experiments/exp1-v1.1-protocol-amendment.md) + +The packet below remains authoritative except for the explicit v1.1 amendment. +The amendment changes the Windows CLI-to-gateway treatment from loopback TCP +to isolated named-pipe IPC, replaces the active product/package/freeze +identities, adds a preregistered 25,000-invocation qualification gate, and +names the new tag `paper-v1.1-freeze`. It does not change the matrix, trials, +seed, timing boundary, metrics, resource cadence, correctness gates, +exclusions, 1,400-second projection limit, or exactly-one-final rule. All v1.0 +freeze and failed-final text below is retained as historical evidence. + +## Objective + +Complete the paper's bounded RQ3 practical-performance campaign on the already +qualified native Windows/Docker Desktop environment: + +1. implement and review the benchmark's released-product-CLI subprocess cohort; +2. validate the planner, runner, correctness, cleanup, provenance, and analysis + paths without collecting paper results; +3. run a five-sample exploratory pilot over every final cell; +4. resolve pilot-discovered protocol or instrumentation defects before freeze; +5. freeze the approved protocol and exact product, benchmark, image, binary, + fixture, and analysis identities; +6. run one complete `paper-good-pass`; +7. deterministically generate the four expected tables and a claim-mapped + evidence handoff. + +This packet must be executed as an evidence campaign, not as a request to obtain +favorable numbers. A negative, partial, slow, or failed result is retained and +reported. + +## Required read-first sequence + +Read every file below completely before editing or running anything: + +1. `C:\Users\yifan\.codex\skills\ai-research-writing\SKILL.md` +2. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\progress.md` +3. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiment_inventory.md` +4. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\environment_setup.md` +5. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\expected_tables.md` +6. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\experiments\experiment_log.md` +7. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\benchmark\PAPER_ARTIFACT.md` +8. `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1\benchmark\docs\phase-3-product-boundary.md` +9. `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\AGENTS.md` +10. `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\CLAUDE.md` +11. `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\README.md` +12. `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\docs\linux-setup.md` + +Also inspect the complete current Git status of both repositories. The paper +working tree is intentionally dirty and contains unrelated user work. Preserve +every pre-existing change. + +## Starting state that must remain true + +### Qualified environment + +| Item | Required value | +|---|---| +| Host | Native Windows x64, `DESKTOP-OLP1ADS`, build 26200 | +| Host capacity | 48 logical CPUs; 137,438,953,472 bytes physical memory | +| Host filesystem | NTFS | +| Docker | Docker Desktop client/server 29.0.1 | +| Docker engine | Linux AMD64, `overlayfs`, cgroup v2 | +| Sandbox image | `ubuntu:24.04@sha256:52df9b1ee71626e0088f7d400d5c6b5f7bb916f8f0c82b474289a4ece6cf3faf` | +| Product | clean `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`, annotated `v0.1.4` | +| Product checkout | `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox` | +| Paper checkout | `C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1` | +| Released package | `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4` | +| Performance workspace | deterministic `paper-100m`; never the product or paper checkout | + +The accepted environment evidence is: + +- directory: + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6`; +- summary: `windows-docker-cli-env-summary.json`; +- archive: + `C:\Users\yifan\code\Ephemeral-AI-Lab\final-host-staging\diagnostics\qualification-windows-docker-20260730-final-6.zip`; +- archive SHA-256: + `eea981665b031846677046d4c211e71ad144f8a32507c09058923241d4d0f7f9`. + +Do not reinterpret Ubuntu as the host. Do not add an SSH, ext4-host, +native-Ubuntu-host, WSL-host, or host-CPython qualification requirement. +Python 3.13 or newer is a benchmark-orchestration dependency, not part of the +already passed product environment gate. + +### Exact released executables + +Use only these native Windows executables from the staged official package: + +```text +C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4\bin\sandbox-gateway.exe +C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4\bin\sandbox-manager-cli.exe +C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4\bin\sandbox-runtime-cli.exe +C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox\target\windows-v0.1.4\bin\sandbox-observability-cli.exe +``` + +Expected SHA-256 values: + +```text +sandbox-gateway.exe 3a96bedcfa9857bd3881155d758ec2d969f6265456ec3b2878eb6dbb26dc9368 +sandbox-manager-cli.exe b43ec520edc2f436adc8aa7e8b2b50680bb9021883fe23d79a85b17afd2e10fe +sandbox-runtime-cli.exe df99f2993a7a9e305d33b656fa239b9e11b61a9e2da6e8dfc2f29ae8953067d4 +sandbox-observability-cli.exe 0e0471e52750805570876a6244868764c44e166ec653627b9ebd490176e2fcbe +``` + +Never resolve these executables from an ambient `PATH`, another build +directory, WSL, or the separately staged Linux raw binaries. + +## Scientific scope + +This packet completes only the focused RQ3 characterization of startup, +workspace/session readiness, command, file read, file write, file edit, and +their resource envelope in the disclosed environment. + +It does not complete the broader RQ1--RQ5 lane. The following are out of scope: + +- competitive superiority or matched-baseline rankings; +- shared-directory, Git-worktree, team, or swarm comparisons; +- general isolation/security claims; +- publication/fault/restart correctness campaigns; +- a universal concurrency ceiling; +- Windows reflink or LayerStack 2.0 performance; +- changes to the manuscript Results section before final evidence handoff. + +Do not place pilot values, qualification elapsed time, simulated preview +values, or partial final-run values in paper tables. + +## Fixed protocol decisions + +These decisions are made before viewing performance measurements: + +1. `product_cli` is the canonical client-cohort identifier. Do not use + `direct_client` or `cli_e2e` for a paper run. +2. Every sandbox lifecycle operation uses `sandbox-manager-cli.exe`. +3. Every workspace/session, command, and file operation uses + `sandbox-runtime-cli.exe`. +4. Every product-level sandbox observation and resource snapshot uses + `sandbox-observability-cli.exe`. +5. Docker commands may verify the engine and perform an independent leak + cross-check; they may not substitute for product CLI operations. +6. Primary latency is end-to-end native CLI subprocess latency. It starts + immediately before process creation and ends after exit, stdout/stderr + capture, JSON parsing, and response validation. +7. Primary latency includes process launch and CLI-to-gateway transport. + Product-reported internal durations are separate secondary observations. +8. Concurrent requests are released from one benchmark barrier. Batch + makespan ends after the last CLI response validates. +9. Setup, verification, resource sampling, and teardown are excluded from the + primary operation interval but must still use the product CLIs for sandbox + operations. +10. Every measured cell uses a fresh benchmark-owned copy of the deterministic + `paper-100m` base: 4,000 files, 104,857,600 logical bytes, maximum depth + 100, seed `20260712`. +11. File-operation targets are prepared inside that base during untimed setup. +12. Sandbox create + base mount is a distinct manager-CLI operation. Session + create is a distinct runtime-CLI `create_workspace_session` operation. +13. The final matrix has 19 cells: one sandbox-create cell, two + workspace/session-create cells, four command cells, and four each for read, + write, and edit. +14. The pilot uses the final two warmups plus five measured trials per cell. +15. The final pass uses two warmups plus 100 measured trials per cell. +16. Concurrency is 1 and 5 where specified. Payloads are 4 KiB and 256 KiB. +17. Scheduling is seeded randomized blocks. Resource sampling is 100 ms. +18. There are no automatic retries and no silent outlier removal. + +Any change to these decisions before protocol lock must be appended to the +experiment log with a reason. Any change after lock requires a protocol version +bump and rerunning every affected final cell. + +## Worktree and mutation rules + +- Preserve the dirty paper working tree and all unrelated files. +- Do not delete or clean existing `__pycache__`, test caches, generated + artifacts, or user changes merely to make status look clean. +- Set `PYTHONDONTWRITEBYTECODE=1` for Python validation to avoid adding more + bytecode noise. +- Keep the product repository on `main`, clean, and at the required commit. +- Do not create a product branch or product worktree. +- Do not modify product source unless a demonstrated CLI-contract defect makes + the campaign impossible. If that occurs, stop, log the evidence, and request + authorization before changing product behavior. +- Paper-local benchmark implementation, tests, presets, analysis, protocols, + and run artifacts are in scope. +- Do not commit, tag, or push unless the active user request explicitly + authorizes that Git mutation. A missing authorization is a Gate 4 blocker, + not permission to weaken the freeze requirement. +- Use `apply_patch` for manual file edits. +- Append every validation, smoke, pilot, final-run, failure, amendment, + analysis, and cleanup attempt to `experiments/experiment_log.md`. + +## Implementation contract + +### 1. Cohort model and planning + +Update the paper-local benchmark so that: + +- `product_cli` is accepted in backend plan types, validation, canonicalization, + manifests, environment metadata, reports, and web types if the web build is + part of validation; +- operation definitions advertise `product_cli` only when the operation has a + complete reviewed CLI implementation; +- `paper-env-smoke`, a new `paper-pilot`, and `paper-good-pass` select + `product_cli`; +- paper presets fail closed if expanded to `direct_client`; +- the expanded final plan contains exactly 19 cells, 1,938 trial batches, and + 5,610 product operation requests before failure or cancellation; +- the pilot contains the same 19 cells with two warmups and five measured + trials per cell. + +Do not rename the paper cohort to the web prototype's existing `cli_e2e` +spelling. Normalize the code and fixtures to `product_cli`. + +### 2. CLI product-access adapter + +Refactor the current direct `GatewayClient`-bound access layer behind an +explicit cohort interface and add a CLI implementation. The runner must choose +the implementation from the expanded plan; it must not construct direct +product access unconditionally. + +The CLI adapter must: + +- use absolute executable paths beneath the checksum-verified package; +- use argument arrays rather than a shell command string; +- launch subprocesses asynchronously so the concurrency barrier is real; +- pass the exact gateway socket and authentication token without logging the + token; +- preserve the benchmark request ID when the CLI contract supports it and + record the resulting response identity; +- capture stdout and stderr separately; +- require exit code zero, empty stderr where the released contract requires + it, exactly one valid JSON response, and the expected operation schema; +- convert the response into the benchmark's timed response/evidence model + without fabricating fields; +- retain raw redacted command metadata, start/end monotonic timestamps, elapsed + nanoseconds, return code, stdout artifact path, stderr artifact path, and + response validation status; +- cancel and reap subprocesses safely on timeout or campaign cancellation; +- never silently fall back to the direct gateway client. + +### 3. CLI operation ownership + +At minimum, implement these mappings: + +| Benchmark action | Required executable | +|---|---| +| list/inspect/create/destroy sandbox | manager CLI | +| create/publish/destroy workspace session | runtime CLI | +| execute command | runtime CLI | +| file read/write/edit and verification reads | runtime CLI | +| product snapshot/resources/topology used by the campaign | observability CLI | + +Use the source-derived CLI contract and the released executable `--help` +outputs as authority. Archive the relevant `--help` output and exact CLI +versions in the implementation-validation evidence. + +### 4. Timing and concurrency + +Use a monotonic high-resolution clock. For each request, start timing +immediately before asynchronous process creation and stop only after the +process has exited and its captured output has passed response validation. + +For concurrent batches: + +1. prepare every immutable argument vector; +2. register every request as waiting; +3. release all subprocess launches from one barrier; +4. retain per-request elapsed time; +5. calculate batch makespan from barrier release through the final validated + response; +6. compute throughput as completed requests divided by batch makespan. + +Do not subtract a separately estimated process-launch cost. Do not mix the +`wall_time_seconds` or `command_total_time_seconds` fields returned by some +commands into the primary latency distribution. + +### 5. Workspace and startup operation + +Extend file read, write, and edit cells so their per-cell workspace is copied +from the cached `paper-100m` seed. Record and verify the fixture manifest and +hash before each campaign. + +Add an explicit `create_sandbox` benchmark operation with: + +- one concurrency-1 cell; +- `paper-100m`; +- the shared network profile; +- manager-CLI process-launch-to-ready timing; +- sandbox identity/readiness validation; +- product-CLI destruction during teardown; +- independent Docker owner-label leak cross-check after cleanup. + +Do not derive this row from generic cell setup time. Do not count session +creation as sandbox creation. + +### 6. Correctness and cleanup + +Every trial is reportable only if: + +- the measured CLI exits successfully and validates; +- operation-specific correctness checks pass; +- verification reads use the runtime CLI; +- observability evidence correlates to the same sandbox/trial; +- session and sandbox teardown use the product CLIs; +- the benchmark-owned workspace is restored or removed according to policy; +- no gateway-owned container, volume, process, or runtime state leaks; +- the product checkout remains clean at the required commit; +- infrastructure did not fail. + +Preserve failed and partial trials. Never convert a failed sample to a retry. + +### 7. Resource and analysis output + +Confirm or implement deterministic emission of: + +- per-request end-to-end latency; +- concurrent batch makespan and throughput; +- p50, p95, and p99 over reportable measured trials; +- daemon CPU-time delta and peak RSS; +- sandbox CPU, peak memory, and block-I/O deltas; +- workspace logical and allocated-byte deltas; +- host free-space minimum; +- unavailable fields represented explicitly as `unavailable`, never zero. + +The analysis must regenerate all four tables in +`experiments/expected_tables.md` from immutable raw artifacts. No table number +may be manually typed or edited. + +## Required tests before any live pilot + +Add or update unit, contract, and integration tests for: + +- plan parsing and canonical `product_cli` serialization; +- rejection of `direct_client` by every paper preset; +- exact executable selection by operation; +- argument-vector construction and token redaction; +- valid JSON, malformed JSON, nonzero exit, unexpected stderr, timeout, and + cancellation; +- request IDs and response schema validation; +- end-to-end timing boundaries and concurrent barrier behavior; +- `paper-100m` use by all operation families; +- sandbox-create separation from session-create; +- correctness verification through CLI calls; +- cleanup and leak detection; +- raw observation, manifest, report, and table regeneration; +- 19-cell/1,938-batch/5,610-request final-plan counts. + +Run the narrow tests first, followed by the complete benchmark backend suite. +Use the prepared Python environment and disable bytecode writes: + +```powershell +$env:PYTHONDONTWRITEBYTECODE = '1' +Set-Location -LiteralPath 'C:\Users\yifan\code\Ephemeral-AI-Lab\research-papers\ephemeral-sandbox-v1' +& .\.venv\Scripts\python.exe -m pytest .\benchmark\backend\tests +``` + +If the virtual environment does not exist, create/install it off-clock and log +the tool versions and commands. Installation is forbidden once a pilot or +final campaign clock begins. + +## Execution phases + +### EXP1-A: audit and protocol review + +1. Read the required inputs and applicable repository instructions. +2. Record both Git states and hashes of every file that will be edited. +3. Audit all current direct-client construction, operation definitions, + manifests, resource sampling, and analysis paths. +4. Verify the fixed decisions in this packet are reflected consistently in the + protocol and table schemas. +5. Add an experiment-log entry. Do not run a live performance preset. + +Acceptance: Gate 0 cross-links/configuration pass, and no unresolved +measurement-boundary ambiguity remains. + +### EXP1-B: implement and validate `product_cli` + +1. Implement the cohort, operation mappings, timing, workspace policy, + sandbox-create cell, raw evidence, and tests. +2. Update `PAPER_ARTIFACT.md` with the complete paper-local modification list. +3. Run targeted and full benchmark tests. +4. Expand and inspect `paper-env-smoke`, `paper-pilot`, and + `paper-good-pass`. +5. Confirm paper presets cannot select or fall back to `direct_client`. +6. Append all attempts and failures to the log. + +Acceptance: deterministic tests pass; final expansion is exactly 19 cells, +1,938 batches, and 5,610 requests; no live performance result has been +collected. + +### EXP1-C: live CLI integration smoke + +1. Recheck the qualified environment and exact hashes. +2. Start a unique released gateway instance. +3. Run only `paper-env-smoke` through `product_cli`. +4. Require all operations and correctness checks to pass. +5. Require no warnings, failures, leaked containers, volumes, processes, or + runtime state. +6. Archive the complete smoke evidence and append the log. + +Acceptance: the benchmark's own CLI cohort, not the standalone qualifier, +passes its minimal live gate. + +### EXP1-D: exploratory pilot + +1. Re-run fast preflight without builds, installs, pulls, or source mutation. +2. Run `paper-pilot` once over all 19 cells with two warmups and five measured + trials per cell. +3. Mark every artifact and report `exploratory`; pilot numbers are ineligible + for manuscript tables. +4. Verify correctness, cleanup, timing separation, resource correlation, and + deterministic table regeneration. +5. Project the full good-pass duration and require no more than 1,400 seconds + (23 minutes 20 seconds). +6. Review anomalies and resolve instrumentation/protocol defects before freeze. +7. Append every attempt and amendment to the log. + +Acceptance: Gate 3 passes. A slow, failed, leaking, or ambiguous pilot blocks +freeze and final measurement. + +### EXP1-E: protocol and source freeze + +1. Resolve every Gate 0--3 item. +2. Record the clean product `main` commit, annotated release tag, tag object, + and binary hashes. +3. Obtain explicit user authorization before creating any new Git commit, + annotated `paper-v1-freeze` tag, or push. +4. Freeze the paper-local benchmark revision and complete plan hash. +5. Freeze the image digest, fixture manifest/hash, table schema, metrics, + exclusions, seed, trials, analysis code, and protocol version `v1.0`. +6. Append the freeze record to the experiment log. + +Acceptance: Gate 4 passes and no scientific decision remains conditional. +Without the required Git authorization, stop at this gate and report that +single exact blocker. + +### EXP1-F: final good pass + +1. Create a new immutable run directory under `experiments\runs`. +2. Re-run the strict fast preflight. +3. Confirm no build, image pull, installation, source mutation, environment + reconfiguration, or unrelated process load is introduced. +4. Run the frozen `paper-good-pass` exactly once. +5. Preserve the run manifest, expanded plan, raw observations, resource + samples, traces, stdout/stderr, failures, and cleanup proof. +6. Require all 19 cells to contain exactly 100 reportable measured trials. +7. Preserve and report the corpus even if partial or failed. Do not rerun + without a logged cause and protocol decision. + +Acceptance: Gate 5 passes with one complete provenance-rich corpus. + +### EXP1-G: deterministic analysis and paper handoff + +1. Validate the immutable raw corpus. +2. Generate all four expected tables exclusively from archived data. +3. Regenerate the tables a second time and require byte-identical output. +4. Record the analysis command, code identity, input selectors, output hashes, + exclusions, anomalies, and unavailable fields. +5. Create or update numeric-evidence v2 records before any number enters + LaTeX. +6. Map every result to its RQ3 claim and record supported and unsafe wording. +7. Update `claim_evidence_map.md`, `experiment_inventory.md`, `progress.md`, + `plan/progress.md`, `paper_state.json`, and the experiment log. +8. Do not draft broader performance, superiority, security, or multi-agent + claims. + +Acceptance: Gates 6 and 7 pass; every displayed number has a raw selector and +recomputable aggregate. + +## Run and artifact contract + +Each smoke, pilot, or final attempt uses a unique run ID and immutable directory: + +```text +experiments/ +|-- runs/ +| `-- RUN_ID/ +| |-- environment-preflight.txt +| |-- run-manifest.json +| |-- intent-plan.yml +| |-- expanded-plan.json +| |-- fixture-manifest.json +| |-- cli-help/ +| |-- raw/ +| |-- resources/ +| |-- logs/ +| |-- cleanup/ +| `-- failures.md +`-- analysis/ + |-- scripts/ + |-- tables/ + |-- numeric-evidence.json + `-- generation-log.txt +``` + +Generated `.benchmark-state` data may be used while the campaign is active, +but the complete retained evidence must be copied or linked into the immutable +run directory before the run is accepted. + +Every run manifest must include: + +- run ID and exploratory/final disposition; +- host, Docker, image, product, release, package, CLI, daemon, benchmark, + protocol, plan, fixture, and analysis identities/hashes; +- exact sanitized command vectors and environment; +- seed, warmups, trials, scheduling, timeout, and resource interval; +- start/end/elapsed timestamps; +- cell and request counts; +- failures, exclusions, cleanup result, and artifact paths. + +Authentication tokens and other secrets must be redacted from manifests, +events, console output, and logs. + +## Stop conditions + +Stop immediately, retain evidence, clean up safely, append the log, and report +the blocker if any of these occurs: + +- host, Docker, image, product, package, CLI, daemon, fixture, plan, or analysis + drift; +- a paper plan expands to `direct_client` or `cli_e2e`; +- any sandbox operation bypasses the product CLIs; +- malformed JSON, nonzero exit, unexpected stderr, request mismatch, or + correctness failure; +- setup/verification/teardown contaminates primary timing; +- missing or uncorrelated resource samples; +- disk pressure below the protocol threshold; +- repeated cleanup leakage or daemon instability; +- product working tree or branch drift; +- a measured cell has fewer than the required reportable trials; +- a requested commit, tag, or push lacks explicit user authorization. + +Do not weaken a gate, remove a slow cell, reduce final trials, retry silently, +or change exclusions after seeing results. + +## Required final report + +The executing agent must finish with: + +1. a Gate 0--7 PASS/FAIL table; +2. exact code and documentation files changed; +3. tests and commands run with exit status; +4. run IDs and absolute artifact paths; +5. hashes for the final run archive and generated tables; +6. failures, partial evidence, exclusions, unavailable metrics, and cleanup + outcome; +7. claim wording supported by the evidence; +8. wording still unsafe; +9. exact remaining blockers and the single next external action, if any. + +Never fabricate or estimate benchmark metrics in the final report. diff --git a/ephemeral-sandbox-v1/plan/task-packets/pw1-foundations.md b/ephemeral-sandbox-v1/plan/task-packets/pw1-foundations.md index 0adfaef..d2439de 100644 --- a/ephemeral-sandbox-v1/plan/task-packets/pw1-foundations.md +++ b/ephemeral-sandbox-v1/plan/task-packets/pw1-foundations.md @@ -161,5 +161,6 @@ No inconsistency in `claim_evidence_map.md` or `plan/terminology.md` was exposed ## Final outcome and remaining scientific risk -- Outcome: in progress. +- Outcome: complete. Both sections satisfy the bounded content requirements; the recorder-generated build and every declared quality, citation, structure, link, JSON, claim, whitespace, hash, and source-baseline check passed. +- Build: `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex`, exit 0; input SHA-256 `8a9d0f97487ccf937efd82eb6a5726b2c7c8b5e3e31be1434945826a00731708`; PDF SHA-256 `ba4963d3d5f6352e1829946290265671599432e9984e301d5626de7316435327`; build-log SHA-256 `719696c0aa0efb9fce4796efaab3cce5b27dd17563c4b60d7f4951b539aa7f30`. - Remaining scientific risk: source links remain provisional until `paper-v1-freeze`; no frozen evaluation establishes isolation strength, publication fault behavior, scaling, resource behavior, useful work, or a concurrency ceiling; publication attribution is best-effort after data commit; protected-drop policy and daemon-restart lease/substitution behavior remain unresolved. diff --git a/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-generation.md b/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-generation.md new file mode 100644 index 0000000..6d7db2d --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-generation.md @@ -0,0 +1,59 @@ +# PW2.5: Concept-Figure Generation and Review + +## Bounded scope + +- Generate four concept-method PNG assets from the approved prompt package: + - `figures/concept/fig_system_architecture.png`; + - `figures/concept/fig_publication_sequence.png`; + - `figures/concept/fig_lifecycle_state_machine.png`; + - `figures/concept/fig_reconciliation_decision.png`. +- Delegate generation to non-overlapping subagents. +- Inspect every output for scientific boundaries, exact terminology, layout, + label readability, and style consistency. +- Request targeted edits or redesigns when an output fails review. +- Update figure planning and review records only after selected assets exist. +- Do not edit manuscript TeX or treat the images as experimental evidence. + +## Shared style + +- Classic Academic precision with Modern Minimal spacing. +- Flat 2D, vector-inspired geometry on white. +- Consistent semantic palette and arrow grammar from + `figures/source/STYLE_GUIDE.md`. +- No isometric art, 3D, gradients, marketing decoration, fake metrics, + security imagery, or performance implications. + +## Review gates + +1. The intended message is readable without the manuscript paragraph. +2. Labels are exact, legible at final-paper scale, and not hallucinated. +3. Shared/private and precommit/post-commit boundaries are visually correct. +4. Active-manifest replacement is the data-visibility point wherever shown. +5. Rejection never becomes partial publication. +6. Audit and cleanup remain outside atomic data publication. +7. The four assets look like one visual family. +8. Lifecycle remains marked provisional until PW3. + +## Delegation + +- Architecture: `/root/architecture_figure`. +- Publication sequence: `/root/publication_sequence_figure`. +- Reconciliation flow: `/root/reconciliation_figure`. +- Lifecycle state machine: assigned when one concurrency slot becomes free. + +The three delegated generation attempts were interrupted and did not supply +the selected assets. The author subsequently supplied four PNGs from +`C:\Users\yifan\Downloads`; generator/model/seed metadata was not available. +Their dimensions, hashes, provenance boundary, and review findings are recorded +in [`../../figures/concept-figure-review.md`](../../figures/concept-figure-review.md). + +## Outcome + +- Draft-generation outcome complete: all four expected PNG paths exist. +- Drafting-stage review complete with recorded disparities. The author directed + that the images be integrated unchanged and that redesign, topology repair, + resolution normalization, and style-family harmonization be deferred to PW7. +- This outcome does not declare the figures submission-final. PW3 later passed + lifecycle/prose consistency with an explicit normal-path caption qualifier, + integrated all four files unchanged, and inspected the compiled color and + grayscale pages. Final visual acceptance remains a PW7 gate. diff --git a/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-prompts.md b/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-prompts.md new file mode 100644 index 0000000..c33a2f5 --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/pw2-5-figure-prompts.md @@ -0,0 +1,63 @@ +# PW2.5: Concept-Figure Prompt Package + +## Bounded scope + +- Create reusable prompt/spec sources for four concept-method figures: + 1. LayerStack and workspace-session architecture; + 2. workspace-to-publication sequence; + 3. session lifecycle state machine; + 4. current-head reconciliation decision flow. +- Define one shared visual style so the four assets form a coherent paper figure family. +- Update only figure planning/prompt sources and this workflow record. +- Do not generate final images, edit manuscript TeX, refresh the PDF attestation, run experiments, or modify source/benchmark artifacts. + +## Evidence and terminology + +- Architecture and execution: C1, D1--D3, Sections 3--4, and `plan/terminology.md`. +- Capture, reconciliation, and publication: C2, D4--D8, Section 5, and `plan/terminology.md`. +- Lifecycle: C4, D1, D6, D8, and D9; the prompt is provisional until PW3 completes Section 6. +- Generated concept figures are explanatory only. They are not correctness, performance, security, or cross-platform evidence. + +## Files allowed to edit + +- `figures/figure_plan.md` +- `figures/source/STYLE_GUIDE.md` +- `figures/source/fig_system_architecture_prompt.md` +- `figures/source/fig_publication_sequence_prompt.md` +- `figures/source/fig_lifecycle_state_machine_prompt.md` +- `figures/source/fig_reconciliation_decision_prompt.md` +- `plan/task-packets/pw2-5-figure-prompts.md` +- `plan/progress.md` + +## Prompt requirements + +Each figure source must contain: + +- the figure contract: class, role, message, conclusion, evidence hierarchy, entities, relationships, layout, backend, source, caption takeaway, and reviewer risk; +- one self-contained copy-paste prompt; +- exact label inventory; +- forbidden visual claims and terminology; +- expected output dimensions and file name; +- a deterministic-label fallback when generated text is unreliable. + +## Acceptance checks + +1. All four prompt files and the shared style guide exist. +2. Every prompt uses canonical manuscript terminology. +3. No prompt requests fake numbers, axes, performance trends, security imagery, or an oversized transaction boundary. +4. Architecture and publication prompts visibly separate shared/private state and data commit/post-commit activity. +5. Lifecycle is marked provisional until PW3. +6. Markdown links resolve and edited files pass whitespace checks. + +## Outcome + +- Complete. One shared style guide and four self-contained prompt/spec files + exist. Each prompt repeats its own style, palette, layout, exact labels, + constraints, output contract, and acceptance checklist, so it can be sent + independently to an image-generator agent. +- Validation passed: four-file existence and structure, shared/self-contained + style checks, canonical-boundary review, 121 relative Markdown links, + full-paper quality gate, and scoped Git whitespace check. +- Remaining boundary: no image asset has been generated or visually inspected. + The lifecycle state-machine prompt remains provisional until PW3 completes + Section 6. diff --git a/ephemeral-sandbox-v1/plan/task-packets/pw2-workspace-publication.md b/ephemeral-sandbox-v1/plan/task-packets/pw2-workspace-publication.md new file mode 100644 index 0000000..2c11348 --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/pw2-workspace-publication.md @@ -0,0 +1,183 @@ +# PW2: Workspace Execution and Capture/Publication + +## Bounded scope + +- Draft only Section 4, **Workspace Execution**, in `sections/04-workspace-execution.tex`. +- Draft only Section 5, **Capture and Publication**, in `sections/05-capture-publication.tex`. +- Preserve Sections 1--3 and 6--10, all experiment artifacts, the bibliography, figures, and planning work outside the authorized execution records. +- Update only this task packet, the skill progress log, the root milestone tracker after acceptance, the machine/build records, and recorder-produced build outputs. +- Do not modify, build, test, benchmark, commit, branch, tag, push, or otherwise mutate the `ephemeral-sandbox` source checkout. + +The paper-writing lane fully specifies PW2, but no standalone `lanes/prompts/pw2.md` exists. This execution therefore treats the PW2 block in `lanes/paper-writing.md` as the authoritative work-package contract and records the missing prompt as a planning inconsistency rather than inventing one. + +## Target venue and template + +- Target: arXiv `cs.OS`. +- Template: the existing 10 pt `article` manuscript in `main.tex`. +- Mode and stage: `full-paper` / `drafting`. +- Source evidence status: provisional baseline evidence at `b22862550e0a7cb4fe61ce581831e9244cc492b5` until an annotated `paper-v1-freeze` exists. + +## Authoritative inputs and evidence IDs + +- Project contract and state: `README.md`, `PRD.md`, root `progress.md`, `paper_state.json`, `plan/progress.md`, and `NEXT_AGENT_PROMPT.md`. +- Story and structure: `paper_story.md`, `paper_skeleton.md`, `lanes/paper-writing.md`, and `lanes/experiments.md`. +- Vocabulary and evidence: `plan/terminology.md`, `project_inventory.md`, `claim_evidence_map.md`, `cli_contract_matrix.md`, `references/related_work.md`, and `complexity_and_evolution.md`. +- Manuscript/build continuity: `main.tex`, `BUILD.md`, `build_check.md`, Sections 2--5, and the PW1 task packet. +- Primary PW2 claim-map scope: C1, C2, and D1--D8. +- Source evidence inspected in the read-only baseline: + - command and workspace-session services; + - workspace construction, snapshot leasing, OverlayFS projection, and namespace-holder/runner paths; + - session and sessionless file-operation paths; + - upper-directory capture; + - publication route, plan, fingerprint, opaque-directory expansion, resolve, and merge modules; + - layer staging, promotion, digest, manifest replacement, post-commit audit attribution, and session finalization. + +No source test was run. Test definitions remain source-derived evidence rather than fresh execution results. + +## Files allowed to edit + +- `sections/04-workspace-execution.tex` +- `sections/05-capture-publication.tex` +- `plan/task-packets/pw2-workspace-publication.md` +- `plan/progress.md` +- root `progress.md` +- `paper_state.json` +- `BUILD.md` +- `build_check.md` +- the recorder-selected build log and ignored build outputs +- only if drafting exposes a real inconsistency: `claim_evidence_map.md` or `plan/terminology.md` + +## Pre-existing worktree state + +PW2 began with authorized PW1 record updates already present in `BUILD.md`, `build_check.md`, `paper_state.json`, `plan/progress.md`, `plan/task-packets/pw1-foundations.md`, and root `progress.md`. It also found unrelated benchmark/experiment modifications and generated Python bytecode under `benchmark/` and `experiments/`. PW2 will preserve all of these changes and validate its own edit scope separately. + +## Section theses and paragraph roles + +### Section 4 thesis + +A workspace session projects the exact LayerStack view held by its creation lease into a Linux OverlayFS with private writable state, then runs commands and file operations through holder-owned namespaces while keeping implicit commands, explicit multi-call sessions, and sessionless file paths semantically distinct. + +Paragraph roles before drafting: + +1. Define workspace creation as the projection of a leased manifest and ordered LayerStack into a session-specific runtime view. Evidence: C1, D3. +2. Explain newest-first lower layers, private upper/work directories, and the Linux/OverlayFS platform boundary. Evidence: C1, D2, D3. +3. Explain the long-lived namespace holder and short-lived runner control path, including namespace join order and the daemon's position outside the command namespaces. Evidence: C1. +4. Explain the implicit-session path for sessionless `exec_command`, including shared networking and publish-then-destroy only after the command ledger drains. Evidence: D1. +5. Explain explicit sessions as multi-call containers whose commands and file operations share one private overlay until explicit publication or destruction. Evidence: D2. +6. Qualify session file operations, sessionless reads, sessionless direct write/edit paths, and selectable network behavior so no “every tool call” or universal-egress claim is made. Evidence: D1, D2, D2a. +7. State creation/finalization failure boundaries and transition from a complete private filesystem delta to capture; explicitly exclude process checkpointing, rollback, and a formal security guarantee. Evidence: C1, D1--D3. + +### Section 5 thesis + +Publication converts a complete private upper-directory delta into a typed candidate changeset, validates and reconciles the entire candidate against the current head, and exposes accepted data only through durable layer promotion followed by atomic active-manifest replacement, with attribution and cleanup outside that atomic data boundary. + +Paragraph roles before drafting: + +1. Explain capture as a typed filesystem delta covering writes, deletions, symlinks, directories, opaque directories, and OverlayFS whiteouts, while preserving literal `.wh.*` names for later rejection and reporting unsupported entries as protected drops. Evidence: C2, D4. +2. Explain planning against the leased base: base identity, protected-path and source/ignored routing, bounded opaque-directory expansion, and base fingerprints. Evidence: D5. +3. Explain current-head reconciliation under the writer lock: re-read the active manifest, accept unchanged fingerprints, reject structural divergence, and admit only eligible exact-file source writes to merge. Evidence: C2, D5. +4. Bound conflict-merge behavior to clean line-oriented text no larger than 8 MiB; when a concurrent source-path change requires merging, reject binary, invalid-UTF-8, oversized, non-file, structurally divergent, or text-conflicting cases. Distinguish this from an unchanged-fingerprint binary or oversized write, which can publish without merging, and do not claim semantic correctness or a fully resource-bounded diff. Evidence: C2, D5. +5. State whole-candidate resolution and rejection: no conflicting subset becomes visible, a no-op leaves the current head in place, and lifecycle handling after pre-commit failure is deferred to Section 6. Evidence: C2, D6. +6. Explain the durable commit sequence: write and sync a staging tree, rename it into the layer store, persist its digest, revalidate the active manifest, and atomically replace the active manifest. Evidence: C2, D7. +7. Delimit atomic data publication from best-effort post-commit attribution and later session cleanup, leaving cleanup-failure outcomes to Section 6. Evidence: C2, D7, D8. +8. State the baseline and platform qualifiers and transition to lifecycle/recovery: source inspection is provisional, no cross-platform crash proof or oversized-transaction claim is made, and cleanup/recovery are separate phases. Evidence: D7, D8. + +## Planned reverse outline + +The final reverse outline will record each paragraph's actual message, evidence IDs, and residual risk. Initial controls are: + +- do not generalize implicit sessions beyond sessionless `exec_command`; +- do not describe sessionless direct file writes/edits as private workspace activity; +- do not call namespace isolation a formal security boundary or claim universal network-egress denial; +- do not describe capture as process-state capture; +- do not interpret ordinary `.wh.*` filenames as capture-time whiteout markers; +- do not hide the explicit-session protected-drop fail-closed behavior or generalize it into an unverified uniform policy; +- do not call optimistic reconciliation serializable snapshot isolation or a full transaction; +- do not claim that a clean textual merge is semantically correct; +- do not claim that binary or oversized writes are categorically rejected: the restriction applies when concurrent source-path divergence requires a merge; +- do not describe the 8 MiB text limit as a complete CPU/RSS bound; +- do not extend atomic data publication to audit attribution, accounting, autosquash notification, session destruction, or cleanup; +- do not claim universal cross-platform crash durability; +- do not present source test definitions as freshly executed validation. + +## Final reverse outline + +### Section 4 + +| Paragraph | Message | Evidence | Residual risk and control | +|---|---|---|---| +| 1 | Creation acquires a lease before opening the workspace, so a live session retains the selected manifest and ordered layer paths even if the active head advances. | C1, D3. | The paragraph calls this a stable leased view, not serializable snapshot isolation. | +| 2 | Linux OverlayFS projects newest-first shared lower layers beneath a session-unique upper/work pair. | C1, D2, D3. | The non-Linux path is explicitly unsupported; no universal platform claim is made. | +| 3 | A long-lived holder owns namespace handles and short-lived runners join them for commands while the daemon stays outside. | C1. | Namespace separation is described as an implementation boundary, not a hostile-code security proof. | +| 4 | Only sessionless `exec_command` creates an implicit shared-network, publish-then-destroy session, and finalization waits for the command ledger to drain. | D1. | The paragraph distinguishes command completion from an initial response and does not generalize to every tool call. | +| 5 | Explicit sessions preserve one leased overlay across multiple command and file calls until deliberate publication or destruction. | D2. | Publication admission with active commands is stated; process-state rollback is not implied. | +| 6 | Session file operations use the live overlay, sessionless reads project the active head, sessionless writes/edits amend it directly, and network profiles have different boundaries. | D1, D2, D2a. | Shared networking is not called egress containment; direct file amendments are not called implicit sessions. | +| 7 | Creation is a fallible multi-stage admission with rollback attempts, while capture begins only from filesystem state in the private upper tree. | C1, D1--D3. | No claim is made that the runtime snapshots registers, memory, sockets, or arbitrary process state. | + +### Section 5 + +| Paragraph | Message | Evidence | Residual risk and control | +|---|---|---|---| +| 1 | Capture maps upper-tree files, links, directories, kernel whiteouts, and opaque metadata to typed changes and reports invalid or unsupported entries as protected drops. | C2, D4. | Literal `.wh.*` names are preserved until protected-path rejection; the explicit/generic protected-drop policy asymmetry is disclosed. | +| 2 | Planning validates the leased-base identity, routes source versus ignored paths, rejects reserved paths, fingerprints source state, and bounds opaque-directory expansion. | C2, D5. | The paragraph calls this a plan rather than accepted or durable data. | +| 3 | Under the writer lock, source fingerprints are checked against the current head and structural divergence rejects, while ignored-route writes deliberately do not receive source reconciliation. | C2, D5. | The source/ignored distinction prevents an overbroad isolation claim. | +| 4 | Concurrent exact-file source writes may use the eligible 8 MiB line merge; other required merge cases reject. | C2, D5. | Unchanged-path binary/oversized writes remain publishable; clean merge is not semantic correctness and the byte gate is not a complete CPU/RSS bound. | +| 5 | Resolution emits one complete changeset, one no-op, or one rejection; no unrelated resolved subset is passed to the writer. | C2, D6. | Lifecycle behavior after pre-commit failure is deferred rather than imported from D9. | +| 6 | Commit stages and syncs the resolved layer, promotes it, persists its digest, rechecks the manifest, and exposes it only through atomic active-manifest replacement. | C2, D7. | Failures may leave unselected artifacts; the claim concerns manifest-selected data, not zero residual files. | +| 7 | Best-effort attribution and subsequent destruction, accounting, notification, and reclamation lie after the atomic data-publication boundary. | C2, D7, D8. | No atomic coupling between data, audit, or cleanup is claimed; cleanup outcomes are left to Section 6. | +| 8 | The private-to-public path is bounded to the provisional Linux baseline and does not establish semantic merge correctness, cross-platform crash proof, or unlimited transaction size. | Boundaries attached to C2, D5, D7, D8. | Frozen fault and resource evidence remains pending. | + +No inconsistency in `claim_evidence_map.md` or `plan/terminology.md` required an edit. The missing standalone PW2 prompt and the protected-drop entry-point asymmetry remain recorded planning/scientific risks. + +## Required artifacts + +- Complete drafts of Sections 4 and 5 with exact headings and stable labels. +- A final paragraph-level reverse outline in this task packet. +- PW2 start/completion records in `plan/progress.md`. +- A root milestone update only after every acceptance check succeeds. +- A fresh recorder-generated build attestation in `paper_state.json`, `BUILD.md`, and `build_check.md`. +- Recorder build log and `main.pdf`. + +## Rejection checks + +- Reject performance, productivity, resource-savings, security, universal-egress, semantic-correctness, serializability, full-transaction, process-rollback, and cross-platform durability claims. +- Reject “every tool call creates a session”; reserve **implicit session** for sessionless `exec_command`. +- Reject wording that makes sessionless direct file writes/edits part of an implicit session. +- Reject any atomicity statement that includes post-commit attribution or cleanup. +- Reject any statement that categorically rejects binary or oversized clean-path writes, or presents the 8 MiB merge threshold as sufficient protection from adversarial diff CPU or memory use. +- Reject any edit to non-target manuscript sections or source/experiment artifacts. +- Reject completion if the build recorder, full-paper quality gate, citation check, links, JSON, section structure/order, headings/labels, whitespace, protected-file hashes, or source-baseline checks fail. + +## Validation commands + +- Declared manuscript command: `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex`. +- Skill build recorder: `python /scripts/record_build.py --run`. +- Full-paper gate: `python /scripts/research_quality_gate.py `. +- Citation checker: `python /scripts/check_citations.py main.tex references.bib`. +- JSON parse and `full-paper` / `drafting` assertions. +- Exact ten-section `main.tex` input count/order check. +- Exact Section 4/5 heading and label check. +- Relative-Markdown-link resolution check. +- Prohibited-claim scan and manual skeptical review. +- PW2-scoped `git diff --check`, plus whole-worktree whitespace inspection without changing unrelated files. +- Protected-file hash comparison. +- Paper/source `git status`, source branch, and source-commit verification. + +## Acceptance criteria + +1. Both target sections satisfy the bounded lane requirements and preserve their headings and labels. +2. Every strong sentence maps to C1, C2, or D1--D8, or is visibly a boundary or limitation. +3. The path from private session state to durable public head is complete and distinguishes capture, planning, reconciliation, commit, attribution, and cleanup. +4. Implicit commands, explicit sessions, session file operations, and sessionless file operations remain distinct. +5. Atomic data publication is bounded to the accepted active-manifest transition. +6. No later manuscript section, source file, benchmark artifact, or experiment artifact is overwritten. +7. This task packet contains the pre-draft plan and final reverse outline. +8. The declared build succeeds through the skill recorder with a fresh attestation. +9. All required quality, citation, link, JSON, section, claim, whitespace, hash, and source checks pass. +10. Root and skill progress records agree, and the source checkout remains clean on `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`. + +## Final outcome and remaining scientific risk + +- Outcome: complete. Sections 4 and 5 satisfy the bounded PW2 contract; the recorder-generated build and every declared quality, citation, structure, link, JSON, claim, whitespace, hash, and source-baseline check passed. +- Build: `latexmk -pdf -interaction=nonstopmode -halt-on-error main.tex`, exit 0; input SHA-256 `36877891dbce1e066589d8a295e436654ccb097970620003cdaf45871f74311b`; PDF SHA-256 `801ac91c302ae3ea7d5827d34dd4da09278e8f537409e3426fd4ff30c8ed36e7`; build-log SHA-256 `9fc4261ab327472e004f4f62466bf2266218d1b17a66cc5b852ee5dbe7b23265`. +- Remaining scientific risk: source links remain provisional until `paper-v1-freeze`; no frozen fault/resource evaluation establishes crash behavior or worst-case merge cost; attribution remains best effort after the data commit; protected-drop policy differs by publication entry point; and lifecycle/recovery failure handling remains for PW3. diff --git a/ephemeral-sandbox-v1/plan/task-packets/pw3-lifecycle-interface.md b/ephemeral-sandbox-v1/plan/task-packets/pw3-lifecycle-interface.md new file mode 100644 index 0000000..7b55f11 --- /dev/null +++ b/ephemeral-sandbox-v1/plan/task-packets/pw3-lifecycle-interface.md @@ -0,0 +1,244 @@ +# PW3: Lifecycle, Recovery, Implementation, and Operational Interface + +## Bounded scope + +- Draft Section 6, `Lifecycle and Recovery`. +- Complete Section 7, `Implementation and Operational Interface`, while + preserving and integrating the existing source-derived cost model. +- Integrate the four supplied concept-method PNGs into Sections 3, 5, and 6 + with evidence-bounded captions. +- Revalidate the lifecycle figure against the completed Section 6 prose. +- Record supplied-asset provenance, review findings, and the author's + instruction to defer visual/style/resolution changes to PW7. +- Refresh the reproducible manuscript build and workflow records. +- Do not modify the source repository, benchmark implementation, experiment + protocol, bibliography, or Sections 1--2 and 8--10. + +## Target venue and template + +- arXiv `cs.OS`. +- Current one-column 10 pt `article` scaffold with 1 inch margins. +- Concept figures are illustrative mechanism diagrams, not correctness, + performance, security, or cross-platform evidence. + +## Authoritative inputs and evidence + +- Claims C3--C4, D8--D13, and K1--K5. +- Sections 3--5 and their PW1/PW2 reverse outlines. +- `project_inventory.md`, `claim_evidence_map.md`, + `cli_contract_matrix.md`, `complexity_and_evolution.md`, and + `plan/terminology.md`. +- Baseline source checkout on `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- Lifecycle source: finalization policy, explicit publish, implicit finalize, + guarded destruction, holder-exit reconciliation, bounded recovery artifact, + shutdown convergence, squash, lease cleanup, and live remount paths. +- Interface source: maintainer component map, three CLI projections, routed + catalog, request construction, gateway discovery, output rendering, + catalog-derived help, and contract tests. + +## Files allowed to edit + +- `main.tex` +- `sections/03-system-model.tex` +- `sections/05-capture-publication.tex` +- `sections/06-lifecycle-recovery.tex` +- `sections/07-implementation-interface.tex` +- `figures/figure_plan.md` +- `figures/concept-figure-review.md` +- `plan/task-packets/pw2-5-figure-generation.md` +- `plan/task-packets/pw3-lifecycle-interface.md` +- `lanes/paper-writing.md` +- `plan/progress.md` +- root `progress.md` +- `paper_state.json` +- `BUILD.md` +- `build_check.md` +- the recorder build log and ignored LaTeX outputs + +The four PNG assets are inputs and must remain byte-identical during PW3. + +## Section theses and planned paragraph roles + +### Section 6 thesis + +The runtime makes precommit retry, accepted publication, discard, cleanup +failure, holder loss, and lease-aware compaction distinct lifecycle outcomes +rather than treating workspace finalization as one rollback-capable +transaction. + +Planned paragraph roles: + +1. Define the lifecycle state and admission boundary. +2. Explain explicit publication, precommit failure, rejection, and retry. +3. Separate commit/no-op from later cleanup and published-but-not-closed. +4. Contrast implicit publish-then-destroy finalization with explicit retry. +5. Explain deliberate discard and convergent resource teardown. +6. Bound holder-exit handling and recovery artifacts without claiming process + restoration or automatic replay. +7. Explain lease-aware squash, live remount, retained/faulty sessions, and GC. +8. State restart, cancellation, recovery, and platform evidence boundaries. + +### Section 7 thesis + +The implementation preserves architectural ownership through a role-separated, +catalog-derived operational contract, while its source-derived cost model +identifies scaling variables without converting them into measured results. + +Planned paragraph roles: + +1. Map the request path and component responsibilities. +2. Describe the management, runtime, and observability client roles and + baseline operation counts. +3. Define scope, request correlation, connection, and transport behavior. +4. Define JSON streams, exits, and catalog-derived help. +5. Bound role separation as an interface property rather than authorization or + a complete coordination plane. +6. Preserve and frame the existing source-derived cost model. +7. State the serialized publication and merge-resource hypotheses. +8. Close with Linux, source-freeze, and contract-test evidence boundaries. + +## Figure placement and caption plan + +- System architecture: Section 3, after shared/private/durable/public state is + defined. +- Publication sequence: Section 5, before the detailed capture-to-commit prose. +- Reconciliation decision flow: Section 5, next to current-head + reconciliation. +- Lifecycle state machine: Section 6, after explicit and implicit lifecycle + outcomes are defined. + +Captions must state the takeaway and any omitted boundary. They must not turn +the diagrams into experimental evidence or erase the post-commit boundary. + +## Supplied-figure handling + +- Preserve exact PNG bytes and record SHA-256, dimensions, and the absence of + generator/model/seed metadata. +- The author accepts the supplied figures for the drafting-stage manuscript. +- Record topology, layout, resolution, and style-family disparities. +- Defer image redesign, normalization, and replacement to PW7. +- Current acceptance is for manuscript use at the drafting stage, not for + submission-final figure quality. + +## Rejection checks + +- Reject wording that lets precommit failure or rejection publish a subset. +- Reject wording that lets post-commit cleanup failure roll back visible data. +- Do not imply that an implicit command session is retained for caller retry + after publication rejection. +- Do not call a bounded recovery artifact a process checkpoint, restored + session, or automatic replay. +- Do not claim daemon-restart correctness for in-memory lease/substitution + registries. +- Do not claim that live remount always succeeds; preserve migrated, retained, + faulty, and gone outcomes. +- Do not call role-separated clients an authorization or security boundary. +- Do not claim a complete task, intent, resource-lease, service-discovery, + handoff, scheduling, or coordination plane. +- Keep 8/10/8 operation counts explicitly tied to the provisional baseline and + final-tag regeneration. +- Keep cost and queueing statements source-derived hypotheses, not measured + performance results. +- Do not modify or regenerate the four supplied PNG files during PW3. + +## Required artifacts + +- Complete Sections 6 and 7. +- Four LaTeX figure inclusions with self-contained captions and stable labels. +- A figure provenance and QA record. +- Final paragraph-level reverse outline with evidence and residual risk. +- Synchronized root/lane/skill trackers. +- Fresh recorder-generated build attestation and inspected PDF. + +## Validation commands + +- `python /scripts/research_quality_gate.py ` +- `python /scripts/check_citations.py main.tex references.bib` +- `python /scripts/record_build.py --run` +- `python /scripts/parse_build_log.py main.log` +- JSON parse and `full-paper` / `drafting` assertions. +- Exact ten-section input count/order check. +- Exact Section 6/7 heading and label check. +- Figure-file existence, hash-preservation, and LaTeX-reference check. +- Relative Markdown link check. +- Prohibited-claim and terminology review. +- Scoped and whole-worktree `git diff --check`. +- Read-only source branch, commit, and clean-status check. +- Rendered-PDF visual inspection, including pages containing figures and + tables. + +## Acceptance criteria + +1. Sections 6 and 7 satisfy the PW3 lane contract and retain stable headings + and labels. +2. Every strong sentence maps to C3--C4, D8--D13, K1--K5, or is explicitly a + boundary or hypothesis. +3. Explicit retry, implicit finalization, no-op, discard, accepted publication, + published-but-not-closed, holder loss, and cleanup failure remain distinct. +4. Recovery wording does not exceed the implemented bounded-artifact and + in-process reconciliation mechanisms. +5. Client roles, scopes, request IDs, JSON streams, exits, help derivation, and + coordination boundary match the baseline contract. +6. The existing cost model remains source-derived and non-numerical in the + empirical sense. +7. All four supplied figures are integrated unchanged and their disparities + are recorded as PW7 debt. +8. The lifecycle figure is consistent with the completed prose or any + discrepancy is explicit in the review record. +9. The recorded build and all declared validation checks pass. +10. The source checkout remains clean at the audited baseline and unrelated + paper/benchmark/experiment changes remain untouched. + +## Final reverse outline + +### Section 6 + +| Paragraph | Message | Evidence | Residual risk | +|---:|---|---|---| +| 1 | Active/finalizing/finalize-failed states, a per-session gate, and the command ledger coordinate in-process lifecycle work without forming a durable transaction log. | C4 | Abrupt restart remains untested. | +| 2 | Explicit precommit capture/publication failure restores an active retained session for deliberate retry or destroy. | C4 | Protected-drop entry-point policy remains nonuniform. | +| 3 | Accepted commit and no-op precede cleanup; committed cleanup failure is published-but-not-closed and cannot roll back the manifest. | C4, D8, D9 | Attribution and cleanup remain outside data commit. | +| 4 | Implicit command sessions publish-then-destroy after ledger drain and do not become caller-managed retry sessions after rejection. | C4 | Capture-after-quiescence and teardown faults still need archived Linux evidence. | +| 5 | Deliberate discard rejects live-command teardown and converges workspace/cgroup cleanup without repeating completed raw teardown. | C4 | Failure interleavings remain source-derived. | +| 6 | Holder-exit handling cancels/joins work and preserves only a bounded diagnostic artifact for eligible implicit sessions, not a restored process/session. | C4 | Partial artifact writes and daemon restart remain unverified. | +| 7 | Squash honors lease boundaries and remount can migrate, retain, fault, disappear, or remain unchanged; GC remains reference-sensitive. | C4, K5 | Physical savings and every substitution ordering are unmeasured. | +| 8 | Shutdown is bounded in-process convergence; in-memory registries prevent a general automatic crash-recovery claim. | C4, K5 | Final fault campaign and source freeze are absent. | + +### Section 7 + +| Paragraph | Message | Evidence | Residual risk | +|---:|---|---|---| +| 1 | Component ownership follows the request path from projected catalog operation through client, gateway, manager, daemon, and low-level crates. | C3 | Baseline links remain provisional. | +| 2 | Three feature-gated clients expose baseline 8/10/8 role-specific operations with projection-integrity coverage. | C3, D10 | Counts require final-tag regeneration. | +| 3 | Request envelopes carry operation, ID, scope, and arguments; only runtime exposes an ID override, and gateway discovery follows explicit precedence. | C3, D11 | Correlation is not causal filesystem attribution. | +| 4 | Help/stdout/stderr/exit behavior forms a machine-readable shell contract, with command exit status remaining response data. | C3, D12 | Final binary fixtures and digests are absent. | +| 5 | Help joins the semantic catalog and projection but does not synchronize external website documentation or prove runtime correctness. | C3, D13 | Live-site drift remains. | +| 6 | Role separation is an orchestration-facing interface property, not authorization, security, or a complete coordination plane. | C3 | External orchestration semantics remain out of scope. | +| 7 | The cost model names manifest, lease, upperdir, changeset, fingerprint, byte, queue, and retained-reference variables without treating them as measurements. | K1--K5 | Empirical validation is absent. | +| 8 | The exclusive publication path creates a queueing/saturation hypothesis while private execution remains parallel. | K3 | No observed limiting regime is claimed. | +| 9 | The 8 MiB input gate does not independently bound Myers trace memory; line count and edit distance remain required stress axes. | K4 | Adversarial CPU/RSS evidence is absent. | +| 10 | Squash can reduce eligible depth but remains constrained by leases and coexistence of old/new representations. | K5 | Latency and physical storage savings are unmeasured. | +| 11 | Linux/OverlayFS, baseline-source, and contract-freeze limits close the implementation account. | C3, C4, K1--K5 | `paper-v1-freeze` and archived final contract tests remain open. | + +## Completed validation and outcome + +- The full-paper quality gate and citation-key check pass. +- All ten section inputs remain present exactly once and in order; Section 6 + and Section 7 headings and labels are stable. +- All four PNGs exist at the declared paths, retain their recorded SHA-256 + hashes, and are referenced once with stable labels and bounded captions. +- Lifecycle/prose consistency passes with the explicit normal-path qualifier; + all topology, layout, resolution, grayscale-contrast, and style-family + exceptions remain recorded for PW7. +- The final recorder run passed with no LaTeX errors, emergency stops, + undefined citations/references, missing files, or overfull boxes. Seven + underfull boxes remain in the two narrow source-derived tables, and the + comment-only bibliography produces the expected empty References warning. +- The 14-page PDF was rendered in color and grayscale. Figures appear on + pages 3, 6, 8, and 10; the Section 7 tables remain after their introductions; + no clipping, overlap, broken glyph, or page-number defect was found. +- The read-only source checkout remains clean on `main` at + `b22862550e0a7cb4fe61ce581831e9244cc492b5`. +- PW3 is complete. The four supplied assets remain drafting-stage review + drafts; their final visual disposition is exclusively a PW7 task. diff --git a/ephemeral-sandbox-v1/progress.md b/ephemeral-sandbox-v1/progress.md index 5fe362c..2256066 100644 --- a/ephemeral-sandbox-v1/progress.md +++ b/ephemeral-sandbox-v1/progress.md @@ -1,7 +1,7 @@ # Ephemeral Sandbox v1 — Paper Progress -**Last updated:** 2026-07-30 -**Overall status:** Baseline design, interface, story, related-work, and source-derived complexity audits are complete; PW0 is reproducibly built and attested, while all frozen evaluation evidence remains pending. +**Last updated:** 2026-07-31 +**Overall status:** Baseline design, interface, story, related-work, and source-derived complexity audits are complete; PW0--PW3 are reproducibly built and attested; the v1.0 final is permanently failed/ineligible; and the focused EXP1 v1.1 local-IPC campaign has completed Gates 0--7, including its sole final, immutable archive, deterministic tables, numeric-evidence v2, and claim handoff. Broader paper experiments and manuscript work remain. **Submission gate:** Not ready for arXiv. Use this tracker as the authoritative task list for the preprint. A checked item means its stated acceptance condition has been met; it does not imply that later claims are validated. @@ -10,7 +10,9 @@ Use this tracker as the authoritative task list for the preprint. A checked item - [x] Identify the paper’s primary contribution: leased, copy-on-write workspaces with conflict-aware atomic publication. - [x] Define the initial experimental baseline: upstream `main` at `b22862550e0a7cb4fe61ce581831e9244cc492b5`. -- [ ] Freeze the final paper snapshot with an annotated `paper-v1-freeze` tag after paper-specific fixes and benchmarks. +- [x] Freeze the active EXP1 v1.1 paper snapshot with an annotated + `paper-v1.1-freeze` tag after qualification, smoke, pilot, and projection. + Preserve the existing v1.0 `paper-v1-freeze` tag unchanged. - [x] Complete an initial design inventory and claim-boundary review. - [ ] Revalidate the design inventory against the final paper snapshot. - [x] Review initial related work: DeltaBox, Shepherd, and AgentBay. @@ -47,22 +49,77 @@ Claim mapping, citation verification, provenance, and skeptical review are share environment preflight, expected table schemas, simulated previews, and an append-only experiment log. - [x] Paper PW0: scaffold, vocabulary, planning artifacts, and later PW3/PW4 evolution of Sections 7--9 are preserved; the declared build completed with an executed tool/version/hash attestation on 2026-07-30. -- [ ] Paper PW1: execute [`lanes/prompts/pw1.md`](./lanes/prompts/pw1.md) to write Sections 2–3, Goals/Non-goals and System Model. -- [ ] Paper PW2: write Sections 4–5, Workspace Execution and Capture/Publication. -- [ ] Paper PW3: write Sections 6–7, Lifecycle/Recovery and Implementation/Interface, including the source-derived operational cost table. +- [x] Paper PW1: Sections 2--3 now define Goals/Non-goals, the System Model, and four evidence-bounded invariants; the recorder-generated build and required PW1 checks pass. +- [x] Paper PW2: Sections 4--5 now explain leased OverlayFS execution, implicit/explicit/sessionless paths, typed capture, current-head reconciliation, whole-changeset rejection, durable manifest publication, and the post-commit attribution/cleanup boundary; the recorder-generated build and required PW2 checks pass. +- [x] Paper PW3: Sections 6–7 now cover Lifecycle/Recovery and Implementation/Interface, preserve the source-derived operational cost model, integrate four unchanged review-draft figures, and pass the recorded build and PDF inspection. - [ ] Paper PW4: write Section 8 methodology and Section 9 related-work/source limitations/future evolution after protocol lock. - [ ] Paper PW5: write results and measured failure analysis after evidence lock. - [ ] Paper PW6: rewrite Introduction, Conclusion, contributions, title, and Abstract. - [ ] Paper PW7: complete whole-paper verification, build, and packaging. -- [ ] Experiment lane: create and review `experiment_inventory.md`. - -The next synchronization gate is **protocol lock**. Before final source freeze, create `experiment_inventory.md` (or an equivalent benchmark protocol) that fixes RQ1–RQ5, the accepted-work unit, workloads, baselines, worker grid, metrics, uncertainty method, seeds/repeats, integration/verification policy, and required runtime instrumentation. +- [x] Experiment lane: create `experiment_inventory.md`. +- [x] Experiment lane: review and approve the focused RQ3 protocol, implement + the `product_cli` benchmark cohort, and complete the five-sample exploratory + pilot defined by + [`plan/task-packets/exp1-cli-performance-campaign.md`](plan/task-packets/exp1-cli-performance-campaign.md). + +EXP1 Gates 0--4 passed and protocol v1.0 was frozen. The sole eligible +`paper-good-pass`, run `019fb6e5-c00b-7b02-8a3c-d76bd1346eb4`, failed after +853 of 1,938 batches when a mandatory observability CLI connection hit Windows +socket error 10048; TCP/IP Event 4227 confirms local endpoint-reuse pressure. +The verified corpus is `failed_ineligible`. Gates 5--7 fail, no numeric result +may enter the paper, and v1.0 must not be rerun. The broader RQ1--RQ5 +experiment lane still requires its own accepted-work, workload, baseline, +fault, and useful-work decisions. + +The author authorized a permanent CLI-focused remedy and campaign resumption. +The product candidate at clean direct `main` commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8` uses Windows named pipes for +local CLI traffic, retains explicit TCP compatibility outside the paper +treatment, and is staged as `target/windows-exp1-5c48dae1`. The earlier +active-store IPv4 proposal is superseded without execution; no host network +setting changed. The +[`v1.1 protocol amendment`](experiments/exp1-v1.1-protocol-amendment.md) +preregisters an ineligible 25,000-call concurrency-5 qualifier before fresh +smoke, pilot, a no-more-than-1,400-second projection, a new freeze, and exactly +one final. + +The exact v1.1 qualifier subsequently passed: all 25,000 invocations completed +successfully in 5,000 concurrency-5 batches, with zero gateway-owned TCP +endpoints, zero new TCP/IP 4227/4231 events, bounded gateway resource growth, +and validated cleanup. Fresh v1.1 smoke +`019fb83a-54bc-79db-b6ac-6189fb28f5f2` then passed all 19 cells, 19 batches, +55 issued requests, correctness, warning, transport, and cleanup gates; its +verified archive content tree is +`sha256:c8e0e872d42c0df2ce2c19c4b030a29b615a7d250c95097dac9bff66fa4405e4`. +Both artifacts remain qualification-only. Fresh five-sample pilot +`019fb84e-aef1-7fdc-9a56-1adbe712f30d` subsequently passed all 19 cells, 133 +trial batches, 95 measured trials, 385 issued requests, correctness, resource +correlation, warning, transport, cleanup, archive, and deterministic +exploratory-analysis gates. The conservative observed-envelope projection is +1303.732241600 seconds against the fixed 1400-second limit. Pilot and projected +values remain exploratory and ineligible. Gate 3 passed and authorized the +subsequent scoped v1.1 freeze. + +The authorized v1.1 freeze then passed at paper measurement commit +`1680b599129532f72e706b6acb12ef62c63759e2`, product commit +`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`, and annotated tag object +`834c84534359f37653fb25ac45304091e82c37a6`. The sole final +`019fb86c-096e-7589-a0a4-a6d6ef5d7f8b` completed all 19 cells and 1,900 +measured trials without a classified failure. Its immutable archive tree is +`sha256:606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. +After a disclosed post-freeze Table-1 host-schema compatibility erratum, two +final nine-file analyses are byte-identical at output tree +`sha256:27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`; +all 153 numeric values have raw selectors. EXP1 Gates 0--7 pass. Author review +before LaTeX import is the next editorial action, not an experiment blocker. Detailed section dependencies, work packages, and evidence gates are in [`paper_skeleton.md`](paper_skeleton.md). ## 3. Correctness evaluation -- [ ] Define reproducible environment, build/image digest, configuration, and random seeds. +- [x] Define and qualify the focused RQ3 environment, product release, binary + and image digests, configuration, and seed. Broader RQ1/RQ2 correctness and + fault campaigns remain unexecuted. - [ ] Run and archive private-write isolation tests for simultaneous workspaces. - [ ] Run and archive three-namespace connectivity/isolation tests. - [ ] Run and archive non-overlapping concurrent-publication tests. @@ -73,7 +130,26 @@ Detailed section dependencies, work packages, and evidence gates are in [`paper_ ## 4. Performance evaluation -- [ ] Choose and document a Linux cgroup-v2 test platform. +- [x] Choose and document the final focused RQ3 platform: native Windows x64 + host with Docker Desktop 29.0.1 providing a Linux AMD64 `overlayfs`, + cgroup-v2 engine and the pinned Ubuntu 24.04 sandbox image. +- [x] Implement and review a `product_cli` subprocess cohort for every paper + performance operation; `direct_client` and `cli_e2e` are prohibited. +- [x] Apply the deterministic `paper-100m` base to every measured cell and add + explicit manager-CLI sandbox-create-to-ready timing. +- [x] Run and archive a five-sample exploratory pilot over all 19 final cells; + its numbers remain outside paper tables. +- [x] Lock protocol `v1.0`, freeze product/benchmark/image/binary identities, + and launch the sole 19-cell `paper-good-pass` after every prior gate passed. + The run failed and therefore produced no eligible final result. +- [x] Pass and retain the preregistered v1.1 local-IPC qualifier. +- [x] Run and archive a fresh v1.1 smoke. +- [x] Run and archive a fresh v1.1 five-sample pilot, deterministically + regenerate exploratory tables, review anomalies, and pass the fixed + no-more-than-1,400-second projection. +- [x] Freeze v1.1, pass strict frozen preflight, execute exactly one eligible + final, verify its immutable archive, regenerate all four tables twice, and + complete numeric-evidence/claim handoff. - [ ] Implement or verify a fair independent-container/worktree-per-agent baseline. - [ ] Measure 1, 5, and 20 agents across 4 KiB, 256 KiB, and 3 MiB payloads. - [ ] Measure layer depths 1, 10, 50, and 100. @@ -91,9 +167,13 @@ Detailed section dependencies, work packages, and evidence gates are in [`paper_ - [x] Create arXiv-compatible `main.tex` and a reproducible local build command; execute it through the skill build recorder and retain the PDF, log, tool versions, and hashes. - [ ] Create verified `references.bib`; check that every citation supports its sentence. -- [ ] Create a source-grounded architecture figure. -- [ ] Create a workspace-to-publication sequence figure. -- [ ] Generate result figures/tables from archived measurement data. +- [x] Create a source-grounded architecture review draft; final visual repair is deferred to PW7. +- [x] Create a workspace-to-publication sequence review draft; final visual repair is deferred to PW7. +- [x] Create lifecycle and reconciliation review drafts; lifecycle is revalidated against Section 6 and final visual repair is deferred to PW7. +- [ ] Complete PW7 submission-final figure normalization, topology review, label audit, and style-family decision. +- [x] Generate the four focused EXP1 v1.1 result tables from archived data; + retain numeric-evidence v2 and a byte-identical independent regeneration. +- [ ] Generate result figures from archived data if the manuscript uses them. - [ ] Write core sections: system model, design, implementation, and evaluation. - [ ] Write framing sections: abstract, introduction, related work, limitations/future evolution, conclusion. - [ ] Create `ARTIFACTS.md` and archive code/data needed to reproduce every result. @@ -111,10 +191,23 @@ Detailed section dependencies, work packages, and evidence gates are in [`paper_ ## Active blockers and constraints -- There are no v1-specific archived performance measurements yet; do not state performance conclusions. +- The focused EXP1 v1.1 final corpus is eligible within its bounded RQ3 scope; + the earlier v1.0 final remains `failed_ineligible` and must not be pooled or + quoted as a result. - Existing test source is not proof of a passing v1 test run; rerun and archive results. - The prior scripted ten-lane run predates v1 and has incomplete provenance; it is exploratory only. - The accepted-work unit, structured-team and exploratory-swarm workloads, matched baseline policy, and final experiment protocol are not yet locked. +- The reviewed `product_cli` cohort and permanent local named-pipe treatment + passed qualification, smoke, pilot, projection, freeze, strict preflight, + sole final, archive, deterministic analysis, and handoff. EXP1 has no + remaining execution/analysis blocker; author review is required before + importing selected result rows into the manuscript. +- Focused RQ3 timing is end-to-end native CLI subprocess latency, including + process launch and CLI-to-gateway transport. Product-reported internal timing + may be retained only as a separate secondary field. +- The selected environment is already qualified. Do not reintroduce the + superseded native-Ubuntu-host, SSH, ext4-host, or CPython-host requirements; + Ubuntu 24.04 is the pinned sandbox image. - Attribution is best-effort after data publication rather than transactionally coupled to it; narrow the claim or strengthen and test the implementation before freeze. - Explicit/implicit protected-drop semantics and lease/substitution behavior across daemon restart need resolution. - The Windows Docker Desktop/WSL 2 reflink experiment failed with `errno=95`; do not make Windows reflink or performance claims. diff --git a/ephemeral-sandbox-v1/project_inventory.md b/ephemeral-sandbox-v1/project_inventory.md index 88e0062..ff77bdb 100644 --- a/ephemeral-sandbox-v1/project_inventory.md +++ b/ephemeral-sandbox-v1/project_inventory.md @@ -1,224 +1,71 @@ -# Ephemeral Sandbox v1: project and evidence inventory +# Ephemeral Sandbox v1 final project inventory -Last audited: 2026-07-30. +**Revalidated:** 2026-08-02 +**Product source for manuscript claims:** +[`5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8), +annotated tag `paper-v1.1-freeze` object +`834c84534359f37653fb25ac45304091e82c37a6` +**Historical comparison only:** baseline `b22862550e0a7cb4fe61ce581831e9244cc492b5` -## Provenance and evidence policy +## Evidence inventory -The source checkout is `C:\Users\yifan\code\Ephemeral-AI-Lab\ephemeral-sandbox`. Its existing `main` was fast-forwarded from `origin/main` and remained clean at the requested initial baseline, commit [`b22862550e0a7cb4fe61ce581831e9244cc492b5`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5). The repository's `AGENTS.md` and `CLAUDE.md` require direct work on `main` and prohibit side branches/worktrees. The requested `paper-v1-freeze` tag does not yet exist. Every source-backed statement below is therefore **baseline evidence, not final paper provenance**; it must be rechecked and relinked to the commit named by the final annotated tag. - -The authoritative [`PRD.md`](./PRD.md) and [`progress.md`](./progress.md) were read from the `research-papers` checkout at commit [`57777979f3988e3005717da833bea37b95552a99`](https://github.com/Ephemeral-AI-Lab/research-papers/tree/57777979f3988e3005717da833bea37b95552a99/ephemeral-sandbox-v1). They define the evidence gates, baseline, required paper artifacts, and progress checklist. An earlier loose-folder audit incorrectly reported them missing; this repository copy supersedes that statement. - -Evidence labels used here: - -- **S — source-proven:** implemented in the audited commit. -- **T — tested:** exercised by a named test at that commit. -- **D — documented:** public or maintainer description; not a substitute for frozen source. -- **E — experiment needed:** a proposed empirical claim or an existing exploratory protocol without v1-freeze results. -- **O — out of scope:** deliberately not claimed. - -The live site is useful product and architecture context, but it is not the frozen v1 interface contract. In particular, its [CLI overview](https://ephemeral-sandbox.com/docs/cli) currently lists 8 management, 7 runtime, and 5 observability operations, whereas the baseline projections contain 8, 10, and 8. The final CLI reference must be regenerated from the tagged projection sources. - -## Repository and evidence map - -| Area | Primary implementation | Principal tests | Paper use | +| Area | Measured-source evidence | Paper use | Boundary | |---|---|---|---| -| Semantic operation catalog and routing | [`crates/sandbox-operations/catalog`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog), especially [`routed.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/routed.rs) | [`catalog/tests/integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/tests/integrity.rs) | Operation identity, authority, visibility, scope, and routing. | -| CLI projections and contracts | [`projection/manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/manager.rs), [`runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/runtime.rs), [`observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/src/projection/observability.rs) | [`projection_integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/projection_integrity.rs), client-specific CLI tests | Frozen operational interface; contract evidence only. | -| Gateway/client protocol | [`sandbox-operations/client`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client), [`sandbox-operations/contract`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/contract) | [`client/tests/request.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/client/tests/request.rs), [`contract/tests/contract.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/contract/tests/contract.rs) | Request envelope, scope, connection discovery, token field, response envelope. | -| Layer history, leases, publish, squash | [`sandbox-runtime/layerstack`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack) | [`layerstack/tests/unit`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit) | Snapshot and publication mechanism; primary systems evidence. | -| Workspace lifecycle and overlay capture | [`sandbox-runtime/workspace`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace) | [`workspace/tests/unit/overlay_capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/tests/unit/overlay_capture.rs) | Private writable view, capture, lifecycle teardown. | -| Namespace execution | [`namespace-process`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-process), [`namespace-execution`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-execution) | Crate unit/integration tests | Holder-owned namespaces and namespace-joined operations. | -| Runtime orchestration and explicit sessions | [`sandbox-runtime/operation`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation) | [`workspace_session_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs), [`layerstack_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs) | Admission, capture/publish/destroy sequencing, retry and partial-success behavior. | -| Architecture narrative | [`docs/maintainer-architecture.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/docs/maintainer-architecture.md) | Source tree above | Maintainer-authored map; use source/tests for individual claims. | - -The request path is CLI or MCP adapter → semantic catalog/projection → operation client → authenticated newline-delimited JSON gateway → manager or sandbox daemon → runtime/workspace/layer services. This ownership map is described in the [maintainer architecture](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/docs/maintainer-architecture.md) and instantiated by the catalog routing sources above. **S/D** - -## System inventory - -### 1. Private executable workspace over shared history - -What is shared: - -- The project base, promoted layer directories, and ordered active history are shared runtime state. A manifest records a version, ordered layer references, and schema version; a root hash is derived from the serialized layer references ([model](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/model/mod.rs)). **S** -- Session creation first acquires a LayerStack snapshot and then opens a workspace over that lease ([create service](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/service/impls/create_workspace.rs), [lifecycle create](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/lifecycle/create.rs)). **S** -- The host checkout is copied into the initial base layer; it is not used as the agents' shared mutable working directory ([base build](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/workspace_base/build.rs), [base layer copy](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/workspace_base/layer.rs)). **S** - -What is private: - -- Each session gets fresh run, upper, and work directories ([overlay directories](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/dirs.rs)). The Linux overlay mounts shared read-only lowers with that private upper/work pair ([kernel mount](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs)). **S** -- A sessionless `exec_command` tool call creates one implicit workspace session with `PublishThenDestroy`; after its command ledger drains, the runtime captures and publishes or rejects the session delta, then destroys the session ([command execution](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/command/service/exec_command.rs), [finalization policy](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs), [implicit publication tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs)). **S/T** -- An explicit workspace session can carry private live state across multiple command and file tool calls until `publish_workspace_session` merges its captured delta back or `destroy_workspace_session` discards it ([session catalog](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime/workspace_session.rs), [explicit-session tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session.rs)). **S/T** -- Each session carries user, mount, and PID namespace handles. A separate network namespace exists only for the isolated-network profile; the shared-network profile joins the host network namespace while retaining the other namespace separations ([workspace network model](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/model.rs), [namespace creation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-process/src/holder/namespace.rs)). Commands join user → mount → PID → network handles ([setns sequence](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/namespace-process/src/runner/setns/namespaces.rs)). **S** -- Commands and file operations using the same explicit session ID observe the same live upperdir. Distinct sessions have distinct writable upperdirs. This is workspace isolation, not a formal security proof. **S/O** - -The “isolated workspace session per tool call” description is exact for sessionless `exec_command`, not for every runtime operation. A sessionless `file_read` projects the current LayerStack without a session, and sessionless `file_write`/`file_edit` amend the current head directly through one-layer operations ([read](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/read.rs), [write](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/write.rs), [edit](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/service/impls/edit.rs)). **S** - -The implementation uses the Linux overlay and namespace APIs; non-Linux overlay mounting returns unsupported ([kernel mount fallback](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs)). No v1 claim should be made for Windows overlay execution, reflink cloning, universal egress denial, or process-state rollback. **O** - -### 2. Snapshot leases and compaction - -- A lease captures the exact manifest value and the resolved ordered layer paths ([LayerStack and Lease](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/mod.rs)). **S** -- Releasing a lease removes a candidate layer only when it is absent from both the active manifest and every remaining lease ([lease cleanup](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/cleanup.rs)). **S** -- Squash plans compactable runs around the base and live-lease boundaries, builds replacements outside the writer lock, then rechecks and commits under an exclusive lock ([squash implementation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/squash.rs)). Replacement acquisition pins a substituted layer before the superseded lease is released ([lease rewrite](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/rewrite.rs)). **S** -- Tests cover lease boundaries, GC safety, racing publication, abort/crash shapes, and retention until the last lease releases ([squash tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/squash.rs)). **T** - -Bounded claim: leases preserve the captured logical snapshot and constrain compaction/garbage collection. They do not forbid all compaction while a session exists, and the audited code does not justify calling the protocol serializable snapshot isolation. The lease/substitution registries are in memory; daemon-restart ordering should be fault-tested before a broad crash-recovery claim. **S/E** - -### 3. Capture - -- Capture walks only the private overlay upperdir and interprets kernel overlay metadata for deletions and opaque directories. It emits regular-file writes, directories, symlinks, deletes, and opaque-directory changes; unsupported special entries and invalid paths are reported as protected drops ([capture implementation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs)). **S** -- Tests distinguish genuine kernel whiteouts and opaque markers from literal `.wh.*` filenames and cover unsupported/non-UTF-8 entries ([capture tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/tests/unit/overlay_capture.rs)). **T** -- Explicit publication serializes capture with command admission, requires no active command, and restores the session to active state after a precommit capture or publication failure ([explicit publish service](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs)). **S** - -An unresolved semantic asymmetry needs a maintainer decision: explicit session publication rejects every protected capture drop, whereas generic LayerStack planning permits an unsupported-special-file drop while retaining safe changes ([explicit drop gate](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs), [publish planner](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/plan.rs)). Confirm whether implicit and explicit policies are intentionally different. **E** - -### 4. Conflict-aware merge/reject - -- Planning checks the caller's base revision, protects runtime-owned paths, classifies source versus ignored changes, and expands opaque directories within a configured bound ([publish plan](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/plan.rs), [route policy](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/route.rs)). **S** -- Under the writer lock, every change resolves or one structured rejection aborts the whole data changeset; no resolved subset is published ([resolver](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs)). **S** -- Source paths compare a fingerprint from the leased base with the active head. Divergent exact-file writes can invoke a line-oriented three-way text merge; structural differences, conflicting edits, binary/invalid UTF-8 input, and files above the 8 MiB merge bound reject ([resolver](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/resolve.rs), [merge implementation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/merge.rs)). **S** -- Tests cover disjoint concurrent publication, stale structural conflicts, all-or-none rejection, binary divergence, and retryable structured merge conflict ([LayerStack publish tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/publish.rs), [explicit publication tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs)). **T** - -“Conflict-aware” in the paper must mean these exact fingerprint, structure, protected-path, and bounded text-merge rules. A clean publication is not evidence of semantic conflict-freedom or test correctness. **O** - -### 5. Atomic durable publication and its boundaries - -- Publication plans outside the writer lock, rereads and resolves against the active manifest under the exclusive lock, writes a staged layer, syncs it, renames it into the layer directory, writes its digest, rechecks the active manifest, prepends the layer, and atomically replaces the manifest ([publish operation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs), [atomic file helper](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/storage/fs.rs)). **S** -- After success, the durable data state is the promoted layer/digest plus the active manifest reference to it. Layer-byte accounting is written best-effort after commit. Audit attribution is also appended after the data commit and is explicitly best-effort; it can resolve to `unknown` ([publish service](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/layerstack/service/impls/publish_changes.rs), [audit implementation](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/file/audit.rs)). **S** -- Session cleanup is a later, fallible phase. If publication commits but destruction fails, the layer remains durable and the session reports partial success/finalize failure; tests prevent retry from duplicating the committed layer ([explicit publication lifecycle tests](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs)). **S/T** - -The defensible paper claim is **atomic durable data publication of one resolved changeset at the manifest visibility boundary**. Do not claim that data, attribution, accounting, and cleanup form one transaction, or make a generic cross-platform crash-durability claim: directory sync is a no-op on Windows in the audited helper. **S/O** - -### 6. Lifecycle and observability - -- Explicit CLI-created sessions use no automatic finalization; implicit command sessions use publish-then-destroy after the final command ([session model](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/model.rs)). **S** -- Explicit publication retains a rejected/precommit-failed session for retry and closes it only after commit/no-op; destruction is separately exposed ([runtime session catalog](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-operations/catalog/src/runtime/workspace_session.rs), [publish service](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/src/workspace_session/service/impls/publish_session.rs)). **S** -- The read-only observability catalog exposes aggregate or sandbox views without placing mutation operations in that client; exact operations and scopes are recorded in `cli_contract_matrix.md`. This is a role-specific interface and reduces accidental exposure, not by itself an authorization or security boundary. **S/O** - -## Public documentation inventory and trust level - -| Page | Useful description | Paper treatment | -|---|---|---| -| [Architecture](https://ephemeral-sandbox.com/architecture) | Layer history, leased snapshot, private overlay, publish/reject pipeline | Framing/diagram source; re-prove each behavior from tagged code and tests. | -| [LayerStack](https://ephemeral-sandbox.com/architecture/layerstack) | Newest-first history, leases, publish, squash | Do not reuse live performance figures. | -| [Overlay mount](https://ephemeral-sandbox.com/architecture/overlay-mount) | Shared lowers, private upper/work, capture | Public description; Linux support boundary agrees with source. | -| [Namespace runtime](https://ephemeral-sandbox.com/architecture/namespace-runtime) | Holder and one-operation runner pattern | Public description; network isolation is optional. | -| [Workspace coordination](https://ephemeral-sandbox.com/architecture/workspace-coordination) | Capture → plan → lock/resolve → commit/attribute | Useful system narrative, but it elides the best-effort attribution boundary found in source. | -| [Squash/remount](https://ephemeral-sandbox.com/architecture/squash-remount) | Background compaction and remount description | Do not use its observed timings without frozen-run provenance. | -| [CLI overview](https://ephemeral-sandbox.com/docs/cli) and [management](https://ephemeral-sandbox.com/docs/cli/management), [runtime](https://ephemeral-sandbox.com/docs/cli/runtime), [observability](https://ephemeral-sandbox.com/docs/cli/observability) | Current public commands and shell contracts | Known operation-count drift; not the v1-freeze contract. | -| [Publish/export guide](https://ephemeral-sandbox.com/docs/guides/publish-and-export) | Operator workflow | Background only; exact semantics come from source/tests. | -| [Operation reference](https://ephemeral-sandbox.com/docs/reference/operations) | Catalog concepts | Regenerate from final tag. | - -### Operating-system and worktree framing sources - -These sources support the problem model, not an empirical claim that native operating systems fail at a particular agent count: - -| Primary source | Source-proven primitive | Paper-safe inference | -|---|---|---| -| Linux [`rename(2)`](https://www.man7.org/linux/man-pages/man2/rename.2.html) | Atomic pathname replacement with existing open descriptors continuing to reference their objects. | Per-path atomicity does not create an atomic multi-file coding changeset or agent publication boundary. | -| Linux [`flock(2)`](https://man7.org/linux/man-pages/man2/flock.2.html) | Advisory file locks. | Locks coordinate agents only when every participating tool follows the same application protocol. | -| Linux [`inotify(7)`](https://man7.org/linux/man-pages/man7/inotify.7.html) | Filesystem event queues can overflow and report `IN_Q_OVERFLOW`. | Native filesystem watching should not be treated as a lossless coordination or audit ledger. | -| Linux [OverlayFS](https://docs.kernel.org/filesystems/overlayfs.html) | Copy-up, whiteout, opaque-directory, and upper/work-directory rules. | Unique session upper/work directories and faithful capture are correctness requirements; OverlayFS alone does not supply shared version history or publication semantics. | -| [Git worktree](https://git-scm.com/docs/git-worktree) | Distinct working trees share repository state. | Worktrees reduce direct file overwrite but do not define service ownership, current-head reconciliation, verification ordering, or agent-level attribution. | - -The paper should say that conventional operating systems expose human/application-oriented process, file, lock, and resource primitives but lack a first-class coding-agent workspace-session and publication abstraction. Avoid the broader historical claim that operating systems were built only for humans or are generally “incompetent.” - -## Interface inventory summary - -At the baseline commit the three source projections expose: - -- 8 system-scoped management operations; -- 10 sandbox-required runtime operations; -- 8 read-only observability operations, of which `snapshot` and `resources` accept system or sandbox scope and the remaining six require sandbox scope. - -The exact operation matrix, CLI usage, mutation status, routing scope, help derivation, request correlation, connection/token discovery, JSON streams, and exit codes are in [`cli_contract_matrix.md`](./cli_contract_matrix.md), with the projection files as primary evidence. - -## Test inventory: do not mix evidence classes - -### CLI/interface contract tests - -These test the adapter contract, not isolation or publication: - -- Catalog/projection completeness and uniqueness: [`projection_integrity.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/projection_integrity.rs). -- Catalog-derived help/search: [`help.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/help.rs). -- Manager operation set, system scope, JSON/progress streams, and errors: [`manager.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/manager.rs). -- Runtime selector, explicit request ID, JSON and exit behavior: [`runtime.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/runtime.rs), [`request_builder.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/request_builder.rs). -- Observability aggregate/scoped routing and rejection of other authority sets: [`observability.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/observability.rs). -- Compatibility fixtures and unknown-operation behavior: [`compatibility.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-cli/tests/compatibility.rs). - -### Runtime correctness tests - -- Capture semantics: [`workspace/tests/unit/overlay_capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/tests/unit/overlay_capture.rs). -- OCC, merge/reject, all-or-none data changeset, and no-op behavior: [`layerstack/tests/unit/publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/publish.rs). -- Explicit publish, retry, commit-versus-cleanup partial success, and admission serialization: [`operation/tests/workspace_session_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/workspace_session_publish.rs). -- Implicit capture/publish/destroy sequencing: [`operation/tests/layerstack_publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/operation/tests/layerstack_publish.rs). -- Lease-constrained squash and GC: [`layerstack/tests/unit/squash.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/unit/squash.rs). - -The six-thread disjoint-publication test is correctness evidence, not a throughput or concurrency-ceiling measurement. - -### Checks run during this audit - -- The source checkout remained clean at `b22862550e0a7cb4fe61ce581831e9244cc492b5`; `paper-v1-freeze` was absent. -- On Windows, 50 distinct CLI contract tests passed across compatibility, help, manager, runtime, observability, projection integrity, and request building. Two exact-fixture comparisons were observed to fail solely because generated output used LF while checked-in fixtures used CRLF: the compatibility aggregate and the management catalog-help snapshot. The analogous runtime and observability snapshot tests were then skipped while all of their other tests ran. The semantic strings, JSON, and exits in the observed diffs matched; no source file was changed. -- The Windows-compatible LayerStack unit subset passed 27 tests. The publication and squash modules are gated by `cfg(unix)`, so this run did not validate those mechanisms. -- The workspace runtime test target did not compile natively because `sandbox-observability-telemetry` imports non-Windows `rustix::fs`. Ubuntu WSL was present but had no Rust toolchain, so Linux-only capture, publication, squash, and operation tests were not rerun in this audit. -- Both abstract drafts in [`paper_story.md`](./paper_story.md) are within the requested 150–200 words (184 and 193 by the current audit's tokenization), and every relative Markdown link in the six maintained paper documents resolves locally. - -These checks do not replace the required frozen Linux test run. The final artifact should archive exact test commands, full logs, OS/kernel/filesystem, toolchain, and binary digests. - -## Source-derived complexity inventory - -The stage-by-stage derivation and symbols are maintained in [`complexity_and_evolution.md`](./complexity_and_evolution.md). The source supports the following cost-driver statements, but no performance conclusion: - -- Lease acquisition clones the active manifest and resolves every layer path; the in-memory registry retains one full manifest per lease ([`stack/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/mod.rs), [`lease/registry.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/registry.rs)). **S** -- Overlay creation validates, opens, and configures each lower layer; direct merged-view reads scan layers newest-first ([`kernel_mount.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/overlay/src/kernel_mount.rs), [`projection/mod.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/projection/mod.rs)). **S** -- Capture walks the private upperdir, sorts each directory, and retains metadata/source-path references rather than copying regular-file payload into memory ([`capture.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/workspace/src/overlay/capture.rs)). **S** -- Validated publication holds the exclusive writer guard across current-head resolution, merge, regular-file hashing/copy/sync, promotion, and manifest replacement ([`ops/publish.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/ops/publish.rs), [`layer/write.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/layer/write.rs)). This motivates writer wait/hold measurement; it does not prove a bottleneck. **S/E** -- Text merge admits at most 8 MiB per input but stores a Myers frontier trace whose memory still depends on line count and edit distance ([`merge.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/publish/merge.rs)). Adversarial CPU/RSS characterization or a tighter implementation bound is required. **S/E** -- Squash hardlinks regular-file winners, while lease boundaries constrain eligible blocks and lease release retains layers referenced by active history or any other lease ([`squash.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/squash.rs), [`flatten.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/squash/flatten.rs), [`lease/cleanup.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/src/stack/lease/cleanup.rs)). **S** - -## Experimental and writing assets - -### In the v1 source repository - -- [`.github/workflows/benchmark-boundary.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/.github/workflows/benchmark-boundary.yml) keeps the benchmark lab outside the runtime repository. -- [`config/bench.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/config/bench.yml) is a benchmark configuration template, not a result. -- [`occ_merge_bench.rs`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/blob/b22862550e0a7cb4fe61ce581831e9244cc492b5/crates/sandbox-runtime/layerstack/tests/occ_merge_bench.rs) is an ignored experimental harness. Its own comments distinguish directly measured scenarios from modeled scenarios; no number from it is paper-ready without a frozen rerun and raw artifact. -- No LaTeX manuscript, BibTeX database, or paper-ready result bundle was found in this checkout. - -### In the paper folder - -- [`progress.md`](./progress.md) is the current readiness and execution tracker, including parallel manuscript/source/benchmark/evidence lanes and synchronization gates. -- [`paper_skeleton.md`](./paper_skeleton.md) maps the planned manuscript sections to claims, source evidence, experiments, figures/tables, work packages, dependencies, and completion gates. -- [`paper_story.md`](./paper_story.md), [`claim_evidence_map.md`](./claim_evidence_map.md), [`complexity_and_evolution.md`](./complexity_and_evolution.md), and [`cli_contract_matrix.md`](./cli_contract_matrix.md) are the current story, evidence, cost-model, and interface sources for that skeleton. - -### External benchmark/test repository - -The sibling checkout `ephemeral-sandbox-test` was clean at commit [`d45618733c8bfe75466947fdb9c47bea67f74b78`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/tree/d45618733c8bfe75466947fdb9c47bea67f74b78). Its [`benchmark/README.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/blob/d45618733c8bfe75466947fdb9c47bea67f74b78/benchmark/README.md) describes an external lab; [`concurrency-scaling.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/blob/d45618733c8bfe75466947fdb9c47bea67f74b78/benchmark/presets/concurrency-scaling.yml) and [`publication.yml`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/blob/d45618733c8bfe75466947fdb9c47bea67f74b78/benchmark/presets/publication.yml) are protocols, not results. - -The deterministic ten-lane FlashCart demo and its recorded runs under [`demo/multi-agent`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-test/tree/d45618733c8bfe75466947fdb9c47bea67f74b78/demo/multi-agent) predate the paper freeze. They may inform workload design, but every current run is **exploratory** until reproduced with the final source commit, exact binaries/images, environment, command line, seeds, raw event log, and analysis version. - -The sibling `ephemeral-sandbox-layerstack-2-experiment` checkout was clean on `windows_experiment` at commit [`6e486fca75cf3afebd091f5ba8a2e48e77d9d05e`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/tree/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e). Its [`README.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/README.md) and [`EXPERIMENT.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/EXPERIMENT.md) preregister a single-storage-domain/reflink candidate and A/B/C acceptance protocol; pending tables and thresholds are not results. The pinned stock Windows Docker Desktop/WSL 2 feasibility run failed direct `FICLONE` with `errno=95`, so A-Windows failed and B-Windows remained blocked ([`CONCLUSION.md`](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/CONCLUSION.md), [sealed report](https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox-layerstack-2-experiment/blob/6e486fca75cf3afebd091f5ba8a2e48e77d9d05e/reports/20260719T100800Z-x86_64-6af011f4-s0001.md)). This is future-work and narrow negative feasibility evidence, not a v1 capability or a universal Windows result. - -## Required final-freeze checklist - -Before a source claim or number enters the paper: - -1. Create the annotated `paper-v1-freeze` tag and record its exact commit, tag object, source-tree status, Rust toolchain, lockfile digest, and build command. -2. Regenerate the CLI catalog/help from that tag and diff it against this matrix and the public site. -3. Replace every baseline source link in the paper artifacts with the final commit link. -4. Run and archive the separated CLI-contract and runtime-correctness test suites. -5. Record for every experiment: source commit/tag, benchmark-repository commit, OS/kernel/filesystem, hardware, cgroup/runtime configuration, compiler/toolchain, binary and container/image digests, command line, environment variables, workload commit, seeds/repeats, raw event data, exclusions, and analysis-code commit. -6. Label pre-freeze scripted data exploratory; do not combine it with final measurements. -7. Re-audit attribution wording, explicit-versus-implicit protected-drop policy, and restart/lease recovery. -8. Re-verify every 2026 preprint version and every sentence-level citation immediately before submission. - -## Present evidence gaps - -- Missing source freeze: no `paper-v1-freeze` tag. -- Missing paper-ready measurements: isolation matrix, publication/fault matrix, latency/resource scaling, and multi-agent workflow comparison. -- Missing source-derived cost validation: no frozen measurements yet test the predicted layer-depth, live-lease, upperdir-entry, changed-path/byte, merge-shape, writer-queue, or retained-history terms. -- Missing formal boundary: no proof of sandbox security, serializability, semantic merge correctness, universal egress denial, or cross-platform durability. -- Attribution is not transactionally coupled to data publication. -- Restart ordering for in-memory leases/substitutions needs explicit fault testing. -- Claim Plane's full design needs a deeper sentence-level audit because of strong novelty overlap. +| LayerStack/history/leases | `crates/sandbox-runtime/layerstack` | Source-grounded logical view, publication, and lease description | No serializable-isolation, crash, or physical-storage claim. | +| Overlay workspace/capture | `crates/sandbox-runtime/workspace`, `overlay` | Private upper/work projection and typed capture description | Linux/OverlayFS implementation; not security or measured isolation. | +| Namespace execution | `namespace-process`, `namespace-execution`, workspace holder | Holder/runner implementation description | Shared-network profile is not egress isolation; restart behavior is unmeasured. | +| Runtime operations | `sandbox-runtime/operation` | Explicit/implicit session and lifecycle behavior | Test presence is not a paper correctness result. | +| CLI projections | `sandbox-cli/src/projection/{manager,runtime,observability}.rs` | Measured-source 8/10/8 contract | Contract evidence only. | +| Endpoint/client/gateway | `sandbox-operations/client/{config,endpoint,client}.rs`, gateway listener/config/lifecycle | Windows named-pipe default, TCP compatibility, URI resolution, optional request IDs | EXP1 treatment is local Windows named pipe per execution block only. | +| Resource sampler | daemon observability resource source | Defines available sampled metrics/`upperdir_bytes` | Does not make unavailable allocated-block or LayerStack metrics zero. | + +## Baseline-to-measured source reconciliation + +The measured revision preserves the core workspace, capture, reconciliation, +and manifest-publication concepts used in the manuscript, while changing the +operational treatment in ways that must be reflected in paper prose: + +1. Gateway endpoints became typed TCP, Windows named-pipe, or Unix-socket + endpoints. Windows defaults to a local named pipe; TCP compatibility remains + explicit rather than removed. +2. `--gateway-endpoint` became the canonical global CLI flag with + `--gateway-socket` as a visible compatibility alias. +3. Optional validated request IDs moved from runtime-only use to all three + CLI clients. +4. Base-workspace handling and resource sampling changed to support the frozen + benchmark treatment; the paper treats these as implementation/provenance, + not general performance mechanisms. +5. Holder/supervisor and Docker-executor changes affect implementation details + but do not justify new correctness/recovery claims. + +`cli_contract_matrix.md` is the authoritative regenerated public-interface +summary. The manuscript must link source-derived statements to this measured +snapshot, not to the historical baseline. + +## Experiment inventory + +| Artifact | Identity and use | +|---|---| +| Protocol | `ephemeral-sandbox-v1-practical-performance-v1.1`; methodology in `experiment_inventory.md`. | +| Sole eligible final | `experiments/runs/019fb86c-096e-7589-a0a4-a6d6ef5d7f8b`; immutable content tree `606863f2843a7b19f04e27e2ba5b736d544dd143f56f6d3626611cb29bb44986`. | +| Measurement commit | paper `1680b599129532f72e706b6acb12ef62c63759e2`; product `5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8`. | +| Final analysis | output trees `final-v11-019fb86c-tables-a` and `-b`, byte-identical tree `27b53ee5acc049899b4e5821f8d92b14488c7d08ed076ba379af4799c765ad04`. | +| Numeric provenance | `numeric-evidence.json` plus `numeric-provenance.csv`: 153 unique selector-bound values. | +| Erratum | analysis-only commit `538f6c98233863957082620329203348ddaa781c`; canonical-host compatibility correction, with Tables 2--4 numeric neutrality validated. | + +The 3.1-GB archive is immutable and local/ignored. The failed v1.0 final, +qualifier, smoke, pilot, projection, setup, verification, teardown, partial, +and unavailable observations are retained for provenance but ineligible for +manuscript results. + +## Known limitations to preserve + +- Post-commit attribution is best effort and may be `unknown`. +- Explicit/implicit protected-drop behavior needs maintainer clarification. +- Lease/substitution state across daemon restart lacks a final fault evaluation. +- Merge trace memory remains line/edit-distance dependent despite the 8 MiB + input gate. +- EXP1 does not evaluate isolation, publication faults, baselines, deeper + scaling, useful work, or multi-agent quality. +- LayerStack 2.0 and reflink remain future work; the stock Windows/Docker/WSL + feasibility cell failed direct `FICLONE` with `errno=95`. diff --git a/ephemeral-sandbox-v1/references.bib b/ephemeral-sandbox-v1/references.bib index 983554c..7d59264 100644 --- a/ephemeral-sandbox-v1/references.bib +++ b/ephemeral-sandbox-v1/references.bib @@ -1,2 +1,138 @@ -% PW0 bibliography scaffold. -% Add entries only after metadata and sentence-level support are verified. +% GENERATED FILE. DO NOT EDIT BY HAND. +% Source: citation_lock.json, generated by scripts/generate_bibliography.py. + +@article{swe_minisandbox_2026, + author = {Danlong Yuan and Wei Wu and Enhan Zhao and Zhengren Wang and Xueliang Zhao and Huishuai Zhang and Dongyan Zhao}, + title = {{SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents}}, + journal = {arXiv preprint arXiv:2602.11210}, + year = {2026}, + url = {https://arxiv.org/abs/2602.11210} +} + +@article{agentbay_2025, + author = {Yun Piao and Hongbo Min and Hang Su and Leilei Zhang and Lei Wang and Yue Yin and Xiao Wu and Zhejing Xu and Liwei Qu and Hang Li and Xinxin Zeng and Wei Tian and Fei Yu and Xiaowei Li and Jiayi Jiang and Tongxu Liu and Hao Tian and Yufei Que and Xiaobing Tu and Bing Suo and Yuebing Li and Xiangting Chen and Zeen Zhao and Jiaming Tang and Wei Huang and Xuguang Li and Jing Zhao and Jin Li and Jie Shen and Jinkui Ren and Xiantao Zhang}, + title = {{AgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systems}}, + journal = {arXiv preprint arXiv:2512.04367}, + year = {2025}, + url = {https://arxiv.org/abs/2512.04367} +} + +@article{deltabox_2026, + author = {Yunpeng Dong and Jingkai He and Shiqi Liu and Yuze Hou and Dong Du and Zhonghu Xu and Si Yu and Baochuan Yang and Yubin Xia and Haibo Chen}, + title = {{DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback}}, + journal = {arXiv preprint arXiv:2605.22781}, + year = {2026}, + url = {https://arxiv.org/abs/2605.22781} +} + +@article{shepherd_2026, + author = {Simon Yu and Derek Chong and Ananjan Nandi and Dilara Soylu and Jiuding Sun and Christopher D Manning and Weiyan Shi}, + title = {{Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Traces}}, + journal = {arXiv preprint arXiv:2605.10913}, + year = {2026}, + url = {https://arxiv.org/abs/2605.10913} +} + +@article{caid_2026, + author = {Jiayi Geng and Graham Neubig}, + title = {{Effective Strategies for Asynchronous Software Engineering Agents}}, + journal = {arXiv preprint arXiv:2603.21489}, + year = {2026}, + url = {https://arxiv.org/abs/2603.21489} +} + +@article{coagent_2026, + author = {Hongtao Lyu and Dingyan Zhang and Mingyu Wu and Xingda Wei and Haibo Chen}, + title = {{CoAgent: Concurrency Control for Multi-Agent Systems}}, + journal = {arXiv preprint arXiv:2606.15376}, + year = {2026}, + url = {https://arxiv.org/abs/2606.15376} +} + +@article{claimplane_2026, + author = {Maxim Nikolaev}, + title = {{Claim Plane: Enforceable Change Intents and Dynamic Scope for Parallel Coding Agents}}, + journal = {arXiv preprint arXiv:2607.21909}, + year = {2026}, + url = {https://arxiv.org/abs/2607.21909} +} + +@misc{palantir_2012, + author = {A. Sarma and D. F. Redmiles and A. van der Hoek}, + title = {{Palantir: Early Detection of Development Conflicts Arising from Parallel Code Changes}}, + year = {2012}, + doi = {10.1109/TSE.2011.64}, + url = {https://doi.org/10.1109/TSE.2011.64} +} + +@misc{crystal_2013, + author = {Yuriy Brun and Reid Holmes and Michael D. Ernst and David Notkin}, + title = {{Early Detection of Collaboration Conflicts and Risks}}, + year = {2013}, + doi = {10.1109/TSE.2013.28}, + url = {https://doi.org/10.1109/TSE.2013.28} +} + +@misc{threewaymerge_2018, + author = {Marcelo Sousa and Isil Dillig and Shuvendu K. Lahiri}, + title = {{Verified three-way program merge}}, + year = {2018}, + doi = {10.1145/3276535}, + url = {https://doi.org/10.1145/3276535} +} + +@misc{occ_1981, + author = {H. T. Kung and John T. Robinson}, + title = {{On optimistic methods for concurrency control}}, + year = {1981}, + doi = {10.1145/319566.319567}, + url = {https://doi.org/10.1145/319566.319567} +} + +@misc{snapshot_isolation_1995, + author = {Hal Berenson and Phil Bernstein and Jim Gray and Jim Melton and Elizabeth O'Neil and Patrick O'Neil}, + title = {{A critique of ANSI SQL isolation levels}}, + year = {1995}, + doi = {10.1145/223784.223785}, + url = {https://doi.org/10.1145/223784.223785} +} + +@article{cooperbench_2026, + author = {Arpandeep Khatua and Hao Zhu and Peter Tran and Arya Prabhudesai and Frederic Sadrieh and Johann K. Lieberwirth and Xinkai Yu and Yicheng Fu and Michael J. Ryan and Jiaxin Pei and Diyi Yang}, + title = {{CooperBench: Why Coding Agents Cannot be Your Teammates Yet}}, + journal = {arXiv preprint arXiv:2601.13295}, + year = {2026}, + url = {https://arxiv.org/abs/2601.13295} +} + +@misc{agenticflict_2026, + author = {Daniel Ogenrwot and John Businge}, + title = {{AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub}}, + year = {2026}, + doi = {10.1145/3805760.3814923}, + url = {https://doi.org/10.1145/3805760.3814923} +} + +@article{teambench_2026, + author = {Yubin Kim and Chanwoo Park and Taehan Kim and Eugene Park and Samuel Schmidgall and Salman Rahman and Chunjong Park and Cynthia Breazeal and Xin Liu and Hamid Palangi and Hae Won Park and Daniel McDuff}, + title = {{TeamBench: Evaluating Agent Coordination under Enforced Role Separation}}, + journal = {arXiv preprint arXiv:2605.07073}, + year = {2026}, + url = {https://arxiv.org/abs/2605.07073} +} + +@article{swebench_2024, + author = {Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan}, + title = {{SWE-bench: Can Language Models Resolve Real-World GitHub Issues?}}, + journal = {arXiv preprint arXiv:2310.06770}, + year = {2023}, + url = {https://arxiv.org/abs/2310.06770} +} + +@article{paperbench_2025, + author = {Giulio Starace and Oliver Jaffe and Dane Sherburn and James Aung and Jun Shern Chan and Leon Maksin and Rachel Dias and Evan Mays and Benjamin Kinsella and Wyatt Thompson and Johannes Heidecke and Amelia Glaese and Tejal Patwardhan}, + title = {{PaperBench: Evaluating AI's Ability to Replicate AI Research}}, + journal = {arXiv preprint arXiv:2504.01848}, + year = {2025}, + url = {https://arxiv.org/abs/2504.01848} +} diff --git a/ephemeral-sandbox-v1/scripts/generate_bibliography.py b/ephemeral-sandbox-v1/scripts/generate_bibliography.py new file mode 100644 index 0000000..833e0b8 --- /dev/null +++ b/ephemeral-sandbox-v1/scripts/generate_bibliography.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +"""Generate BibTeX only from terminally verified citation-lock metadata.""" + +from __future__ import annotations + +import json +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +LOCK = ROOT / "citation_lock.json" +OUT = ROOT / "references.bib" + + +def escape(value: object) -> str: + text = str(value) + return ( + text.replace("\\", r"\textbackslash{}") + .replace("&", r"\&") + .replace("%", r"\%") + .replace("_", r"\_") + .replace("#", r"\#") + ) + + +def entry(record: dict[str, object]) -> str: + if record["status"] != "verified": + raise RuntimeError(f"citation lock is not terminal: {record['key']}") + metadata = record["metadata"] + key = record["key"] + identifier = record["identifier"] + authors = " and ".join(escape(author) for author in metadata["authors"]) + title = escape(metadata["title"]) + year = metadata["year"] + if identifier.startswith("arxiv:"): + arxiv_id = identifier.split(":", 1)[1] + return f"""@article{{{key}, + author = {{{authors}}}, + title = {{{{{title}}}}}, + journal = {{arXiv preprint arXiv:{arxiv_id}}}, + year = {{{year}}}, + url = {{https://arxiv.org/abs/{arxiv_id}}} +}}""" + doi = identifier.split(":", 1)[1] + return f"""@misc{{{key}, + author = {{{authors}}}, + title = {{{{{title}}}}}, + year = {{{year}}}, + doi = {{{doi}}}, + url = {{https://doi.org/{doi}}} +}}""" + + +def main() -> None: + lock = json.loads(LOCK.read_text(encoding="utf-8")) + records = lock.get("records") + if not isinstance(records, list) or not records: + raise RuntimeError("citation lock contains no records") + output = [ + "% GENERATED FILE. DO NOT EDIT BY HAND.", + "% Source: citation_lock.json, generated by scripts/generate_bibliography.py.", + "", + ] + output.extend(entry(record) + "\n" for record in records) + OUT.write_text("\n".join(output), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/ephemeral-sandbox-v1/scripts/generate_latex_results.py b/ephemeral-sandbox-v1/scripts/generate_latex_results.py new file mode 100644 index 0000000..ee43faa --- /dev/null +++ b/ephemeral-sandbox-v1/scripts/generate_latex_results.py @@ -0,0 +1,258 @@ +#!/usr/bin/env python3 +"""Render immutable EXP1 table artifacts as provenance-linked LaTeX tables. + +This is a paper-side projection: it reads the frozen Table-A output directory +and writes only the manuscript-facing registry, binding report, and TeX input. +It never writes into the experiment archive or analysis output directory. +""" + +from __future__ import annotations + +import json +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +FROZEN_REL = Path("experiments/analysis/final-v11-019fb86c-tables-a") +FROZEN = ROOT / FROZEN_REL +OUT_TEX = ROOT / "sections/generated_results_tables.tex" +OUT_REGISTRY = ROOT / "numeric_evidence.json" +OUT_BINDINGS = ROOT / "sections/results_numeric_bindings.md" + + +def tex(value: object) -> str: + return str(value).replace("_", r"\_").replace("%", r"\%") + + +def fixed(value: float, places: int = 3) -> str: + rendered = f"{value:.{places}f}" + return rendered.rstrip("0").rstrip(".") if "." in rendered else rendered + + +def metric_row(values: dict[str, float], sample_word: str = "100") -> str: + return " & ".join( + ( + sample_word, + fixed(values["p50_ms"]), + fixed(values["p95_ms"]), + fixed(values["p99_ms"]), + fixed(values["throughput_ops_s"], 2), + ) + ) + + +def operation_case(row: dict[str, object]) -> str: + raw_operation = str(row["operation_label"]) + if raw_operation.startswith("`") and raw_operation.endswith("`"): + operation = r"\texttt{" + tex(raw_operation[1:-1]) + "}" + else: + operation = tex(raw_operation) + case = str(row["case"]) + if row["key"][1] == 4096: + case += ", small payload" + elif row["key"][1] == 262144: + case += ", large payload" + concurrency = "single client" if row["concurrency"] == 1 else "five clients" + return f"{operation}; {tex(case)}; {concurrency}" + + +def resource_case(row: dict[str, object]) -> str: + operation = row["key"][0] + if operation == "create_workspace": + return "Workspace create" + if operation == "exec_command": + return "Command no-op" + if operation == "file_read": + return "Read, large payload" + if operation == "file_write": + return "Write, large payload" + if operation == "file_edit": + return "Edit, large payload" + raise ValueError(f"unexpected resource operation: {operation}") + + +def evidence_id(entry: dict[str, object]) -> str: + where = entry["selector"].get("where", {}) + return str(where["evidence_id"]) + + +def ids_for_prefix(entries: list[dict[str, object]], prefix: str) -> list[str]: + return [evidence_id(entry) for entry in entries if evidence_id(entry).startswith(prefix)] + + +def render_tables(tables: dict[str, object], entries: list[dict[str, object]]) -> str: + environment = tables["environment"]["fields"] + startup = tables["startup"]["rows"] + operations = tables["public_cli_operations"]["rows"] + resources = tables["resources"]["rows"] + + environment_lines = [ + ("Host", "Windows host; see frozen preflight record"), + ("Container engine OS", environment[1]["display"]), + ("Architecture", environment[2]["display"]), + ("CPU logical processors", "48"), + ("Memory bytes", "137,438,953,472"), + ("Sandbox limits", "one vCPU / 512 MiB / 256 PIDs"), + ("Workspace fixture", "100 MiB / 4,000 files / depth 100"), + ("Trial plan", "two warm-ups + 100 measured"), + ("Gateway transport", environment[14]["display"]), + ] + rendered_environment = "\n".join(f"{tex(field)} & {tex(value)} \\\\" for field, value in environment_lines) + + startup_lines = "\n".join( + f"{tex(row['stage'])} & {'single client' if row['concurrency'] == 1 else 'five clients'} & " + + metric_row(row["values"]) + + r" \\" + for row in startup + ) + operation_lines = "\n".join( + operation_case(row) + " & " + metric_row(row["values"]) + r" \\" for row in operations + ) + resource_compute_lines = "\n".join( + " & ".join( + ( + resource_case(row), + "single client" if row["concurrency"] == 1 else "five clients", + fixed(row["values"]["daemon_rss_bytes"]["value"]), + fixed(row["values"]["sandbox_memory_peak_bytes"]["value"]), + fixed(row["values"]["sandbox_cpu_time_ns"]["value"]), + ) + ) + + r" \\" + for row in resources + ) + resource_io_lines = "\n".join( + " & ".join( + ( + resource_case(row), + "single client" if row["concurrency"] == 1 else "five clients", + fixed(row["values"]["sandbox_block_read_bytes"]["value"]), + fixed(row["values"]["sandbox_block_write_bytes"]["value"]), + fixed(row["values"]["upperdir_bytes"]["value"]), + ) + ) + + r" \\" + for row in resources + ) + ids = { + "environment": ids_for_prefix(entries, "table1."), + "startup": ids_for_prefix(entries, "table2."), + "operations": ids_for_prefix(entries, "table3."), + "resources": ids_for_prefix(entries, "table4."), + } + comments = "\n".join( + "% numeric-evidence ids for " + name + ": " + ", ".join(values) + for name, values in ids.items() + ) + return rf"""% GENERATED FILE. DO NOT EDIT. +% Run scripts/generate_latex_results.py from the paper root. +% The frozen source is {FROZEN_REL.as_posix()}. +{comments} + +\begin{{table}}[t] +\centering +\scriptsize +\begin{{tabular}}{{p{{0.31\linewidth}}p{{0.57\linewidth}}}} +\hline +Field & Frozen campaign value \\ +\hline +{rendered_environment} +\hline +\end{{tabular}} +\caption{{Measured campaign environment and protocol. Values are descriptive context, not cross-platform claims.}} +\label{{tab:campaign-environment}} +\end{{table}} + +\begin{{table*}}[t] +\centering +\scriptsize +\begin{{tabular}}{{p{{0.28\textwidth}}p{{0.14\textwidth}}rrrrr}} +\hline +Stage & Load & Samples & p50 (ms) & p95 (ms) & p99 (ms) & Ready/s \\ +\hline +{startup_lines} +\hline +\end{{tabular}} +\caption{{Startup and session readiness in the frozen campaign.}} +\label{{tab:startup}} +\end{{table*}} + +\begin{{table*}}[t] +\centering +\scriptsize +\begin{{tabular}}{{p{{0.36\textwidth}}rrrrr}} +\hline +Public CLI operation / load & Samples & p50 (ms) & p95 (ms) & p99 (ms) & Ops/s \\ +\hline +{operation_lines} +\hline +\end{{tabular}} +\caption{{Public CLI operation timings in the frozen campaign. ``Small'' and ``large'' identify the two archived file-size cases without generalizing beyond the fixture.}} +\label{{tab:operations}} +\end{{table*}} + +\begin{{table}}[t] +\centering +\scriptsize +\begin{{tabular}}{{p{{0.24\linewidth}}p{{0.13\linewidth}}rrr}} +\hline +Operation/case & Load & Daemon (MiB) & Sandbox (MiB) & CPU (ms) \\ +\hline +{resource_compute_lines} +\hline +\end{{tabular}} +\caption{{Observed peak memory and CPU from selected frozen campaign cells. CPU is milliseconds per trial; these are not resource budgets or capacity guarantees.}} +\label{{tab:resources-compute}} +\end{{table}} + +\begin{{table}}[t] +\centering +\scriptsize +\begin{{tabular}}{{p{{0.24\linewidth}}p{{0.13\linewidth}}rrr}} +\hline +Operation/case & Load & Read (MiB) & Write (MiB) & Upper (MiB) \\ +\hline +{resource_io_lines} +\hline +\end{{tabular}} +\caption{{Observed I/O and workspace-upper deltas from the same selected frozen cells.}} +\label{{tab:resources-io}} +\end{{table}} +""" + + +def main() -> None: + tables = json.loads((FROZEN / "tables.json").read_text(encoding="utf-8"))["tables"] + registry = json.loads((FROZEN / "numeric-evidence.json").read_text(encoding="utf-8")) + entries = registry["entries"] + + # The source values and selectors remain byte-for-byte source-equivalent. + # A five-thousandths display tolerance is only for the frozen tables' three- + # decimal rounded rendering; it still verifies every source selector exactly. + projected_entries = [] + for original in entries: + entry = dict(original) + entry["source"] = (FROZEN_REL / "numeric-provenance.csv").as_posix() + entry["tolerance"] = max(float(entry.get("tolerance", 0)), 0.005) + projected_entries.append(entry) + OUT_REGISTRY.write_text( + json.dumps({"schema_version": registry["schema_version"], "entries": projected_entries}, indent=2) + "\n", + encoding="utf-8", + ) + OUT_TEX.write_text(render_tables(tables, entries), encoding="utf-8") + binding_lines = [ + "# Results numeric bindings", + "", + "Generated by `scripts/generate_latex_results.py` from the immutable Table-A output.", + "Each identifier below resolves through `numeric_evidence.json` to the frozen `numeric-provenance.csv` selector.", + "", + ] + for prefix, name in (("table1.", "Campaign environment"), ("table2.", "Startup"), ("table3.", "Public CLI operations"), ("table4.", "Resources")): + binding_lines.extend((f"## {name}", "")) + binding_lines.extend(f"- `{item}`" for item in ids_for_prefix(entries, prefix)) + binding_lines.append("") + OUT_BINDINGS.write_text("\n".join(binding_lines), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/ephemeral-sandbox-v1/sections/01-introduction.tex b/ephemeral-sandbox-v1/sections/01-introduction.tex index aa55052..9b6f344 100644 --- a/ephemeral-sandbox-v1/sections/01-introduction.tex +++ b/ephemeral-sandbox-v1/sections/01-introduction.tex @@ -1,3 +1,37 @@ \section{Introduction} \label{sec:introduction} -\emph{Draft.} + +Coding agents increasingly inspect, edit, test, and run software in parallel. +That parallelism creates a systems problem separate from task planning: several +workers need executable state without allowing one worker's unfinished +filesystem effects to become another worker's live premise. Giving every +worker a copy avoids immediate interference, but it merely postpones the +question of which changes may become durable, relative to which project state, +and with which failure outcome. + +Ephemeral Sandbox addresses that narrow runtime question. It maintains a +durable project history as LayerStack layers and gives a command or explicit +workspace session a leased private view. The private overlay accumulates the +session's filesystem effects. On publication, the runtime captures those +effects, checks them against the current active head, applies its bounded +reconciliation rules, and either publishes one new layer through an +active-manifest transition or returns a structured rejection. The protocol +separates this atomic data transition from later attribution, accounting, and +cleanup work, which may have different failure behavior. + +The paper contributes a source-defined account of that composition: the +private-session and lease model; the capture, validation, reconciliation, and +publication boundary; lifecycle and recovery outcomes; and a role-separated +operational interface. It also reports a reproducible local treatment covering +startup, public-CLI, and selected resource observations. The evaluation is +deliberately descriptive. It has no competing system, heterogeneous workload +suite, end-to-end coding-task outcome, fault campaign, or basis for a universal +concurrency claim. + +This scope positions the system between isolated execution environments and +agent coordination frameworks. It does not introduce copy-on-write +filesystems, validation after tentative work, three-way merge, or task-intent +admission. Rather, it specifies how private executable workspace state is +allowed to become one durable project update under a bounded runtime contract. +The remainder of the paper states that contract, its limitations, the frozen +observations, and its relationship to prior systems and coordination work. diff --git a/ephemeral-sandbox-v1/sections/02-goals-nongoals.tex b/ephemeral-sandbox-v1/sections/02-goals-nongoals.tex index 4559918..a51b512 100644 --- a/ephemeral-sandbox-v1/sections/02-goals-nongoals.tex +++ b/ephemeral-sandbox-v1/sections/02-goals-nongoals.tex @@ -37,7 +37,7 @@ \section{Goals, Non-goals, and Threat-model Boundary} measurable. Evaluation must test private-view and publication correctness, partial failures, resource and latency scaling, and verification-passing durably accepted work under stated workloads and runtime configurations. -These are evaluation goals rather than achieved results: the audited baseline +These are evaluation goals rather than achieved results: the audited source snapshot does not establish a performance advantage, a resource advantage, improved agent productivity, or a higher workload-dependent concurrency ceiling. diff --git a/ephemeral-sandbox-v1/sections/03-system-model.tex b/ephemeral-sandbox-v1/sections/03-system-model.tex index 2790d0e..8d471b2 100644 --- a/ephemeral-sandbox-v1/sections/03-system-model.tex +++ b/ephemeral-sandbox-v1/sections/03-system-model.tex @@ -56,6 +56,18 @@ \section{System Model and Invariants} are outside that atomic data boundary and can have different persistence and failure behavior. +\begin{figure}[t] +\centering +\includegraphics[width=\linewidth]{../figures/concept/fig_system_architecture.png} +\caption{System architecture. One active manifest selects the shared durable +LayerStack history leased into session-private overlays. The publication path +summarizes capture, current-head reconciliation, accepted head advancement, +and the later attribution/cleanup phase; the separate inset distinguishes +sessionless file operations. The diagram is explanatory rather than +experimental evidence.} +\label{fig:system-architecture} +\end{figure} + An \emph{orchestrator} is an external controller that selects work, invokes role-appropriate operations, and interprets outcomes; a \emph{worker} performs one or more coding tasks through those operations. These roles do not imply @@ -95,7 +107,7 @@ \section{System Model and Invariants} \item Compaction or garbage collection must not reclaim layers still pinned by a live lease unless a safe substitute preserves the leased logical - view. The baseline source and lease-aware tests establish the in-process + view. The audited source snapshot and lease-aware tests establish the in-process mechanism, but the lease/substitution registries are in memory and their behavior across daemon restart still requires fault testing. \end{enumerate} diff --git a/ephemeral-sandbox-v1/sections/04-workspace-execution.tex b/ephemeral-sandbox-v1/sections/04-workspace-execution.tex index 8e46a0b..e2c28ba 100644 --- a/ephemeral-sandbox-v1/sections/04-workspace-execution.tex +++ b/ephemeral-sandbox-v1/sections/04-workspace-execution.tex @@ -1,3 +1,78 @@ \section{Workspace Execution} \label{sec:workspace-execution} -\emph{Draft.} + +Workspace creation begins by acquiring a snapshot lease, not by reading an +unprotected copy of the current head. The lease records the selected manifest +and its resolved ordered layer paths; the workspace handle carries that +snapshot together with session-unique runtime directories. Only after this +view has been fixed does the runtime open the executable workspace. A later +publication may advance the active manifest, but it does not change the +manifest or layer ordering observed through the live session's lease. This is +the concrete mechanism behind the stable-view invariant in +Section~\ref{sec:system-model}, without importing serializable snapshot +isolation. + +On the supported Linux path, the runtime projects the leased LayerStack with +OverlayFS. The manifest's layers become read-only lower directories in +newest-first precedence order, while the session receives a unique upper +directory and work directory. The resulting mount therefore reads through +shared history but records regular writes, copied-up files, and deletion or +opacity metadata only in the private upper tree. Lower layers are shared +runtime state; the upper/work pair is session-private state. This mechanism +is Linux- and OverlayFS-centered in the audited source snapshot: the non-Linux mount +path reports the operation as unsupported, and the paper makes no universal +cross-platform execution claim. + +Each workspace also has a long-lived namespace-holder process. The holder +creates and keeps open the session's user, mount, and PID namespaces and, when +selected, its network namespace. The daemon retains file descriptors for +these namespaces rather than moving its own control process into them. For +each command, a short-lived runner joins the recorded user, mount, PID, and +optional network namespaces in the runtime's required order and launches the +command against the already mounted workspace. This holder/runner split lets +successive commands enter the same filesystem and process-namespace context +while the daemon remains outside the command namespaces. It is an +implementation isolation boundary, not a proof that hostile code cannot +escape or interfere through every host resource. + +A sessionless \texttt{exec\_command} uses the \emph{implicit session} path. +The operation service creates a workspace with the shared-network profile and +a publish-then-destroy finalization policy, admits the requested command, and +tracks it in the session's command ledger. Publication is eligible only after +the active-command set becomes empty and the ledger reaches its drained +terminal state; it is therefore tied to command completion, not merely to +returning an initial command handle or response. The finalizer then attempts +the capture, publication, and destruction phases described below. This +automatic lifecycle is specific to sessionless command execution. + +An \emph{explicit session} instead keeps one leased view, namespace holder, +mount, and private upper/work pair across multiple calls. Commands admitted +with its session identifier and file operations directed to that session +observe the same live overlay, so later operations can consume earlier private +writes before anything reaches the active head. The explicit session uses a +no-op automatic finalization policy: its caller deliberately requests +publication or destruction. Publication admission fails while commands are +still active, preventing capture from racing an admitted command through the +normal service path. + +File and network qualifiers matter to this account. A session-scoped read, +write, or edit operates on the live mounted workspace and therefore on the +private overlay. A sessionless read instead projects the latest published +LayerStack, while a sessionless write or edit directly amends the current head +with a one-layer operation under the LayerStack writer lock; neither +sessionless file path creates an implicit workspace session. Likewise, +shared networking joins the host network namespace while retaining separate +user, mount, and PID namespaces, whereas the isolated profile creates a +separate network namespace and its configured connectivity. Shared +networking is consequently not an egress-containment guarantee, and selectable +network isolation is not evidence of universal denial policy. + +Workspace creation has several externally fallible steps---lease acquisition, +directory allocation, holder startup, namespace access, overlay mounting, and +optional network setup. The service attempts to roll back resources created +before a failed admission, while a successfully admitted session retains its +lease and private tree until its lifecycle reaches cleanup. Neither path +captures command registers, memory, sockets, or an arbitrary process tree for +later rollback. Publication begins only from the filesystem delta left in the +complete private upper directory; the next section follows that delta from +capture to the active head. diff --git a/ephemeral-sandbox-v1/sections/05-capture-publication.tex b/ephemeral-sandbox-v1/sections/05-capture-publication.tex index 3a35d45..c38c87e 100644 --- a/ephemeral-sandbox-v1/sections/05-capture-publication.tex +++ b/ephemeral-sandbox-v1/sections/05-capture-publication.tex @@ -1,3 +1,123 @@ \section{Capture and Publication} \label{sec:capture-publication} -\emph{Draft.} + +Capture walks the complete private upper tree and converts OverlayFS +representations into typed layer changes. Regular files become write records +that retain a source path and size so publication can stream their bytes; +symbolic links, directory entries, and empty directories remain typed +entries. Kernel whiteouts---character devices with device number zero or the +supported whiteout extended attribute---become deletions, while the supported +opaque-directory extended attributes become opaque-directory changes. An +ordinary name beginning with \texttt{.wh.} is not reinterpreted as a +capture-time marker; it remains a path and is rejected later because that name +space is reserved by the layer format. Invalid layer paths and unsupported +special files are reported as protected drops rather than silently converted. +Explicit-session publication rejects any such drop. The lower-level planner +has a narrower exception for an unsupported-special-file drop, so the source snapshot +does not yet justify one uniform protected-drop policy across every +publication entry point. + +\begin{figure}[t] +\centering +\includegraphics[width=\linewidth]{../figures/concept/fig_publication_sequence.png} +\caption{Workspace-session publication sequence. Execution, ledger drain, +capture, and current-head reconciliation remain private or precommit. Public +data visibility begins at active-manifest replacement; best-effort attribution, +workspace destruction, and lease release follow outside the atomic data +publication boundary. The sequence does not describe sessionless direct file +amendment.} +\label{fig:publication-sequence} +\end{figure} + +Planning evaluates the candidate against the leased base before taking the +commit lock. It verifies that the supplied manifest version, root hash, and +layer count describe the same base manifest. Each path is then routed as +source or ignored according to the base view's ignore policy, and reserved +runtime paths---including the manifest, layer and staging directories, +\texttt{.layer-metadata}, and \texttt{.wh.*} components---cause rejection. +Source-routed paths receive content fingerprints from the leased base. +Planning an opaque directory additionally enumerates the lower descendants it +would hide: a protected descendant, a mixture of source and ignored routes, or +an expansion beyond the fixed bound rejects the candidate. These checks turn +a captured delta into an explicit plan; they do not yet make it durable. + +The service then acquires the LayerStack writer lock, rereads the active +manifest, and resolves source validations against that current head. A source +path whose current fingerprint still matches its base fingerprint is accepted +as planned; an idempotent concurrent creation of the same directory is also +compatible. Changes that replace or obscure a directory trigger descendant +checks so a concurrent change below that path cannot be erased unnoticed. +Other structural differences reject the publication. Ignored-route changes +are carried as wholesale writes and do not receive the source fingerprint and +merge treatment. The policy is thus conflict-aware for source state while +making its separate ignored-state route explicit, rather than claiming a +general database isolation level. Figure~\ref{fig:reconciliation-decision} +expands these route and resolution decisions. + +\begin{figure}[t] +\centering +\includegraphics[height=0.82\textheight]{../figures/concept/fig_reconciliation_decision.png} +\caption{Current-head reconciliation decision flow. Source-routed changes +receive base/current fingerprint and structural checks, and only eligible +exact-file textual divergence reaches three-way line merge. Every planned +change must resolve before a nonempty changeset is committed; otherwise the +whole candidate is rejected and the active manifest remains unchanged. A +no-op retains the current head and is not a new data publication.} +\label{fig:reconciliation-decision} +\end{figure} + +When a source fingerprint has diverged, the only merge-eligible candidate is a +regular-file write at that exact path. The resolver reads the leased-base, +current-head, and command versions and applies a three-way line merge only +when each is valid UTF-8 text without NUL bytes and no larger than 8 MiB. +Disjoint edits, or overlapping edits with byte-identical replacement lines, +can yield a clean merged file; differing overlapping edits reject it. A +non-file transition, binary content, invalid UTF-8, an oversized input, or a +text conflict is ineligible when reconciliation requires this merge and +therefore becomes a source conflict. By contrast, a binary or oversized +write whose source fingerprint has not diverged can still publish without a +merge. A clean line merge establishes only structural compatibility: it is +not a semantic-correctness guarantee, and the byte threshold alone is not a +claim that worst-case diff CPU or memory is adequately bounded. + +Resolution constructs one final changeset or returns one rejection; no +resolved subset escapes to the layer writer. Protected paths, invalid base +identity, opaque-directory violations, structural source changes, and +ineligible or conflicting required merges therefore leave the active manifest +unchanged. An empty resolved changeset is a no-op at the current head. For an +explicit session, lifecycle handling after a capture or publication failure is +separate from this resolution decision and belongs to +Section~\ref{sec:lifecycle-recovery}. The data outcomes provide all-or-none +handling without calling the session a full transaction. + +For a nonempty resolved changeset, the layer writer first computes its digest +and recognizes an identical current head as a no-op. Otherwise it writes the +changes beneath a newly allocated staging directory, synchronizes regular +files and the staging tree, renames that tree into the durable layer store, and +synchronizes the layer directory's parent. It then writes the layer digest, +rereads the active manifest to detect an intervening replacement, and builds a +new manifest whose first entry is the promoted layer. The manifest writer +creates and synchronizes a temporary file, renames it over the active manifest, +and synchronizes the containing directory. Only this final replacement makes +the resolved layer part of the active head. Failures before it can leave +recoverable staging or unreferenced artifacts, but they do not select the new +layer in the active manifest. + +The successful manifest replacement is the end of \emph{atomic data +publication}, not the end of the session lifecycle. After the layer commits, +the operation service maps structural line origins to an owner and appends +per-path audit events. That attribution is deliberately best effort: an audit +append failure does not fail or roll back the visible data. The service then +begins the separate workspace-destruction and lease-release phase. Accounting, +observability events, autosquash notification, session cleanup, and later +garbage collection likewise remain outside the all-or-none data boundary; +Section~\ref{sec:lifecycle-recovery} defines their failure outcomes. + +This path explains how one private filesystem delta can become one durable +public head at the audited Linux source snapshot. It does not establish +semantic merge correctness, a cross-platform crash proof, or an unlimited +transaction size. Staging remnants, retained sessions, lease release, cleanup +failure, compaction, and reclamation are lifecycle and recovery concerns, +which Section~\ref{sec:lifecycle-recovery} treats separately. + +\clearpage diff --git a/ephemeral-sandbox-v1/sections/06-lifecycle-recovery.tex b/ephemeral-sandbox-v1/sections/06-lifecycle-recovery.tex index 82953c7..2638355 100644 --- a/ephemeral-sandbox-v1/sections/06-lifecycle-recovery.tex +++ b/ephemeral-sandbox-v1/sections/06-lifecycle-recovery.tex @@ -1,3 +1,99 @@ \section{Lifecycle and Recovery} \label{sec:lifecycle-recovery} -\emph{Draft.} + +The operation layer represents each admitted workspace session as +\emph{active}, \emph{finalizing}, or \emph{finalize failed}. A per-session +admission gate serializes lifecycle-changing operations, while the command +ledger records active runners. For a live holder, explicit publication and +destruction reject admission while that ledger is nonempty. These states +coordinate in-process service work; they are not a durable transaction log or +a promise that arbitrary process state can be restored after failure. + +An explicit session has no automatic action when a command finishes. A +publish request moves an active session to finalizing, captures the upper tree, +and submits the complete candidate to the publication path in +Section~\ref{sec:capture-publication}. Capture error, protected-drop +rejection, invalid base, or another precommit publication failure restores the +session to active and returns a retained-session result. Its leased view and +private overlay therefore remain available for a deliberate retry or +destruction; rejection neither publishes a subset nor silently closes the +session. + +An accepted explicit publication and an empty no-op both proceed to session +destruction. A nonempty acceptance has already become visible at the +active-manifest transition before this cleanup begins; a no-op adds no layer +and leaves the current head selected. If workspace or cgroup cleanup then +fails, the service retains a finalize-failed record and reports +\emph{published but not closed}. A later guarded cleanup can finish resource +release, but it does not rerun publication or roll back the manifest. The +label applies to a committed path; cleanup failure after a no-op or discard +does not retroactively create a publication. + +Implicit command sessions follow a deliberately different policy. When a +sessionless \texttt{exec\_command}'s ledger drains, the service quiesces the +namespace holder, captures the completed filesystem state, attempts +publication, and then proceeds toward destruction. An empty capture skips +publication. A publish rejection is recorded in the completing command's +finalization outcome, but the implicit session is not converted into a +caller-managed retryable session. Capture-after-quiescence or destruction +failure instead produces a finalization-failure classification and may retain +cleanup or recovery state. Figure~\ref{fig:lifecycle-state-machine} +summarizes the normal explicit and implicit paths. + +\begin{figure}[t] +\centering +\includegraphics[width=\linewidth]{../figures/concept/fig_lifecycle_state_machine.png} +\caption{Workspace-session lifecycle. Explicit precommit failure or rejection +can return to an active session for retry or discard, whereas an accepted +active-manifest replacement precedes the published state and later cleanup. +Cleanup failure cannot undo visible data. This normal-path abstraction omits +holder-exit recovery artifacts, shutdown convergence, and lease-aware remount +outcomes; ``published but not closed'' applies only when publication committed +before cleanup failed.} +\label{fig:lifecycle-state-machine} +\end{figure} + +Deliberate destruction is the discard operation for unpublished private +state. A live session with active commands is rejected rather than torn down +through the normal public path. Once admitted, teardown tracks workspace and +workload-cgroup completion separately: if one resource is released and another +fails, the retained cleanup record prevents a later retry from repeating the +completed raw teardown solely because the other resource remains unreleased. +Only complete teardown removes the session record, commits workspace +destruction, releases terminal command state, and allows lease release. This +is convergent resource cleanup, not rollback of commands already executed. + +Unexpected holder exit uses a separate in-process reconciliation path. The +supervisor marks the holder dead, one teardown owner cancels and joins the +recorded commands, and the service then destroys the workspace. For an +implicit publish-then-destroy session, it first preserves a bounded, +identity-keyed copy of the upper tree plus a recovery manifest; truncation, +missing source state, links, and unsupported entries are recorded. The +artifact supports later diagnosis or manual recovery but is not automatically +published, replayed, or mounted as a restored session. An explicit session +whose holder dies is destroyed under its no-automatic-finalization policy +rather than being silently published. + +LayerStack compaction changes physical representation without changing the +intended leased logical view. Squash planning treats live-lease boundaries as +constraints, builds flattened replacements outside the exclusive writer +section, and commits only after rechecking the planned runs. A post-commit +sweep then attempts to remount each live session onto an identity or +substituted chain. A session can be reported migrated, still leased because a +safe switch was unavailable or uncertain, faulty, gone, or unchanged. Faulty +sessions can be destroyed through the ordinary gated teardown path, and +garbage collection removes a superseded layer only when neither the active +manifest nor another lease still references it. + +Daemon shutdown attempts bounded convergence by joining existing teardown +work, cancelling active commands, finalizing eligible implicit sessions, and +destroying the remaining sessions. The audited source snapshot and tests define +these in-process paths, but final Linux fault evidence has not yet established +their behavior under abrupt daemon restart, partial recovery-artifact writes, +or every lease-substitution ordering. The lease and session registries remain +in memory, and the recovery artifact does not close that gap. Consequently, +the paper claims explicit partial-failure outcomes and conservative retention, +not automatic recovery from every crash. The next section exposes those +outcomes through the operational interface. + +\clearpage diff --git a/ephemeral-sandbox-v1/sections/07-implementation-interface.tex b/ephemeral-sandbox-v1/sections/07-implementation-interface.tex index 74f3334..cb88914 100644 --- a/ephemeral-sandbox-v1/sections/07-implementation-interface.tex +++ b/ephemeral-sandbox-v1/sections/07-implementation-interface.tex @@ -1,6 +1,103 @@ \section{Implementation and Operational Interface} \label{sec:implementation-interface} +The Rust implementation follows the ownership boundaries of the mechanism. +Product adapters obtain semantic operations from +\texttt{sandbox-operation-catalog}, add adapter-specific shell projections, +and construct an adapter-neutral request through the shared client. The +authenticated newline-framed request traverses the gateway and manager to one +sandbox daemon, where \texttt{sandbox-runtime} orchestrates command, file, +workspace-session, LayerStack, and observability applications. Low-level +crates separately own LayerStack storage and leases, workspace lifecycle, +OverlayFS mounting, namespace execution, namespace processes, and telemetry. +This separation keeps transport and presentation code from owning publication +or workspace semantics. The audited source snapshot is the commit peeled from +the annotated \texttt{paper-v1.1-freeze} tag; path-level anchors and the +revalidation record are preserved in the +\href{https://github.com/Ephemeral-AI-Lab/ephemeral-sandbox/tree/5c48dae10847fb9e46ba2bea7675bcf2f5a6f4c8}{tagged source tree} +and \path{plan/source_revalidation.md}. This section is source-derived +interface evidence, not a runtime-correctness or performance result. + +\subsection{Role-Separated Operational Contract} + +The measured source snapshot builds three feature-gated CLI executables and no combined +client. The semantic catalog defines operation names, descriptions, +arguments, visibility, and routing; each CLI projection defines shell paths, +flags, positionals, usage, and examples. A bidirectional projection-integrity +test checks that every public catalog route has exactly one matching projection +and that no projected operation lacks a public route. Table~\ref{tab:cli-roles} +summarizes the measured contract. Management covers lifecycle, image and +workspace inspection, compaction, and export. Runtime covers commands, +standard input/output, files, attribution, and workspace sessions. +Observability covers snapshots, traces, events, resources, diagnostics, and +LayerStack inspection. The complete command shapes are recorded in +\path{cli_contract_matrix.md}. + +\begin{table}[H] +\centering +\small +\begin{tabular}{p{0.17\linewidth}p{0.12\linewidth}p{0.23\linewidth}p{0.37\linewidth}} +\hline +Client & Count & Request scope & Responsibility \\ +\hline +Management & 8 & System envelope & Sandbox lifecycle, selection, compaction, +and export; a target sandbox can remain an operation argument \\ +Runtime & 10 & Sandbox required & Commands, file operations, attribution, +and explicit workspace-session lifecycle inside one selected sandbox \\ +Observability & 8 & Two system-or-sandbox; six sandbox-only & Read-only +snapshots, traces, events, resources, daemon and topology diagnostics, cgroup +counters, and LayerStack/session views \\ +\hline +\end{tabular} +\caption{Role-separated public CLI contract at the measured tagged source +snapshot. Counts and exact projections are source-derived interface evidence, +not runtime-correctness or performance results.} +\label{tab:cli-roles} +\end{table} + +The runtime selector becomes the request-envelope sandbox scope rather than a +duplicated operation argument. Every client constructs an envelope containing +an operation, request identifier, scope, and arguments. The shared builder +generates a UUIDv4 request identifier by default, and all three clients expose +a validated \texttt{--request-id} override for correlation across +orchestration and diagnostics. Correlation identifies one requested operation, +but it does not +establish causal ownership of every resulting filesystem line. Gateway +connection discovery prefers CLI socket/token overrides, then environment +variables, and finally a platform-local default; the client does not discover +a token file automatically. The canonical \texttt{--gateway-endpoint} accepts +TCP, Windows named-pipe, and Unix-socket URIs; the visible +\texttt{--gateway-socket} spelling is an alias. On Windows, the default is a +local named pipe, while explicit TCP remains a compatibility choice. + +Shell behavior is also part of the contract. Help is human-readable text on +standard output with exit status 0. A successful operation writes one JSON +result line to standard output and exits 0. A gateway operation failure or +transport/protocol failure writes a JSON error envelope to standard error and +exits 1, while local syntax, configuration, unknown-operation, or argument +failure writes a local JSON error envelope and exits 2. Requested manager +progress may share standard error with a successful final result on standard +output. For command execution, the command's own exit status is response +data; the CLI process status reports whether the runtime request and protocol +succeeded. + +Help is derived by joining the same semantic catalog with the selected CLI +projection, so descriptions, declared arguments, defaults, usage, examples, +and related operations remain mechanically connected. This does not keep +external website documentation synchronized: the audited live pages expose +different operation counts and are not the authority for the paper source snapshot. +Likewise, catalog/projection/help tests establish an adapter contract; they do +not prove workspace isolation, publication correctness, or recovery behavior. + +Role separation gives an orchestrator smaller, testable operation sets, but it +is not by itself an authorization or sandbox-security boundary. The source snapshot +does not declare coding intent, task dependencies, ownership transfers, +service or port leases, general resource-budget admission, semantic +compatibility, integration scheduling, or handoff acceptance. The clients +are typed building blocks through which an external orchestrator can create, +operate, inspect, publish, and destroy work; they are not a complete agent-team +or exploratory-swarm coordination plane. + \subsection{Source-Derived Operational Cost Model} \label{sec:cost-model} @@ -13,7 +110,7 @@ \subsection{Source-Derived Operational Cost Model} references retained across live leases. Table~\ref{tab:cost-model} records source-derived drivers; it is not a performance result. -\begin{table}[t] +\begin{table}[H] \centering \small \begin{tabular}{p{0.19\linewidth}p{0.36\linewidth}p{0.34\linewidth}} @@ -64,8 +161,14 @@ \subsection{Source-Derived Operational Cost Model} retains an $O(KD)$ frontier trace and can approach an $O(K^2)$ trace envelope. The evaluation must stress line count and edit shape in addition to file bytes. -\subsection{Operational Interface} +Squash reduces eligible history depth but does not erase lease constraints. +Its planning and flattening phases scale with manifest structure and directory +contents; live leases can retain old layers until a safe substitution and +release. These terms, together with the serialized publication path, are +mechanism-level hypotheses for the evaluation rather than evidence of a +particular latency, storage saving, or concurrency ceiling. -\emph{Draft pending PW3. This subsection will describe the role-separated -management, runtime, and read-only observability clients and the final tagged -catalog-derived contract.} +The audited execution mechanism is Linux- and OverlayFS-centered. The tagged +source contract defines responsibility, lifecycle, and cost drivers without +establishing cross-platform behavior, security, semantic merge correctness, or +measured efficiency beyond the bounded campaign in Section~\ref{sec:evaluation}. diff --git a/ephemeral-sandbox-v1/sections/08-evaluation.tex b/ephemeral-sandbox-v1/sections/08-evaluation.tex index d7ac560..fbd30b7 100644 --- a/ephemeral-sandbox-v1/sections/08-evaluation.tex +++ b/ephemeral-sandbox-v1/sections/08-evaluation.tex @@ -1,25 +1,67 @@ \section{Evaluation} \label{sec:evaluation} -\subsection{RQ3: Latency, Storage, and Resource Scaling} - -RQ3 tests the source-derived cost model rather than assuming that worker count -alone explains scaling. The protocol independently varies worker/publisher -count, manifest depth, live-session count and lease age, private-upper entry -count and fan-out, changed and validated paths, published logical bytes, -conflict class, and eligible-merge bytes, lines, similarity, and edit distance. -Storage cases include small edits to large files, many small files, and -incompressible controls. All factors, warmups, repeats, and stopping rules -must be fixed before the final source freeze. - -For each stage, we report latency distributions, CPU, RSS/PSS, I/O, failures, -and exact phase boundaries. Publication additionally reports writer-lock wait -and hold time and decomposes planning, resolution/merge, hashing, copy, -synchronization, and manifest transition. Storage reports logical, allocated, -shared, and exclusive bytes for upper, work, staging, published layers, and -lease-retained history. Squash reports plan, build, commit, and garbage -collection separately. No cost-model term becomes a paper result until it is -reproduced from frozen raw artifacts and analysis code. - -\emph{The remaining RQ1--RQ5 methodology and all numerical results are pending -protocol and evidence lock.} +\subsection{Question and Claim Boundary} + +The evaluation asks a deliberately narrow question: in the archived local +campaign, what startup, public-CLI, and selected resource observations are +obtained for the frozen implementation and fixture? It tests only the +mechanism's operational envelope under that protocol. It does not compare +against another sandbox, establish an application-level speedup, quantify a +useful-work concurrency ceiling, validate publication correctness, or support +a cross-platform or security claim. Those boundaries matter because a +private workspace can make execution independent while durable publication +remains serialized. + +The sole eligible final campaign is preserved as an immutable archive. Its +archive manifest, raw reports, analysis inputs, and deterministic table output +are linked through the run archive and frozen Table-A projection listed in +\path{ARTIFACTS.md}. +The manuscript-side generator reads that projection without modifying it and +records every displayed number's selector in +\path{sections/results_numeric_bindings.md}. Thus the tables below are +evidence displays, rather than fresh analysis or a rerun. + +\subsection{Protocol} + +The campaign used the product CLI against a local gateway, a pinned container +image, and a fixed workspace fixture. It measures sandbox creation, session +readiness, a first command, and public command, read, write, and edit paths +under single-client and small concurrent-client loads. The archive records +warm-ups separately from measured trials, preserves the complete trial +reports, and marks qualification and exploratory runs ineligible. The +resource rows report observed daemon and sandbox peaks, block I/O, and +workspace-upper deltas for selected cells; they are not capacity reservations +or fleet-wide measurements. + +The protocol is intentionally a treatment, not a universal benchmark. It has +no external baseline, no end-to-end coding task, no adversarial merge or +recovery campaign, no layer-depth sweep, and no measurement of team-level +coordination quality. We therefore use descriptive distributions and state +only what the frozen records directly support. + +\subsection{Frozen Observations} + +The evidence displays follow this discussion in the manuscript source. + +The startup table separates sandbox creation from session readiness and the +first command, rather than collapsing them into one headline time. The +operation table retains all archived public-CLI cases and both recorded load +levels. The resource table keeps its selection rule visible: it reports +observed peaks and deltas for the specified cells, not a claim that every +operation consumes the same resources. This organization makes the negative +result as important as the positive one: the campaign supplies a reproducible +local treatment, but it cannot answer the omitted comparison, scaling, or +useful-work questions. + +\subsection{Threats to Validity} + +Internal validity is bounded by the archive and its deterministic analysis, +but a single host, engine configuration, fixture, and transport prevent a +hardware- or deployment-independent inference. Construct validity is also +narrow: command and file operations are not representative of a whole coding +agent workflow, and resource counters do not establish isolation strength or +fairness. External validity is limited by the absence of heterogeneous +projects, competing systems, fault injection, and collaborative outcomes. +The evaluation is consequently evidence for this implementation and protocol, +not a comparative systems result. diff --git a/ephemeral-sandbox-v1/sections/09-limitations-related-work.tex b/ephemeral-sandbox-v1/sections/09-limitations-related-work.tex index 3be9b48..8ee6e0d 100644 --- a/ephemeral-sandbox-v1/sections/09-limitations-related-work.tex +++ b/ephemeral-sandbox-v1/sections/09-limitations-related-work.tex @@ -1,52 +1,86 @@ \section{Limitations and Related Work} \label{sec:limitations-related-work} -\subsection{Cost and Implementation Limits} - -Workspace isolation removes one source of interference; it does not solve task -decomposition, model quality, semantic compatibility, verification, resource -ownership, or general agent coordination. v1 also exposes concrete scaling -risks. It serializes publication and performs current-head reconciliation plus -payload hashing, copying, and synchronization inside the writer critical -section. It retains a full manifest per live lease, repeatedly probes layered -history during path validation, and can retain obsolete history while -long-lived leases protect session views. Its text merge is byte-limited but -does not yet have an independently small line/edit-distance trace bound. -These are source-derived limitations whose practical severity remains to be -measured. - -The copy-on-write workspace abstraction is not a claim of constant or negligible -physical storage. Kernel copy-up, sparse allocation, compression, extent -sharing, page cache, and filesystem metadata are backend-dependent. A small -edit to a large lower file can produce a large logical upper file, and v1 -publication copies the resulting regular-file payload into a new layer. -Accordingly, the evaluation distinguishes logical from allocated, shared, and -exclusive bytes. - -\subsection{LayerStack 2.0 as a Candidate Evolution} - -LayerStack 2.0 is future work, not part of the v1 contribution. Its migration -protocol proposes a qualified single-filesystem storage domain and compares a -copy-based v1 layout, a co-located copy control, and a reflink-required -candidate. The target is to reduce byte-transfer and allocated-storage -amplification without changing image compatibility, blame, squash, remount, -active execution, crash, memory, daemon-health, or privilege behavior. A -reflink is not $O(1)$: clone work depends on filesystem and extent structure, -and later writes allocate changed extents. - -The completed Windows feasibility evidence is negative and narrowly scoped. -On the pinned stock Windows Docker Desktop/WSL~2 storage cell, direct -\texttt{FICLONE} returned \texttt{errno=95}; the integrated Windows candidate -therefore remained blocked. A future design requires capability detection and -a correctness-preserving copy fallback, and this paper makes no Windows -reflink storage or performance claim. - -Other targets follow directly from the v1 model: shorten the writer critical -section while preserving all-or-none and crash visibility semantics; share or -compact immutable manifest state across leases; index revision-keyed path, -fingerprint, and ignore lookups; and reject merge work through independently -bounded CPU/memory admission. - -\subsection{Related Work} - -\emph{Draft pending PW4 and final citation verification.} +\subsection{Limits of the Present System and Evidence} + +The system isolates workspace effects; it does not solve task decomposition, +model quality, semantic compatibility, verification, service ownership, or +general agent coordination. Publication is intentionally serialized, and its +critical path includes active-head reconciliation, eligible text handling, +payload hashing and copying, synchronization, and manifest replacement. +Long-lived leases can retain historical layers, while path validation and merge +work depend on the shape of the changed tree. The implementation's bounded +text merge is not an independently bounded line- or edit-trace memory +algorithm. These are design limitations, not measurements of a general +concurrency ceiling. + +The copy-on-write workspace abstraction also does not imply negligible physical +storage. Copy-up, sparse allocation, compression, extent sharing, page cache, +and metadata are backend dependent; a small edit can materialize a large +logical upper file. The frozen evaluation consequently reports only its +observed resource counters and workspace deltas. It does not establish a +capacity reservation, a storage-saving claim, a security boundary, or behavior +on operating systems and filesystems not covered by the campaign. + +LayerStack evolution remains future work. In particular, the paper does not +claim reflink support, process-state rollback, transactional publication +attribution, daemon-restart recovery, or semantic merge correctness. A future +evaluation needs heterogeneous projects, controlled competing systems, +publication and recovery faults, larger concurrency and history sweeps, and +end-to-end collaborative coding outcomes. + +\subsection{Isolation, Reversibility, and Agent Runtime Substrates} + +Several systems already provide isolated environments for software agents. +SWE-MiniSandbox develops container-free isolated workspaces for software +engineering-agent training, while AgentBay exposes hybrid human and agent +control over sandbox sessions \cite{swe_minisandbox_2026,agentbay_2025}. They +motivate isolated execution but do not make this paper's runtime-publication +claim. DeltaBox focuses on change-based sandbox checkpoint and rollback, +including process state, and Shepherd records reversible agentic execution +traces for fork and replay \cite{deltabox_2026,shepherd_2026}. Ephemeral +Sandbox does not implement complete process-state checkpointing, rollback, or +replay; its claim is restricted to private filesystem workspace state followed +by controlled durable publication. + +\subsection{Concurrent Coding and Integration} + +CAID uses a central manager, isolated workspaces, and executable verification +to coordinate asynchronous software-engineering agents +\cite{caid_2026}. CoAgent instead maintains speculative in-place shared-state +effects under a predetermined order and asks agents to repair affected plans +\cite{coagent_2026}. Claim Plane treats concurrent change as pre-write intent +admission and dynamic scope control \cite{claimplane_2026}. These approaches +address important coordination problems outside this paper's boundary. In +contrast, Ephemeral Sandbox gives a runtime contract for a private executable +workspace view and validates its captured delta only when attempting a durable +publish-or-reject transition; it does not plan tasks, admit semantic intent, +or repair a shared live workspace. + +Earlier developer tools also delimit the claim. Palantir detects development +conflicts arising from parallel code changes, and Crystal detects collaboration +conflicts and risks before integration \cite{palantir_2012,crystal_2013}. +Those systems support awareness and diagnosis. The bounded text reconciliation +here should also not be confused with semantic merge correctness, which is the +subject of verified three-way program merge work \cite{threewaymerge_2018}. +Union-mount designs already establish writable-upper/shared-lower namespace +mechanics, including copy-up and private views over shared work areas.\footnote{J.-S. +Pendry and M. K. McKusick, ``Union Mounts in 4.4BSD-Lite,'' USENIX Technical +Conference. \url{https://www.usenix.org/conference/usenix-1995-technical-conference/union-mounts-44bsd-lite}} +Likewise, validation after tentative work is a foundational optimistic +concurrency-control pattern \cite{occ_1981}. We therefore do not call the +lease behavior serializable snapshot isolation; snapshot terminology carries +stronger semantic obligations \cite{snapshot_isolation_1995}. + +\subsection{Evaluation Context} + +CooperBench provides collaborative coding tasks for studying coordination +failures, while TeamBench studies role-separated agent coordination +\cite{cooperbench_2026,teambench_2026}. SWE-bench supplies executable +repository-level issue-resolution tasks, and PaperBench provides long-horizon +research-replication tasks \cite{swebench_2024,paperbench_2025}. These are +useful candidates for future workloads, but none is a result in this paper. +AgenticFlict is similarly relevant only as narrowly scoped motivation about +textual Git-conflict outcomes in its selected dataset +\cite{agenticflict_2026}; it cannot establish the causal interference rate of +concurrent tool-call sessions or the benefit of this publication protocol. diff --git a/ephemeral-sandbox-v1/sections/10-conclusion.tex b/ephemeral-sandbox-v1/sections/10-conclusion.tex index ea62b1d..bff8fff 100644 --- a/ephemeral-sandbox-v1/sections/10-conclusion.tex +++ b/ephemeral-sandbox-v1/sections/10-conclusion.tex @@ -1,3 +1,19 @@ \section{Conclusion} \label{sec:conclusion} -\emph{Draft.} + +Ephemeral Sandbox defines a bounded runtime publication protocol for coding +agents: leased LayerStack history provides a private executable workspace; +capture turns its filesystem effects into a candidate delta; validation and +bounded reconciliation decide whether that delta may advance the active head; +and the durable data transition is separated from later lifecycle work. This +contract provides a precise place to discuss private execution and controlled +integration without treating either as a substitute for planning, verification, +or general multi-agent coordination. + +The frozen local treatment supplies reproducible observations for the pinned +implementation and fixture, but it is not a comparison or a team-productivity +result. Semantic merge correctness, process-state rollback, security, +cross-platform behavior, recovery under faults, and workload-dependent useful +work remain outside the demonstrated boundary. These limits are not incidental: +they define the evidence required before stronger claims about collaborative +agent systems would be warranted. diff --git a/ephemeral-sandbox-v1/sections/generated_results_tables.tex b/ephemeral-sandbox-v1/sections/generated_results_tables.tex new file mode 100644 index 0000000..a9f2004 --- /dev/null +++ b/ephemeral-sandbox-v1/sections/generated_results_tables.tex @@ -0,0 +1,115 @@ +% GENERATED FILE. DO NOT EDIT. +% Run scripts/generate_latex_results.py from the paper root. +% The frozen source is experiments/analysis/final-v11-019fb86c-tables-a. +% numeric-evidence ids for environment: table1.cpu.logical_processors, table1.memory_bytes, table1.sandbox_limits.memory_bytes, table1.sandbox_limits.pids_limit, table1.sandbox_limits.vcpus, table1.seed, table1.trials.measured, table1.trials.warmups, table1.workspace_fixture.file_count, table1.workspace_fixture.logical_mib, table1.workspace_fixture.maximum_depth +% numeric-evidence ids for startup: table2.create_sandbox.none.c1.p50_ms, table2.create_sandbox.none.c1.p95_ms, table2.create_sandbox.none.c1.p99_ms, table2.create_sandbox.none.c1.samples, table2.create_sandbox.none.c1.throughput_ops_s, table2.create_workspace.none.c1.p50_ms, table2.create_workspace.none.c1.p95_ms, table2.create_workspace.none.c1.p99_ms, table2.create_workspace.none.c1.samples, table2.create_workspace.none.c1.throughput_ops_s, table2.create_workspace.none.c5.p50_ms, table2.create_workspace.none.c5.p95_ms, table2.create_workspace.none.c5.p99_ms, table2.create_workspace.none.c5.samples, table2.create_workspace.none.c5.throughput_ops_s, table2.exec_command.noop.c1.p50_ms, table2.exec_command.noop.c1.p95_ms, table2.exec_command.noop.c1.p99_ms, table2.exec_command.noop.c1.samples, table2.exec_command.noop.c1.throughput_ops_s +% numeric-evidence ids for operations: table3.exec_command.fixture_read.c1.p50_ms, table3.exec_command.fixture_read.c1.p95_ms, table3.exec_command.fixture_read.c1.p99_ms, table3.exec_command.fixture_read.c1.samples, table3.exec_command.fixture_read.c1.throughput_ops_s, table3.exec_command.fixture_read.c5.p50_ms, table3.exec_command.fixture_read.c5.p95_ms, table3.exec_command.fixture_read.c5.p99_ms, table3.exec_command.fixture_read.c5.samples, table3.exec_command.fixture_read.c5.throughput_ops_s, table3.exec_command.noop.c1.p50_ms, table3.exec_command.noop.c1.p95_ms, table3.exec_command.noop.c1.p99_ms, table3.exec_command.noop.c1.samples, table3.exec_command.noop.c1.throughput_ops_s, table3.exec_command.noop.c5.p50_ms, table3.exec_command.noop.c5.p95_ms, table3.exec_command.noop.c5.p99_ms, table3.exec_command.noop.c5.samples, table3.exec_command.noop.c5.throughput_ops_s, table3.file_edit.262144.c1.p50_ms, table3.file_edit.262144.c1.p95_ms, table3.file_edit.262144.c1.p99_ms, table3.file_edit.262144.c1.samples, table3.file_edit.262144.c1.throughput_ops_s, table3.file_edit.262144.c5.p50_ms, table3.file_edit.262144.c5.p95_ms, table3.file_edit.262144.c5.p99_ms, table3.file_edit.262144.c5.samples, table3.file_edit.262144.c5.throughput_ops_s, table3.file_edit.4096.c1.p50_ms, table3.file_edit.4096.c1.p95_ms, table3.file_edit.4096.c1.p99_ms, table3.file_edit.4096.c1.samples, table3.file_edit.4096.c1.throughput_ops_s, table3.file_edit.4096.c5.p50_ms, table3.file_edit.4096.c5.p95_ms, table3.file_edit.4096.c5.p99_ms, table3.file_edit.4096.c5.samples, table3.file_edit.4096.c5.throughput_ops_s, table3.file_read.262144.c1.p50_ms, table3.file_read.262144.c1.p95_ms, table3.file_read.262144.c1.p99_ms, table3.file_read.262144.c1.samples, table3.file_read.262144.c1.throughput_ops_s, table3.file_read.262144.c5.p50_ms, table3.file_read.262144.c5.p95_ms, table3.file_read.262144.c5.p99_ms, table3.file_read.262144.c5.samples, table3.file_read.262144.c5.throughput_ops_s, table3.file_read.4096.c1.p50_ms, table3.file_read.4096.c1.p95_ms, table3.file_read.4096.c1.p99_ms, table3.file_read.4096.c1.samples, table3.file_read.4096.c1.throughput_ops_s, table3.file_read.4096.c5.p50_ms, table3.file_read.4096.c5.p95_ms, table3.file_read.4096.c5.p99_ms, table3.file_read.4096.c5.samples, table3.file_read.4096.c5.throughput_ops_s, table3.file_write.262144.c1.p50_ms, table3.file_write.262144.c1.p95_ms, table3.file_write.262144.c1.p99_ms, table3.file_write.262144.c1.samples, table3.file_write.262144.c1.throughput_ops_s, table3.file_write.262144.c5.p50_ms, table3.file_write.262144.c5.p95_ms, table3.file_write.262144.c5.p99_ms, table3.file_write.262144.c5.samples, table3.file_write.262144.c5.throughput_ops_s, table3.file_write.4096.c1.p50_ms, table3.file_write.4096.c1.p95_ms, table3.file_write.4096.c1.p99_ms, table3.file_write.4096.c1.samples, table3.file_write.4096.c1.throughput_ops_s, table3.file_write.4096.c5.p50_ms, table3.file_write.4096.c5.p95_ms, table3.file_write.4096.c5.p99_ms, table3.file_write.4096.c5.samples, table3.file_write.4096.c5.throughput_ops_s +% numeric-evidence ids for resources: table4.create_workspace.none.c1.daemon_rss_bytes, table4.create_workspace.none.c1.sandbox_block_read_bytes, table4.create_workspace.none.c1.sandbox_block_write_bytes, table4.create_workspace.none.c1.sandbox_cpu_time_ns, table4.create_workspace.none.c1.sandbox_memory_peak_bytes, table4.create_workspace.none.c1.upperdir_bytes, table4.create_workspace.none.c5.daemon_rss_bytes, table4.create_workspace.none.c5.sandbox_block_read_bytes, table4.create_workspace.none.c5.sandbox_block_write_bytes, table4.create_workspace.none.c5.sandbox_cpu_time_ns, table4.create_workspace.none.c5.sandbox_memory_peak_bytes, table4.create_workspace.none.c5.upperdir_bytes, table4.exec_command.noop.c1.daemon_rss_bytes, table4.exec_command.noop.c1.sandbox_block_read_bytes, table4.exec_command.noop.c1.sandbox_block_write_bytes, table4.exec_command.noop.c1.sandbox_cpu_time_ns, table4.exec_command.noop.c1.sandbox_memory_peak_bytes, table4.exec_command.noop.c1.upperdir_bytes, table4.exec_command.noop.c5.daemon_rss_bytes, table4.exec_command.noop.c5.sandbox_block_read_bytes, table4.exec_command.noop.c5.sandbox_block_write_bytes, table4.exec_command.noop.c5.sandbox_cpu_time_ns, table4.exec_command.noop.c5.sandbox_memory_peak_bytes, table4.exec_command.noop.c5.upperdir_bytes, table4.file_edit.262144.c5.daemon_rss_bytes, table4.file_edit.262144.c5.sandbox_block_read_bytes, table4.file_edit.262144.c5.sandbox_block_write_bytes, table4.file_edit.262144.c5.sandbox_cpu_time_ns, table4.file_edit.262144.c5.sandbox_memory_peak_bytes, table4.file_edit.262144.c5.upperdir_bytes, table4.file_read.262144.c5.daemon_rss_bytes, table4.file_read.262144.c5.sandbox_block_read_bytes, table4.file_read.262144.c5.sandbox_block_write_bytes, table4.file_read.262144.c5.sandbox_cpu_time_ns, table4.file_read.262144.c5.sandbox_memory_peak_bytes, table4.file_read.262144.c5.upperdir_bytes, table4.file_write.262144.c5.daemon_rss_bytes, table4.file_write.262144.c5.sandbox_block_read_bytes, table4.file_write.262144.c5.sandbox_block_write_bytes, table4.file_write.262144.c5.sandbox_cpu_time_ns, table4.file_write.262144.c5.sandbox_memory_peak_bytes, table4.file_write.262144.c5.upperdir_bytes + +\begin{table}[t] +\centering +\scriptsize +\begin{tabular}{p{0.31\linewidth}p{0.57\linewidth}} +\hline +Field & Frozen campaign value \\ +\hline +Host & Windows host; see frozen preflight record \\ +Container engine OS & linux \\ +Architecture & x64 \\ +CPU logical processors & 48 \\ +Memory bytes & 137,438,953,472 \\ +Sandbox limits & one vCPU / 512 MiB / 256 PIDs \\ +Workspace fixture & 100 MiB / 4,000 files / depth 100 \\ +Trial plan & two warm-ups + 100 measured \\ +Gateway transport & windows\_named\_pipe; local\_only; per\_execution\_block \\ +\hline +\end{tabular} +\caption{Measured campaign environment and protocol. Values are descriptive context, not cross-platform claims.} +\label{tab:campaign-environment} +\end{table} + +\begin{table*}[t] +\centering +\scriptsize +\begin{tabular}{p{0.28\textwidth}p{0.14\textwidth}rrrrr} +\hline +Stage & Load & Samples & p50 (ms) & p95 (ms) & p99 (ms) & Ready/s \\ +\hline +Sandbox create + base mount & single client & 100 & 1659.811 & 1749.739 & 1794.902 & 0.62 \\ +Session create to ready & single client & 100 & 32.897 & 35.685 & 36.096 & 30.36 \\ +Session create to ready & five clients & 100 & 109.119 & 120.107 & 131.588 & 45.32 \\ +First no-op command & single client & 100 & 26.719 & 28.291 & 28.996 & 37.28 \\ +\hline +\end{tabular} +\caption{Startup and session readiness in the frozen campaign.} +\label{tab:startup} +\end{table*} + +\begin{table*}[t] +\centering +\scriptsize +\begin{tabular}{p{0.36\textwidth}rrrrr} +\hline +Public CLI operation / load & Samples & p50 (ms) & p95 (ms) & p99 (ms) & Ops/s \\ +\hline +\texttt{exec\_command}; no-op; single client & 100 & 26.719 & 28.291 & 28.996 & 37.28 \\ +\texttt{exec\_command}; no-op; five clients & 100 & 45.998 & 53.282 & 60.092 & 107.61 \\ +\texttt{exec\_command}; fixture read; single client & 100 & 27.235 & 28.938 & 32.666 & 36.44 \\ +\texttt{exec\_command}; fixture read; five clients & 100 & 45.441 & 50.973 & 56.303 & 108.88 \\ +Read; snapshot, small payload; single client & 100 & 11.425 & 15.773 & 19.964 & 85 \\ +Read; snapshot, small payload; five clients & 100 & 26.575 & 30.592 & 39.738 & 185.3 \\ +Read; snapshot, large payload; single client & 100 & 20.781 & 24.367 & 25.484 & 47.68 \\ +Read; snapshot, large payload; five clients & 100 & 37.788 & 57.794 & 60.156 & 125.54 \\ +Write; session-local, small payload; single client & 100 & 38.813 & 41.384 & 51.562 & 25.55 \\ +Write; session-local, small payload; five clients & 100 & 108.756 & 111.98 & 116.067 & 46 \\ +Write; session-local, large payload; single client & 100 & 45.066 & 49.947 & 53.544 & 22.03 \\ +Write; session-local, large payload; five clients & 100 & 125.969 & 137.255 & 147.374 & 39.35 \\ +Edit; one replacement, small payload; single client & 100 & 38.647 & 42.615 & 46.878 & 25.61 \\ +Edit; one replacement, small payload; five clients & 100 & 134.61 & 141.758 & 146.725 & 37.22 \\ +Edit; one replacement, large payload; single client & 100 & 48.935 & 53.329 & 63.097 & 20.17 \\ +Edit; one replacement, large payload; five clients & 100 & 187.357 & 208.008 & 294.392 & 26.36 \\ +\hline +\end{tabular} +\caption{Public CLI operation timings in the frozen campaign. ``Small'' and ``large'' identify the two archived file-size cases without generalizing beyond the fixture.} +\label{tab:operations} +\end{table*} + +\begin{table}[t] +\centering +\scriptsize +\begin{tabular}{p{0.24\linewidth}p{0.13\linewidth}rrr} +\hline +Operation/case & Load & Daemon (MiB) & Sandbox (MiB) & CPU (ms) \\ +\hline +Workspace create & single client & 25.5 & 28.172 & 30.471 \\ +Workspace create & five clients & 34.125 & 43.527 & 79.502 \\ +Command no-op & single client & 13.875 & 20.031 & 35.003 \\ +Command no-op & five clients & 13.125 & 21.555 & 63.077 \\ +Read, large payload & five clients & 14.473 & 14.977 & 38.431 \\ +Write, large payload & five clients & 51.043 & 54.27 & 95.115 \\ +Edit, large payload & five clients & 62.602 & 66.715 & 148.732 \\ +\hline +\end{tabular} +\caption{Observed peak memory and CPU from selected frozen campaign cells. CPU is milliseconds per trial; these are not resource budgets or capacity guarantees.} +\label{tab:resources-compute} +\end{table} + +\begin{table}[t] +\centering +\scriptsize +\begin{tabular}{p{0.24\linewidth}p{0.13\linewidth}rrr} +\hline +Operation/case & Load & Read (MiB) & Write (MiB) & Upper (MiB) \\ +\hline +Workspace create & single client & 0 & 0.011 & 0.004 \\ +Workspace create & five clients & 0 & 0.027 & 0.02 \\ +Command no-op & single client & 0 & 0.001 & 0 \\ +Command no-op & five clients & 0 & 0.003 & 0 \\ +Read, large payload & five clients & 0 & 0 & 0 \\ +Write, large payload & five clients & 0 & 1.251 & 1.258 \\ +Edit, large payload & five clients & 0 & 1.251 & 1.258 \\ +\hline +\end{tabular} +\caption{Observed I/O and workspace-upper deltas from the same selected frozen cells.} +\label{tab:resources-io} +\end{table} diff --git a/ephemeral-sandbox-v1/sections/results_numeric_bindings.md b/ephemeral-sandbox-v1/sections/results_numeric_bindings.md new file mode 100644 index 0000000..6ad1113 --- /dev/null +++ b/ephemeral-sandbox-v1/sections/results_numeric_bindings.md @@ -0,0 +1,169 @@ +# Results numeric bindings + +Generated by `scripts/generate_latex_results.py` from the immutable Table-A output. +Each identifier below resolves through `numeric_evidence.json` to the frozen `numeric-provenance.csv` selector. + +## Campaign environment + +- `table1.cpu.logical_processors` +- `table1.memory_bytes` +- `table1.sandbox_limits.memory_bytes` +- `table1.sandbox_limits.pids_limit` +- `table1.sandbox_limits.vcpus` +- `table1.seed` +- `table1.trials.measured` +- `table1.trials.warmups` +- `table1.workspace_fixture.file_count` +- `table1.workspace_fixture.logical_mib` +- `table1.workspace_fixture.maximum_depth` + +## Startup + +- `table2.create_sandbox.none.c1.p50_ms` +- `table2.create_sandbox.none.c1.p95_ms` +- `table2.create_sandbox.none.c1.p99_ms` +- `table2.create_sandbox.none.c1.samples` +- `table2.create_sandbox.none.c1.throughput_ops_s` +- `table2.create_workspace.none.c1.p50_ms` +- `table2.create_workspace.none.c1.p95_ms` +- `table2.create_workspace.none.c1.p99_ms` +- `table2.create_workspace.none.c1.samples` +- `table2.create_workspace.none.c1.throughput_ops_s` +- `table2.create_workspace.none.c5.p50_ms` +- `table2.create_workspace.none.c5.p95_ms` +- `table2.create_workspace.none.c5.p99_ms` +- `table2.create_workspace.none.c5.samples` +- `table2.create_workspace.none.c5.throughput_ops_s` +- `table2.exec_command.noop.c1.p50_ms` +- `table2.exec_command.noop.c1.p95_ms` +- `table2.exec_command.noop.c1.p99_ms` +- `table2.exec_command.noop.c1.samples` +- `table2.exec_command.noop.c1.throughput_ops_s` + +## Public CLI operations + +- `table3.exec_command.fixture_read.c1.p50_ms` +- `table3.exec_command.fixture_read.c1.p95_ms` +- `table3.exec_command.fixture_read.c1.p99_ms` +- `table3.exec_command.fixture_read.c1.samples` +- `table3.exec_command.fixture_read.c1.throughput_ops_s` +- `table3.exec_command.fixture_read.c5.p50_ms` +- `table3.exec_command.fixture_read.c5.p95_ms` +- `table3.exec_command.fixture_read.c5.p99_ms` +- `table3.exec_command.fixture_read.c5.samples` +- `table3.exec_command.fixture_read.c5.throughput_ops_s` +- `table3.exec_command.noop.c1.p50_ms` +- `table3.exec_command.noop.c1.p95_ms` +- `table3.exec_command.noop.c1.p99_ms` +- `table3.exec_command.noop.c1.samples` +- `table3.exec_command.noop.c1.throughput_ops_s` +- `table3.exec_command.noop.c5.p50_ms` +- `table3.exec_command.noop.c5.p95_ms` +- `table3.exec_command.noop.c5.p99_ms` +- `table3.exec_command.noop.c5.samples` +- `table3.exec_command.noop.c5.throughput_ops_s` +- `table3.file_edit.262144.c1.p50_ms` +- `table3.file_edit.262144.c1.p95_ms` +- `table3.file_edit.262144.c1.p99_ms` +- `table3.file_edit.262144.c1.samples` +- `table3.file_edit.262144.c1.throughput_ops_s` +- `table3.file_edit.262144.c5.p50_ms` +- `table3.file_edit.262144.c5.p95_ms` +- `table3.file_edit.262144.c5.p99_ms` +- `table3.file_edit.262144.c5.samples` +- `table3.file_edit.262144.c5.throughput_ops_s` +- `table3.file_edit.4096.c1.p50_ms` +- `table3.file_edit.4096.c1.p95_ms` +- `table3.file_edit.4096.c1.p99_ms` +- `table3.file_edit.4096.c1.samples` +- `table3.file_edit.4096.c1.throughput_ops_s` +- `table3.file_edit.4096.c5.p50_ms` +- `table3.file_edit.4096.c5.p95_ms` +- `table3.file_edit.4096.c5.p99_ms` +- `table3.file_edit.4096.c5.samples` +- `table3.file_edit.4096.c5.throughput_ops_s` +- `table3.file_read.262144.c1.p50_ms` +- `table3.file_read.262144.c1.p95_ms` +- `table3.file_read.262144.c1.p99_ms` +- `table3.file_read.262144.c1.samples` +- `table3.file_read.262144.c1.throughput_ops_s` +- `table3.file_read.262144.c5.p50_ms` +- `table3.file_read.262144.c5.p95_ms` +- `table3.file_read.262144.c5.p99_ms` +- `table3.file_read.262144.c5.samples` +- `table3.file_read.262144.c5.throughput_ops_s` +- `table3.file_read.4096.c1.p50_ms` +- `table3.file_read.4096.c1.p95_ms` +- `table3.file_read.4096.c1.p99_ms` +- `table3.file_read.4096.c1.samples` +- `table3.file_read.4096.c1.throughput_ops_s` +- `table3.file_read.4096.c5.p50_ms` +- `table3.file_read.4096.c5.p95_ms` +- `table3.file_read.4096.c5.p99_ms` +- `table3.file_read.4096.c5.samples` +- `table3.file_read.4096.c5.throughput_ops_s` +- `table3.file_write.262144.c1.p50_ms` +- `table3.file_write.262144.c1.p95_ms` +- `table3.file_write.262144.c1.p99_ms` +- `table3.file_write.262144.c1.samples` +- `table3.file_write.262144.c1.throughput_ops_s` +- `table3.file_write.262144.c5.p50_ms` +- `table3.file_write.262144.c5.p95_ms` +- `table3.file_write.262144.c5.p99_ms` +- `table3.file_write.262144.c5.samples` +- `table3.file_write.262144.c5.throughput_ops_s` +- `table3.file_write.4096.c1.p50_ms` +- `table3.file_write.4096.c1.p95_ms` +- `table3.file_write.4096.c1.p99_ms` +- `table3.file_write.4096.c1.samples` +- `table3.file_write.4096.c1.throughput_ops_s` +- `table3.file_write.4096.c5.p50_ms` +- `table3.file_write.4096.c5.p95_ms` +- `table3.file_write.4096.c5.p99_ms` +- `table3.file_write.4096.c5.samples` +- `table3.file_write.4096.c5.throughput_ops_s` + +## Resources + +- `table4.create_workspace.none.c1.daemon_rss_bytes` +- `table4.create_workspace.none.c1.sandbox_block_read_bytes` +- `table4.create_workspace.none.c1.sandbox_block_write_bytes` +- `table4.create_workspace.none.c1.sandbox_cpu_time_ns` +- `table4.create_workspace.none.c1.sandbox_memory_peak_bytes` +- `table4.create_workspace.none.c1.upperdir_bytes` +- `table4.create_workspace.none.c5.daemon_rss_bytes` +- `table4.create_workspace.none.c5.sandbox_block_read_bytes` +- `table4.create_workspace.none.c5.sandbox_block_write_bytes` +- `table4.create_workspace.none.c5.sandbox_cpu_time_ns` +- `table4.create_workspace.none.c5.sandbox_memory_peak_bytes` +- `table4.create_workspace.none.c5.upperdir_bytes` +- `table4.exec_command.noop.c1.daemon_rss_bytes` +- `table4.exec_command.noop.c1.sandbox_block_read_bytes` +- `table4.exec_command.noop.c1.sandbox_block_write_bytes` +- `table4.exec_command.noop.c1.sandbox_cpu_time_ns` +- `table4.exec_command.noop.c1.sandbox_memory_peak_bytes` +- `table4.exec_command.noop.c1.upperdir_bytes` +- `table4.exec_command.noop.c5.daemon_rss_bytes` +- `table4.exec_command.noop.c5.sandbox_block_read_bytes` +- `table4.exec_command.noop.c5.sandbox_block_write_bytes` +- `table4.exec_command.noop.c5.sandbox_cpu_time_ns` +- `table4.exec_command.noop.c5.sandbox_memory_peak_bytes` +- `table4.exec_command.noop.c5.upperdir_bytes` +- `table4.file_edit.262144.c5.daemon_rss_bytes` +- `table4.file_edit.262144.c5.sandbox_block_read_bytes` +- `table4.file_edit.262144.c5.sandbox_block_write_bytes` +- `table4.file_edit.262144.c5.sandbox_cpu_time_ns` +- `table4.file_edit.262144.c5.sandbox_memory_peak_bytes` +- `table4.file_edit.262144.c5.upperdir_bytes` +- `table4.file_read.262144.c5.daemon_rss_bytes` +- `table4.file_read.262144.c5.sandbox_block_read_bytes` +- `table4.file_read.262144.c5.sandbox_block_write_bytes` +- `table4.file_read.262144.c5.sandbox_cpu_time_ns` +- `table4.file_read.262144.c5.sandbox_memory_peak_bytes` +- `table4.file_read.262144.c5.upperdir_bytes` +- `table4.file_write.262144.c5.daemon_rss_bytes` +- `table4.file_write.262144.c5.sandbox_block_read_bytes` +- `table4.file_write.262144.c5.sandbox_block_write_bytes` +- `table4.file_write.262144.c5.sandbox_cpu_time_ns` +- `table4.file_write.262144.c5.sandbox_memory_peak_bytes` +- `table4.file_write.262144.c5.upperdir_bytes` diff --git a/ephemeral-sandbox-v1/submission_readiness.md b/ephemeral-sandbox-v1/submission_readiness.md new file mode 100644 index 0000000..2cef226 --- /dev/null +++ b/ephemeral-sandbox-v1/submission_readiness.md @@ -0,0 +1,16 @@ +# Submission readiness + +| Gate | Status | Evidence or action | +| --- | --- | --- | +| Source and frozen result provenance | pass | Archive, Table-A tree, numeric registry, and tagged source anchors are recorded in `ARTIFACTS.md`. | +| Results integration | pass | Deterministic LaTeX projection and numeric checker pass. | +| Literature and citations | pass | Citation key, lock, and terminal primary-provider checks pass. | +| Manuscript build and visual QA | pass | LaTeX build has no overfull boxes or unresolved references; color and grayscale PDF review is recorded. | +| Figure review | pass with waiver | Explanatory layout/style exceptions are recorded in `figures/concept-figure-review.md`. | +| Author and venue metadata | blocked | Paper owner must provide real authors, affiliations, category/venue, and disclosures. | +| Merge and publication authority | gated | No merge or external submission is authorized by this file. | + +**Disposition: FAIL/GATED.** The remaining gate is external editorial metadata, +not an experiment or build failure. This package must not be described as +submission-ready until that owner input is supplied and the final checks are +rerun. diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-01.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-01.png new file mode 100644 index 0000000..3a6154a Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-01.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-02.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-02.png new file mode 100644 index 0000000..03c7457 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-02.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-03.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-03.png new file mode 100644 index 0000000..6d5fbdb Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-03.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-04.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-04.png new file mode 100644 index 0000000..d942105 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-04.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-05.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-05.png new file mode 100644 index 0000000..b7ed0a1 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-05.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-06.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-06.png new file mode 100644 index 0000000..f7f6869 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-06.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-07.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-07.png new file mode 100644 index 0000000..05fb0cc Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-07.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-08.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-08.png new file mode 100644 index 0000000..8a4955f Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-08.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-09.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-09.png new file mode 100644 index 0000000..0a77d75 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-09.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-10.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-10.png new file mode 100644 index 0000000..16c1dcb Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-10.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-11.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-11.png new file mode 100644 index 0000000..da806fb Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-11.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-12.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-12.png new file mode 100644 index 0000000..55642d6 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-12.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-13.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-13.png new file mode 100644 index 0000000..4123168 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-20260730/page-13.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-color.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-color.png new file mode 100644 index 0000000..4c38815 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-color.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-gray.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-gray.png new file mode 100644 index 0000000..330fba6 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/contact-gray.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-01.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-01.png new file mode 100644 index 0000000..de72214 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-01.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-02.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-02.png new file mode 100644 index 0000000..65f97e4 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-02.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-03.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-03.png new file mode 100644 index 0000000..0e1ae1e Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-03.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-04.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-04.png new file mode 100644 index 0000000..45376b0 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-04.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-05.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-05.png new file mode 100644 index 0000000..84efcb2 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-05.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-06.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-06.png new file mode 100644 index 0000000..d03943c Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-06.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-07.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-07.png new file mode 100644 index 0000000..ad472dd Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-07.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-08.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-08.png new file mode 100644 index 0000000..7920832 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-08.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-09.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-09.png new file mode 100644 index 0000000..1b34505 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-09.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-10.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-10.png new file mode 100644 index 0000000..6322683 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-10.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-11.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-11.png new file mode 100644 index 0000000..eb8bb2b Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-11.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-12.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-12.png new file mode 100644 index 0000000..9faf445 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-12.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-13.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-13.png new file mode 100644 index 0000000..002f7ae Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-13.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-14.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-14.png new file mode 100644 index 0000000..dad9be4 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/gray-14.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-01.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-01.png new file mode 100644 index 0000000..3a6154a Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-01.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-02.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-02.png new file mode 100644 index 0000000..03c7457 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-02.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-03.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-03.png new file mode 100644 index 0000000..6d5fbdb Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-03.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-04.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-04.png new file mode 100644 index 0000000..d942105 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-04.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-05.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-05.png new file mode 100644 index 0000000..b7ed0a1 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-05.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-06.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-06.png new file mode 100644 index 0000000..f7f6869 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-06.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-07.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-07.png new file mode 100644 index 0000000..c5deab9 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-07.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-08.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-08.png new file mode 100644 index 0000000..6e6b0df Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-08.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-09.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-09.png new file mode 100644 index 0000000..b619c6f Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-09.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-10.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-10.png new file mode 100644 index 0000000..69ee1ac Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-10.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-11.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-11.png new file mode 100644 index 0000000..8296f08 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-11.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-12.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-12.png new file mode 100644 index 0000000..94d6b9d Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-12.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-13.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-13.png new file mode 100644 index 0000000..e7defa1 Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-13.png differ diff --git a/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-14.png b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-14.png new file mode 100644 index 0000000..068c9ff Binary files /dev/null and b/ephemeral-sandbox-v1/tmp/pdfs/pw3-final-20260730/page-14.png differ