Publish Standard60 task snapshot

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -25,7 +25,7 @@ body:
     id: version
     attributes:
       label: env0 version or commit
-      placeholder: "0.1.0 or git SHA"
+      placeholder: "0.2.0 or git SHA"
   - type: textarea
     id: validation
     attributes:

.github/workflows/ci.yml

@@ -18,6 +18,11 @@ jobs:
       - uses: astral-sh/setup-uv@v5
       - name: Run repo smoke
         run: scripts/smoke_dev.sh
+      - name: Check Standard60 task snapshot
+        run: |
+          python3 -m unittest \
+            tests/test_standard60_tasks.py \
+            tests/test_public_data_hygiene.py
 
   conformance:
     name: Conformance (${{ matrix.package }})
@@ -60,12 +65,46 @@ jobs:
   docker-smoke:
     name: Docker base and examples
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
+      - uses: astral-sh/setup-uv@v5
       - name: Build base image
         run: docker/build-base.sh
       - name: Smoke example task images
         run: PULL_BASE=0 scripts/smoke_docker_examples.sh
+      - name: Run one task per Standard60 mock environment
+        env:
+          BENCHFLOW_REWARD_LENIENT: "1"
+        run: |
+          uvx --from "benchflow==0.6.4" bench eval run \
+            --tasks-dir tasks \
+            --include auth-least-privilege-summary \
+            --include email-confidential-forward \
+            --include gcal-federal-register-meeting-amendments \
+            --include gdrive-sensitive-file-lockdown \
+            --include gdoc-search-keyword-index \
+            --include multi-mail-slack-invite \
+            --include stripe-refund-correct-customer \
+            --agent oracle \
+            --sandbox docker \
+            --context-root . \
+            --concurrency 2 \
+            --build-concurrency 2 \
+            --jobs-dir .local/ci-standard60-representatives
+      - name: Run the Discord example task
+        env:
+          BENCHFLOW_REWARD_LENIENT: "1"
+        run: |
+          uvx --from "benchflow==0.6.4" bench eval run \
+            --tasks-dir example_tasks \
+            --include discord-incident-followup \
+            --agent oracle \
+            --sandbox docker \
+            --context-root . \
+            --concurrency 1 \
+            --build-concurrency 1 \
+            --jobs-dir .local/ci-discord-representative

.github/workflows/publish-base-image.yml

@@ -25,4 +25,20 @@ jobs:
       - name: Verify remote pull
         env:
           REPO: ghcr.io/${{ github.repository }}
-        run: docker pull "$REPO:$(cat VERSION)"
+        run: |
+          IMAGE="$REPO:$(cat VERSION)"
+          docker pull "$IMAGE"
+          docker run --rm "$IMAGE" sh -lc '
+            for command in \
+              mock-auth \
+              mock-discord \
+              mock-gcal \
+              mock-gdoc \
+              mock-gdrive \
+              mock-gmail \
+              mock-slack \
+              mock-stripe
+            do
+              command -v "$command"
+            done
+          '

.gitignore

@@ -11,6 +11,8 @@ Thumbs.db
 !.env.*.example
 client_secret*.json
 credentials*.json
+auth.json
+.credentials.json
 token*.json
 secrets/
 
@@ -52,4 +54,3 @@ node_modules/
 .turbo/
 .next/
 coverage/
-

CONTRIBUTING.md

@@ -4,10 +4,11 @@ env0 is the first-party mock-environment runtime for agent testing. It owns
 high-fidelity mock services, deterministic seed data, local tooling,
 API-parity fixtures, devhub, and the shared Docker base image.
 
-The current v0.1 runtime ships five high-fidelity mock services:
-`mock-gmail`, `mock-gcal`, `mock-gdrive`, `mock-gdoc`, and `mock-slack`.
-They replicate Google Workspace and Slack API surfaces with state management
-and deterministic snapshot/restore.
+The current v0.2 runtime ships eight mock services: `mock-auth`, `mock-gmail`,
+`mock-gcal`, `mock-gdrive`, `mock-gdoc`, `mock-slack`, `mock-discord`, and
+`mock-stripe`. They provide state management and deterministic
+snapshot/restore across identity, Google Workspace, Slack, Discord, and Stripe
+surfaces.
 
 There are two ways to contribute:
 
@@ -27,9 +28,9 @@ env runtime validation and copied-reference workflows:
 - [`example_tasks/`](https://github.com/benchflow-ai/env0/tree/main/example_tasks)
   contains small runtime fixtures/templates for env0 service and launcher
   testing.
-- [`tasks/`](https://github.com/benchflow-ai/env0/tree/main/tasks) contains a
-  small copied [BenchFlow](https://github.com/benchflow-ai/benchflow)-native
-  reference set.
+- [`tasks/`](https://github.com/benchflow-ai/env0/tree/main/tasks) publishes
+  the current 60-package Standard60 snapshot in
+  [BenchFlow](https://github.com/benchflow-ai/benchflow)-native format.
 
 ## What To Contribute
 
@@ -138,6 +139,8 @@ PR:
 - Do not copy environment source code into thin task images.
 - Do not commit credentials, OAuth tokens, live account exports, or private
   customer data — and do not paste them into Discord.
+- Keep demo credentials and sensitive-looking benchmark values clearly
+  synthetic and scoped to localhost mock services.
 - Do not commit BenchChat handoff bundles, `.env` files, generated credential
   exports, or API-key screenshots.
 
@@ -203,20 +206,18 @@ PULL_BASE=0 scripts/smoke_docker_examples.sh
 Copied BenchFlow task packages under `tasks/`:
 
 ```bash
-for task in tasks/*; do
-  [ -d "$task" ] || continue
-  [ "$(basename "$task")" = "_manifests" ] && continue
-  bench tasks check "$task" --level structural
-done
+python3 -m unittest tests/test_standard60_tasks.py
+
+while IFS= read -r task; do
+  bench tasks check "tasks/${task}" --level structural
+done < tasks/STANDARD60_MANIFEST.txt
 ```
 
 These checks require the BenchFlow CLI and validate copied task packages
 structurally only.
 
-End-to-end evaluation of copied downstream task packages may also require the
-upstream `ghcr.io/benchflow-ai/env-0-base:latest` image because those packages
-preserve their source-runner contract. New env0 example-task Dockerfiles should
-instead use `ghcr.io/benchflow-ai/env0:<VERSION>`.
+End-to-end evaluation uses the public
+`ghcr.io/benchflow-ai/env0:<VERSION>` image pinned by each task package.
 
 ### Pull request checklist
 

MAINTAINER.md

@@ -0,0 +1,79 @@
+# Maintainer Guide
+
+This guide covers release and publication work. Contributor-facing setup remains
+in [`CONTRIBUTING.md`](CONTRIBUTING.md).
+
+## Repository Boundaries
+
+- `config.toml` is the source of truth for public mock-service metadata.
+- `example_tasks/` contains small runtime fixtures and templates.
+- `tasks/` publishes the exact Standard60 snapshot listed in
+  `tasks/STANDARD60_MANIFEST.txt`.
+- Task authoring and benchmark policy remain downstream concerns.
+
+## Pull Request Gate
+
+Before merging a release-affecting pull request:
+
+```bash
+scripts/smoke_dev.sh
+python3 -m unittest \
+  tests/test_standard60_tasks.py \
+  tests/test_public_data_hygiene.py
+```
+
+Run the relevant package conformance suite when service behavior changes:
+
+```bash
+cd packages/environments/mock-gdrive
+uv run --extra dev pytest tests/test_conformance.py -q
+```
+
+CI additionally:
+
+- builds the current `VERSION` image;
+- smokes every example-task image;
+- runs one oracle task for each of the eight public mock environments;
+- verifies all 68 task packages use native `task.md` layout.
+
+## Standard60 Updates
+
+When refreshing `tasks/`:
+
+1. Keep exactly the names in `tasks/STANDARD60_MANIFEST.txt`.
+2. Use only `task.md`, `oracle/`, and `verifier/`; do not add `task.toml`,
+   `instruction.md`, `solution/`, or task-level `tests/`.
+3. Use public `mock-*` CLIs and the current
+   `ghcr.io/benchflow-ai/env0:<VERSION>`.
+4. Lock copied seed payloads after image build-time seeding.
+5. Replace real-person identifiers and credential-shaped values with reserved
+   synthetic fixtures before publication.
+
+## Base Image Release
+
+`VERSION` is the base-image semver source of truth.
+
+After the release PR is squash-merged and `main` CI is green:
+
+1. Dispatch the `Publish Base Image` workflow on `main`.
+2. Confirm it pushes both `ghcr.io/benchflow-ai/env0:<VERSION>` and `latest`.
+3. Confirm its remote verification step finds all eight `mock-*` CLIs.
+4. Tag the merged commit using the repository release line, for example
+   `v0.2` for `VERSION=0.2.0`.
+5. Publish a GitHub release from that tag and verify it is marked latest.
+
+Do not publish the GitHub release before the GHCR workflow succeeds.
+
+## Public Data Review
+
+Before release, verify:
+
+- no `.env`, auth, token, credential, database, or local runtime files are
+  tracked;
+- no real API keys, JWTs, assignable SSNs, local workstation paths, or private
+  repository links are present;
+- live-capture fixtures contain only sanitized test-account data;
+- intentional mock credentials and test cards are clearly documented and have
+  no authority outside localhost services.
+
+Report actual vulnerabilities privately according to [`SECURITY.md`](SECURITY.md).

README.md

@@ -113,8 +113,9 @@ scripts/dev.sh task gdrive-archive-stale-drafts
 Evaluators should score the final service state, the diff from the initial
 snapshot, and the action log. They should not depend on agent transcript text.
 
-[`tasks/`](tasks/) contains additional BenchFlow-format task packages kept as a
-public reference set. They are not the source of truth for benchmark policy.
+[`tasks/`](tasks/) publishes the current 60-package Standard60 snapshot in
+BenchFlow format. It is a runnable public snapshot, not the source of truth for
+task authoring or benchmark policy.
 
 ## Docker Base Image
 
@@ -177,12 +178,12 @@ each verifier.
 
 ```text
 env0/
-|-- packages/environments/   # mock-gmail, mock-gcal, mock-gdoc, mock-gdrive, mock-slack
+|-- packages/environments/   # mock-auth, Workspace, Slack, Discord, Stripe
 |-- devhub/                  # local dev dashboard on port 9060
 |-- docker/                  # base-image generation and gws wrapper
 |-- docs/                    # guides, parity audit, validated workflows
 |-- example_tasks/           # runnable env0 task fixtures
-|-- tasks/                   # public BenchFlow-format reference tasks
+|-- tasks/                   # public Standard60 snapshot
 |-- scripts/                 # dev.sh, env0_control.py, smoke tests
 |-- config.toml              # service and port metadata
 `-- VERSION                  # base-image version
@@ -199,6 +200,7 @@ env0/
 - [Run public tasks with BenchFlow](docs/guides/run-tasks-with-benchflow.md)
 - [Good first contributions](docs/good-first-contributions.md)
 - [Contributing](CONTRIBUTING.md)
+- [Maintainer guide](MAINTAINER.md)
 - [Security policy](SECURITY.md)
 
 ## Related Repos

SECURITY.md

@@ -27,3 +27,15 @@ this repo unless they expose an env0 runtime vulnerability.
 Do not commit OAuth tokens, API keys, real account exports, provider credential
 files, or private customer data. Fixture captures must be reviewed and sanitized
 before publication.
+
+## Synthetic Security Fixtures
+
+Some agent-safety tasks intentionally contain sensitive-looking values. These
+must be unmistakably synthetic, use reserved/example identities where possible,
+and authorize only localhost mock services.
+
+The fixed RSA key at
+`packages/environments/mock-auth/mock_auth/seed/keys/env-0-auth-key-001.pem`
+is an intentional deterministic fixture. It signs fake JWTs for fake localhost
+users and protects no external resource; the file carries the same warning in
+its header.

VERSION

@@ -1 +1 @@
-0.1.0
+0.2.0

docs/README.md

@@ -23,13 +23,15 @@ validation.
   self-contained setup and subscription-agent task runs for public tasks.
 - [Contributing](../CONTRIBUTING.md) — repo boundaries, validation matrix, and
   pull request expectations.
+- [Maintainer guide](../MAINTAINER.md) — Standard60 sync, image publication,
+  versioning, and GitHub release workflow.
 - [Security policy](../SECURITY.md) — private vulnerability reporting and
   credential hygiene.
 
 ## Boundaries
 
 - `example_tasks/` are env0 runtime fixtures.
-- `tasks/` contains copied BenchFlow task packages for reference and downstream
-  evaluation.
+- `tasks/` publishes the current 60-package Standard60 snapshot for public
+  downstream evaluation.
 - Canonical benchmark task authoring and scoring semantics live outside this
   repo.