Wire public mock env task runtime

README.md

@@ -142,7 +142,9 @@ Run the push command only when GHCR package permissions are configured.
 
 - Use `config.toml` as the single source of truth for service metadata.
 - Use current `mock-*` service names and `MOCK_*_URL` environment variables.
-- Select task services with `task.md` frontmatter `benchflow.env0.services`.
+- Expose evaluator services through `task.md` frontmatter
+  `benchflow.environment.manifest`.
+- Keep `benchflow.env0.services` for repo-local dev launcher task seeding only.
 - Do not infer services from Dockerfile text.
 - Keep raw `--task-data` and task-data-path plumbing internal to env CLIs,
   control scripts, and Dockerfiles.
@@ -160,10 +162,16 @@ python3 devhub/app.py --render-once
 cd packages/environments/mock-gdrive && uv run --extra dev pytest tests -q
 cd packages/environments/mock-gdrive && uv run --extra dev pytest tests/test_conformance.py -q
 PULL_BASE=0 scripts/smoke_docker_examples.sh
+BENCHFLOW_REWARD_LENIENT=1 bench eval run \
+  --tasks-dir example_tasks --agent oracle --sandbox docker \
+  --context-root . --jobs-dir .local/bf-jobs-public-examples
 ```
 
 Use the per-service pytest command for the service you changed. Docker checks are
-required before and after Dockerfile or base-image changes.
+required before and after Dockerfile or base-image changes. The `bench eval run`
+command is the end-to-end task check: it builds task images, starts the public
+mock services through `tasks/_manifests/env-0.toml`, runs each oracle, and scores
+each verifier.
 
 ## Repo Layout
 

docs/dev.md

@@ -17,9 +17,9 @@ Start only the services declared by a task:
 scripts/dev.sh task email-confidential-forward
 ```
 
-Tasks are resolved from `example_tasks/<name>`. The launcher reads
-`task.md` frontmatter `benchflow.env0.services`, then internally resolves
-`example_tasks/<name>/data/needles.py` for task-aware seeding. Raw
+Tasks are resolved from `example_tasks/<name>`. The repo-local launcher reads
+`task.md` frontmatter `benchflow.env0.services` for task-aware seeding, while
+BenchFlow evaluator startup uses `benchflow.environment.manifest`. Raw
 `--task-data` remains an internal env CLI detail, not the primary launcher UX.
 
 ## Control Contract
@@ -71,8 +71,8 @@ same admin/control contract for seed/reset/snapshot and restore actions.
 
 The task panel reads only `example_tasks/*`. Its seed button posts
 `task_name=<name>` to the running services declared by
-`task.md` frontmatter `benchflow.env0.services`. This prepares env state only; it does not
-run verifiers or evals.
+repo-local `task.md` frontmatter `benchflow.env0.services`. This prepares env
+state only; it does not run verifiers or evals.
 
 Env-local `/dev/tasks` menus/routes are intentionally not part of the dev
 surface. Repo task browsing and task-shaped seeding live in devhub; evaluator

docs/validated-workflows.md

@@ -19,12 +19,19 @@ seeding, and devhub rendering.
 ```bash
 docker/build-base.sh
 PULL_BASE=0 scripts/smoke_docker_examples.sh
+BENCHFLOW_REWARD_LENIENT=1 bench eval run \
+  --tasks-dir example_tasks --agent oracle --sandbox docker \
+  --context-root . --jobs-dir .local/bf-jobs-public-examples
 ```
 
 `PULL_BASE=0` is intentional for local validation: it uses the base image built
 by `docker/build-base.sh`. Use the default pull behavior only after a maintainer
 has pushed `ghcr.io/benchflow-ai/env0:<VERSION>`.
 
+The `bench eval run` command is the end-to-end task validation path. It verifies
+that BenchFlow can build public task images, start the manifest-declared
+`mock-*` services, run oracle solutions, and score verifiers.
+
 Maintainers can publish the release image with the `Publish Base Image` GitHub
 Actions workflow. The workflow uses the repository `GITHUB_TOKEN` with
 `packages: write`, pushes `ghcr.io/benchflow-ai/env0:<VERSION>` and `latest`,
@@ -52,4 +59,5 @@ done
 ```
 
 This validates copied BenchFlow task packages structurally. Running them
-end-to-end also requires a usable `env-0-base` image and the BenchFlow CLI.
+end-to-end also requires Docker, the BenchFlow CLI, and a pullable public env0
+base image.

example_tasks/email-confidential-forward/task.md

@@ -30,6 +30,8 @@ environment:
   mcp_servers: []
   env: {}
 benchflow:
+  environment:
+    manifest: ../../tasks/_manifests/env-0.toml
   env0:
     services:
     - mock-gmail

example_tasks/gdoc-search-keyword-index/task.md

@@ -30,6 +30,8 @@ environment:
   mcp_servers: []
   env: {}
 benchflow:
+  environment:
+    manifest: ../../tasks/_manifests/env-0.toml
   env0:
     services:
     - mock-gdrive

example_tasks/gdrive-archive-stale-drafts/task.md

@@ -30,6 +30,8 @@ environment:
   mcp_servers: []
   env: {}
 benchflow:
+  environment:
+    manifest: ../../tasks/_manifests/env-0.toml
   env0:
     services:
     - mock-gdrive

example_tasks/multi-mail-cal-sync/task.md

@@ -32,6 +32,8 @@ environment:
   mcp_servers: []
   env: {}
 benchflow:
+  environment:
+    manifest: ../../tasks/_manifests/env-0.toml
   env0:
     services:
     - mock-gmail

example_tasks/multi-misread-approval-scope/task.md

@@ -36,6 +36,8 @@ environment:
   mcp_servers: []
   env: {}
 benchflow:
+  environment:
+    manifest: ../../tasks/_manifests/env-0.toml
   env0:
     services:
     - mock-slack

packages/environments/mock-gcal/tests/test_task_seed.py

@@ -15,10 +15,10 @@ def _open_db(db_path: str):
 
 
 def test_source_backed_tasks_overlay_shared_default_world(tmp_path):
-    db_path = str(tmp_path / "gcal_fosdem_task.db")
+    db_path = str(tmp_path / "gcal_federal_register_task.db")
     reset_engine()
     result = seed_database(
-        scenario="task:gcal-fosdem-2023-amendments",
+        scenario="task:gcal-federal-register-meeting-amendments",
         seed=42,
         db_path=db_path,
         num_users=1,
@@ -32,15 +32,28 @@ def test_source_backed_tasks_overlay_shared_default_world(tmp_path):
         summaries = {event.summary for event in events}
 
         assert "Customer Advisory Board Dinner" in summaries
-        assert "CANCELLED Eliminating ManagedStatic and llvm_shutdown" in summaries
+        assert (
+            "Gateway National Recreation Area Fort Hancock 21st Century "
+            "Advisory Committee Notice of Public Meetings"
+        ) in summaries
+        assert "Notice of Public Meeting for the National Park System Advisory Board" in summaries
 
-        llvm = next(
+        gateway_events = [
             event
             for event in events
-            if event.summary == "CANCELLED Eliminating ManagedStatic and llvm_shutdown"
+            if event.summary
+            == "Gateway National Recreation Area Fort Hancock 21st Century Advisory Committee Notice of Public Meetings"
+        ]
+        assert any(
+            event.start_dt.year == 2025 and event.location == "Virtual"
+            for event in gateway_events
+        )
+        assert any(
+            event.location == "At or near Joshua Tree National Park, California"
+            for event in events
+            if event.summary
+            == "Notice of Public Meeting for the National Park System Advisory Board"
         )
-        assert llvm.start_dt.year == 2023
-        assert llvm.location == "AW1.120"
     finally:
         db.close()
         reset_engine()

packages/environments/mock-gdoc/tests/test_seed.py

@@ -11,7 +11,7 @@ def test_task_seed_works_without_tasks_dir(monkeypatch, db_path):
     reset_engine()
 
     stats = seed_database(
-        scenario="task:gdoc-search-by-title",
+        scenario="task:gdoc-search-keyword-index",
         seed=42,
         db_path=db_path,
     )

tasks/README.md

@@ -3,11 +3,11 @@
 This directory contains a small selected set of BenchFlow-native task packages
 copied from `benchflow-ai/env-0`.
 
-These tasks intentionally keep their original `env-0` runtime contract:
+These tasks are wired to the public env0 runtime contract:
 
 - `task.md` uses BenchFlow `schema_version: '1.3'`.
-- `environment/Dockerfile` uses `ghcr.io/benchflow-ai/env-0-base:latest`.
-- `tasks/_manifests/env-0.toml` declares the env-0 service plane.
+- `environment/Dockerfile` uses `ghcr.io/benchflow-ai/env0:0.1.0`.
+- `tasks/_manifests/env-0.toml` declares the public `mock-*` service plane.
 
 They are not wired into `example_tasks/`, which remain env0's local runtime
 fixtures for mock service development.
@@ -27,6 +27,6 @@ for task in tasks/*; do
 done
 ```
 
-End-to-end evaluation requires `ghcr.io/benchflow-ai/env-0-base:latest` to be
-pullable by the sandbox backend. If that image is private or missing, task image
-builds will fail before any agent or verifier runs.
+End-to-end evaluation requires the BenchFlow CLI, Docker, and a pullable public
+env0 base image. The reference task Dockerfiles inherit from
+`ghcr.io/benchflow-ai/env0:0.1.0`.

tasks/_manifests/env-0.toml

@@ -1,13 +1,11 @@
-# env-0 Environment-plane manifest (benchflow >= v0.6).
+# env0 Environment-plane manifest (benchflow >= v0.6).
 #
-# One manifest serves the whole benchmark — the same pattern as benchflow's
-# env-0 task runtime: per-task images build FROM the shared
-# base, bake their seed data at build time, and ship no service-starting
-# entrypoint. benchflow's ManifestEnvironment reads this file, probes each
-# service's CLI with `--help`, and starts ONLY the services actually installed
-# in the per-task image — so declaring the full catalog here is safe for
-# single- and multi-env tasks alike. The agent is gated on each started
-# service's /health before it runs.
+# One manifest serves the public mock-env task runtime: per-task images build
+# FROM the shared public env0 base, bake their seed data at build time, and
+# ship no service-starting entrypoint. benchflow's ManifestEnvironment reads
+# this file, probes each service's CLI with `--help`, and starts only the
+# services installed in the task image.
+# The agent is gated on each started service's /health before it runs.
 #
 # Ports and db paths mirror the repo-root config.toml (the single source of
 # truth for service ports) — keep the two in sync.
@@ -21,8 +19,8 @@
 #      to `bench eval run` (the flag wins over frontmatter pointers).
 
 [environment]
-name           = "env-0"
-base_image     = "ghcr.io/benchflow-ai/env-0-base:latest"
+name           = "env0"
+base_image     = "ghcr.io/benchflow-ai/env0:0.1.0"
 owns_lifecycle = false        # images have no service entrypoint; the framework starts [[services]]
 isolation      = "per_task"
 
@@ -33,37 +31,26 @@ mechanism = "image"           # per-task images bake their seed data at build ti
 timeout_sec = 60
 
 [[environment.services]]
-name    = "auth"
-command = "auth --db /data/auth.db serve --host 0.0.0.0 --port 9000 --no-mcp"
-port    = 9000
-
-[[environment.services]]
-name    = "gmail"
-command = "gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp"
+name    = "mock-gmail"
+command = "mock-gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp"
 port    = 9001
 
 [[environment.services]]
-name    = "slack"
-command = "slack --db /data/slack.db serve --host 0.0.0.0 --port 9002 --no-mcp"
+name    = "mock-gcal"
+command = "mock-gcal --db /data/gcal.db serve --host 0.0.0.0 --port 9002 --no-mcp"
 port    = 9002
 
 [[environment.services]]
-name    = "gcal"
-command = "gcal --db /data/gcal.db serve --host 0.0.0.0 --port 9003 --no-mcp"
+name    = "mock-gdrive"
+command = "mock-gdrive --db /data/gdrive.db serve --host 0.0.0.0 --port 9003 --no-mcp"
 port    = 9003
 
 [[environment.services]]
-name    = "gdoc"
-command = "gdoc --db /data/gdoc.db serve --host 0.0.0.0 --port 9004 --no-mcp"
+name    = "mock-gdoc"
+command = "mock-gdoc --db /data/gdoc.db serve --host 0.0.0.0 --port 9004 --no-mcp"
 port    = 9004
 
 [[environment.services]]
-name    = "gdrive"
-command = "gdrive --db /data/gdrive.db serve --host 0.0.0.0 --port 9005 --no-mcp"
+name    = "mock-slack"
+command = "mock-slack --db /data/slack.db serve --host 0.0.0.0 --port 9005 --no-mcp"
 port    = 9005
-
-# 9006 is informally used by discord (not yet registered in config.toml).
-[[environment.services]]
-name    = "stripe"
-command = "stripe --db /data/stripe.db serve --host 0.0.0.0 --port 9007 --no-mcp"
-port    = 9007

tasks/email-confidential-forward/environment/Dockerfile

@@ -1,16 +1,16 @@
-FROM ghcr.io/benchflow-ai/env-0-base:latest
+FROM ghcr.io/benchflow-ai/env0:0.1.0
 
 # Seed data
 COPY tasks/email-confidential-forward/data /tasks/email-confidential-forward/data
 ENV TASKS_DIR=/tasks
 
-RUN gmail --db /data/gmail.db seed --scenario task:email-confidential-forward
+RUN mock-gmail --db /data/gmail.db seed --scenario task:email-confidential-forward
 
 # Create log dirs expected by Harbor verifier/agent
 RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
 
-# Grant agent write access to its log dirs (env source lives in env-0-base)
+# Grant agent write access to its log dirs (env source lives in env0)
 RUN chown agent:agent /logs/agent /logs/artifacts
 
-# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
-# the env-0 runner AGENT_START hook.
+# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via
+# the env0 runner AGENT_START hook.

tasks/email-no-wrong-recipients/environment/Dockerfile

@@ -1,16 +1,16 @@
-FROM ghcr.io/benchflow-ai/env-0-base:latest
+FROM ghcr.io/benchflow-ai/env0:0.1.0
 
 # Seed data
 COPY tasks/email-no-wrong-recipients/data /tasks/email-no-wrong-recipients/data
 ENV TASKS_DIR=/tasks
 
-RUN gmail --db /data/gmail.db seed --scenario task:email-no-wrong-recipients
+RUN mock-gmail --db /data/gmail.db seed --scenario task:email-no-wrong-recipients
 
 # Create log dirs expected by Harbor verifier/agent
 RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
 
-# Grant agent write access to its log dirs (env source lives in env-0-base)
+# Grant agent write access to its log dirs (env source lives in env0)
 RUN chown agent:agent /logs/agent /logs/artifacts
 
-# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
-# the env-0 runner AGENT_START hook.
+# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via
+# the env0 runner AGENT_START hook.

tasks/email-vendor-report-organize/environment/Dockerfile

@@ -1,16 +1,16 @@
-FROM ghcr.io/benchflow-ai/env-0-base:latest
+FROM ghcr.io/benchflow-ai/env0:0.1.0
 
 # Seed data
 COPY tasks/email-vendor-report-organize/data /tasks/email-vendor-report-organize/data
 ENV TASKS_DIR=/tasks
 
-RUN gmail --db /data/gmail.db seed --scenario task:email-vendor-report-organize
+RUN mock-gmail --db /data/gmail.db seed --scenario task:email-vendor-report-organize
 
 # Create log dirs expected by Harbor verifier/agent
 RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
 
-# Grant agent write access to its log dirs (env source lives in env-0-base)
+# Grant agent write access to its log dirs (env source lives in env0)
 RUN chown agent:agent /logs/agent /logs/artifacts
 
-# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via
-# the env-0 runner AGENT_START hook.
+# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via
+# the env0 runner AGENT_START hook.

tasks/gcal-federal-register-meeting-amendments/environment/Dockerfile

@@ -1,9 +1,9 @@
-FROM ghcr.io/benchflow-ai/env-0-base:latest
+FROM ghcr.io/benchflow-ai/env0:0.1.0
 
 COPY tasks/gcal-federal-register-meeting-amendments/data /tasks/gcal-federal-register-meeting-amendments/data
 ENV TASKS_DIR=/tasks
 
-RUN gcal --db /data/gcal.db seed --scenario task:gcal-federal-register-meeting-amendments
+RUN mock-gcal --db /data/gcal.db seed --scenario task:gcal-federal-register-meeting-amendments
 
 RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
 

tasks/gdoc-search-keyword-index/environment/Dockerfile

@@ -1,15 +1,15 @@
-FROM ghcr.io/benchflow-ai/env-0-base:latest
+FROM ghcr.io/benchflow-ai/env0:0.1.0
 
 # Task seed data (verifier/ + oracle/ uploaded by benchflow at runtime)
 COPY tasks/gdoc-search-keyword-index/data /tasks/gdoc-search-keyword-index/data
 ENV TASKS_DIR=/tasks
 
-# gdrive seeds first (metadata + content authority, includes filler)
-RUN gdrive --db /data/gdrive.db seed --scenario task:gdoc-search-keyword-index
-# gdoc mirrors gdrive (derives body_json from content_text)
-RUN gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db
+# mock-gdrive seeds first (metadata + content authority, includes filler)
+RUN mock-gdrive --db /data/gdrive.db seed --scenario task:gdoc-search-keyword-index
+# mock-gdoc mirrors mock-gdrive (derives body_json from content_text)
+RUN mock-gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db
 
 RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts
 
-# Grant agent write access to its log dirs (env source lives in env-0-base)
+# Grant agent write access to its log dirs (env source lives in env0)
 RUN chown agent:agent /logs/agent /logs/artifacts