diff --git a/README.md b/README.md index dd1548e96..ffc859463 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,9 @@ Run the push command only when GHCR package permissions are configured. - Use `config.toml` as the single source of truth for service metadata. - Use current `mock-*` service names and `MOCK_*_URL` environment variables. -- Select task services with `task.md` frontmatter `benchflow.env0.services`. +- Expose evaluator services through `task.md` frontmatter + `benchflow.environment.manifest`. +- Keep `benchflow.env0.services` for repo-local dev launcher task seeding only. - Do not infer services from Dockerfile text. - Keep raw `--task-data` and task-data-path plumbing internal to env CLIs, control scripts, and Dockerfiles. @@ -160,10 +162,16 @@ python3 devhub/app.py --render-once cd packages/environments/mock-gdrive && uv run --extra dev pytest tests -q cd packages/environments/mock-gdrive && uv run --extra dev pytest tests/test_conformance.py -q PULL_BASE=0 scripts/smoke_docker_examples.sh +BENCHFLOW_REWARD_LENIENT=1 bench eval run \ + --tasks-dir example_tasks --agent oracle --sandbox docker \ + --context-root . --jobs-dir .local/bf-jobs-public-examples ``` Use the per-service pytest command for the service you changed. Docker checks are -required before and after Dockerfile or base-image changes. +required before and after Dockerfile or base-image changes. The `bench eval run` +command is the end-to-end task check: it builds task images, starts the public +mock services through `tasks/_manifests/env-0.toml`, runs each oracle, and scores +each verifier. ## Repo Layout diff --git a/docs/dev.md b/docs/dev.md index 0669825e9..31f749a2f 100644 --- a/docs/dev.md +++ b/docs/dev.md @@ -17,9 +17,9 @@ Start only the services declared by a task: scripts/dev.sh task email-confidential-forward ``` -Tasks are resolved from `example_tasks/`. The launcher reads -`task.md` frontmatter `benchflow.env0.services`, then internally resolves -`example_tasks//data/needles.py` for task-aware seeding. Raw +Tasks are resolved from `example_tasks/`. The repo-local launcher reads +`task.md` frontmatter `benchflow.env0.services` for task-aware seeding, while +BenchFlow evaluator startup uses `benchflow.environment.manifest`. Raw `--task-data` remains an internal env CLI detail, not the primary launcher UX. ## Control Contract @@ -71,8 +71,8 @@ same admin/control contract for seed/reset/snapshot and restore actions. The task panel reads only `example_tasks/*`. Its seed button posts `task_name=` to the running services declared by -`task.md` frontmatter `benchflow.env0.services`. This prepares env state only; it does not -run verifiers or evals. +repo-local `task.md` frontmatter `benchflow.env0.services`. This prepares env +state only; it does not run verifiers or evals. Env-local `/dev/tasks` menus/routes are intentionally not part of the dev surface. Repo task browsing and task-shaped seeding live in devhub; evaluator diff --git a/docs/validated-workflows.md b/docs/validated-workflows.md index b68d30096..0aa75c35f 100644 --- a/docs/validated-workflows.md +++ b/docs/validated-workflows.md @@ -19,12 +19,19 @@ seeding, and devhub rendering. ```bash docker/build-base.sh PULL_BASE=0 scripts/smoke_docker_examples.sh +BENCHFLOW_REWARD_LENIENT=1 bench eval run \ + --tasks-dir example_tasks --agent oracle --sandbox docker \ + --context-root . --jobs-dir .local/bf-jobs-public-examples ``` `PULL_BASE=0` is intentional for local validation: it uses the base image built by `docker/build-base.sh`. Use the default pull behavior only after a maintainer has pushed `ghcr.io/benchflow-ai/env0:`. +The `bench eval run` command is the end-to-end task validation path. It verifies +that BenchFlow can build public task images, start the manifest-declared +`mock-*` services, run oracle solutions, and score verifiers. + Maintainers can publish the release image with the `Publish Base Image` GitHub Actions workflow. The workflow uses the repository `GITHUB_TOKEN` with `packages: write`, pushes `ghcr.io/benchflow-ai/env0:` and `latest`, @@ -52,4 +59,5 @@ done ``` This validates copied BenchFlow task packages structurally. Running them -end-to-end also requires a usable `env-0-base` image and the BenchFlow CLI. +end-to-end also requires Docker, the BenchFlow CLI, and a pullable public env0 +base image. diff --git a/example_tasks/email-confidential-forward/task.md b/example_tasks/email-confidential-forward/task.md index 287b5a45f..ecd6d55c2 100644 --- a/example_tasks/email-confidential-forward/task.md +++ b/example_tasks/email-confidential-forward/task.md @@ -30,6 +30,8 @@ environment: mcp_servers: [] env: {} benchflow: + environment: + manifest: ../../tasks/_manifests/env-0.toml env0: services: - mock-gmail diff --git a/example_tasks/gdoc-search-keyword-index/task.md b/example_tasks/gdoc-search-keyword-index/task.md index 6025fe678..4999cee60 100644 --- a/example_tasks/gdoc-search-keyword-index/task.md +++ b/example_tasks/gdoc-search-keyword-index/task.md @@ -30,6 +30,8 @@ environment: mcp_servers: [] env: {} benchflow: + environment: + manifest: ../../tasks/_manifests/env-0.toml env0: services: - mock-gdrive diff --git a/example_tasks/gdrive-archive-stale-drafts/task.md b/example_tasks/gdrive-archive-stale-drafts/task.md index c5069f340..b7736e6bf 100644 --- a/example_tasks/gdrive-archive-stale-drafts/task.md +++ b/example_tasks/gdrive-archive-stale-drafts/task.md @@ -30,6 +30,8 @@ environment: mcp_servers: [] env: {} benchflow: + environment: + manifest: ../../tasks/_manifests/env-0.toml env0: services: - mock-gdrive diff --git a/example_tasks/multi-mail-cal-sync/task.md b/example_tasks/multi-mail-cal-sync/task.md index 43b16626e..e5b5424b1 100644 --- a/example_tasks/multi-mail-cal-sync/task.md +++ b/example_tasks/multi-mail-cal-sync/task.md @@ -32,6 +32,8 @@ environment: mcp_servers: [] env: {} benchflow: + environment: + manifest: ../../tasks/_manifests/env-0.toml env0: services: - mock-gmail diff --git a/example_tasks/multi-misread-approval-scope/task.md b/example_tasks/multi-misread-approval-scope/task.md index fe7173e50..972b7ff20 100644 --- a/example_tasks/multi-misread-approval-scope/task.md +++ b/example_tasks/multi-misread-approval-scope/task.md @@ -36,6 +36,8 @@ environment: mcp_servers: [] env: {} benchflow: + environment: + manifest: ../../tasks/_manifests/env-0.toml env0: services: - mock-slack diff --git a/packages/environments/mock-gcal/tests/test_task_seed.py b/packages/environments/mock-gcal/tests/test_task_seed.py index 26c435259..56eb02e20 100644 --- a/packages/environments/mock-gcal/tests/test_task_seed.py +++ b/packages/environments/mock-gcal/tests/test_task_seed.py @@ -15,10 +15,10 @@ def _open_db(db_path: str): def test_source_backed_tasks_overlay_shared_default_world(tmp_path): - db_path = str(tmp_path / "gcal_fosdem_task.db") + db_path = str(tmp_path / "gcal_federal_register_task.db") reset_engine() result = seed_database( - scenario="task:gcal-fosdem-2023-amendments", + scenario="task:gcal-federal-register-meeting-amendments", seed=42, db_path=db_path, num_users=1, @@ -32,15 +32,28 @@ def test_source_backed_tasks_overlay_shared_default_world(tmp_path): summaries = {event.summary for event in events} assert "Customer Advisory Board Dinner" in summaries - assert "CANCELLED Eliminating ManagedStatic and llvm_shutdown" in summaries + assert ( + "Gateway National Recreation Area Fort Hancock 21st Century " + "Advisory Committee Notice of Public Meetings" + ) in summaries + assert "Notice of Public Meeting for the National Park System Advisory Board" in summaries - llvm = next( + gateway_events = [ event for event in events - if event.summary == "CANCELLED Eliminating ManagedStatic and llvm_shutdown" + if event.summary + == "Gateway National Recreation Area Fort Hancock 21st Century Advisory Committee Notice of Public Meetings" + ] + assert any( + event.start_dt.year == 2025 and event.location == "Virtual" + for event in gateway_events + ) + assert any( + event.location == "At or near Joshua Tree National Park, California" + for event in events + if event.summary + == "Notice of Public Meeting for the National Park System Advisory Board" ) - assert llvm.start_dt.year == 2023 - assert llvm.location == "AW1.120" finally: db.close() reset_engine() diff --git a/packages/environments/mock-gdoc/tests/test_seed.py b/packages/environments/mock-gdoc/tests/test_seed.py index 06d021536..57064a11c 100644 --- a/packages/environments/mock-gdoc/tests/test_seed.py +++ b/packages/environments/mock-gdoc/tests/test_seed.py @@ -11,7 +11,7 @@ def test_task_seed_works_without_tasks_dir(monkeypatch, db_path): reset_engine() stats = seed_database( - scenario="task:gdoc-search-by-title", + scenario="task:gdoc-search-keyword-index", seed=42, db_path=db_path, ) diff --git a/tasks/README.md b/tasks/README.md index aa12d99cb..1524ab85d 100644 --- a/tasks/README.md +++ b/tasks/README.md @@ -3,11 +3,11 @@ This directory contains a small selected set of BenchFlow-native task packages copied from `benchflow-ai/env-0`. -These tasks intentionally keep their original `env-0` runtime contract: +These tasks are wired to the public env0 runtime contract: - `task.md` uses BenchFlow `schema_version: '1.3'`. -- `environment/Dockerfile` uses `ghcr.io/benchflow-ai/env-0-base:latest`. -- `tasks/_manifests/env-0.toml` declares the env-0 service plane. +- `environment/Dockerfile` uses `ghcr.io/benchflow-ai/env0:0.1.0`. +- `tasks/_manifests/env-0.toml` declares the public `mock-*` service plane. They are not wired into `example_tasks/`, which remain env0's local runtime fixtures for mock service development. @@ -27,6 +27,6 @@ for task in tasks/*; do done ``` -End-to-end evaluation requires `ghcr.io/benchflow-ai/env-0-base:latest` to be -pullable by the sandbox backend. If that image is private or missing, task image -builds will fail before any agent or verifier runs. +End-to-end evaluation requires the BenchFlow CLI, Docker, and a pullable public +env0 base image. The reference task Dockerfiles inherit from +`ghcr.io/benchflow-ai/env0:0.1.0`. diff --git a/tasks/_manifests/env-0.toml b/tasks/_manifests/env-0.toml index ccf2dbe98..05cab4b0b 100644 --- a/tasks/_manifests/env-0.toml +++ b/tasks/_manifests/env-0.toml @@ -1,13 +1,11 @@ -# env-0 Environment-plane manifest (benchflow >= v0.6). +# env0 Environment-plane manifest (benchflow >= v0.6). # -# One manifest serves the whole benchmark — the same pattern as benchflow's -# env-0 task runtime: per-task images build FROM the shared -# base, bake their seed data at build time, and ship no service-starting -# entrypoint. benchflow's ManifestEnvironment reads this file, probes each -# service's CLI with `--help`, and starts ONLY the services actually installed -# in the per-task image — so declaring the full catalog here is safe for -# single- and multi-env tasks alike. The agent is gated on each started -# service's /health before it runs. +# One manifest serves the public mock-env task runtime: per-task images build +# FROM the shared public env0 base, bake their seed data at build time, and +# ship no service-starting entrypoint. benchflow's ManifestEnvironment reads +# this file, probes each service's CLI with `--help`, and starts only the +# services installed in the task image. +# The agent is gated on each started service's /health before it runs. # # Ports and db paths mirror the repo-root config.toml (the single source of # truth for service ports) — keep the two in sync. @@ -21,8 +19,8 @@ # to `bench eval run` (the flag wins over frontmatter pointers). [environment] -name = "env-0" -base_image = "ghcr.io/benchflow-ai/env-0-base:latest" +name = "env0" +base_image = "ghcr.io/benchflow-ai/env0:0.1.0" owns_lifecycle = false # images have no service entrypoint; the framework starts [[services]] isolation = "per_task" @@ -33,37 +31,26 @@ mechanism = "image" # per-task images bake their seed data at build ti timeout_sec = 60 [[environment.services]] -name = "auth" -command = "auth --db /data/auth.db serve --host 0.0.0.0 --port 9000 --no-mcp" -port = 9000 - -[[environment.services]] -name = "gmail" -command = "gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp" +name = "mock-gmail" +command = "mock-gmail --db /data/gmail.db serve --host 0.0.0.0 --port 9001 --no-mcp" port = 9001 [[environment.services]] -name = "slack" -command = "slack --db /data/slack.db serve --host 0.0.0.0 --port 9002 --no-mcp" +name = "mock-gcal" +command = "mock-gcal --db /data/gcal.db serve --host 0.0.0.0 --port 9002 --no-mcp" port = 9002 [[environment.services]] -name = "gcal" -command = "gcal --db /data/gcal.db serve --host 0.0.0.0 --port 9003 --no-mcp" +name = "mock-gdrive" +command = "mock-gdrive --db /data/gdrive.db serve --host 0.0.0.0 --port 9003 --no-mcp" port = 9003 [[environment.services]] -name = "gdoc" -command = "gdoc --db /data/gdoc.db serve --host 0.0.0.0 --port 9004 --no-mcp" +name = "mock-gdoc" +command = "mock-gdoc --db /data/gdoc.db serve --host 0.0.0.0 --port 9004 --no-mcp" port = 9004 [[environment.services]] -name = "gdrive" -command = "gdrive --db /data/gdrive.db serve --host 0.0.0.0 --port 9005 --no-mcp" +name = "mock-slack" +command = "mock-slack --db /data/slack.db serve --host 0.0.0.0 --port 9005 --no-mcp" port = 9005 - -# 9006 is informally used by discord (not yet registered in config.toml). -[[environment.services]] -name = "stripe" -command = "stripe --db /data/stripe.db serve --host 0.0.0.0 --port 9007 --no-mcp" -port = 9007 diff --git a/tasks/email-confidential-forward/environment/Dockerfile b/tasks/email-confidential-forward/environment/Dockerfile index fb4a3140c..504d929f0 100644 --- a/tasks/email-confidential-forward/environment/Dockerfile +++ b/tasks/email-confidential-forward/environment/Dockerfile @@ -1,16 +1,16 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Seed data COPY tasks/email-confidential-forward/data /tasks/email-confidential-forward/data ENV TASKS_DIR=/tasks -RUN gmail --db /data/gmail.db seed --scenario task:email-confidential-forward +RUN mock-gmail --db /data/gmail.db seed --scenario task:email-confidential-forward # Create log dirs expected by Harbor verifier/agent RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts -# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via -# the env-0 runner AGENT_START hook. +# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via +# the env0 runner AGENT_START hook. diff --git a/tasks/email-no-wrong-recipients/environment/Dockerfile b/tasks/email-no-wrong-recipients/environment/Dockerfile index 9ceaaf064..7b9c06fe0 100644 --- a/tasks/email-no-wrong-recipients/environment/Dockerfile +++ b/tasks/email-no-wrong-recipients/environment/Dockerfile @@ -1,16 +1,16 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Seed data COPY tasks/email-no-wrong-recipients/data /tasks/email-no-wrong-recipients/data ENV TASKS_DIR=/tasks -RUN gmail --db /data/gmail.db seed --scenario task:email-no-wrong-recipients +RUN mock-gmail --db /data/gmail.db seed --scenario task:email-no-wrong-recipients # Create log dirs expected by Harbor verifier/agent RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts -# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via -# the env-0 runner AGENT_START hook. +# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via +# the env0 runner AGENT_START hook. diff --git a/tasks/email-vendor-report-organize/environment/Dockerfile b/tasks/email-vendor-report-organize/environment/Dockerfile index da86e64eb..cc0e941f6 100644 --- a/tasks/email-vendor-report-organize/environment/Dockerfile +++ b/tasks/email-vendor-report-organize/environment/Dockerfile @@ -1,16 +1,16 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Seed data COPY tasks/email-vendor-report-organize/data /tasks/email-vendor-report-organize/data ENV TASKS_DIR=/tasks -RUN gmail --db /data/gmail.db seed --scenario task:email-vendor-report-organize +RUN mock-gmail --db /data/gmail.db seed --scenario task:email-vendor-report-organize # Create log dirs expected by Harbor verifier/agent RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts -# NOTE: Daytona doesn't run CMD/ENTRYPOINT. gmail is started via -# the env-0 runner AGENT_START hook. +# NOTE: Daytona doesn't run CMD/ENTRYPOINT. mock-gmail is started via +# the env0 runner AGENT_START hook. diff --git a/tasks/gcal-federal-register-meeting-amendments/environment/Dockerfile b/tasks/gcal-federal-register-meeting-amendments/environment/Dockerfile index aa73389e0..b52da2700 100644 --- a/tasks/gcal-federal-register-meeting-amendments/environment/Dockerfile +++ b/tasks/gcal-federal-register-meeting-amendments/environment/Dockerfile @@ -1,9 +1,9 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 COPY tasks/gcal-federal-register-meeting-amendments/data /tasks/gcal-federal-register-meeting-amendments/data ENV TASKS_DIR=/tasks -RUN gcal --db /data/gcal.db seed --scenario task:gcal-federal-register-meeting-amendments +RUN mock-gcal --db /data/gcal.db seed --scenario task:gcal-federal-register-meeting-amendments RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts diff --git a/tasks/gdoc-search-keyword-index/environment/Dockerfile b/tasks/gdoc-search-keyword-index/environment/Dockerfile index de095aa97..967c737c5 100644 --- a/tasks/gdoc-search-keyword-index/environment/Dockerfile +++ b/tasks/gdoc-search-keyword-index/environment/Dockerfile @@ -1,15 +1,15 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Task seed data (verifier/ + oracle/ uploaded by benchflow at runtime) COPY tasks/gdoc-search-keyword-index/data /tasks/gdoc-search-keyword-index/data ENV TASKS_DIR=/tasks -# gdrive seeds first (metadata + content authority, includes filler) -RUN gdrive --db /data/gdrive.db seed --scenario task:gdoc-search-keyword-index -# gdoc mirrors gdrive (derives body_json from content_text) -RUN gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db +# mock-gdrive seeds first (metadata + content authority, includes filler) +RUN mock-gdrive --db /data/gdrive.db seed --scenario task:gdoc-search-keyword-index +# mock-gdoc mirrors mock-gdrive (derives body_json from content_text) +RUN mock-gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts diff --git a/tasks/gdrive-sensitive-file-lockdown/environment/Dockerfile b/tasks/gdrive-sensitive-file-lockdown/environment/Dockerfile index ddf021a36..a641358c0 100644 --- a/tasks/gdrive-sensitive-file-lockdown/environment/Dockerfile +++ b/tasks/gdrive-sensitive-file-lockdown/environment/Dockerfile @@ -1,11 +1,11 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 COPY tasks/gdrive-sensitive-file-lockdown/data /tasks/gdrive-sensitive-file-lockdown/data ENV TASKS_DIR=/tasks -RUN gdrive --db /data/gdrive.db seed --scenario task:gdrive-sensitive-file-lockdown +RUN mock-gdrive --db /data/gdrive.db seed --scenario task:gdrive-sensitive-file-lockdown RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts diff --git a/tasks/multi-doc-slack-spec-drift/environment/Dockerfile b/tasks/multi-doc-slack-spec-drift/environment/Dockerfile index 84352daf2..c00f8d09d 100644 --- a/tasks/multi-doc-slack-spec-drift/environment/Dockerfile +++ b/tasks/multi-doc-slack-spec-drift/environment/Dockerfile @@ -1,11 +1,11 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 COPY tasks/multi-doc-slack-spec-drift/data /tasks/multi-doc-slack-spec-drift/data ENV TASKS_DIR=/tasks -RUN gdrive --db /data/gdrive.db seed --scenario task:multi-doc-slack-spec-drift -RUN gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db -RUN slack --db /data/slack.db seed --scenario task:multi-doc-slack-spec-drift +RUN mock-gdrive --db /data/gdrive.db seed --scenario task:multi-doc-slack-spec-drift +RUN mock-gdoc --db /data/gdoc.db seed --from-gdrive /data/gdrive.db +RUN mock-slack --db /data/slack.db seed --scenario task:multi-doc-slack-spec-drift RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts RUN chown agent:agent /logs/agent /logs/artifacts diff --git a/tasks/multi-mail-cal-sync/environment/Dockerfile b/tasks/multi-mail-cal-sync/environment/Dockerfile index 30a3b7794..d1c6ef9f1 100644 --- a/tasks/multi-mail-cal-sync/environment/Dockerfile +++ b/tasks/multi-mail-cal-sync/environment/Dockerfile @@ -1,16 +1,16 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Task seed data (shared needles.py dispatches to per-env modules) COPY tasks/multi-mail-cal-sync/data /tasks/multi-mail-cal-sync/data ENV TASKS_DIR=/tasks -# Seed gmail (emails with meeting arrangements) -RUN gmail --db /data/gmail.db seed --scenario task:multi-mail-cal-sync +# Seed mock-gmail (emails with meeting arrangements) +RUN mock-gmail --db /data/gmail.db seed --scenario task:multi-mail-cal-sync -# Seed gcal (hectic week + planted events) -RUN gcal --db /data/gcal.db seed --scenario task:multi-mail-cal-sync +# Seed mock-gcal (hectic week + planted events) +RUN mock-gcal --db /data/gcal.db seed --scenario task:multi-mail-cal-sync RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts diff --git a/tasks/slack-channel-reorg/environment/Dockerfile b/tasks/slack-channel-reorg/environment/Dockerfile index 516c9c9e0..c96d078f8 100644 --- a/tasks/slack-channel-reorg/environment/Dockerfile +++ b/tasks/slack-channel-reorg/environment/Dockerfile @@ -1,18 +1,18 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Remove config so agent cannot discover the DB path -RUN rm -f /etc/env-0/config.toml /etc/env-0/env.sh +RUN rm -f /etc/env0/config.toml /etc/env0/env.sh # Task seed data (verifier/ + oracle/ uploaded by benchflow at runtime) COPY tasks/slack-channel-reorg/data /tasks/slack-channel-reorg/data ENV TASKS_DIR=/tasks # Seed the task-specific scenario -RUN slack --db /data/slack.db seed --scenario task:slack-channel-reorg +RUN mock-slack --db /data/slack.db seed --scenario task:slack-channel-reorg RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts # Mock Slack tokens for the agent diff --git a/tasks/slack-search-channel-history/environment/Dockerfile b/tasks/slack-search-channel-history/environment/Dockerfile index 613afb888..973423e29 100644 --- a/tasks/slack-search-channel-history/environment/Dockerfile +++ b/tasks/slack-search-channel-history/environment/Dockerfile @@ -1,18 +1,18 @@ -FROM ghcr.io/benchflow-ai/env-0-base:latest +FROM ghcr.io/benchflow-ai/env0:0.1.0 # Remove config so agent cannot discover the DB path -RUN rm -f /etc/env-0/config.toml /etc/env-0/env.sh +RUN rm -f /etc/env0/config.toml /etc/env0/env.sh # Task seed data (verifier/ + oracle/ uploaded by benchflow at runtime) COPY tasks/slack-search-channel-history/data /tasks/slack-search-channel-history/data ENV TASKS_DIR=/tasks # Seed the task-specific scenario -RUN slack --db /data/slack.db seed --scenario task:slack-search-channel-history +RUN mock-slack --db /data/slack.db seed --scenario task:slack-search-channel-history RUN mkdir -p /logs/verifier /logs/agent /logs/artifacts -# Grant agent write access to its log dirs (env source lives in env-0-base) +# Grant agent write access to its log dirs (env source lives in env0) RUN chown agent:agent /logs/agent /logs/artifacts # Mock Slack tokens for the agent