Wire public mock env task runtime#18
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bec6860153
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| FROM ghcr.io/benchflow-ai/env0:0.1.0 | ||
|
|
||
| # Seed data | ||
| COPY tasks/email-confidential-forward/data /tasks/email-confidential-forward/data | ||
| ENV TASKS_DIR=/tasks |
There was a problem hiding this comment.
Hide the baked task payload from agents
When these tasks/* Dockerfiles are evaluated on the public env0 image, the seed payload is still copied under /tasks and exposed through TASKS_DIR; Docker COPY leaves those files world-readable, so the normal agent user can inspect data/needles.py and recover target IDs/answers before acting. The same pattern appears across the updated imported task Dockerfiles; move the payload under /var/lib/task with root-only permissions, or otherwise remove agent read access after seeding.
Useful? React with 👍 / 👎.
Summary
bench eval runcan start mock servicesValidation
BENCHFLOW_REWARD_LENIENT=1 bench eval run --tasks-dir example_tasks --include gdrive-archive-stale-drafts --agent oracle --sandbox docker --jobs-dir .local/bf-jobs-public-example-smoke --context-root . --concurrency 1 --build-concurrency 1BENCHFLOW_REWARD_LENIENT=1 bench eval run --tasks-dir example_tasks --agent oracle --sandbox docker --jobs-dir .local/bf-jobs-public-examples-oracle --context-root . --concurrency 1 --build-concurrency 1tasks/*/environment/Dockerfileimages againstghcr.io/benchflow-ai/env0:0.1.0scripts/smoke_dev.shPULL_BASE=0 scripts/smoke_docker_examples.shbench tasks checkover allexample_tasks/*andtasks/*cd packages/environments/mock-gdoc && uv run --extra dev pytest tests -qcd packages/environments/mock-gcal && uv run --extra dev pytest tests -qgit diff --check