Add public mock stripe runtime#20
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1bfccc1326
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| ENV STRIPE_URL=http://localhost:9007 | ||
|
|
||
| COPY tasks/stripe-refund-correct-customer/data /tasks/stripe-refund-correct-customer/data |
There was a problem hiding this comment.
In the public task image, this COPY leaves /tasks/stripe-refund-correct-customer/data/needles.py readable by the normal agent user because Docker's default copy mode is readable and this Dockerfile never locks /tasks afterward. The base image only locks /var/lib/task and /data, and the example-task Dockerfile adds chmod -R 700 /tasks; without that here, agents can read the exact target and decoy customers from the seed payload, corrupting the benchmark result.
Useful? React with 👍 / 👎.
| needles = _load_needles() | ||
|
|
||
| for entry in needles.ALL: | ||
| created = ANCHOR - entry["days_ago"] * _DAY |
There was a problem hiding this comment.
This computes the task charge timestamps from the fixed mock-stripe ANCHOR of 2026-05-28, so days_ago: 1 seeds the target charge on 2026-05-27 rather than the actual “yesterday” described in the prompt when the task is run later. Agents that use Stripe's created[...] filters to find yesterday's charge will miss the target; either make the seed relative to runtime or change the prompt to the absolute seeded date.
Useful? React with 👍 / 👎.
| and target_ch.get("amount_refunded", 0) == target_ch.get("amount") | ||
| and target_refunds | ||
| and all(r.get("status") == "succeeded" for r in target_refunds) |
There was a problem hiding this comment.
When an agent refunds the correct charge without reason=duplicate, the API still creates a succeeded refund and this predicate awards full credit because it only checks the amount/status. Since the prompt explicitly says to process it as a duplicate refund, runs that omit or use a different reason currently pass; include the refund reason in the success condition.
Useful? React with 👍 / 👎.
1 participant
Summary
mock-stripeenvironment package with CLI/runtime wiring and testsstripe-refund-correct-customerunder bothtasks/andexample_tasks/Validation
uv run --extra dev pytest tests -qinpackages/environments/mock-stripe(291 passed)uv run --extra dev pytest tests/test_conformance.py tests/test_auth_integration.py -qinpackages/environments/mock-stripe(62 passed)python3 -m unittest tests/test_env0_control.py(23 tests OK)all-public-task-packages-valid)scripts/smoke_dev.shpython3 scripts/env0_control.py seed-task stripe-refund-correct-customer mock-stripedocker/build-base.sh && PULL_BASE=0 scripts/smoke_docker_examples.shBENCHFLOW_REWARD_LENIENT=1 bench eval run --tasks-dir example_tasks --include stripe-refund-correct-customer --agent oracle --sandbox docker --jobs-dir .local/bf-jobs-public-stripe-example-oracle --context-root . --concurrency 1 --build-concurrency 1(1/1, reward1.0)BENCHFLOW_REWARD_LENIENT=1 bench eval run --tasks-dir example_tasks --agent oracle --sandbox docker --jobs-dir .local/bf-jobs-public-examples-oracle-stripe --context-root . --concurrency 1 --build-concurrency 1(7/7, score100.0%)git diff --check