Make it impossible for a test to spend inference tokens - #79
Merged
Conversation
Adopts the project's dividing line as a mechanism rather than a
convention: anything that does not require inference is a TEST
(automated, fast, every push); anything that costs tokens is an EVAL
(deliberate, periodic, never in CI).
The suite already complied — the live e2e suites use fake keys and point
ROTE_BASE_URL_<NODE_ID> at a local Messages stub. It complied by
convention, which is not the same as being unable to do otherwise. Two
DBOS e2e suites previously burned real subscription inference on every
run, and that was caught by wall-clock timing rather than by any
assertion, because the failure mode of this bug is silent spending.
Two independent leaks, both closed in conftest:
1. **Credentials in the environment.** A developer running the suite
with real keys exported would have any SDK constructed without an
explicit key bill them. All vendor credential vars are now scrubbed
per-test.
2. **A subscription CLI on PATH.** This is the one with nothing to
scrub: `claude -p` authenticates from an OAuth session, so having no
API key is not protection — not finding the binary is. Bare-name
`shutil.which("claude" | "codex")` now returns None.
Verified adversarially rather than by inspection. With the guard, a
probe test cannot find the CLI and sees no credentials; without it, both
succeed — there IS a claude binary on PATH on this machine, so this
closes a live hole, not a theoretical one.
Deliberately narrow in two directions, because an over-broad guard gets
disabled:
- Only BARE-NAME lookups are blocked. Several driver tests write a fake
`claude` script under tmp_path and pass its absolute path as
`executable=`; that is a stub, not a subscription, and it still
resolves. The first attempt blocked those too and broke 9 tests.
- Every other `which` delegates, so the e2e suites' node/npm/docker
probes are untouched — blocking those would trade a spending bug for a
silent-skip coverage bug, which is the trade this project keeps having
to undo.
`tests/test_no_inference_in_tests.py` verifies the guard itself,
including that an opt-in test can still patch the lookup back: the guard
is a default, not a cage.
CLAUDE.md gains a "Tests vs. evals" section stating the rule, what
enforces it, and the two consequences that are easy to get wrong — an
e2e test is still a test (real npm, wrangler, Postgres, no inference →
automated), and `@pytest.mark.slow` says what toolchain a suite needs,
never whether it should run.
1175 fast + 27 slow pass; ruff, strict mypy, sanity-check clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SBuSDL7Svm7iDzgmhTpDSW
trevhud
force-pushed
the
chore/test-eval-taxonomy
branch
from
July 30, 2026 23:50
de8eacd to
92e7cd6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts the dividing line as a mechanism rather than a convention:
The suite already complied — by convention
The live e2e suites use fake keys and point
ROTE_BASE_URL_<NODE_ID>at a local Messages stub. But complying and being unable to do otherwise are different things. Two DBOS e2e suites previously burned real subscription inference on every run, and that was caught by wall-clock timing, not by any assertion — because the failure mode of this bug is silent spending.Two leaks, both closed in conftest
claude -pauthenticates from an OAuth session, so having no API key is not protection — not finding the binary is. Bare-nameshutil.which("claude" | "codex")now returnsNone.Verified adversarially
claudeThere is a
claudebinary on PATH on this machine, so this closes a live hole rather than a theoretical one.Deliberately narrow
An over-broad guard gets disabled, so it isn't one:
claudestub and pass its absolute path asexecutable=— that still resolves. My first attempt blocked those too and broke 9 tests.whichdelegates, so the e2e suites' node/npm/docker probes are untouched. Blocking those would trade a spending bug for a silent-skip coverage bug — the exact trade this repo keeps having to undo.tests/test_no_inference_in_tests.pyverifies the guard itself, including that an opt-in test can patch the lookup back. The guard is a default, not a cage.Docs
CLAUDE.md gains a Tests vs. evals section with the rule, what enforces it, and the two consequences that are easy to get wrong:
npm,wrangler, Postgres, no inference → automated.@pytest.mark.slowsays what toolchain a suite needs, never whether it should run.1175 fast + 27 slow pass; ruff, strict mypy, sanity-check clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SBuSDL7Svm7iDzgmhTpDSW