Skip to content

Fix main: invert the reporter-session assertions, and widen the test trigger - #433

Merged
TennyZhuang merged 3 commits into
mainfrom
fix/production-config-test
Aug 9, 2026
Merged

Fix main: invert the reporter-session assertions, and widen the test trigger#433
TennyZhuang merged 3 commits into
mainfrom
fix/production-config-test

Conversation

@TennyZhuang

@TennyZhuang TennyZhuang commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

main is red on 076cbf3. This fixes it.

What broke

production_config.test.ts asserted that the deploy workflow wires the
reporter-session keyring through the secret store. The cleanup merged in 076cbf3
removed exactly that wiring, so the test was describing a state we deliberately
left behind.

What this does

Inverts the assertions rather than deleting them — they are the record of what
was removed, and a deleted test cannot notice the wiring coming back. Since a suite
of doesNotMatch passes on an empty string, on a renamed file, and on a read that
returned nothing, it keeps a positive control anchoring it to a workflow that was
actually read.

Splits the environment binding into its own test. It outlived the case it lived
in: the environment is named for the retired feature but is not part of it. It is now
matched at job-key indentation rather than requiring jobs:deploy:
environment: to be adjacent — that adjacency is what broke, because a comment was
added above the line explaining why it must never be deleted. A test that goes red
when someone documents the thing it protects gets "fixed" by deleting the
documentation.

Both new cases were mutation-checked:

mutation result
remove the environment: binding binding test reds
re-add a keyring secret reference absence test reds
restore both green

Why it wasn't caught before the merge

The pull_request paths list enumerated the files this job reads in order to run.
That is not the same set as the files its tests make assertions about.
production_config.test.ts opens deploy-hands-server.yml; that path matched nothing
in the list; a PR touching only it ran no tests at all. Six green checks, none of them
this suite.

The push-to-main trigger has no paths filter — an earlier deliberate decision — and
that is what caught it, one merge too late. It is a net, not a gate.

The fix is a wildcard, not a longer list

The first version of this PR named deploy-hands-server.yml. That is still a derived
list — "which files do these tests assert on" — and a derivation that misses one is
silent in the direction of green, which is the same failure this entry exists to fix,
one level up. .github/workflows/** has nothing to miss.

Cost measured rather than assumed: the last eight runs of this job took 28–42s,
median 35
. Thirty-five seconds on an unrelated workflow edit is affordable waste; a
guard that silently does not run is an unobservable failure. paths is a performance
optimisation and was being used as a correctness boundary.

(.github/workflows/worker-tests.yml is now subsumed by the wildcard and left in place
as documentation of why this job watches its own definition.)

Known residual, stated rather than papered over: this still does not cover a test
asserting on a file outside worker/** and .github/workflows/**. That is better
settled by a one-off repo-wide census than by a standing mechanism that misses silently.

"No test ran" and "every test passed" are the same green on a pull request page.

Verification

  • full worker suite: 403 passed, 35 files
  • workflow lint: exit 0
  • mutation controls above

Rhea Rafferty and others added 2 commits August 9, 2026 10:34
…file they read

main went red on 076cbf3. `production_config.test.ts` asserted that the deploy
workflow wires the reporter-session keyring through the secret store, and the
cleanup that merged in 076cbf3 removed exactly that wiring. The test was
describing a state we deliberately left behind.

Inverted rather than deleted. The assertions are the record of what was removed,
and a deleted test cannot notice the wiring coming back:

  HANDS_FEEDBACK_REPORTER_SESSION_KEYS         must not appear
  wrangler secret put FEEDBACK_REPORTER_...    must not appear
  vars.HANDS_FEEDBACK_REPORTER_SESSION_        must not appear
  Validate reporter-session rollout inputs     must not appear

A suite of `doesNotMatch` passes on an empty string, on a renamed file, and on a
read that quietly returned nothing, so it keeps a positive control anchoring it to
a workflow that was actually read.

The environment binding moves to its own test, because it outlived the case it
was living in: the environment is named for the retired feature but is not part
of it, and its branch policy is what confines this deploy to main. It is now
matched at job-key indentation instead of requiring `jobs:\n  deploy:\n
environment:` to be adjacent. That adjacency is what broke here - a comment was
added above the line explaining why it must never be deleted, and the test failed
because of the explanation. A test that goes red when someone documents the thing
it protects gets "fixed" by deleting the documentation.

Both new cases were mutation-checked: removing the environment binding reds the
binding test, re-adding a keyring reference reds the absence test, and restoring
the file greens both.

Why nobody saw it before the merge: the pull_request paths list here enumerated
the files this job *reads in order to run*, which is not the same set as the files
its tests *make assertions about*. production_config.test.ts opens
deploy-hands-server.yml, that path matched nothing in the list, and a PR touching
only it therefore ran no tests at all - six green checks, none of them this suite.
The push-to-main trigger has no paths filter, by an earlier deliberate decision,
and that is what caught it, one merge too late. The deploy workflow is now in the
list with the distinction written down.

"No test ran" and "every test passed" are the same green on a pull request page.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deriving 'which files do these tests assert on' is itself a list that can miss
one, and it misses in the direction of green - the same failure this entry
exists to fix, one level up. A wildcard has nothing to miss.

Cost measured rather than assumed: the last eight runs of this job took 28-42s,
median 35. Thirty-five seconds on an unrelated workflow edit is affordable
waste; a guard that silently does not run is an unobservable failure. `paths`
is a performance optimisation and was being used as a correctness boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TennyZhuang TennyZhuang changed the title Fix main: invert the reporter-session assertions, and trigger on the file they read Fix main: invert the reporter-session assertions, and widen the test trigger Aug 9, 2026
…ector

cors_allow_headers.test.ts scans container/src and container/package.json
because the container has no cors() today and its own suite does not run here.
Without container/** a PR touching only container/ does not run it - and a PR
introducing the first cors() call is, almost by definition, a PR touching only
container/. The instrument would have been absent from exactly the change it
exists to catch, which is the trigger condition task #97 currently rests on.

Its own `registrations.length > 0` guard is correctly written and would not
have helped. A denominator only means anything in a run that happens.

Second instance in one census, which is why these entries are wildcards rather
than a list derived again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TennyZhuang
TennyZhuang merged commit 6ae3b74 into main Aug 9, 2026
7 checks passed
@TennyZhuang
TennyZhuang deleted the fix/production-config-test branch August 9, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant