fix: harden spec validation and field-path evaluation#2
Merged
Conversation
Addresses several silent-failure and trust-break issues in the core
assertion engine, so invalid tests error out instead of passing.
- Strict YAML decoding with KnownFields(true) + validateSpec: typos
like `toEquals:` now fail parse instead of silently passing.
- Require at least one operator per assertion (toExist/toBeNull/value
ops); empty describe blocks and missing should/expect also rejected.
- ParseManifests uses yaml.Decoder instead of strings.Split("---"),
so literal blocks containing "---" stay in one document.
- extractFieldValues probes the parent with has() to distinguish
"present but null" from "missing"; fixes both toExist and toBeNull
semantics.
- `[*]` wildcard paths now iterate over ALL produced values instead of
checking only the first (documented-but-broken behavior).
- Reject --fail-fast combined with --workers > 1 (was silently ignored
on the parallel path).
- Add --pre-run-timeout flag (default 60s) wrapping each pre_run in
context.WithTimeout, preventing indefinite hangs.
- Sort \`list --tags\` output deterministically.
Adds 16 new tests covering each of the above cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TestE2E_PreRunTimeout flaked on CI (Ubuntu) because exec.CommandContext only SIGKILLs the immediate `sh`, leaving `sleep` as an orphan holding inherited fds — cmd.Wait() then blocked past the deadline. - Start the shell in a new process group (Setpgid on unix) - cmd.Cancel kills the whole group with SIGKILL(-pgid) - cmd.WaitDelay = 2s as a belt-and-braces Wait return guarantee - Windows build tag uses default Process.Kill (no pgroup concept) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7403dcf to
93db647
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses silent-failure and trust-break issues in the core assertion
engine, so invalid tests error out instead of passing. All existing
fixtures still pass; 16 new tests lock in the new behavior.
Correctness fixes
KnownFields(true)plus avalidateSpecstep means typos liketoEquals:now fail parseinstead of silently passing. Also rejects empty describes, missing
should/expect, and assertions with zero operators.ParseManifestsusesyaml.Decoderinstead of
strings.Split("---"), so a literal block that happens tocontain
---no longer creates phantom documents.extractFieldValuesprobes theparent object with
has()to tell "present but null" from "missing".Fixes both
toExist(now passes for an explicitly-null field) andtoBeNull(now fails for a missing field, as it should).[*]wildcard checks all results — previously only the firstvalue from a wildcard path was evaluated; now the operator runs
against every produced value, matching the documented semantics.
Ergonomics
--fail-fastwith--workers > 1— the parallel pathsilently ignored fail-fast before; now it errors clearly.
--pre-run-timeoutflag (default 60s) — wraps eachpre_runcommand in
context.WithTimeout, preventing indefinite hangs.list --tagsoutput — deterministic alphabetical order.Incidental
go 1.25.0→go 1.25.8ingo.modto pick up the stdlibosfix (GO-2026-4602).Test plan
make test— 118 tests pass (82 unit, 36 integration)go vet ./...cleangofmt -l .cleangovulncheck ./...— no vulns in our codeNew tests added:
empty describe, toExist/toBeNull as sole operators, null-vs-missing,
wildcard-checks-all, literal block with
---🤖 Generated with Claude Code