Skip to content

fix: harden spec validation and field-path evaluation#2

Merged
machado144 merged 2 commits into
mainfrom
fix/correctness-hardening
Apr 24, 2026
Merged

fix: harden spec validation and field-path evaluation#2
machado144 merged 2 commits into
mainfrom
fix/correctness-hardening

Conversation

@machado144

Copy link
Copy Markdown
Contributor

Summary

Addresses silent-failure and trust-break issues in the core assertion
engine, so invalid tests error out instead of passing. All existing
fixtures still pass; 16 new tests lock in the new behavior.

Correctness fixes

  • Strict YAML decoding + spec validationKnownFields(true) plus a
    validateSpec step means typos like toEquals: now fail parse
    instead of silently passing. Also rejects empty describes, missing
    should/expect, and assertions with zero operators.
  • Real multi-doc YAML parsingParseManifests uses yaml.Decoder
    instead of strings.Split("---"), so a literal block that happens to
    contain --- no longer creates phantom documents.
  • Missing vs null distinctionextractFieldValues probes the
    parent object with has() to tell "present but null" from "missing".
    Fixes both toExist (now passes for an explicitly-null field) and
    toBeNull (now fails for a missing field, as it should).
  • [*] wildcard checks all results — previously only the first
    value from a wildcard path was evaluated; now the operator runs
    against every produced value, matching the documented semantics.

Ergonomics

  • Reject --fail-fast with --workers > 1 — the parallel path
    silently ignored fail-fast before; now it errors clearly.
  • --pre-run-timeout flag (default 60s) — wraps each pre_run
    command in context.WithTimeout, preventing indefinite hangs.
  • Sorted list --tags output — deterministic alphabetical order.

Incidental

  • Bumped go 1.25.0go 1.25.8 in go.mod to pick up the stdlib
    os fix (GO-2026-4602).

Test plan

  • make test — 118 tests pass (82 unit, 36 integration)
  • go vet ./... clean
  • gofmt -l . clean
  • govulncheck ./... — no vulns in our code
  • Pre-commit hooks green (gofmt, structlint, govet, golangci-lint)

New tests added:

  • Unit: unknown operator, missing operator, missing should/expect,
    empty describe, toExist/toBeNull as sole operators, null-vs-missing,
    wildcard-checks-all, literal block with ---
  • Integration: fail-fast+workers rejection, pre-run-timeout, tags-sorted

🤖 Generated with Claude Code

machado144 and others added 2 commits April 24, 2026 10:28
Addresses several silent-failure and trust-break issues in the core
assertion engine, so invalid tests error out instead of passing.

- Strict YAML decoding with KnownFields(true) + validateSpec: typos
  like `toEquals:` now fail parse instead of silently passing.
- Require at least one operator per assertion (toExist/toBeNull/value
  ops); empty describe blocks and missing should/expect also rejected.
- ParseManifests uses yaml.Decoder instead of strings.Split("---"),
  so literal blocks containing "---" stay in one document.
- extractFieldValues probes the parent with has() to distinguish
  "present but null" from "missing"; fixes both toExist and toBeNull
  semantics.
- `[*]` wildcard paths now iterate over ALL produced values instead of
  checking only the first (documented-but-broken behavior).
- Reject --fail-fast combined with --workers > 1 (was silently ignored
  on the parallel path).
- Add --pre-run-timeout flag (default 60s) wrapping each pre_run in
  context.WithTimeout, preventing indefinite hangs.
- Sort \`list --tags\` output deterministically.

Adds 16 new tests covering each of the above cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TestE2E_PreRunTimeout flaked on CI (Ubuntu) because exec.CommandContext
only SIGKILLs the immediate `sh`, leaving `sleep` as an orphan holding
inherited fds — cmd.Wait() then blocked past the deadline.

- Start the shell in a new process group (Setpgid on unix)
- cmd.Cancel kills the whole group with SIGKILL(-pgid)
- cmd.WaitDelay = 2s as a belt-and-braces Wait return guarantee
- Windows build tag uses default Process.Kill (no pgroup concept)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@machado144 machado144 force-pushed the fix/correctness-hardening branch from 7403dcf to 93db647 Compare April 24, 2026 08:52
@machado144 machado144 merged commit 6354dae into main Apr 24, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant