Acceptance .prologos files are not part of the test suite — silent regression risk

[hierophantos]

Summary

.prologos acceptance files are not run as part of the test suite. As a result, regressions in user-visible language behavior can land silently — verified bugs include a recently-discovered ground->prologos-expr regression where solver bindings were rendered as unknown instead of their actual values (introduced by PUnify Phase 0a, commit 32d62c67, never caught by Racket-level tests).

This issue tracks closing the gap between Racket-implementation testing and .prologos-source-language testing.

What's currently covered vs not

Layer	Coverage	Mechanism
Racket implementation	High	tests/test-*.rkt — 7900+ tests across ~250 files via tools/run-affected-tests.rkt
.prologos syntax/elaboration	Medium	Some tests use process-string-ws or process-file indirectly
.prologos end-to-end behavior	LOW	Acceptance files exist (examples/*.prologos) but are NOT run automatically

Specific evidence of the gap

The bug that motivated this issue: solver returned unknown instead of "red" for (color "apple" c). This bug:

• Existed for an unknown duration since commit 32d62c67
• Was not caught by any of 7900+ tests
• WOULD have been caught by examples/2026-03-14-wfle-acceptance.prologos if that file were run as part of CI
• WAS caught manually by user inspection during exploration

Other dormant issues found in the same exploration:

• examples/2026-03-14-wfle-acceptance.prologos:298 references parent which is never defined in the file — pre-existing broken reference, not currently caught by anything

Why this happened

.prologos acceptance files are documents — they're read for design discussion, used as targets for "feature X is not done until this section runs," but not enforced as regression tests. The architecture works:

1. tests/test-*.rkt uses process-string / process-string-ws for inline test fixtures, but each test file constructs its own scenarios — none source from examples/*.prologos.
2. run-affected-tests.rkt discovers tests/test-*.rkt, not examples/*.prologos.
3. Acceptance files have informal expected-output comments (;; => [...]) but these are documentation, not assertions.

The result: a .prologos file can elaborate, run to completion (or to expected boundaries), and produce wrong output, without any test failing.

Proposed shape

Three layers of fix, ordered by impact:

Layer 1 (immediate): Auto-run acceptance files in CI

Add a test harness that:

• Discovers examples/*.prologos files (excluding files marked draft / WIP)
• Runs each via process-file
• Asserts the run completes without errors
• Reports counter changes (cell allocs, prop firings, etc.) for benchmarking

Doesn't validate output values, but catches:

• Crashes / aborts
• Type errors
• Undefined references (caught the parent bug above)
• Missing primitives

This is the lowest-cost win — picks up bugs that cause the file to abort or error.

Layer 2 (medium): Assertion-checking from ;; => comments

The acceptance files already have comments like ;; => [{:c "red"}]. Convert these into assertions: expected-output comments are checked against actual output during test runs.

Mechanism options:

• A simple comment-extraction parser that pulls ;; => annotations from .prologos files and matches them against process-file output
• A more structured .prologos annotation form (e.g., (check ...) keyword) that the parser recognizes and the elaborator/runtime asserts

The first is lighter-touch (works with existing files); the second is more principled (could integrate with property-based testing later).

Layer 3 (long-term): Property-based testing for .prologos code

Property tests written in .prologos itself, e.g.:

property color-roundtrip
  &> (color item hue) => non-empty

Or generative testing with QuickCheck-style generators. This is a real language feature and deserves its own track; mentioned here only for completeness — not blocking on this issue.

Suggested scope for this issue

• In scope: Layer 1 (auto-run acceptance files). Add to tools/run-affected-tests.rkt or a sibling tool. Include in the standard CI gate.
• In scope: Curate the examples/ set — distinguish "WIP / commented-out work-in-progress" files from "should run clean today" acceptance files. Add a header convention (e.g., ;; STATUS: ACCEPTANCE vs ;; STATUS: WIP).
• In scope: Triage existing files — fix the parent reference in 2026-03-14-wfle-acceptance.prologos, audit other examples for similar latent issues.
• Out of scope: Layer 2 + Layer 3 — separate follow-up work once Layer 1 is in place and we have data on which files need stronger checks.

Examples currently in racket/prologos/examples/

A non-exhaustive sample:

• 2026-03-14-wfle-acceptance.prologos — Well-Founded Logic Engine (motivated this issue)
• 2026-04-17-ppn-track4c.prologos — PPN Track 4C acceptance
• eigentrust.prologos — kumavis's EigenTrust implementation (PR Add EigenTrust reputation algorithm: implementation, tests, benchmark #2 territory)
• Many 2026-MM-DD-*.prologos files for various tracks

A baseline audit (run each, see which abort) is the first deliverable.

References

• Bug discovery: 2026-04-26 session, ground->prologos-expr regression in reduction.rkt
• Fix commit: pending
• Related: kumavis's PR Add EigenTrust reputation algorithm: implementation, tests, benchmark #2 (EigenTrust draft) had its OWN acceptance file that exposed pitfalls multi-line spec with comments between tokens confused the preparser #1-pvec: add Int-indexed helpers (eigentrust pitfall #12) #15 — same pattern of "writing real code finds bugs that test fixtures miss"
• Related: .claude/rules/testing.md § "Three-level WS validation" — already documents this gap conceptually but the Layer 3 (WS file) part isn't enforced

Priority

Medium-high. The bug class this prevents is "silent wrong output for working programs" — exactly the kind that erodes user trust without explicit signal. Compounds with each new feature that touches the elaboration → runtime path.