Remove test theater, add state/snapshot/domain-oracle tests - #2
Merged
Conversation
P0 - EFFECT_HARNESS.md mapping table: - Fixed 3 'Active' entries for non-existent CLI commands - Updated to point to actual commands (vp, veto-cascade, ec --diff) P1 - Rename stale vocab script: - evaluate_vocab_effect.py → evaluate_quale_effect.py - Updated all references in docs, tests, and the script itself P2 - Rename stale naming in tests: - VocabCliTests → QualeCliTests - run_vocab → run_quale (4 test files: test_cli.py, test_adversarial.py, test_smoke.py, test_commands.py) - vocab@example.test → quale@example.test - Created tests/helpers.py with shared QualeTestCase base class P5 - Structural guardrail tests (test_structure.py): - TestModuleTestMapping: core source modules mapped to test coverage - TestNoStaleVocabNaming: no stale 'run_vocab' defs, 'VocabCli' refs, or import-level 'vocab' references outside vocabulary.py - TestTestHelperConsistency: new test files must use CLI run helper Total: 365 passing, 0 failing
…es of theater Theater removed (-132 lines): - test_cli_smoke.py: removed 8 individual smoke tests (review_smoke, onboard_smoke, refactor_cost_smoke, agent orient/edit/guard smoke, ci init/check/trend smoke) — all duplicated by data-driven COMMANDS_WITH_PATH loop - test_human_ci.py: removed 7 redundant tests (review_basic, review_json, onboard_basic, onboard_json, refactor_cost_basic, refactor_cost_json, ci_trend_after_report) — all duplicated by test_output_contracts.py - test_commands.py: replaced 6 'assertIn key' theater checks with real domain value oracles (isinstance checks, range checks, type checks) Real tests added (+368 lines): - test_state.py (5 tests): scan cache hit on repeat, cache invalidation on new commit, CI history accumulation, scan limit enforcement, contract invalidation after file change - test_snapshots.py (4 tests): agent_orient, test_gaps, extinct_exports, health_score with golden file comparison (UPDATE_SNAPSHOTS=1 to refresh) - test_commands.py: 6 'assertIn key' → isinstance/range/type assertions Quality shift: - LOW bug-finding tests: ~110 → ~55 - HIGH bug-finding tests: ~50 → ~75 - Genuine regression coverage: 0 → 5 state transition tests + 4 snapshot tests
- Enable branch protection on master via gh api (requires: test, guardrails, lint, security status checks; no direct pushes; squash merge) - Clean up 6 stale merged branches from remote + local - Add snapshot, state, structure tests to CI guardrails workflow - Update CONTRIBUTING.md: merge strategy, branch naming, CI gate matrix, snapshot update instructions, stale branch cleanup
Real bugs found by ruff: - Dead code: orphaned 'return concerns[:5]' at end of _count_new_identifiers - Duplicate _safe_islands_data definition in __init__.py (analysis.py version was imported, but a stale local copy also existed at line 383) - Unused variable 'base' in mcp_server.py - 'not ... ==' -> '!=' simplification in mcp_server.py - Import sorting in reports/__init__.py Also updated health_score snapshot that drifted during bug fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaced 132 lines of testing theater (keyword-presence checks, duplicate smoke tests) with 368 lines of real tests:
assertIn("key", data)→isinstance(data[key], list) + range checksAlso:
Branch protection is now active
Direct pushes to master are blocked. All changes go through PRs with
test,guardrails,lint,securityrequired to pass.