Skip to content

Commit 4904c31

Browse files
TimelordUKclaude
andcommitted
feat: Improve test output to clearly show JSON validation
Makes it explicit when formal tests are validating against JSON expectations and when JSON validation succeeds. Changes to output format: - FORMAL tests now show: [FORMAL (validating against expectations/file.json)] - Pass message shows: ✓ PASS (JSON validated) for formal tests - Fail message shows: ✗ FAIL - JSON output mismatch for formal tests - SMOKE tests remain: [SMOKE] ... ✓ PASS (runs) Before: [FORMAL] qualified_names ... ✓ PASS [SMOKE] accounting_format ... ✓ PASS (runs) After: [FORMAL (validating against expectations/qualified_names.json)] qualified_names ... ✓ PASS (JSON validated) [SMOKE] accounting_format ... ✓ PASS (runs) This makes it immediately clear which tests are doing real validation vs just smoke testing that queries run without errors. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 7ee89d3 commit 4904c31

1 file changed

Lines changed: 5 additions & 4 deletions

File tree

tests/integration/test_examples.py

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -224,12 +224,13 @@ def run_test(cli_path: str, sql_file: Path, expectations_dir: Path, result: Test
224224

225225
# Determine test mode
226226
is_formal = expectation_file.exists()
227-
test_mode = "FORMAL" if is_formal else "SMOKE"
228227

229228
if is_formal:
230229
result.formal_tests += 1
230+
test_mode = f"FORMAL (validating against expectations/{base_name}.json)"
231231
else:
232232
result.smoke_tests += 1
233+
test_mode = "SMOKE"
233234

234235
print(f"[{test_mode}] {base_name} ... ", end='', flush=True)
235236

@@ -267,13 +268,13 @@ def run_test(cli_path: str, sql_file: Path, expectations_dir: Path, result: Test
267268
# Compare
268269
matches, diff = compare_json(expected_json, actual_json)
269270
if matches:
270-
print(f"{Color.GREEN}✓ PASS{Color.NC}")
271+
print(f"{Color.GREEN}✓ PASS (JSON validated){Color.NC}")
271272
result.passed += 1
272273
result.passed_tests.append(base_name)
273274
else:
274-
print(f"{Color.RED}✗ FAIL - Output mismatch{Color.NC}")
275+
print(f"{Color.RED}✗ FAIL - JSON output mismatch{Color.NC}")
275276
result.failed += 1
276-
result.failed_tests.append((base_name, "Output mismatch"))
277+
result.failed_tests.append((base_name, "JSON output mismatch"))
277278
if '--verbose' in sys.argv and diff:
278279
print(f"{Color.YELLOW}{diff}{Color.NC}")
279280
else:

0 commit comments

Comments
 (0)