test: add unit tests for parser, selector, entity coverage, setcover, and dedup#1
Open
hai-pilgrim wants to merge 19 commits into
Open
test: add unit tests for parser, selector, entity coverage, setcover, and dedup#1hai-pilgrim wants to merge 19 commits into
hai-pilgrim wants to merge 19 commits into
Conversation
55 tests covering the core compaction pipeline — previously untested outside the Codex CLI plugin: - tests/test_parser.py (29 tests): _is_user_message classification, parse_jsonl turn grouping (empty files, multi-exchange, SKIP_TYPES, malformed JSON, trailing system turns), extract_text for all block types (text, thinking, tool_use, tool_result) - tests/test_selector.py (10 tests): select_turns guarantees (user turns always kept, short system turns always kept, last system turn always kept), greedy scoring, budget cap, original-order output, token accounting, dropped turn tracking - tests/test_entity_coverage.py (16 tests): ENTITY_TYPES weights, entity extraction for file paths, exceptions, URLs, ports, packages, and deduplication behaviour All tests are self-contained (no external files, no network, no models).
Runs pytest on Python 3.11 and 3.12 using only the base dependencies (no torch/transformers required — the unit tests are model-free). Triggers on push and PRs to main.
Covers the enhanced EITF scorer (setcover.py): - Empty input returns empty list - One ScoredTurn produced per system turn - Scores normalized to [0, 1] with max == 1.0 - Entity-rich turns score higher than content-free turns - Turns with exclusive entities score >= turns with shared entities (verifying the 1.2x exclusivity bonus has the intended effect) - Token count and turn reference preserved in ScoredTurn 65 total tests across the suite.
Covers SuffixAutomaton internals (extend, propagate_counts, match_repeated_length) and dedup_scores contracts (normalization, unique vs duplicate scoring, min_repeat_len parameter, ordering). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers write_summary_text (labels, truncation, separators, empty turns), write_compacted_jsonl (valid JSONL, multi-turn), and write_scores_csv (header, score values, kept flag, sorted output). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers normalization, entity-rich vs empty scoring, rare vs common entity ITF behavior, BM25-style length normalization, and edge cases (single turn, all-zero entities). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers get_scorer (known/unknown methods), LOCAL_METHODS/ALL_METHODS invariants, and smoke tests for DedupScorer/EitfScorer/SetcoverScorer .score() via the lazy-import pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers ScoredTurn field access, build_query turn selection (last 3, fewer than 3, separator, truncation), and random_scores token/reference handling. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- `test_llm_compact.py`: 23 tests covering `make_synthetic_turn` (9 tests) and `llm_compact` (10 tests) with mocked httpx calls, plus 4 tests for `COMPACT_SYSTEM_PROMPT` content - `llm_compact.py`: remove unused `from .tokenizer import estimate_tokens` import — `tokenizer` depends on `transformers` which is not installed in CI, causing import errors when the test file is collected Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion Tests cover: - `_dcg`: empty list, single item, weight sorting, zero/max scores - `aggregate`: list return type, one result per model key, method/budget propagation, empty answers, dimension count, probe_count, normalised mean scores (0 and 1), composite/NDCG ranges, perfect-score NDCG = 1, raw_scores storage, dimension_map property, model label, difficulty weight application Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests cover export_json (11 tests: file creation, valid JSON, per-result entries, method/budget/composite/NDCG preservation, dimensions schema, empty list, speed/token counts) and export_trace (9 tests: file creation, filename pattern, auto-mkdir, JSON validity, method/budget, entries count, required entry fields, score value, unknown probe ID skipped). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests cover Probe field access and defaults, DIMENSIONS constant (expected keys, weights sum to 1, all positive), and ProbeSet round-trip serialization (to_dict/from_dict): structure, probes count, field preservation (id, dimension, tier, question, gold_answer, evidence_turns, difficulty), conv_hash/split_ratio/version preservation, missing-field defaults, and empty probes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests cover conv_hash (16-char hex output, deterministic, sensitivity to ratio and content changes, large-file handling), _cache_path (key/version in filename, json extension, correct parent dir), and save_probes/ load_probes round-trip (file creation, mkdir, return type, missing key returns None, conv_hash/version/probes/split_ratio preservation, wrong version returns None). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- 43 tests for evidence_coverage: _dcg, compute_evidence_coverage, ProbeCoverage/DimensionCoverage fields, composite/NDCG ranges, perfect coverage → 1.0, empty probes, EvidenceCoverageResult.to_dict - 31 tests for judge: ProbeAnswer/JudgeResult dataclasses, module constants, _openrouter_generate_async (no key, success, URL, model), _score_one_answer (JSON parsing, score clamping 0-3, markdown fencing, invalid JSON error handling, missing keys) Total suite: 326 tests, all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…Result.f1 Tests cover: - _extract_vocab: empty input, short word exclusion, lowercasing, repeated word counts, file path extraction, identifiers with underscores - _idf: zero for missing terms, formula verification, rare > common terms, empty doc list, total_docs scaling - FitnessResult.f1: edge cases (recall=0, compression=1), harmonic mean formula, balanced inputs, full range grid check - FitnessResult dataclass: field access, budget, token counts Uses sys.modules stub for transformers (not installed in CI). Total suite: 355 tests, all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- 5 tests for _instruct pure function: format, instruction/query inclusion, empty strings - 3 constant checks: non-empty instructions, positive MAX_DOC_CHARS - 11 tests for LlamaEmbedScorer: health check call, trailing slash stripping, score_turns return type, cosine sim math (orthogonal → 0.5, identical → 1.0), multi-turn, index sort - 9 tests for LlamaRerankScorer: health check, score preservation, token counts, multi-turn, missing index defaults to 0.0, request body validation (query + documents) Total suite: 382 tests, all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uses matplotlib Agg backend (no display required). - 3 tests for METHOD_STYLES: known methods present, color/marker/label fields, claude-code entry - 13 tests for plot_entity_coverage: single result, empty list, multiple methods, same-method multiple budgets, unknown method fallback, axes labels, legend presence/absence, budget/token annotations, duplicate method legend deduplication - 6 tests for plot_type_breakdown: with and without type_coverage, largest-budget selection, axis labels, empty type_coverage dict Total suite: 404 tests, all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_scorer.py (16 tests): _last_token_pool pooling logic (right-padding, left-padding, batch, no-padding), _format_instruct formatting, and module-level constants - test_tokenizer.py (6 tests): estimate_tokens and turn_tokens with mocked tokenizer, including verifying add_special_tokens=False is passed Total test count: 416 (up from 394) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_resolve_methods, build_parser, and _export_eval_json had zero test coverage. Adds 30 tests: - _resolve_methods: 'all' expansion, single-method wrapping, return type - build_parser: all three subcommands, default values for every flag, boolean flags, method/budget overrides - _export_eval_json: writes valid JSON, evidence-only, entity-only, merging entity into evidence entries, empty inputs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_parser.py(29 tests):_is_user_message,parse_jsonl,extract_texttest_selector.py(10 tests):select_turnsinvariants, greedy scoring, budget, token accountingtest_entity_coverage.py(16 tests): ENTITY_TYPES weights, extraction (paths, exceptions, URLs, ports, packages)test_setcover.py(10 tests):setcover_scoresnormalization, exclusivity bonus, edge casestest_dedup.py(18 tests):SuffixAutomatoninternals +dedup_scorescontractstest_eitf.py: eitf scorertest_fitness.py(29 tests):_extract_vocab,_idf,FitnessResult.f1test_evidence_coverage.py+test_judge.py(74 tests): eval frameworktest_pareto.py(22 tests): Pareto plot helperstest_scorer.py+test_tokenizer.py: scorer/tokenizer pure functionstest_llama_embed.py(27 tests): llama_embed and llama_rerank scorerstest_compact_helpers.py(30 tests):_resolve_methods,build_parser,_export_eval_jsonTest plan
uv run pytest tests/ -v→ 446 passed, 1 warning🤖 Generated with Claude Code