test(eval): textbook collection for TECH-005 (ground truth pending review)

[fvadicamo]

Status: DRAFT, awaiting a human review of the ground truth

The material is published here so the evaluation can be reproduced and, above all, reviewed. It is not ready to merge: the human spot-check of the LLM-generated ground truth is an explicit acceptance criterion of TECH-005 and has not run yet. Merging it as-is would mark a box we have no right to mark.

What this adds

A second evaluation corpus. The existing ones are short, self-contained legal articles: the excerpt corpus saturates (hit rate near 100% whatever the configuration, because 20 candidates cover most of a 12-chunk corpus) and eval-full, while discriminative, understates anything that depends on surrounding context, which is precisely what parent chunk expansion (FEAT-017) exists for. A textbook chunk, taken alone, is genuinely ambiguous: that is the bench we were missing.

• tests/eval/dataset-textbook.jsonl — 60 Italian questions (24 factual, 15 reasoning, 12 multi-chunk, 9 adversarial) over namespace eval-textbook. Each carries a source_section field (ignored by the harness) so a human can audit where it came from.
• scripts/eval_clean_textbook.py — reproducible cleaning of the source PDF: font-size classification (body 11pt, footnotes 9pt, running headers 10pt, headings 14/18pt), footnote and header removal, hyphenation repair, chapter/section structure preserved as markdown headings for the dual chunker. Verified to reproduce the corpus byte for byte.
• tests/eval/README.md — collection matrix and when to use which, corpus provenance, the ground-truth methodology, and a validation-status table that states plainly what has and has not been verified.

Corpus

Carlo Smuraglia, Diritto penale del lavoro, Milano University Press 2025, DOI 10.54103/milanoup.236, CC BY-SA 4.0. Prose, no formulas, no tables, 13 chapters with numbered sections and dense implicit cross-references. Free download, sha256 recorded in the README.

A licensing note that cost some time and is worth keeping: sources with an ND clause cannot be used at all, because a cleaned and chunked corpus is a derivative work. That rules out most Italian university lecture notes (CC BY-NC-ND). OpenStax advertises CC BY but its live per-book metadata says CC BY-NC-SA, so the licence has to be checked on the artefact, not the marketing page.

How the ground truth was built

Questions were authored from full chapters, never from chunks: writing a question while looking at a chunk guarantees it is answerable from that chunk, which biases the dataset towards trivially retrievable passages. Keywords are verbatim phrases lifted from the source section, each verified to occur in exactly one chapter, because the harness counts a hit when any one keyword matches: a generic keyword would score a wrong chunk as correct.

Closed-book control (new, and now documented as a required step for every dataset): each question was put to the answering LLM with no context and an instruction to answer only if certain. Result: 0 expected keywords leaked, 53/60 answered "NON SO". The dataset measures retrieval, not the model's memory. Three factual questions that the model guessed from general legal knowledge are flagged in the review sample for rewriting.

First baseline

MiniLM, dual chunking, top_k=5: hit rate 78.4%, MRR 0.5873 (factual 79%, multi-chunk 92%, reasoning 67%), against 82.6% / 0.7029 on eval-full under the same configuration. The textbook is harder, which is exactly what it is for.

What is still open

1. Human spot-check of a stratified sample (13 questions: the riskiest adversarials, the weak factuals, and the multi-chunk ones that might collapse to a single section). Machine checks cannot catch a multi-chunk question that one section already answers, nor an adversarial question the corpus covers under a synonym.
2. Once the verdicts are in: apply them, record the outcome, run the end-to-end baseline, and take this out of draft.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 85e5695b-3751-4d46-ba8c-d626d0ef13a8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch test/tech-005-textbook-collection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request introduces a new evaluation dataset (dataset-textbook.jsonl) based on the Italian legal textbook "Diritto penale del lavoro" by Carlo Smuraglia, along with a dedicated PDF extraction and cleaning script (scripts/eval_clean_textbook.py) and updated documentation in tests/eval/README.md. The feedback suggests a minor optimization in the sorting logic of the line grouping function in the cleaning script to improve efficiency and avoid variable shadowing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Since lc is already sorted by the top coordinate, the first element lc[0] is guaranteed to have the minimum top value. Using lc[0]['top'] is more efficient and avoids shadowing the outer variable c in the generator expression.

Suggested change

	clusters.sort(key=lambda lc: min(c["top"] for c in lc))
	clusters.sort(key=lambda lc: lc[0]["top"])