Add reviewed German domain relevance corpus (driving-license law)#37
Merged
Conversation
Scopes two new judged domain corpora (non-English + faceted-content) and a real-query-evidence phase for the existing GOV.UK suite, per the roadmap's open relevance-coverage item. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Splits Domain A (German snapshot corpus, straightforward) from Domain B (faceted-content corpus, needs new evaluator facet support) per research findings; amends the design spec to record the split. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds 23 normalized snapshot documents for the de-fahrerlaubnisrecht suite, sourced from German Wikipedia's Kategorie:Fahrerlaubnisrecht (Deutschland), CC BY-SA 4.0. queries: [] pending Task 4. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit adds three changes: 1. Adds de-fahrerlaubnisrecht-fixture-policy.test.ts: A policy test that locks the German domain fixture shape, verifying 23 documents, 19 queries, all topics, content hashes, and judgment/rationale consistency. 2. Fixes de-fahrerlaubnisrecht.json: URL-encodes document IDs and judgment/rationale keys to match URL pathname handling (e.g., Ü becomes %C3%BC). This was a fixture defect introduced in Task 3 that prevented validation. 3. Updates domain-fixture-policy.test.ts: Replaces dynamic topic computation with explicit SEARCHABLE_DOCS_TOPICS list, preventing false failures when new domain topics are added to the schema. All 153 tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…test Remove unused DOMAIN_QUERY_TOPICS and GOVUK_DOMAIN_QUERY_TOPICS imports from domain-fixture-policy.test.ts. These became unused after a recent change replaced dynamic topic computation with explicit SEARCHABLE_DOCS_TOPICS. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…vocabulary
Live-evaluator review found these queries returned zero or missing
grade-3 results because the lexical search does strict boolean AND
across query terms with no compound-word splitting or stopword
filtering: a query fails entirely if even one term is absent from the
target document's text, regardless of whether the concept is present
under different wording.
Rewrote query text (not judgments/rationales/corpus) for:
- sperrfrist-nach-entzug: "Fahrerlaubnisentzug" -> "Führerscheinentzug"
(target doc uses Entzug/Führerscheinentzug, never that compound)
- alkoholverbot-fahranfaenger: "Alkoholverbot" -> "Null-Promille-Grenze"
(the exact term the target doc's section is titled)
- wann-mpu-noetig: "Wann muss ich..." -> "Anordnung einer MPU"
(wann/ich occur 0x anywhere in the corpus)
- mofa-pruefbescheinigung-beantragen: "beantragen" -> "erwerben"
(target doc frames the process as Erwerb, not Antrag)
- strafe-fahren-ohne-fuehrerschein: dropped "Strafe" (0x in either
grade-3 doc) for shared vocabulary ("Fahren ohne Führerschein oder
Fahrerlaubnis") since the two docs describe different penalty
concepts with no shared punishment vocabulary
Verified via the live evaluator that all 5 now retrieve their grade-3
doc(s) at rank 1, the other 14 queries' results are byte-identical
before/after, and the full relevance test suite (153 tests) still
passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flip de-fahrerlaubnisrecht to reviewed status and publish its k=5 baseline metrics alongside the existing documentation and GOV.UK domain corpora.
CI's project-identity-policy test forbids tracking internal agent planning documents under docs/superpowers/. Untrack the spec/plan/notes files this branch added and gitignore the directory going forward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
packages/relevance:de-fahrerlaubnisrecht, a German-language snapshot corpus (23 German Wikipedia articles on driving-license law, CC BY-SA 4.0) with 19 native-German judged queries across 5 topics.docs/superpowers/specs/2026-07-17-relevance-evidence-expansion-design.md. Domain B (a faceted-content corpus) is intentionally deferred to a separate future plan, since it needs new facet-filter support that doesn't exist in the evaluator yet.docs/project/relevance-baselines.md.validateDomainSuite'surl.pathname === idcheck; all 23 ids and their judgment/rationale keys are now consistently percent-encoded. Also fixes an unrelated regression indomain-fixture-policy.test.ts(broken by this branch's new topic registrations) without weakening its assertions on thesearchable-docssuite.Test plan
pnpm --filter @ktjn/searchable-relevance test— 153/153 passingnpx biome checkclean across changed packages/docsde-fahrerlaubnisrechtreproduces the published baseline exactly🤖 Generated with Claude Code