Add real-query-derived evidence to govuk-learn-to-drive (Phase 2)#39
Merged
Conversation
…ve (draft, v1.1.0)
…unts Updates the hard-coded version, query count, word-count distribution, and draft-review assertions in govuk-fixture-policy.test.ts to match Task 3's fixture (20->28 queries, 1.0.0->1.1.0, review reset to draft). Also fixes a hardcoded version:1.0.0 match in load-domain-suite.test.ts, which broke as a direct consequence of the same version bump.
…topword search
Task 5 live-evaluator review found 6 of the 8 recently-added real-query
fixtures failed to retrieve their judged primary document, because this
project's search does strict-AND term matching with zero stopword removal
and a real Porter stemmer. The autocomplete-derived question phrasing used
words ("legal", "does", "how", "lessons", "learners") that never literally
appear (or stem-match) in the target GOV.UK document text.
Rewrote the 6 query texts to use vocabulary genuinely present in their
primary judged document while keeping them natural, real-query-flavored
phrases in the same topic:
- glasses-need-to-drive: "do i" -> "do you" (doc uses "you", not "i")
- eyesight-legal-standard: "legal" -> "minimum" (doc says "minimum eyesight
standard for driving", never "legal")
- provisional-licence-cost: drop "does" (absent from doc); "how much is a
provisional driving licence" matches the doc's "How much it costs" heading
- how-to-book-theory-test: "how to book" -> "book a ... online" (doc has no
"how"; matches its "Book a test online" heading)
- lessons-after-passing: "lessons" -> "course" (doc says "practical training
course", never "lesson(s)")
- learners-motorway: "learners" -> "learning" (doc uses "learning"/"learn";
the real Porter stemmer does not fold "learners" to "learn")
All 6 queries now retrieve their grade-3 primary document within the top 5
(5 of 6 at rank 1; how-to-book-theory-test at rank 2). learners-motorway's
grade-1 secondary judgment (/guidance/the-highway-code) no longer retrieves
as a side effect of the vocabulary swap, but its judgment is left as-is
since the document remains genuinely topically relevant — an accepted
recall gap per task guidance, and a net improvement since the grade-3
primary now surfaces directly instead of not at all.
Verified via evaluator JSON diff that the other 22 queries' returnedIds are
byte-identical before/after. Word-count distribution (18 short/10 long)
is unchanged, so govuk-fixture-policy.test.ts needed no updates. Full
policy/relevance test suite (165 tests) and biome lint both pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…derived evidence Flips the govuk-learn-to-drive suite's review status from draft to reviewed (reviewer ktjn, 2026-07-18) now that all 28 queries (20 original + 8 Google-autocomplete-derived) have been reviewed. Documents the new baseline metrics at k=5, the autocomplete provenance and its limits (not a licensed dataset, not query-log evidence), and updates the roadmap's real-query-evidence tracking accordingly.
…gal-standard rationale 6-12 -> 6/12, per the source document's actual notation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
govuk-learn-to-driverelevance domain suite (20→28 queries, v1.0.0→v1.1.0), without touching the 22-document corpus. This is Phase 2 of the relevance-evidence-expansion effort (Domain A/German PR Add reviewed German domain relevance corpus (driving-license law) #37, Domain B/Gutenberg-facets PR Add reviewed Gutenberg facets domain relevance corpus #38 already merged).govuk-refresh.ts/pnpm relevance:refresh— that pipeline is document-refresh-only and would force an unwanted live GOV.UK re-fetch with no queries-only path (confirmed by reading its source during planning).Test plan
pnpm --filter @ktjn/searchable-relevance test— 165/165 passingnpx biome checkclean across changed packages/docscontentHash/provenanceconfirmed byte-identical to before this branch; all 20 original queries' individual results confirmed bit-for-bit identical to the recorded v1.0.0 baseline🤖 Generated with Claude Code