Add real-query-derived evidence to govuk-learn-to-drive (Phase 2)

CHANGELOG.md

@@ -79,6 +79,20 @@ below was built and tested across prior unreleased commits.
   genre-and-year intersection query — alongside the existing documentation,
   GOV.UK, and German corpora (docs/project/relevance-baselines.md).
 
+### Changed
+
+- **GOV.UK relevance corpus expanded to real-query-derived phrasing**:
+  `govuk-learn-to-drive` bumped `1.0.0` → `1.1.0` (20 → 28 queries, 22
+  documents unchanged). The 8 added queries are sourced from Google's public
+  autocomplete suggestion endpoint as real-search-language phrasing
+  inspiration — not a licensed dataset, and not GOV.UK query-log evidence
+  (GOV.UK does not publish query logs) — and are hand-graded and reviewed
+  like the rest of the suite. The reviewed `k = 5` baseline improves on every
+  metric but recall versus `1.0.0` (MRR 0.650000 → 0.732143, Precision@5
+  0.160000 → 0.171429, Recall@5 0.475000 → 0.464286, nDCG@5 0.585738 →
+  0.642698, zero-result rate 0.300000 → 0.214286) despite the added queries
+  being harder and more naturally phrased (docs/project/relevance-baselines.md).
+
 ### Notes
 
 - The core runtime bundle (`index.js` + `worker.js` + `sw.js`) is held

docs/project/relevance-baselines.md

@@ -114,26 +114,48 @@ does not determine whether a ranking change is acceptable.
 
 ## Reviewed GOV.UK learner-driving corpus
 
-`govuk-learn-to-drive@1.0.0` contains the 22 public pages in GOV.UK's “Learn to
-drive a car” journey: the journey hub and its 21 internal destinations. It has
-20 English task-oriented queries, comprising 16 concise queries and four
-longer natural-language queries, across seven topics: eligibility and eyesight,
-provisional licences, lessons and practice, theory preparation, theory-test
-management, practical-test management, and after passing.
+`govuk-learn-to-drive@1.1.0` contains the 22 public pages in GOV.UK's “Learn to
+drive a car” journey: the journey hub and its 21 internal destinations
+(unchanged from `1.0.0`). It has 28 English task-oriented queries across seven
+topics: eligibility and eyesight, provisional licences, lessons and practice,
+theory preparation, theory-test management, practical-test management, and
+after passing.
+
+Twenty of the 28 queries are the original authored queries from `1.0.0`. The
+remaining 8 are sourced from Google's public, unauthenticated autocomplete
+suggestion endpoint (`suggestqueries.google.com/complete/search`), used as
+phrasing inspiration for authoring genuine evaluation queries — **this is not
+a licensed dataset**, unlike the GOV.UK OGL content itself, and it is not a
+GOV.UK search-query log: GOV.UK does not publish query-log data. Each
+candidate suggestion was screened against the corpus for genuine
+answerability and non-duplication with existing queries, hand-graded against
+the real documents, and reviewed for vocabulary grounding (several needed
+rewording so their significant terms appear in the judged documents' literal
+text before being added). Example new query ids: `glasses-need-to-drive`,
+`learners-motorway`, and `lessons-after-passing`. Full sourcing detail,
+including rejected candidates and the vocabulary-grounding review, is in
+`docs/superpowers/notes/2026-07-17-govuk-real-query-candidates.md`.
 
 The suite uses the same graded judgment and page-specific rationale policy as
 the documentation corpus. Maintainer `ktjn` reviewed every normalized
-document, query, grade, rationale, and measured top-five result on 2026-07-14.
+document, query, grade, rationale, and measured top-five result on 2026-07-18.
 
 The reviewed baseline at `k = 5` is:
 
-| Metric | Value |
-|---|---:|
-| MRR | 0.650000 |
-| Precision@5 | 0.160000 |
-| Recall@5 | 0.475000 |
-| nDCG@5 | 0.585738 |
-| Zero-result rate | 0.300000 |
+| Metric | `1.0.0` (20 queries) | `1.1.0` (28 queries) |
+|---|---:|---:|
+| MRR | 0.650000 | 0.732143 |
+| Precision@5 | 0.160000 | 0.171429 |
+| Recall@5 | 0.475000 | 0.464286 |
+| nDCG@5 | 0.585738 | 0.642698 |
+| Zero-result rate | 0.300000 (6/20) | 0.214286 (6/28) |
+
+The `1.1.0` aggregate is measurably better than `1.0.0` on every metric except
+a small dip in recall, despite the 8 added queries being harder and more
+naturally phrased (several are longer, informally-worded questions rather
+than concise keyword searches) — see the source note above for the per-query
+vocabulary-grounding risks that were identified and, where needed, addressed
+before this baseline was recorded.
 
 The source snapshot was retrieved on 2026-07-13. It contains public sector
 information licensed under the [Open Government Licence
@@ -260,7 +282,13 @@ catalog — four domains across two languages (English and German). Only the
 Gutenberg facets corpus exercises facet-filtered search under judged
 relevance; the other three domains judge free-text search only. Facet
 *counts* (`facetValues()`) remain unexercised by any relevance suite. Their
-queries are authored intents rather than user-query logs. These scores are
+queries are authored intents rather than user-query logs, with one exception:
+8 of the 28 `govuk-learn-to-drive` queries are sourced from Google
+autocomplete suggestions as real-world phrasing inspiration (see that
+section above). This is a proxy for real search language, not query-log
+evidence — GOV.UK does not publish query logs, and an informal, unauthenticated
+autocomplete endpoint is not equivalent to production search analytics.
+These scores are
 not latency or memory evidence and do not establish web-scale quality or
 superiority over another engine. Do not compare scores across suites because
 their source material, query sets, and judgments differ.

docs/project/roadmap.md

@@ -18,7 +18,7 @@ This page is the single current list of shipped capability and remaining work; d
 
 ## Near-term work
 
-- Expand relevance coverage beyond the documentation, learner-driving, German driving-license-law, and Gutenberg faceted-fiction domains — now spanning English and German, and now including judged-relevance coverage of terms and range facet filtering — with broader judged sets and real query evidence before defining thresholds or making production-scale claims.
+- Expand relevance coverage beyond the documentation, learner-driving, German driving-license-law, and Gutenberg faceted-fiction domains — now spanning English and German, and now including judged-relevance coverage of terms and range facet filtering — with broader judged sets and real query evidence before defining thresholds or making production-scale claims. Real query evidence is now partially addressed for one suite: `govuk-learn-to-drive@1.1.0` adds 8 queries sourced from Google's public autocomplete suggestion endpoint as a real-search-language proxy (see [Relevance baselines](relevance-baselines.md)). This is not full production query-log evidence — GOV.UK does not publish query logs, and no equivalent source is currently available — and it remains open for the documentation, German, and Gutenberg suites, and for a more rigorous real-log source if one ever becomes available.
 - Add a semantic-search example that states model, download, memory, and latency costs clearly; keep lexical search as the default for content sites.
 - Expand full language profiles only with representative corpora, analyzer fixtures, relevance queries, and cross-implementation conformance tests.
 - Refine loading priority, memory controls, and prefetching from measured browser behavior rather than fixed speculative policies.

packages/relevance/fixtures/domains/govuk-learn-to-drive.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": 2,
   "id": "govuk-learn-to-drive",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "language": "en",
   "provenance": {
     "publisher": "Government Digital Service and GOV.UK publishing organisations",
@@ -17,7 +17,7 @@
     "status": "reviewed",
     "method": "Maintainer review of every normalized document, query, grade, rationale, and measured top-five result.",
     "reviewer": "ktjn",
-    "reviewedAt": "2026-07-14"
+    "reviewedAt": "2026-07-18"
   },
   "corpus": {
     "kind": "snapshot",
@@ -458,6 +458,116 @@
         "/pass-plus": "Explains optional post-test training and possible insurance discounts.",
         "/legal-obligations-drivers-riders": "Supplies continuing legal obligations for drivers."
       }
+    },
+    {
+      "id": "glasses-need-to-drive",
+      "text": "do you need glasses to drive",
+      "topic": "eligibility-eyesight",
+      "judgments": {
+        "/driving-eyesight-rules": 3,
+        "/learn-to-drive-a-car": 1,
+        "/legal-obligations-drivers-riders": 1
+      },
+      "rationales": {
+        "/driving-eyesight-rules": "States you must wear glasses or contact lenses every time you drive if you need them to meet the standards of vision for driving.",
+        "/learn-to-drive-a-car": "Links 'Driving eyesight rules' from the eligibility step of the learner journey.",
+        "/legal-obligations-drivers-riders": "Lists meeting the minimum eyesight rules as one of the legal requirements before driving or riding."
+      }
+    },
+    {
+      "id": "eyesight-legal-standard",
+      "text": "minimum eyesight standard for driving test",
+      "topic": "eligibility-eyesight",
+      "judgments": {
+        "/driving-eyesight-rules": 3,
+        "/apply-first-provisional-driving-licence": 1,
+        "/learn-to-drive-a-car": 1
+      },
+      "rationales": {
+        "/driving-eyesight-rules": "Gives the Snellen-scale visual-acuity standard (decimal 0.5 / 6/12) and the number-plate-at-20-metres check used at the start of the practical driving test.",
+        "/apply-first-provisional-driving-licence": "Restates the number-plate-at-20-metres eyesight check as a first-licence application requirement.",
+        "/learn-to-drive-a-car": "Links 'Driving eyesight rules' from the eligibility step of the learner journey."
+      }
+    },
+    {
+      "id": "provisional-licence-cost",
+      "text": "how much is a provisional driving licence",
+      "topic": "provisional-licence",
+      "judgments": {
+        "/apply-first-provisional-driving-licence": 3,
+        "/learn-to-drive-a-car": 1
+      },
+      "rationales": {
+        "/apply-first-provisional-driving-licence": "States the £34 online fee and the £43 cheque/postal-order fee for a first provisional licence.",
+        "/learn-to-drive-a-car": "Repeats the '£34 to £43' fee range under the 'Get a provisional licence' step of the journey hub."
+      }
+    },
+    {
+      "id": "learners-motorway",
+      "text": "driving on the motorway when learning to drive",
+      "topic": "lessons-practice",
+      "judgments": {
+        "/driving-lessons-learning-to-drive": 3,
+        "/guidance/the-highway-code": 1
+      },
+      "rationales": {
+        "/driving-lessons-learning-to-drive": "States the exact motorway rule for learners: only allowed if driving in England, Scotland or Wales, with an approved driving instructor, in a car fitted with dual controls.",
+        "/guidance/the-highway-code": "Indexes the Highway Code's dedicated motorways chapter (rules 253-274) as further reading on motorway driving."
+      }
+    },
+    {
+      "id": "prepare-theory-test",
+      "text": "how to prepare for theory test",
+      "topic": "theory-preparation",
+      "judgments": {
+        "/theory-test/revision-and-practice": 3,
+        "/take-practice-theory-test": 2,
+        "/guidance/the-highway-code": 1
+      },
+      "rationales": {
+        "/theory-test/revision-and-practice": "Explains preparing for the theory test using the three official study books, DVSA apps, free practice tests, and hazard-perception prep guidance.",
+        "/take-practice-theory-test": "Provides the free official DVSA practice tests referenced by the general revision guidance.",
+        "/guidance/the-highway-code": "Is one of the three named study books ('The Highway Code') used to prepare for the theory test's multiple-choice questions."
+      }
+    },
+    {
+      "id": "how-to-book-theory-test",
+      "text": "book a theory test online",
+      "topic": "theory-test-management",
+      "judgments": {
+        "/book-theory-test": 3,
+        "/learn-to-drive-a-car": 1
+      },
+      "rationales": {
+        "/book-theory-test": "Provides the official theory-test booking service for cars, motorcycles, lorries, buses and coaches.",
+        "/learn-to-drive-a-car": "Shows theory-test booking within the learner journey, including the £23 fee and the provisional-licence prerequisite."
+      }
+    },
+    {
+      "id": "theory-before-practical",
+      "text": "how long after theory test can you book practical",
+      "topic": "practical-test-management",
+      "judgments": {
+        "/book-driving-test": 2,
+        "/learn-to-drive-a-car": 2
+      },
+      "rationales": {
+        "/book-driving-test": "States 'You must usually have passed your theory test before booking' — confirms the required order but gives no minimum wait time, so it only partially answers the literal 'how long' framing of the query.",
+        "/learn-to-drive-a-car": "States the same ordering rule in the journey hub ('You must pass your theory test before you can book your driving test') but likewise gives no duration or cooling-off period."
+      }
+    },
+    {
+      "id": "lessons-after-passing",
+      "text": "driving course after passing test",
+      "topic": "after-passing",
+      "judgments": {
+        "/pass-plus": 3,
+        "/learn-to-drive-a-car": 2
+      },
+      "rationales": {
+        "/pass-plus": "Describes Pass Plus, a practical training course of at least 6 hours, as 'most useful to new drivers in the year after passing their test'.",
+        "/learn-to-drive-a-car": "Links to Pass Plus training courses from the 'When you pass' step and states the insurance-without-supervision requirement once you pass."
+      }
     }
   ]
 }

packages/relevance/test/govuk-fixture-policy.test.ts

@@ -30,19 +30,19 @@ describe("committed GOV.UK driving relevance fixture", () => {
       "govuk-learn-to-drive" as never,
     );
     expect(suite.id).toBe("govuk-learn-to-drive");
-    expect(suite.version).toBe("1.0.0");
+    expect(suite.version).toBe("1.1.0");
     expect(suite.corpus.kind).toBe("snapshot");
     if (suite.corpus.kind !== "snapshot") throw new Error("unreachable");
     expect(suite.corpus.documents).toHaveLength(22);
-    expect(suite.queries).toHaveLength(20);
+    expect(suite.queries).toHaveLength(28);
 
     const wordCounts = suite.queries.map(
       (query) => query.text.trim().split(/\s+/u).length,
     );
     expect(wordCounts.filter((count) => count >= 2 && count <= 5)).toHaveLength(
-      16,
+      18,
     );
-    expect(wordCounts.filter((count) => count >= 6)).toHaveLength(4);
+    expect(wordCounts.filter((count) => count >= 6)).toHaveLength(10);
     expect(new Set(suite.queries.map((query) => query.topic))).toEqual(
       new Set(GOVUK_DOMAIN_QUERY_TOPICS),
     );
@@ -51,7 +51,7 @@ describe("committed GOV.UK driving relevance fixture", () => {
       method:
         "Maintainer review of every normalized document, query, grade, rationale, and measured top-five result.",
       reviewer: "ktjn",
-      reviewedAt: "2026-07-14",
+      reviewedAt: "2026-07-18",
     });
     expect(suite.corpus.documents.map((document) => document.id)).toEqual(
       GOVUK_EXPECTED_ROUTES,

packages/relevance/test/load-domain-suite.test.ts

@@ -71,7 +71,7 @@ it("loads all committed allowlisted domain suites", async () => {
   ]);
   await expect(
     loadDomainSuite(fixtureDirectory, "govuk-learn-to-drive"),
-  ).resolves.toMatchObject({ id: "govuk-learn-to-drive", version: "1.0.0" });
+  ).resolves.toMatchObject({ id: "govuk-learn-to-drive", version: "1.1.0" });
   await expect(
     loadDomainSuite(fixtureDirectory, "de-fahrerlaubnisrecht"),
   ).resolves.toMatchObject({ id: "de-fahrerlaubnisrecht", version: "1.0.0" });