diff --git a/CHANGELOG.md b/CHANGELOG.md index 8091ea6..974508a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,6 +79,20 @@ below was built and tested across prior unreleased commits. genre-and-year intersection query — alongside the existing documentation, GOV.UK, and German corpora (docs/project/relevance-baselines.md). +### Changed + +- **GOV.UK relevance corpus expanded to real-query-derived phrasing**: + `govuk-learn-to-drive` bumped `1.0.0` → `1.1.0` (20 → 28 queries, 22 + documents unchanged). The 8 added queries are sourced from Google's public + autocomplete suggestion endpoint as real-search-language phrasing + inspiration — not a licensed dataset, and not GOV.UK query-log evidence + (GOV.UK does not publish query logs) — and are hand-graded and reviewed + like the rest of the suite. The reviewed `k = 5` baseline improves on every + metric but recall versus `1.0.0` (MRR 0.650000 → 0.732143, Precision@5 + 0.160000 → 0.171429, Recall@5 0.475000 → 0.464286, nDCG@5 0.585738 → + 0.642698, zero-result rate 0.300000 → 0.214286) despite the added queries + being harder and more naturally phrased (docs/project/relevance-baselines.md). + ### Notes - The core runtime bundle (`index.js` + `worker.js` + `sw.js`) is held diff --git a/docs/project/relevance-baselines.md b/docs/project/relevance-baselines.md index 9cc54a5..0107653 100644 --- a/docs/project/relevance-baselines.md +++ b/docs/project/relevance-baselines.md @@ -114,26 +114,48 @@ does not determine whether a ranking change is acceptable. ## Reviewed GOV.UK learner-driving corpus -`govuk-learn-to-drive@1.0.0` contains the 22 public pages in GOV.UK's “Learn to -drive a car” journey: the journey hub and its 21 internal destinations. It has -20 English task-oriented queries, comprising 16 concise queries and four -longer natural-language queries, across seven topics: eligibility and eyesight, -provisional licences, lessons and practice, theory preparation, theory-test -management, practical-test management, and after passing. +`govuk-learn-to-drive@1.1.0` contains the 22 public pages in GOV.UK's “Learn to +drive a car” journey: the journey hub and its 21 internal destinations +(unchanged from `1.0.0`). It has 28 English task-oriented queries across seven +topics: eligibility and eyesight, provisional licences, lessons and practice, +theory preparation, theory-test management, practical-test management, and +after passing. + +Twenty of the 28 queries are the original authored queries from `1.0.0`. The +remaining 8 are sourced from Google's public, unauthenticated autocomplete +suggestion endpoint (`suggestqueries.google.com/complete/search`), used as +phrasing inspiration for authoring genuine evaluation queries — **this is not +a licensed dataset**, unlike the GOV.UK OGL content itself, and it is not a +GOV.UK search-query log: GOV.UK does not publish query-log data. Each +candidate suggestion was screened against the corpus for genuine +answerability and non-duplication with existing queries, hand-graded against +the real documents, and reviewed for vocabulary grounding (several needed +rewording so their significant terms appear in the judged documents' literal +text before being added). Example new query ids: `glasses-need-to-drive`, +`learners-motorway`, and `lessons-after-passing`. Full sourcing detail, +including rejected candidates and the vocabulary-grounding review, is in +`docs/superpowers/notes/2026-07-17-govuk-real-query-candidates.md`. The suite uses the same graded judgment and page-specific rationale policy as the documentation corpus. Maintainer `ktjn` reviewed every normalized -document, query, grade, rationale, and measured top-five result on 2026-07-14. +document, query, grade, rationale, and measured top-five result on 2026-07-18. The reviewed baseline at `k = 5` is: -| Metric | Value | -|---|---:| -| MRR | 0.650000 | -| Precision@5 | 0.160000 | -| Recall@5 | 0.475000 | -| nDCG@5 | 0.585738 | -| Zero-result rate | 0.300000 | +| Metric | `1.0.0` (20 queries) | `1.1.0` (28 queries) | +|---|---:|---:| +| MRR | 0.650000 | 0.732143 | +| Precision@5 | 0.160000 | 0.171429 | +| Recall@5 | 0.475000 | 0.464286 | +| nDCG@5 | 0.585738 | 0.642698 | +| Zero-result rate | 0.300000 (6/20) | 0.214286 (6/28) | + +The `1.1.0` aggregate is measurably better than `1.0.0` on every metric except +a small dip in recall, despite the 8 added queries being harder and more +naturally phrased (several are longer, informally-worded questions rather +than concise keyword searches) — see the source note above for the per-query +vocabulary-grounding risks that were identified and, where needed, addressed +before this baseline was recorded. The source snapshot was retrieved on 2026-07-13. It contains public sector information licensed under the [Open Government Licence @@ -260,7 +282,13 @@ catalog — four domains across two languages (English and German). Only the Gutenberg facets corpus exercises facet-filtered search under judged relevance; the other three domains judge free-text search only. Facet *counts* (`facetValues()`) remain unexercised by any relevance suite. Their -queries are authored intents rather than user-query logs. These scores are +queries are authored intents rather than user-query logs, with one exception: +8 of the 28 `govuk-learn-to-drive` queries are sourced from Google +autocomplete suggestions as real-world phrasing inspiration (see that +section above). This is a proxy for real search language, not query-log +evidence — GOV.UK does not publish query logs, and an informal, unauthenticated +autocomplete endpoint is not equivalent to production search analytics. +These scores are not latency or memory evidence and do not establish web-scale quality or superiority over another engine. Do not compare scores across suites because their source material, query sets, and judgments differ. diff --git a/docs/project/roadmap.md b/docs/project/roadmap.md index 689efaa..2b59c60 100644 --- a/docs/project/roadmap.md +++ b/docs/project/roadmap.md @@ -18,7 +18,7 @@ This page is the single current list of shipped capability and remaining work; d ## Near-term work -- Expand relevance coverage beyond the documentation, learner-driving, German driving-license-law, and Gutenberg faceted-fiction domains — now spanning English and German, and now including judged-relevance coverage of terms and range facet filtering — with broader judged sets and real query evidence before defining thresholds or making production-scale claims. +- Expand relevance coverage beyond the documentation, learner-driving, German driving-license-law, and Gutenberg faceted-fiction domains — now spanning English and German, and now including judged-relevance coverage of terms and range facet filtering — with broader judged sets and real query evidence before defining thresholds or making production-scale claims. Real query evidence is now partially addressed for one suite: `govuk-learn-to-drive@1.1.0` adds 8 queries sourced from Google's public autocomplete suggestion endpoint as a real-search-language proxy (see [Relevance baselines](relevance-baselines.md)). This is not full production query-log evidence — GOV.UK does not publish query logs, and no equivalent source is currently available — and it remains open for the documentation, German, and Gutenberg suites, and for a more rigorous real-log source if one ever becomes available. - Add a semantic-search example that states model, download, memory, and latency costs clearly; keep lexical search as the default for content sites. - Expand full language profiles only with representative corpora, analyzer fixtures, relevance queries, and cross-implementation conformance tests. - Refine loading priority, memory controls, and prefetching from measured browser behavior rather than fixed speculative policies. diff --git a/packages/relevance/fixtures/domains/govuk-learn-to-drive.json b/packages/relevance/fixtures/domains/govuk-learn-to-drive.json index 533cec3..082524e 100644 --- a/packages/relevance/fixtures/domains/govuk-learn-to-drive.json +++ b/packages/relevance/fixtures/domains/govuk-learn-to-drive.json @@ -1,7 +1,7 @@ { "schemaVersion": 2, "id": "govuk-learn-to-drive", - "version": "1.0.0", + "version": "1.1.0", "language": "en", "provenance": { "publisher": "Government Digital Service and GOV.UK publishing organisations", @@ -17,7 +17,7 @@ "status": "reviewed", "method": "Maintainer review of every normalized document, query, grade, rationale, and measured top-five result.", "reviewer": "ktjn", - "reviewedAt": "2026-07-14" + "reviewedAt": "2026-07-18" }, "corpus": { "kind": "snapshot", @@ -458,6 +458,116 @@ "/pass-plus": "Explains optional post-test training and possible insurance discounts.", "/legal-obligations-drivers-riders": "Supplies continuing legal obligations for drivers." } + }, + { + "id": "glasses-need-to-drive", + "text": "do you need glasses to drive", + "topic": "eligibility-eyesight", + "judgments": { + "/driving-eyesight-rules": 3, + "/learn-to-drive-a-car": 1, + "/legal-obligations-drivers-riders": 1 + }, + "rationales": { + "/driving-eyesight-rules": "States you must wear glasses or contact lenses every time you drive if you need them to meet the standards of vision for driving.", + "/learn-to-drive-a-car": "Links 'Driving eyesight rules' from the eligibility step of the learner journey.", + "/legal-obligations-drivers-riders": "Lists meeting the minimum eyesight rules as one of the legal requirements before driving or riding." + } + }, + { + "id": "eyesight-legal-standard", + "text": "minimum eyesight standard for driving test", + "topic": "eligibility-eyesight", + "judgments": { + "/driving-eyesight-rules": 3, + "/apply-first-provisional-driving-licence": 1, + "/learn-to-drive-a-car": 1 + }, + "rationales": { + "/driving-eyesight-rules": "Gives the Snellen-scale visual-acuity standard (decimal 0.5 / 6/12) and the number-plate-at-20-metres check used at the start of the practical driving test.", + "/apply-first-provisional-driving-licence": "Restates the number-plate-at-20-metres eyesight check as a first-licence application requirement.", + "/learn-to-drive-a-car": "Links 'Driving eyesight rules' from the eligibility step of the learner journey." + } + }, + { + "id": "provisional-licence-cost", + "text": "how much is a provisional driving licence", + "topic": "provisional-licence", + "judgments": { + "/apply-first-provisional-driving-licence": 3, + "/learn-to-drive-a-car": 1 + }, + "rationales": { + "/apply-first-provisional-driving-licence": "States the £34 online fee and the £43 cheque/postal-order fee for a first provisional licence.", + "/learn-to-drive-a-car": "Repeats the '£34 to £43' fee range under the 'Get a provisional licence' step of the journey hub." + } + }, + { + "id": "learners-motorway", + "text": "driving on the motorway when learning to drive", + "topic": "lessons-practice", + "judgments": { + "/driving-lessons-learning-to-drive": 3, + "/guidance/the-highway-code": 1 + }, + "rationales": { + "/driving-lessons-learning-to-drive": "States the exact motorway rule for learners: only allowed if driving in England, Scotland or Wales, with an approved driving instructor, in a car fitted with dual controls.", + "/guidance/the-highway-code": "Indexes the Highway Code's dedicated motorways chapter (rules 253-274) as further reading on motorway driving." + } + }, + { + "id": "prepare-theory-test", + "text": "how to prepare for theory test", + "topic": "theory-preparation", + "judgments": { + "/theory-test/revision-and-practice": 3, + "/take-practice-theory-test": 2, + "/guidance/the-highway-code": 1 + }, + "rationales": { + "/theory-test/revision-and-practice": "Explains preparing for the theory test using the three official study books, DVSA apps, free practice tests, and hazard-perception prep guidance.", + "/take-practice-theory-test": "Provides the free official DVSA practice tests referenced by the general revision guidance.", + "/guidance/the-highway-code": "Is one of the three named study books ('The Highway Code') used to prepare for the theory test's multiple-choice questions." + } + }, + { + "id": "how-to-book-theory-test", + "text": "book a theory test online", + "topic": "theory-test-management", + "judgments": { + "/book-theory-test": 3, + "/learn-to-drive-a-car": 1 + }, + "rationales": { + "/book-theory-test": "Provides the official theory-test booking service for cars, motorcycles, lorries, buses and coaches.", + "/learn-to-drive-a-car": "Shows theory-test booking within the learner journey, including the £23 fee and the provisional-licence prerequisite." + } + }, + { + "id": "theory-before-practical", + "text": "how long after theory test can you book practical", + "topic": "practical-test-management", + "judgments": { + "/book-driving-test": 2, + "/learn-to-drive-a-car": 2 + }, + "rationales": { + "/book-driving-test": "States 'You must usually have passed your theory test before booking' — confirms the required order but gives no minimum wait time, so it only partially answers the literal 'how long' framing of the query.", + "/learn-to-drive-a-car": "States the same ordering rule in the journey hub ('You must pass your theory test before you can book your driving test') but likewise gives no duration or cooling-off period." + } + }, + { + "id": "lessons-after-passing", + "text": "driving course after passing test", + "topic": "after-passing", + "judgments": { + "/pass-plus": 3, + "/learn-to-drive-a-car": 2 + }, + "rationales": { + "/pass-plus": "Describes Pass Plus, a practical training course of at least 6 hours, as 'most useful to new drivers in the year after passing their test'.", + "/learn-to-drive-a-car": "Links to Pass Plus training courses from the 'When you pass' step and states the insurance-without-supervision requirement once you pass." + } } ] } diff --git a/packages/relevance/test/govuk-fixture-policy.test.ts b/packages/relevance/test/govuk-fixture-policy.test.ts index f63fac0..df0b89b 100644 --- a/packages/relevance/test/govuk-fixture-policy.test.ts +++ b/packages/relevance/test/govuk-fixture-policy.test.ts @@ -30,19 +30,19 @@ describe("committed GOV.UK driving relevance fixture", () => { "govuk-learn-to-drive" as never, ); expect(suite.id).toBe("govuk-learn-to-drive"); - expect(suite.version).toBe("1.0.0"); + expect(suite.version).toBe("1.1.0"); expect(suite.corpus.kind).toBe("snapshot"); if (suite.corpus.kind !== "snapshot") throw new Error("unreachable"); expect(suite.corpus.documents).toHaveLength(22); - expect(suite.queries).toHaveLength(20); + expect(suite.queries).toHaveLength(28); const wordCounts = suite.queries.map( (query) => query.text.trim().split(/\s+/u).length, ); expect(wordCounts.filter((count) => count >= 2 && count <= 5)).toHaveLength( - 16, + 18, ); - expect(wordCounts.filter((count) => count >= 6)).toHaveLength(4); + expect(wordCounts.filter((count) => count >= 6)).toHaveLength(10); expect(new Set(suite.queries.map((query) => query.topic))).toEqual( new Set(GOVUK_DOMAIN_QUERY_TOPICS), ); @@ -51,7 +51,7 @@ describe("committed GOV.UK driving relevance fixture", () => { method: "Maintainer review of every normalized document, query, grade, rationale, and measured top-five result.", reviewer: "ktjn", - reviewedAt: "2026-07-14", + reviewedAt: "2026-07-18", }); expect(suite.corpus.documents.map((document) => document.id)).toEqual( GOVUK_EXPECTED_ROUTES, diff --git a/packages/relevance/test/load-domain-suite.test.ts b/packages/relevance/test/load-domain-suite.test.ts index 50fbd54..6a71a76 100644 --- a/packages/relevance/test/load-domain-suite.test.ts +++ b/packages/relevance/test/load-domain-suite.test.ts @@ -71,7 +71,7 @@ it("loads all committed allowlisted domain suites", async () => { ]); await expect( loadDomainSuite(fixtureDirectory, "govuk-learn-to-drive"), - ).resolves.toMatchObject({ id: "govuk-learn-to-drive", version: "1.0.0" }); + ).resolves.toMatchObject({ id: "govuk-learn-to-drive", version: "1.1.0" }); await expect( loadDomainSuite(fixtureDirectory, "de-fahrerlaubnisrecht"), ).resolves.toMatchObject({ id: "de-fahrerlaubnisrecht", version: "1.0.0" });