Harden WPT evidence validation

docs/verification-audit.md

@@ -1,78 +1,67 @@
 # Verification Audit
 
-This audit maps the original goal and success criteria to current repository
-evidence. It is intentionally conservative: stale local artifacts and targeted
-checks do not prove the full Linux/macOS/Windows matrix.
-
-## Current Local Evidence
-
-Audited from `C:\Users\mertu\Desktop\webrtc-node` on 2026-05-28.
-
-| Gate | Evidence |
-| --- | --- |
-| Quality gate | `npm run check`, `npm run types:check`, `npm run api:check`, and `npm run pack:check` passed after Biome was added and the API surface checker was fixed for multiline TypeScript declarations. |
-| Native integration | `npm run native:check` passed; it verifies Node-API/node-addon-api usage, TSFN dispatch, and the pinned libdatachannel commit. |
-| Native build | `npm run build` passed on Windows with Visual Studio 2022 Build Tools. |
-| Unit tests | `npm test` passed 20/20 Node `node:test` tests. The remote-close data-channel test also passed 20 serial stress iterations after one parallel-load timeout. |
-| API surface | `npm run api:check` passed for 17 classes and 1 nonstandard member. |
-| Types | `npm run types:check` passed. |
-| WPT checkout | `npm run wpt:ensure` verified WPT commit `03169f171c797d0953b21d7388561b454fde0ad4`. |
-| WPT selection | `npm run wpt:selection:check` verified 620 selected subtests. |
-| Targeted WPT | The current Windows build passed `webrtc/RTCDataChannel-close.html`, the selected `RTCDataChannel-send.html` subset, and `RTCPeerConnection-ondatachannel.html` together as 46/46 subtests after the remote-close message-grace change. |
-| Docker Linux smoke | `scripts/run-docker-linux-ci.ps1 -NodeImage node:20-bookworm -SkipWpt` passed build/unit/API/types/WPT-selection using the snapshot-backed Docker helper. Docker helpers now exist for PowerShell and POSIX shells, but remain optional local reproduction only. |
-| Docker Linux targeted stress | Node 24 Docker passed 20 repeated runs of `webrtc/RTCDataChannel-close.html#Repeated open/send/echo/close datachannel works` with retries=0 after the remote-close message-grace change. |
-| Superseded full WPT artifacts | Earlier Docker Linux Node 20 and Node 22 artifacts reached 620/620 with retries=0, and a later Node 24 full run reached 619/620 before the close-race fix. These predate the current close-path change and must not be treated as current full-suite evidence. |
-| Local Docker CI | `scripts/run-docker-linux-ci.ps1` documents a reproducible Linux CI slice for Docker Desktop/WSL and rewrites Debian image apt sources to pinned snapshot URLs to reduce mirror instability. |
-
-## Requirement Status
-
-| Requirement | Current status |
-| --- | --- |
-| Phase 0 analysis before coding | Satisfied by `docs/phase0-analysis.md`, including upstream files reviewed, lifecycle/state/callback analysis, mismatch analysis, binding design, and WPT subset plan. |
-| Data-channel-first WebRTC package | Implemented in `lib/index.js`, `src/native/addon.cc`, `index.d.ts`, and tested by local/WPT gates. |
-| Node-API/N-API, no direct V8 addon API | Locally verified by `npm run native:check`; native source uses node-addon-api and `NODE_API_MODULE`. |
-| Reproducible libdatachannel integration | Implemented in `CMakeLists.txt` with upstream commit `502ae351495792192ef21788e093b48e34ab393e`, including the OpenSSL DTLS and TLS input BIO synchronization fixes from upstream PRs #1584 and #1585; repository and commit are verified by `native:check`. |
-| W3C-compatible JS facade | Covered by API/type checks, local tests, targeted WPT, and targeted Docker stress. Fresh full selected-WPT evidence is still pending after the latest close-path change. |
-| RTCDataChannel selected WPT coverage | Targeted close/send/datachannel coverage is green locally; Node 24 Docker close-race stress is green. Fresh full 620-subtest evidence is still required. |
-| RTCPeerConnection selected WPT coverage | Targeted datachannel and state coverage is green locally. Fresh full 620-subtest evidence is still required. |
-| Safe callback dispatch | Locally verified by `native:check`; native callbacks dispatch through a thread-safe function. |
-| Safe object lifetime | Covered by local tests and selected WPT close/GC cases; still needs continued stress coverage as the API expands. |
-| TypeScript declarations | `index.d.ts` checked by `npm run types:check` and API surface verification. |
-| CI builds/tests/WPT/report | Workflow exists in `.github/workflows/ci.yml` for Linux, macOS, and Windows on Node 20/22/24. Each matrix job writes `ci-evidence.json` and uploads it with WPT artifacts. A final `verify-ci-evidence` job downloads all matrix artifacts and runs `npm run ci:evidence:check`. |
-
-## Current Known Gap
-
-Fresh hosted selected-WPT evidence is still pending after the latest close-path
-message-grace change. GitHub Actions is the authoritative conformance gate for
-the public repository. Docker Linux runs are useful for local reproduction, but
-they are no longer treated as release-blocking evidence because they cannot
-prove macOS or Windows behavior.
-
-## Remaining Completion Evidence
-
-The active goal should not be marked complete until hosted CI or equivalent
-authoritative logs prove the full matrix:
-
-- `ubuntu-latest` on Node 20, 22, and 24
-- `macos-latest` on Node 20, 22, and 24
-- `windows-latest` on Node 20, 22, and 24
-
-The Quality job must pass `npm ci`, `check`, `types:check`, and `pack:check`.
-Each matrix job must pass `npm ci`, `native:check`, `build`, `test`, `api:check`,
-`types:check`, `wpt:ensure`, `wpt:selection:check`, `wpt:test:sharded`,
-`wpt:check:strict`, `wpt:report`, and `ci:evidence`.
-
-After downloading all workflow artifacts into `ci-artifacts/`, run
-`npm run ci:evidence:check`. The verifier requires `ci-evidence.json`,
-`wpt-results.json`, `wpt-report.md`, `wpt-manifest.json`, and
-`wpt-manifest.txt` for each OS/Node matrix entry and rejects missing jobs,
-pin mismatches, WPT failures, and WPT retries.
-
-The GitHub Actions workflow also runs this verifier automatically in the
-`verify-ci-evidence` job. That job uses `always()` so failed or incomplete
-matrix runs are reported as missing or non-green evidence instead of leaving the
-final conformance verifier skipped.
-
-Local Docker evidence is useful before pushing, but it only proves the Linux
-Node image used by `scripts/run-docker-linux-ci.ps1`. It does not replace the
-required macOS and Windows hosted matrix evidence.
+This document records authoritative hosted evidence and the limits of local
+validation. Generated artifacts and local checkouts are not committed.
+
+## Hosted Conformance Evidence
+
+GitHub Actions Conformance run `27392464467` completed successfully on
+2026-06-12. It tested PR #11 head
+`f4c9edf438291e432fcc024cea80198abfe08717`, which was squash-merged as
+`e6a3cfca4beee3163806908c433807354f384c42`.
+
+The run completed:
+
+- the Quality job;
+- Linux, macOS, and Windows on Node.js 20, 22, and 24;
+- all 620 selected WPT subtests with strict retry rejection;
+- the final `Verify CI evidence` job.
+
+The run is available at:
+`https://github.com/mertushka/webrtc-node/actions/runs/27392464467`.
+
+This evidence applies to the tested commit. Later WebRTC semantic, native,
+lifecycle, SDP, ICE, buffering, or event-timing changes require new applicable
+conformance evidence.
+
+## Conformance Contract
+
+`wpt-manifest.json` is the selected compatibility contract. It pins:
+
+- the libdatachannel commit;
+- the WPT commit;
+- the expected selected subtest count;
+- a SHA-256 digest of the sorted `{file, name}` test identities.
+
+`npm run wpt:selection:check` discovers the selected tests without executing
+them and rejects count, identity, duplicate, or digest changes. Updating the
+digest requires deliberate review of the changed selection.
+
+## Workflow Evidence
+
+`.github/workflows/conformance.yml` runs the full matrix separately from normal
+push and pull-request CI. Each matrix job produces:
+
+- `ci-evidence.json`;
+- `wpt-results.json`;
+- `wpt-report.md`;
+- `wpt-manifest.json`;
+- `wpt-manifest.txt`.
+
+The final evidence verifier requires every OS and Node.js matrix entry,
+recomputes WPT status and retry counts, rejects duplicate or inconsistent test
+identities, verifies manifest equality, and binds all artifacts to one GitHub
+workflow run and commit.
+
+After downloading artifacts into `ci-artifacts/`, maintainers can run:
+
+```sh
+npm run ci:evidence:check
+```
+
+## Local Validation Boundary
+
+Focused local tests and Docker runs are useful for development and
+reproduction. They do not replace hosted macOS and Windows evidence. The full
+selected WPT suite is intentionally separate from ordinary local and push CI
+because of its runtime cost.

scripts/check-ci-evidence.js

@@ -2,17 +2,32 @@
 
 const fs = require("node:fs");
 const path = require("node:path");
+const { isDeepStrictEqual } = require("node:util");
+const { wptSelectionDigest } = require("./wpt-sharding");
 
 const root = path.resolve(__dirname, "..");
 const args = process.argv.slice(2);
 const artifactsIndex = args.indexOf("--artifacts");
+const manifestIndex = args.indexOf("--manifest");
 const artifactsRoot =
   artifactsIndex === -1
     ? path.join(root, "ci-artifacts")
     : path.resolve(root, args[artifactsIndex + 1] || "");
-const manifestPath = path.join(root, "wpt-manifest.json");
+const manifestPath =
+  manifestIndex === -1
+    ? path.join(root, "wpt-manifest.json")
+    : path.resolve(root, args[manifestIndex + 1] || "");
 const requiredOs = ["Linux", "macOS", "Windows"];
 const requiredNodeMajors = [20, 22, 24];
+const requiredGithubFields = ["workflow", "job", "runId", "runAttempt", "repository", "ref", "sha"];
+const currentGithub = {
+  workflow: process.env.GITHUB_WORKFLOW,
+  runId: process.env.GITHUB_RUN_ID,
+  runAttempt: process.env.GITHUB_RUN_ATTEMPT,
+  repository: process.env.GITHUB_REPOSITORY,
+  ref: process.env.GITHUB_REF,
+  sha: process.env.GITHUB_SHA,
+};
 
 function fail(message) {
   console.error(`CI evidence check failed: ${message}`);
@@ -41,7 +56,116 @@ function nodeMajor(version) {
   return match ? Number(match[1]) : null;
 }
 
+function validateResults(results, key) {
+  if (!Array.isArray(results.results)) fail(`${key} WPT result artifact is invalid`);
+  if (!Number.isInteger(results.total) || results.total < 1) {
+    fail(`${key} WPT total is invalid`);
+  }
+  if (!Number.isInteger(results.pass) || results.pass < 0) {
+    fail(`${key} WPT pass count is invalid`);
+  }
+  if (!Number.isInteger(results.fail) || results.fail < 0) {
+    fail(`${key} WPT fail count is invalid`);
+  }
+  if (results.results.length !== results.total) fail(`${key} result length mismatch`);
+  if (results.total !== manifest.expectedSelectedSubtests) fail(`${key} WPT total mismatch`);
+
+  let pass = 0;
+  let failCount = 0;
+  let retries = 0;
+  const identities = new Set();
+  const files = new Set();
+
+  for (const result of results.results) {
+    if (
+      !result ||
+      typeof result.file !== "string" ||
+      result.file.length === 0 ||
+      typeof result.name !== "string" ||
+      result.name.length === 0
+    ) {
+      fail(`${key} contains an invalid WPT result identity`);
+    }
+
+    const identity = `${result.file}\0${result.name}`;
+    if (identities.has(identity))
+      fail(`${key} contains duplicate WPT result ${result.file}#${result.name}`);
+    identities.add(identity);
+    files.add(result.file);
+
+    if (result.status === "PASS") pass += 1;
+    else if (result.status === "FAIL") failCount += 1;
+    else fail(`${key} contains unexpected WPT status ${result.status}`);
+
+    const retryCount = result.retries === undefined ? 0 : result.retries;
+    if (!Number.isInteger(retryCount) || retryCount < 0) {
+      fail(`${key} contains an invalid retry count for ${result.file}#${result.name}`);
+    }
+    if (retryCount > 0) retries += 1;
+  }
+
+  if (results.pass !== pass) fail(`${key} WPT pass summary mismatch`);
+  if (results.fail !== failCount) fail(`${key} WPT fail summary mismatch`);
+  if (failCount !== 0 || pass !== results.total || retries !== 0) {
+    fail(
+      `${key} WPT is not strict-green: pass=${pass} total=${results.total} fail=${failCount} retries=${retries}`,
+    );
+  }
+
+  const selectedSubtestsSha256 = wptSelectionDigest(identities);
+  if (selectedSubtestsSha256 !== manifest.selectedSubtestsSha256) {
+    fail(`${key} WPT result identities do not match the manifest digest`);
+  }
+
+  return {
+    pass,
+    failCount,
+    retries,
+    identities,
+    fileCount: files.size,
+    selectedSubtestsSha256,
+  };
+}
+
+function validateGithubEvidence(evidence, key, baseline) {
+  if (evidence.source !== "write-ci-evidence.js") fail(`${key} evidence source is invalid`);
+  if (evidence.github?.actions !== true) fail(`${key} is not GitHub Actions evidence`);
+
+  for (const field of requiredGithubFields) {
+    if (typeof evidence.github[field] !== "string" || evidence.github[field].length === 0) {
+      fail(`${key} evidence GitHub ${field} is missing`);
+    }
+  }
+
+  if (baseline) {
+    for (const field of requiredGithubFields) {
+      if (evidence.github[field] !== baseline[field]) {
+        fail(`${key} evidence GitHub ${field} does not match the matrix run`);
+      }
+    }
+  }
+
+  if (process.env.GITHUB_ACTIONS === "true") {
+    for (const [field, expected] of Object.entries(currentGithub)) {
+      if (!expected || evidence.github[field] !== expected) {
+        fail(`${key} evidence GitHub ${field} does not match the current workflow run`);
+      }
+    }
+  }
+
+  return evidence.github;
+}
+
+function sameSet(left, right) {
+  if (left.size !== right.size) return false;
+  for (const value of left) {
+    if (!right.has(value)) return false;
+  }
+  return true;
+}
+
 if (artifactsIndex !== -1 && !args[artifactsIndex + 1]) fail("--artifacts requires a directory");
+if (manifestIndex !== -1 && !args[manifestIndex + 1]) fail("--manifest requires a file");
 if (!fs.existsSync(artifactsRoot)) {
   fail(`${artifactsRoot} does not exist; download CI artifacts there or pass --artifacts <dir>`);
 }
@@ -53,6 +177,8 @@ const evidenceFiles = walk(artifactsRoot);
 if (!evidenceFiles.length) fail(`no ci-evidence.json files found under ${artifactsRoot}`);
 
 const byMatrix = new Map();
+let githubBaseline = null;
+let identityBaseline = null;
 
 for (const evidencePath of evidenceFiles) {
   const evidence = readJson(evidencePath);
@@ -72,39 +198,42 @@ for (const evidencePath of evidenceFiles) {
   for (const requiredPath of [resultsPath, reportPath, artifactManifestPath, manifestTextPath]) {
     if (!fs.existsSync(requiredPath)) fail(`${path.relative(root, requiredPath)} is missing`);
   }
+  for (const requiredPath of [reportPath, manifestTextPath]) {
+    if (fs.statSync(requiredPath).size === 0) {
+      fail(`${path.relative(root, requiredPath)} is empty`);
+    }
+  }
 
   const artifactManifest = readJson(artifactManifestPath);
   const results = readJson(resultsPath);
-  const retries = Array.isArray(results.results)
-    ? results.results.filter((result) => Number(result.retries) > 0).length
-    : null;
 
-  if (artifactManifest.libdatachannelCommit !== manifest.libdatachannelCommit) {
-    fail(`${key} libdatachannel pin mismatch`);
+  if (!isDeepStrictEqual(artifactManifest, manifest)) {
+    fail(`${key} WPT manifest does not match the repository manifest`);
   }
-  if (artifactManifest.wptCommit !== manifest.wptCommit) fail(`${key} WPT pin mismatch`);
-  if (artifactManifest.expectedSelectedSubtests !== manifest.expectedSelectedSubtests) {
-    fail(`${key} selected subtest count mismatch`);
-  }
-  if (!Array.isArray(results.results)) fail(`${key} WPT result artifact is invalid`);
-  if (results.results.length !== results.total) fail(`${key} result length mismatch`);
-  if (results.total !== manifest.expectedSelectedSubtests) fail(`${key} WPT total mismatch`);
-  if (results.pass !== results.total || results.fail !== 0 || retries !== 0) {
-    fail(
-      `${key} WPT is not strict-green: pass=${results.pass} total=${results.total} fail=${results.fail} retries=${retries}`,
-    );
+
+  const validated = validateResults(results, key);
+  if (identityBaseline && !sameSet(validated.identities, identityBaseline)) {
+    fail(`${key} WPT result identities do not match the matrix run`);
   }
+  identityBaseline ??= validated.identities;
+
+  const github = validateGithubEvidence(evidence, key, githubBaseline);
+  githubBaseline ??= github;
+
   if (evidence.pins?.libdatachannel !== manifest.libdatachannelCommit) {
     fail(`${key} evidence libdatachannel pin mismatch`);
   }
   if (evidence.pins?.wpt !== manifest.wptCommit) fail(`${key} evidence WPT pin mismatch`);
   if (
+    evidence.wpt?.expectedSelectedSubtests !== manifest.expectedSelectedSubtests ||
     evidence.wpt?.total !== manifest.expectedSelectedSubtests ||
-    evidence.wpt?.pass !== manifest.expectedSelectedSubtests ||
-    evidence.wpt?.fail !== 0 ||
-    evidence.wpt?.retries !== 0
+    evidence.wpt?.pass !== validated.pass ||
+    evidence.wpt?.fail !== validated.failCount ||
+    evidence.wpt?.retries !== validated.retries ||
+    evidence.wpt?.resultFiles !== validated.fileCount ||
+    evidence.wpt?.selectedSubtestsSha256 !== validated.selectedSubtestsSha256
   ) {
-    fail(`${key} evidence WPT summary is not strict-green`);
+    fail(`${key} evidence WPT summary does not match the result artifact`);
   }
 
   byMatrix.set(key, { os, major, evidencePath });