Skip to content

Replace the serialized retrieval generalization wrapper with one bounded Node suite #1601

Description

@TheGreenCedar

Context

CodeStory’s retrieval generalization lint prevents benchmark names, holdout paths, and expected-answer shapes from entering production retrieval behavior. Its production scan is sub-minute, but crates/codestory-runtime/tests/retrieval_generalization_guard.rs wraps the lint in 48 Rust tests that mostly start a new Node process.

Those tests share checkout mutations behind a cross-process file lock. cargo-nextest starts them concurrently, so they occupy workers while waiting for the lock and starve unrelated workspace tests. Existing CI evidence shows the Rust wrapper taking 10m25s in the dedicated retrieval job and dominating a 38m33s workspace test phase, while the production lint itself took 39s and the existing Node contract tests took 19s.

This issue owns the first release-blocking repair to that test architecture. It starts from dev/codestory-next at 10344e9b3e871ce0a5b4c0b2a70201aede5dfa28 and must not modify the frozen calibration branch.

Refs #1179

Outcome

Run the full hostile generalization fixture matrix once in an isolated Node process without checkout mutation or global serialization, while retaining one read-only smoke against the real repository.

Scope

  • Extract the lint implementation behind an importable Node module and keep the command-line entry point thin.
  • Load the benchmark corpus and derive policy once per Node test process.
  • Move every hostile fixture into temporary synthetic trees and execute the matrix in one Node test suite.
  • Keep one read-only production-repository smoke.
  • Remove the Rust subprocess wrapper from the workspace nextest graph.
  • Run the Node matrix as a separate source-proof lane in parallel with Rust workspace tests.
  • Update workflow policy and its mutation tests for every workflow change; update release-claims.json only if the job or claim graph changes.

Acceptance criteria

  • The hostile fixture matrix covers every contract currently pinned by retrieval_generalization_guard.rs, including environment inheritance, scan-root containment, task-root policy, non-Rust surfaces, benchmark self-exemption, and stale-fixture recovery without touching the checkout.
  • One Node process loads the corpus/policy once and evaluates all synthetic cases from temporary roots.
  • One read-only smoke runs the actual repository scan and rejects any attempted checkout write.
  • cargo nextest run --workspace --all-targets --all-features no longer builds or executes the Rust generalization wrapper.
  • The dedicated Node suite completes in at most 90 seconds on hosted Linux; its own work stays under five minutes locally.
  • Source proof schedules the Node suite independently of the Rust workspace test phase so it cannot occupy Rust test workers.
  • An independent execution pass mutates the shape around the migrated examples and proves the suite fails before each relevant protection is restored.

Verification

  • Focused Node contract and hostile-fixture suite with measured wall time.
  • Production lint once against the real repository.
  • Workflow-policy script and mutation tests.
  • Retrieval smoke on the exact head.
  • One full exact-head source proof after independent review; do not repeat it for an unchanged commit.

Non-goals

  • Do not weaken or delete any generalization ban to gain speed.
  • Do not add runners, caches, or test sharding as a substitute for removing serialization.
  • Do not redesign reverse-dependency CI selection, package/hardware proof reuse, or repository-wide duration reporting in this PR.
  • Do not touch product retrieval behavior or the active calibration source.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingphase:promotion-readinessPromotion readiness and post-contract proof wavepriority:p1Release blocker or high-risk reliability/security issuerisk:publishable-blockerBlocks publishable benchmark or product claim readinesstype:harnessBenchmark or verification harness work

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions