diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4d376817..e63ff709 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,7 +37,9 @@ jobs: run: npm run typecheck:surface - name: 'Gate 6: Markdown lint (fenced code blocks require language)' run: npm run lint:md - - name: 'Gate 7: Security audit (runtime deps, advisory)' + - name: 'Gate 7: Markdown JS/TS code-sample syntax check' + run: npm run lint:md:code + - name: 'Gate 8: Security audit (runtime deps, advisory)' continue-on-error: true run: npm audit --omit=dev --audit-level=high diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c92e921..2876c3c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,21 +14,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Fluent `WarpStateV5` test builder** — Added `createStateBuilder()` in `test/helpers/stateBuilder.js` so state-heavy tests can seed nodes, edges, removals, properties, frontier state, and graph materialization through one fluent helper instead of ad hoc OR-Set/LWW mutation. - **Seeded tree-construction determinism fuzzer** — Added property-based coverage for patch and checkpoint tree construction, proving stable tree OIDs across internal content-anchor permutations in `PatchBuilderV2` and shuffled content-property insertion order in `CheckpointService.createV5()`. - **Focused markdownlint gate** — Added `npm run lint:md` backed by `markdownlint-cli` and a repo config that enforces fenced code-block languages (`MD040`) across Markdown files. +- **Markdown JS/TS code-sample linter** — Added `npm run lint:md:code`, which scans fenced JavaScript and TypeScript blocks in Markdown and syntax-checks them with the TypeScript parser for file/line-accurate diagnostics. ### Changed - **Large-graph traversal memory profile** — `topologicalSort()` now has a lightweight mode that avoids retaining discovery adjacency when callers do not need it. `levels()` and `transitiveReduction()` were refactored to re-fetch neighbors on demand instead of pinning full topo adjacency in memory, reducing steady-state large-graph working sets. - **Surface validation accounting** — The declaration surface checker now distinguishes runtime-backed exports from type-only manifest entries and understands namespace declarations, which makes the type-surface contract tighter without forcing runtime exports for pure types. -- **Local push firewall now matches CI surface checks** — `scripts/hooks/pre-push` now runs `npm run typecheck:surface` alongside lint, strict typecheck, policy, and consumer surface checks before unit tests, so declaration-surface drift is blocked locally instead of waiting for CI. +- **Local push firewall now matches CI surface and docs checks** — `scripts/hooks/pre-push` now runs `npm run typecheck:surface`, `npm run lint:md`, and `npm run lint:md:code` alongside lint, strict typecheck, policy, and consumer surface checks before unit tests, so declaration-surface drift and Markdown sample regressions are blocked locally instead of waiting for CI. - **Trust test infrastructure deduplicated** — The TrustRecordService suites now share a single in-memory ref/blob/tree/commit fixture and JSON codec via `test/helpers/trustTestUtils.js`, eliminating the four forked mock implementations that had started to drift. - **Explicit type-only export manifest section** — `type-surface.m8.json` now separates runtime `exports` from declaration-only `typeExports`, and the surface checker now fails on misplaced or duplicate entries across those sections. - **Constructor option-bag defaults made explicit** — Added an ESLint rule banning `constructor({ ... } = {})` in source files and rewrote the remaining constructors to destructure an explicit `options` bag inside the constructor body. This avoids accidentally marking required constructor params optional in JSDoc and strict type checking. - **Checkpoint content-anchor batching** — `CheckpointService.createV5()` now folds content blob OIDs into sorted anchor entries in batches instead of building one monolithic `Set` before tree serialization. Added direct checkpoint coverage for anchor dedupe, deterministic ordering, and load-path indifference to `_content_*` anchor entries. - **CI gate dedupe** — Folded the duplicate `lint` workflow job into `type-firewall` and carried forward the advisory runtime `npm audit` step there, leaving one authoritative lint/type gate in the main CI workflow. - **Markdown fence labeling sweep** — Unlabeled Markdown code fences now declare a language such as `text`, letting the new markdown gate verify docs/examples without broad style-rule churn. +- **Docs static firewall extended** — The CI fast gate now runs both markdown fence-language checks and JavaScript/TypeScript code-sample syntax validation before the runtime matrix jobs. ### Fixed +- **Deno CI resolver drift** — The Deno test image now imports a Node 22 npm toolchain from `node:22-slim`, installs dependencies with `npm ci`, and runs tests with `--node-modules-dir=manual`, avoiding runtime npm re-resolution of `cbor-extract` optional platform packages while keeping the container on the repo’s supported Node engine line. +- **Markdown code-sample linter edge cases** — The Markdown JS/TS sample linter now recognizes fenced code blocks indented by up to three spaces, rejects malformed mixed-marker fences, fails on unterminated JS/TS fences, and parses snippets with the repository’s configured TypeScript target from `tsconfig.base.json`. +- **B87 review follow-ups** — Clarified the ADR folds snippet as a wholly proposed `graph.view()` sketch, corrected the pre-push quick-mode gate label to Gate 8, aligned the local hook’s gate numbers with CI for faster failure triage, and removed the self-expiring `pending merge` wording from the completed-roadmap archive entry. - **Signed trust verification now performs real crypto checks** — Trust evaluation now verifies Ed25519 signatures and key fingerprints during evidence processing instead of stopping at envelope/shape validation. - **Browser/WebSocket serve payload parity for edge properties** — `WarpServeService` state payloads now include edge properties, so served graph views no longer drop part of the graph model. - **`attachContent()` / `attachEdgeContent()` orphan blob writes** — Content attachment now validates the target node/edge before writing blob content, preventing orphaned blob storage on invalid mutations. diff --git a/ROADMAP.md b/ROADMAP.md index 74a190fd..881d5664 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -204,7 +204,7 @@ P1 is complete on `v15`: B36 and B37 landed as the shared test-foundation pass, ### P2 — CI & Tooling (one batch PR) -`B83`, `B85`, `B57`, and `B86` are complete on `v15`: the redundant `lint` workflow job was folded into `type-firewall`, the declaration surface manifest now splits runtime `exports` from type-only `typeExports`, the local pre-push firewall now runs `typecheck:surface` alongside the other type gates, and CI now runs markdownlint for fenced-code language coverage. Remaining P2 work now starts at B87. B123 is still the largest item and may need to split out if the PR gets too big. +`B83`, `B85`, `B57`, and `B86` are already merged. `B87` is complete in `feature/b87-markdown-code-lint`: the new script syntax-checks fenced JavaScript/TypeScript samples in Markdown, the CI fast gate runs it after markdownlint, the local pre-push firewall mirrors that coverage, and malformed/unterminated JS/TS fences now fail with file/line diagnostics. If this branch lands, remaining P2 work starts at B88. B123 is still the largest item and may need to split out if the PR gets too big. | ID | Item | Depends on | Effort | | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------ | @@ -212,7 +212,7 @@ P1 is complete on `v15`: B36 and B37 landed as the shared test-foundation pass, | B85 | ✅ **TYPE-ONLY EXPORT MANIFEST SECTION** — Added explicit `typeExports` to `type-surface.m8.json` and taught `check-dts-surface` to fail on misplaced or duplicate entries across `exports` and `typeExports`, so type-only declaration drift is validated directly instead of inferred from `kind`. | B97 (P0) | S | | B57 | ✅ **AUTO-VALIDATE `type-surface.m8.json` AGAINST `index.d.ts`** — `typecheck:surface` now runs in CI, release preflight, and the local `scripts/hooks/pre-push` firewall, so declaration-surface drift is blocked before push instead of only after CI starts. | B97, B85 | M | | B86 | ✅ **MARKDOWNLINT CI GATE** — Added `npm run lint:md` with focused `MD040` enforcement and wired it into `.github/workflows/ci.yml`, so fenced code blocks in Markdown must declare a language before CI passes. | — | S | -| B87 | **CODE SAMPLE LINTER** — syntax-check JS/TS code blocks in markdown files via `eslint-plugin-markdown` or custom extractor. From B-DOC-2. **Files:** new script, `docs/**/*.md` | — | M | +| B87 | ✅ **CODE SAMPLE LINTER** — Added `scripts/lint-markdown-code-samples.js` plus `npm run lint:md:code`, which extracts fenced JavaScript/TypeScript samples from Markdown and syntax-checks them with file/line diagnostics. Wired into `.github/workflows/ci.yml` and the local `scripts/hooks/pre-push` firewall, with explicit failures for malformed mixed-marker fences and unterminated JS/TS blocks. | — | M | | B88 | **MERMAID RENDERING SMOKE TEST** — parse all ` ```mermaid ` blocks with `@mermaid-js/mermaid-cli` in CI. From B-DIAG-2. **File:** `.github/workflows/ci.yml` or `scripts/` | — | S | | B119 | **`scripts/pr-ready` MERGE-READINESS CLI** — single tool aggregating unresolved review threads, pending/failed checks, CodeRabbit status/cooldown, and human-review count into one deterministic verdict. From BACKLOG 2026-02-27/28. | — | M | | B123 | **BENCHMARK BUDGETS + CI REGRESSION GATE** — define perf thresholds for eager post-commit and materialize hash cost; fail CI on agreed regression. From BACKLOG 2026-02-27. | — | L | @@ -335,9 +335,9 @@ Complete on `v15`: **B80** and **B99**. #### Wave 2: CI & Tooling (P2, one batch PR) -3. **B87, B88, B119, B123, B128, B12, B43** +3. **B88, B119, B123, B128, B12, B43** -Internal chain: **B97 already resolved on v15** → B85 → B57. That chain is complete on `v15`; B86 closed the first docs-quality gate in the remaining P2 pack. B123 remains the largest remaining item and may need to split out. +Internal chain: **B97 already resolved** → B85 → B57. That chain is complete, and this branch adds B87 on top of the existing B86 markdown gate to cover JS/TS sample syntax. B123 remains the largest remaining item and may need to split out. #### Wave 3: Type Surface (P3) @@ -395,11 +395,11 @@ B158 (P7) ──→ B159 (P7) CDC seek cache | **Milestone (M12)** | 18 | B66, B67, B70, B73, B75, B105–B115, B117, B118 | | **Milestone (M13)** | 1 | B116 (internal: DONE; wire-format: DEFERRED) | | **Milestone (M14)** | 16 | B130–B145 | -| **Standalone** | 25 | B12, B28, B34–B35, B43, B53, B54, B76, B79, B87–B88, B96, B98, B102–B104, B119, B123, B127–B129, B147, B152, B155–B156 | -| **Standalone (done)** | 60 | B19, B22, B26, B36–B37, B44, B46, B47, B48–B52, B55, B57, B71, B72, B77, B78, B80–B86, B89–B95, B97, B99–B100, B120–B122, B124, B125, B126, B146, B148–B151, B153, B154, B157–B165, B167 | +| **Standalone** | 24 | B12, B28, B34–B35, B43, B53, B54, B76, B79, B88, B96, B98, B102–B104, B119, B123, B127–B129, B147, B152, B155–B156 | +| **Standalone (done)** | 61 | B19, B22, B26, B36–B37, B44, B46, B47, B48–B52, B55, B57, B71, B72, B77, B78, B80–B87, B89–B95, B97, B99–B100, B120–B122, B124, B125, B126, B146, B148–B151, B153, B154, B157–B165, B167 | | **Deferred** | 7 | B4, B7, B16, B20, B21, B27, B101 | | **Rejected** | 7 | B5, B6, B13, B17, B18, B25, B45 | -| **Total tracked** | **144** total; 60 standalone done | | +| **Total tracked** | **144** total; 61 standalone done | | ### STANK.md Cross-Reference @@ -503,7 +503,7 @@ B158 (P7) ──→ B159 (P7) CDC seek cache Every milestone has a hard gate. No milestone blurs into the next. All milestones are complete: M10 → M12 → M13 (internal) → M11 → M14. M13 wire-format cutover remains deferred by ADR 3 readiness gates. -The active backlog is **25 standalone items** sorted into **8 priority tiers** (P0–P7) with **6 execution waves**. Wave 1 is complete on `v15`, and Wave 2 now starts at B87 in the CI & Tooling pack. See [Execution Order](#execution-order) for the full sequence. +The active backlog is **24 standalone items** sorted into **8 priority tiers** (P0–P7) with **6 execution waves**. Wave 1 is complete, and with B87 in this branch, Wave 2 now starts at B88 in the CI & Tooling pack. See [Execution Order](#execution-order) for the full sequence. Rejected items live in `GRAVEYARD.md`. Resurrections require an RFC. `BACKLOG.md` retired — all intake goes directly into this file (policy in `CLAUDE.md`). diff --git a/docker-compose.test.yml b/docker-compose.test.yml index 6ec20fb8..817a1a1f 100644 --- a/docker-compose.test.yml +++ b/docker-compose.test.yml @@ -52,5 +52,5 @@ services: environment: - GIT_STUNTS_DOCKER=1 # Keep allow-scripts package names aligned with package.json dependencies. - command: ["deno", "test", "--allow-all", "--allow-scripts=npm:roaring,npm:cbor-extract", "--node-modules-dir=auto", "test/runtime/deno/"] + command: ["deno", "test", "--allow-all", "--allow-scripts=npm:roaring,npm:cbor-extract", "--node-modules-dir=manual", "test/runtime/deno/"] profiles: [deno, full] diff --git a/docker/Dockerfile.deno b/docker/Dockerfile.deno index 374a89c8..0526f8ee 100644 --- a/docker/Dockerfile.deno +++ b/docker/Dockerfile.deno @@ -2,18 +2,30 @@ # Runs: API integration tests using Deno.test() wrappers. # CLI and unit tests are excluded — they depend on vitest / node: built-ins. # Build context is the parent monorepo directory (context: ..). +FROM node:22-slim AS node22 + FROM denoland/deno:2.1.9 ARG DENO_ALLOW_SCRIPTS="npm:roaring,npm:cbor-extract" USER root +COPY --from=node22 /usr/local/bin/node /usr/local/bin/node +COPY --from=node22 /usr/local/bin/npm /usr/local/bin/npm +COPY --from=node22 /usr/local/bin/npx /usr/local/bin/npx +COPY --from=node22 /usr/local/lib/node_modules /usr/local/lib/node_modules RUN apt-get update && apt-get install -y --no-install-recommends \ git \ python3 \ make \ g++ \ - # Needed on PATH for Deno npm lifecycle fallback builds (roaring/cbor-extract). - node-gyp \ && rm -rf /var/lib/apt/lists/* +RUN ln -sf /usr/local/lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm \ + && ln -sf /usr/local/lib/node_modules/npm/bin/npx-cli.js /usr/local/bin/npx \ + # Needed on PATH for Deno npm lifecycle fallback builds (roaring/cbor-extract). + && ln -sf /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js /usr/local/bin/node-gyp WORKDIR /app +COPY git-warp/package*.json ./ +COPY git-warp/scripts ./scripts +COPY git-warp/patches ./patches +RUN npm ci COPY git-warp/ . # Init a git repo so plumbing operations work inside the container. RUN git init -q \ @@ -21,21 +33,10 @@ RUN git init -q \ && git config user.name "Git Warp Container" \ && git add -A \ && git commit --allow-empty -m "seed git-warp" >/dev/null -# Install npm dependencies for Deno test entrypoints and allow native addon -# lifecycle scripts so roaring/cbor-extract are prepared before test execution. -RUN printf '%s\n' \ - "import 'npm:roaring';" \ - "import 'npm:cbor-extract';" \ - > /tmp/deno-npm-bootstrap.ts \ - && deno install \ - --allow-scripts=${DENO_ALLOW_SCRIPTS} \ - --node-modules-dir=auto \ - --entrypoint /tmp/deno-npm-bootstrap.ts \ - && rm /tmp/deno-npm-bootstrap.ts # Run tests as non-root to mirror CI and catch permission issues. # Also chown Deno's global cache so the deno user can fetch npm packages at runtime. RUN chown -R deno:deno /app /deno-dir USER deno ENV GIT_STUNTS_DOCKER=1 ENV DENO_ALLOW_SCRIPTS=${DENO_ALLOW_SCRIPTS} -CMD ["sh", "-lc", "deno test --allow-all --allow-scripts=${DENO_ALLOW_SCRIPTS} --node-modules-dir=auto test/runtime/deno/"] +CMD ["sh", "-lc", "deno test --allow-all --allow-scripts=${DENO_ALLOW_SCRIPTS} --node-modules-dir=manual test/runtime/deno/"] diff --git a/docs/ADR-001-Folds.md b/docs/ADR-001-Folds.md index 7871b784..e3a3a95f 100644 --- a/docs/ADR-001-Folds.md +++ b/docs/ADR-001-Folds.md @@ -119,12 +119,14 @@ Folds should be exposed as explicit view operations, not implicit traversal surp #### 3.1 View API -```javascript -const view = graph.view({ fold: { mode: "shallow", maxDepth: 1 } }); +_Proposed API sketch: the entire `graph.view()` flow shown below, including `view.traverse()`, `view.query().match().run()`, and `view.renderAscii()`, is not implemented yet._ + +```ts +const view = graph.view({ fold: { mode: 'shallow', maxDepth: 1 } }); await view.traverse(startNodeId); -await view.query(...); -await view.renderAscii(...); +await view.query().match('doc:*').run(); +await view.renderAscii(); ``` `graph.view()` returns a wrapper that: @@ -134,7 +136,7 @@ await view.renderAscii(...); #### 3.2 Attachment Graph Accessors -```javascript +```text // returns fold root id (even if fold is empty) graph.getFoldRootForNode(nodeId) -> string diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 63387533..38cd5ab9 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -371,7 +371,7 @@ const result = await graph.query() **Object shorthand** — strict equality on primitive values. Multiple properties use AND semantics: -```javascript +```text .where({ role: 'admin' }) .where({ role: 'admin', active: true }) .where({ status: null }) @@ -379,7 +379,7 @@ const result = await graph.query() **Function form** — arbitrary predicates: -```javascript +```text .where(({ props }) => props.age >= 18) .where(({ edgesOut }) => edgesOut.length > 0) ``` @@ -400,7 +400,7 @@ const result = await graph.query() `outgoing()` and `incoming()` follow edges with optional depth control: -```javascript +```text // Single hop (default) .outgoing('manages') diff --git a/docs/ROADMAP/COMPLETED.md b/docs/ROADMAP/COMPLETED.md index 62febf98..11329dd8 100644 --- a/docs/ROADMAP/COMPLETED.md +++ b/docs/ROADMAP/COMPLETED.md @@ -396,6 +396,7 @@ Investigation revealed the correct approach is a two-phase split: | B83 | ~~**DEDUP CI `type-firewall` AND `lint` JOBS**~~ — **DONE (`v15`).** Folded the duplicate `lint` workflow job into `type-firewall` and carried forward the advisory runtime `npm audit` step there so CI exposes one authoritative lint/type gate. | | B85 | ~~**TYPE-ONLY EXPORT MANIFEST SECTION**~~ — **DONE (`v15`).** Added explicit `typeExports` to `type-surface.m8.json` and taught `check-dts-surface` to fail on misplaced or duplicate entries across `exports` and `typeExports`. | | B86 | ~~**MARKDOWNLINT CI GATE**~~ — **DONE (`v15`).** Added `npm run lint:md` with focused `MD040` enforcement and wired it into CI, then labeled the repo’s bare fenced code blocks so Markdown docs/examples pass the new gate immediately. | +| B87 | ~~**CODE SAMPLE LINTER**~~ — **DONE.** Added `scripts/lint-markdown-code-samples.js` and `npm run lint:md:code`, which extract fenced JavaScript/TypeScript samples from Markdown and syntax-check them with line-accurate diagnostics. Wired into the CI fast gate and local pre-push firewall alongside markdownlint, with failures for malformed mixed-marker fences and unterminated JS/TS blocks. | | B95 | ~~**NAMESPACE EXPORT SUPPORT**~~ — **DONE (`v15`).** The declaration surface validator now handles namespace exports in `.d.ts` parsing. | | B97 | ~~**AUDIT MANIFEST vs `index.js` DRIFT**~~ — **DONE (`v15`).** Runtime exports were reconciled with the public surface, and the surface checker now distinguishes runtime-backed vs type-only manifest entries. | | B99 | ~~**DETERMINISM FUZZER FOR TREE CONSTRUCTION**~~ — **DONE (`v15`).** Added seeded property tests that prove stable tree OIDs when `PatchBuilderV2` content anchor order is permuted internally and when `CheckpointService.createV5()` sees the same content properties in different insertion orders. | diff --git a/docs/design/MIGRATION_PROBLEM.md b/docs/design/MIGRATION_PROBLEM.md index e42ea4a9..750a2180 100644 --- a/docs/design/MIGRATION_PROBLEM.md +++ b/docs/design/MIGRATION_PROBLEM.md @@ -64,7 +64,7 @@ All three are semantically equivalent, but the inconsistency means a reader must Introduce a new operation type `EdgePropSet` at schema version 4: -```javascript +```text // New op type { type: 'EdgePropSet', diff --git a/docs/design/joinreducer-split.md b/docs/design/joinreducer-split.md index 51918581..cb9fb4d9 100644 --- a/docs/design/joinreducer-split.md +++ b/docs/design/joinreducer-split.md @@ -151,7 +151,7 @@ the more complex usage is; ReceiptBuilder imports it. After extraction, JoinReducer retains: ### Imports -```javascript +```text import { createEventId } from '../utils/EventId.js'; import { normalizeRawOp } from './OpNormalizer.js'; import { createEmptyDiff, mergeDiffs } from '../types/PatchDiff.js'; diff --git a/docs/design/warpgraph-decomposition.md b/docs/design/warpgraph-decomposition.md index 5f27d75d..096a451e 100644 --- a/docs/design/warpgraph-decomposition.md +++ b/docs/design/warpgraph-decomposition.md @@ -300,7 +300,7 @@ Instead of reaching back into the host for every field update, `_setMaterializedState()` returns a result object that the host (WarpGraph) uses to update its own core fields: -```javascript +```text // In CacheCoordinator: setMaterializedState(state, opts) { // ... build adjacency, view, index ... diff --git a/package.json b/package.json index 2e371025..f9fe30ed 100644 --- a/package.json +++ b/package.json @@ -74,6 +74,7 @@ "scripts": { "lint": "eslint .", "lint:md": "markdownlint \"**/*.md\" --ignore node_modules", + "lint:md:code": "node scripts/lint-markdown-code-samples.js", "lint:links": "lychee --config .lychee.toml '**/*.md'", "format": "prettier --write .", "test": "sh -c 'if [ \"$GIT_STUNTS_DOCKER\" = \"1\" ]; then vitest run test/unit \"$@\"; else docker compose run --build --rm test npm run test:local -- \"$@\"; fi' --", diff --git a/scripts/hooks/pre-push b/scripts/hooks/pre-push index cdf6ea1e..a15cf115 100755 --- a/scripts/hooks/pre-push +++ b/scripts/hooks/pre-push @@ -2,7 +2,7 @@ # ═══════════════════════════════════════════════════════════════════════════ # IRONCLAD M9 — pre-push gate # -# Six gates in parallel, then unit tests. ALL must pass or push is blocked. +# Seven gates in parallel, then unit tests. ALL must pass or push is blocked. # This is the last local line of defense before CI. # ═══════════════════════════════════════════════════════════════════════════ set -e @@ -18,7 +18,7 @@ cd "$ROOT" QUICK=0 if [ "$WARP_QUICK_PUSH" = "1" ] || [ "$WARP_QUICK_PUSH" = "true" ]; then QUICK=1 - echo "WARP_QUICK_PUSH: quick mode active — Gate 6 (unit tests) will be skipped" + echo "WARP_QUICK_PUSH: quick mode active — Gate 8 (unit tests) will be skipped" fi echo "══════════════════════════════════════════════════════════" @@ -33,8 +33,8 @@ else echo "[Gate 0] Link check skipped (lychee not installed)" fi -# ── Gates 1-5 in parallel (all are read-only) ───────────────────────────── -echo "[Gates 1-5] Running lint + typecheck + policy + consumer type test + surface validator..." +# ── Gates 1-7 in parallel (all are read-only) ───────────────────────────── +echo "[Gates 1-7] Running lint + typecheck + policy + consumer type test + surface validator + markdown gates..." npm run lint & LINT_PID=$! @@ -46,20 +46,26 @@ npm run typecheck:consumer & CONSUMER_PID=$! npm run typecheck:surface & SURFACE_PID=$! +npm run lint:md & +MD_PID=$! +npm run lint:md:code & +MD_CODE_PID=$! -wait $LINT_PID || { echo ""; echo "BLOCKED — Gate 1 FAILED: ESLint (includes no-explicit-any, no-unsafe-*)"; exit 1; } -wait $TC_PID || { echo ""; echo "BLOCKED — Gate 2 FAILED: TypeScript compiler (strict mode)"; exit 1; } -wait $POLICY_PID || { echo ""; echo "BLOCKED — Gate 3 FAILED: IRONCLAD policy (any/wildcard/ts-ignore ban)"; exit 1; } -wait $CONSUMER_PID || { echo ""; echo "BLOCKED — Gate 4 FAILED: Consumer type surface test"; exit 1; } +wait $LINT_PID || { echo ""; echo "BLOCKED — Gate 4 FAILED: ESLint (includes no-explicit-any, no-unsafe-*)"; exit 1; } +wait $TC_PID || { echo ""; echo "BLOCKED — Gate 1 FAILED: TypeScript compiler (strict mode)"; exit 1; } +wait $POLICY_PID || { echo ""; echo "BLOCKED — Gate 2 FAILED: IRONCLAD policy (any/wildcard/ts-ignore ban)"; exit 1; } +wait $CONSUMER_PID || { echo ""; echo "BLOCKED — Gate 3 FAILED: Consumer type surface test"; exit 1; } wait $SURFACE_PID || { echo ""; echo "BLOCKED — Gate 5 FAILED: Declaration surface validator"; exit 1; } +wait $MD_PID || { echo ""; echo "BLOCKED — Gate 6 FAILED: Markdown lint"; exit 1; } +wait $MD_CODE_PID || { echo ""; echo "BLOCKED — Gate 7 FAILED: Markdown JS/TS code-sample syntax check"; exit 1; } -echo "[Gates 1-5] All type gates passed." +echo "[Gates 1-7] All static gates passed." -# ── Gate 6: Unit tests ───────────────────────────────────────────────────── +# ── Gate 8: Unit tests ───────────────────────────────────────────────────── if [ "$QUICK" = "1" ]; then - echo "[Gate 6] Skipped (WARP_QUICK_PUSH quick mode)" + echo "[Gate 8] Skipped (WARP_QUICK_PUSH quick mode)" else - echo "[Gate 6] Running unit tests..." + echo "[Gate 8] Running unit tests..." npm run test:local fi diff --git a/scripts/lint-markdown-code-samples.js b/scripts/lint-markdown-code-samples.js new file mode 100644 index 00000000..89a86bba --- /dev/null +++ b/scripts/lint-markdown-code-samples.js @@ -0,0 +1,278 @@ +#!/usr/bin/env node + +import { readdirSync, readFileSync, statSync } from 'node:fs'; +import { resolve, dirname, extname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import ts from 'typescript'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const root = resolve(__dirname, '..'); +const repoTsconfigPath = resolve(root, 'tsconfig.base.json'); + +const IGNORED_DIRS = new Set(['.git', 'node_modules', 'coverage']); +const CODE_SAMPLE_LANGUAGES = new Set(['js', 'javascript', 'ts', 'typescript']); +const ANY_FENCE_PATTERN = /^ {0,3}([`~]{3,})(.*)$/; +const OPENING_FENCE_PATTERN = /^ {0,3}((?:`{3,}|~{3,}))(.*)$/; + +/** + * @typedef {{ + * filePath: string, + * language: 'js'|'javascript'|'ts'|'typescript', + * code: string, + * fenceLine: number, + * startLine: number, + * }} MarkdownCodeSample + */ + +/** + * @typedef {{ + * filePath: string, + * line: number, + * column: number, + * message: string, + * language: string, + * }} MarkdownCodeSampleIssue + */ + +/** + * @param {string} info + * @returns {string | null} + */ +export function parseFenceLanguage(info) { + const language = info.trim().split(/\s+/, 1)[0]?.toLowerCase() || ''; + return CODE_SAMPLE_LANGUAGES.has(language) ? language : null; +} + +/** + * @returns {ts.ScriptTarget} + */ +export function resolveRepoScriptTarget() { + try { + const configText = readFileSync(repoTsconfigPath, 'utf8'); + const parsed = ts.parseConfigFileTextToJson(repoTsconfigPath, configText); + if (parsed.error) { + return ts.ScriptTarget.Latest; + } + const converted = ts.convertCompilerOptionsFromJson(parsed.config?.compilerOptions || {}, root, repoTsconfigPath); + return typeof converted.options.target === 'number' + ? converted.options.target + : ts.ScriptTarget.Latest; + } catch { + return ts.ScriptTarget.Latest; + } +} + +const repoScriptTarget = resolveRepoScriptTarget(); + +/** + * @param {string} filePath + * @param {number} line + * @param {string} language + * @param {string} message + * @returns {MarkdownCodeSampleIssue} + */ +function createMarkdownCodeSampleIssue(filePath, line, language, message) { + return { + filePath, + line, + column: 1, + language, + message, + }; +} + +/** + * @param {string} markdown + * @param {string} filePath + * @returns {{ samples: MarkdownCodeSample[], issues: MarkdownCodeSampleIssue[] }} + */ +function extractMarkdownCodeSamplesWithIssues(markdown, filePath) { + const lines = markdown.split('\n'); + /** @type {MarkdownCodeSample[]} */ + const samples = []; + /** @type {MarkdownCodeSampleIssue[]} */ + const issues = []; + /** @type {{ marker: string, markerLength: number, language: string|null, fenceLine: number, codeLines: string[] } | null} */ + let activeFence = null; + + for (let index = 0; index < lines.length; index += 1) { + const line = lines[index]; + const anyFenceMatch = line.match(ANY_FENCE_PATTERN); + const fenceMatch = line.match(OPENING_FENCE_PATTERN); + if (!activeFence) { + if (!anyFenceMatch) { + continue; + } + if (!fenceMatch) { + const language = parseFenceLanguage(anyFenceMatch[2]); + if (language) { + issues.push( + createMarkdownCodeSampleIssue( + filePath, + index + 1, + language, + 'Malformed Markdown fence marker; use only backticks or only tildes.' + ) + ); + } + continue; + } + activeFence = { + marker: fenceMatch[1][0], + markerLength: fenceMatch[1].length, + language: parseFenceLanguage(fenceMatch[2]), + fenceLine: index + 1, + codeLines: [], + }; + continue; + } + + const closePattern = new RegExp(`^ {0,3}${activeFence.marker}{${activeFence.markerLength},}\\s*$`); + if (closePattern.test(line)) { + if (activeFence.language) { + samples.push({ + filePath, + language: /** @type {'js'|'javascript'|'ts'|'typescript'} */ (activeFence.language), + code: activeFence.codeLines.join('\n'), + fenceLine: activeFence.fenceLine, + startLine: activeFence.fenceLine + 1, + }); + } + activeFence = null; + continue; + } + + activeFence.codeLines.push(line); + } + + if (activeFence?.language) { + issues.push( + createMarkdownCodeSampleIssue( + filePath, + activeFence.fenceLine, + activeFence.language, + 'Unterminated Markdown code fence.' + ) + ); + } + + return { samples, issues }; +} + +/** + * @param {string} markdown + * @param {string} filePath + * @returns {MarkdownCodeSample[]} + */ +export function extractMarkdownCodeSamples(markdown, filePath) { + return extractMarkdownCodeSamplesWithIssues(markdown, filePath).samples; +} + +/** + * @param {string} startPath + * @returns {string[]} + */ +export function collectMarkdownFiles(startPath = root) { + const resolved = resolve(startPath); + const stats = statSync(resolved); + if (stats.isFile()) { + return extname(resolved) === '.md' ? [resolved] : []; + } + + /** @type {string[]} */ + const files = []; + for (const entry of readdirSync(resolved, { withFileTypes: true })) { + if (entry.isDirectory()) { + if (IGNORED_DIRS.has(entry.name) || entry.name.startsWith('.')) { + continue; + } + files.push(...collectMarkdownFiles(join(resolved, entry.name))); + continue; + } + if (entry.isFile() && extname(entry.name) === '.md') { + files.push(join(resolved, entry.name)); + } + } + return files.sort(); +} + +/** + * @param {MarkdownCodeSample} sample + * @returns {MarkdownCodeSampleIssue[]} + */ +export function lintMarkdownCodeSample(sample) { + const scriptKind = sample.language === 'ts' || sample.language === 'typescript' + ? ts.ScriptKind.TS + : ts.ScriptKind.JS; + const sourceFile = ts.createSourceFile( + sample.language.startsWith('ts') ? 'sample.ts' : 'sample.js', + sample.code, + repoScriptTarget, + true, + scriptKind + ); + // `parseDiagnostics` is an internal SourceFile property; we read it here so + // the linter can report parse-only syntax errors without building a Program. + const diagnostics = /** @type {ReadonlyArray} */ ( + /** @type {ts.SourceFile & { parseDiagnostics?: ReadonlyArray }} */ (sourceFile) + .parseDiagnostics || [] + ); + + return diagnostics.map((diagnostic) => { + const start = diagnostic.start ?? 0; + const location = ts.getLineAndCharacterOfPosition(sourceFile, start); + const message = ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n'); + return { + filePath: sample.filePath, + line: sample.startLine + location.line, + column: location.character + 1, + message, + language: sample.language, + }; + }); +} + +/** + * @param {string[]} markdownFiles + * @returns {MarkdownCodeSampleIssue[]} + */ +export function lintMarkdownCodeSamples(markdownFiles) { + /** @type {MarkdownCodeSampleIssue[]} */ + const issues = []; + for (const filePath of markdownFiles) { + const markdown = readFileSync(filePath, 'utf8'); + const { samples, issues: extractionIssues } = extractMarkdownCodeSamplesWithIssues(markdown, filePath); + issues.push(...extractionIssues); + for (const sample of samples) { + issues.push(...lintMarkdownCodeSample(sample)); + } + } + return issues; +} + +const isMain = process.argv[1] && fileURLToPath(import.meta.url) === resolve(process.argv[1]); + +if (isMain) { + const targets = process.argv.slice(2); + const markdownFiles = targets.length === 0 + ? collectMarkdownFiles(root) + : targets.flatMap((target) => collectMarkdownFiles(resolve(process.cwd(), target))); + const issues = lintMarkdownCodeSamples(markdownFiles); + + if (issues.length === 0) { + process.stdout.write( + `Markdown code sample lint passed: ${markdownFiles.length} Markdown files checked.\n` + ); + process.exit(0); + } + + for (const issue of issues) { + process.stderr.write( + `${issue.filePath}:${issue.line}:${issue.column} [${issue.language}] ${issue.message}\n` + ); + } + process.stderr.write( + `Markdown code sample lint failed: ${issues.length} issue(s) across ${markdownFiles.length} Markdown files.\n` + ); + process.exit(1); +} diff --git a/test/unit/scripts/lint-markdown-code-samples.test.js b/test/unit/scripts/lint-markdown-code-samples.test.js new file mode 100644 index 00000000..959852c3 --- /dev/null +++ b/test/unit/scripts/lint-markdown-code-samples.test.js @@ -0,0 +1,254 @@ +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { afterEach, describe, expect, it } from 'vitest'; +import ts from 'typescript'; +import { + collectMarkdownFiles, + extractMarkdownCodeSamples, + lintMarkdownCodeSample, + lintMarkdownCodeSamples, + parseFenceLanguage, + resolveRepoScriptTarget, +} from '../../../scripts/lint-markdown-code-samples.js'; + +/** @type {string[]} */ +const tempDirs = []; + +afterEach(() => { + while (tempDirs.length > 0) { + rmSync(/** @type {string} */ (tempDirs.pop()), { force: true, recursive: true }); + } +}); + +/** + * @returns {string} + */ +function createTempDir() { + const dir = mkdtempSync(join(tmpdir(), 'git-warp-md-code-')); + tempDirs.push(dir); + return dir; +} + +describe('parseFenceLanguage', () => { + it('accepts JavaScript and TypeScript fence labels', () => { + expect(parseFenceLanguage('javascript')).toBe('javascript'); + expect(parseFenceLanguage('ts title="example"')).toBe('ts'); + }); + + it('ignores non-code-sample fence labels', () => { + expect(parseFenceLanguage('bash')).toBeNull(); + expect(parseFenceLanguage('')).toBeNull(); + }); +}); + +describe('extractMarkdownCodeSamples', () => { + it('extracts JS and TS fences with accurate start lines', () => { + const markdown = [ + '# Demo', + '```javascript', + 'const answer = 42;', + '```', + '', + '```ts title="types"', + 'export const value: number = 1;', + '```', + ].join('\n'); + + expect(extractMarkdownCodeSamples(markdown, 'README.md')).toEqual([ + { + filePath: 'README.md', + language: 'javascript', + code: 'const answer = 42;', + fenceLine: 2, + startLine: 3, + }, + { + filePath: 'README.md', + language: 'ts', + code: 'export const value: number = 1;', + fenceLine: 6, + startLine: 7, + }, + ]); + }); + + it('ignores non-JS/TS fences', () => { + const markdown = [ + '```bash', + 'npm run test', + '```', + '', + '```text', + 'plain text block', + '```', + ].join('\n'); + + expect(extractMarkdownCodeSamples(markdown, 'GUIDE.md')).toEqual([]); + }); + + it('extracts fences indented by up to three spaces', () => { + const markdown = [ + ' ```ts', + ' export const answer = 42;', + ' ```', + ].join('\n'); + + expect(extractMarkdownCodeSamples(markdown, 'INDENTED.md')).toEqual([ + { + filePath: 'INDENTED.md', + language: 'ts', + code: ' export const answer = 42;', + fenceLine: 1, + startLine: 2, + }, + ]); + }); + + it('rejects mixed fence markers instead of parsing them as samples', () => { + const markdown = [ + '``~ts', + 'export const bad = 1;', + '``~', + ].join('\n'); + + expect(extractMarkdownCodeSamples(markdown, 'MIXED.md')).toEqual([]); + }); +}); + +describe('resolveRepoScriptTarget', () => { + it('uses the repository TypeScript target from tsconfig.base.json', () => { + expect(resolveRepoScriptTarget()).toBe(ts.ScriptTarget.ES2022); + }); +}); + +describe('lintMarkdownCodeSample', () => { + it('accepts valid JavaScript and TypeScript snippets', () => { + const jsSample = { + filePath: 'README.md', + language: /** @type {'javascript'} */ ('javascript'), + code: 'export const answer = 42;', + fenceLine: 10, + startLine: 11, + }; + const tsSample = { + filePath: 'GUIDE.md', + language: /** @type {'ts'} */ ('ts'), + code: 'export const answer: number = 42;', + fenceLine: 20, + startLine: 21, + }; + + expect(lintMarkdownCodeSample(jsSample)).toEqual([]); + expect(lintMarkdownCodeSample(tsSample)).toEqual([]); + }); + + it('accepts documentation-style top-level await TypeScript snippets', () => { + const sample = { + filePath: 'docs/ADR-001-Folds.md', + language: /** @type {'ts'} */ ('ts'), + code: [ + "const view = graph.view({ fold: { mode: 'shallow', maxDepth: 1 } });", + '', + 'await view.traverse(startNodeId);', + "await view.query().match('doc:*').run();", + 'await view.renderAscii();', + ].join('\n'), + fenceLine: 122, + startLine: 123, + }; + + expect(lintMarkdownCodeSample(sample)).toEqual([]); + }); + + it('reports syntax errors with Markdown-relative locations', () => { + const sample = { + filePath: 'README.md', + language: /** @type {'javascript'} */ ('javascript'), + code: 'export const broken = ;', + fenceLine: 5, + startLine: 6, + }; + + expect(lintMarkdownCodeSample(sample)).toEqual([ + { + filePath: 'README.md', + line: 6, + column: 23, + language: 'javascript', + message: 'Expression expected.', + }, + ]); + }); +}); + +describe('collectMarkdownFiles', () => { + it('walks Markdown files and ignores hidden directories', () => { + const root = createTempDir(); + mkdirSync(join(root, 'docs')); + mkdirSync(join(root, '.hidden')); + writeFileSync(join(root, 'README.md'), '# root\n'); + writeFileSync(join(root, 'docs', 'GUIDE.md'), '# guide\n'); + writeFileSync(join(root, '.hidden', 'SECRET.md'), '# secret\n'); + + expect(collectMarkdownFiles(root)).toEqual([ + join(root, 'README.md'), + join(root, 'docs', 'GUIDE.md'), + ]); + }); +}); + +describe('lintMarkdownCodeSamples', () => { + it('checks extracted samples across Markdown files', () => { + const root = createTempDir(); + const goodDoc = join(root, 'README.md'); + const badDoc = join(root, 'docs', 'broken.md'); + mkdirSync(join(root, 'docs')); + writeFileSync(goodDoc, '```js\nexport const ok = 1;\n```\n'); + writeFileSync(badDoc, '```ts\nexport const broken: = 1;\n```\n'); + + expect(lintMarkdownCodeSamples([goodDoc, badDoc])).toEqual([ + { + filePath: badDoc, + line: 2, + column: 22, + language: 'ts', + message: 'Type expected.', + }, + ]); + }); + + it('reports unterminated JavaScript and TypeScript fences', () => { + const root = createTempDir(); + const badDoc = join(root, 'docs', 'unterminated.md'); + mkdirSync(join(root, 'docs')); + writeFileSync(badDoc, '```ts\nexport const broken = 1;\n'); + + expect(lintMarkdownCodeSamples([badDoc])).toEqual([ + { + filePath: badDoc, + line: 1, + column: 1, + language: 'ts', + message: 'Unterminated Markdown code fence.', + }, + ]); + }); + + it('reports malformed mixed-marker JavaScript and TypeScript fences', () => { + const root = createTempDir(); + const badDoc = join(root, 'docs', 'mixed-marker.md'); + mkdirSync(join(root, 'docs')); + writeFileSync(badDoc, '``~ts\nexport const broken = 1;\n``~\n'); + + expect(lintMarkdownCodeSamples([badDoc])).toEqual([ + { + filePath: badDoc, + line: 1, + column: 1, + language: 'ts', + message: 'Malformed Markdown fence marker; use only backticks or only tildes.', + }, + ]); + }); +});