From cf13c4207be88c237e94cfb97a983137792696bb Mon Sep 17 00:00:00 2001 From: Aarne Laur Date: Fri, 6 Mar 2026 11:10:29 +0100 Subject: [PATCH 1/5] feat: expand fuzz testing coverage across the Bridge codebase - Introduced a comprehensive fuzz testing expansion plan in `fuzz-expansion.md`, outlining current coverage, identified gaps, and a prioritized implementation roadmap. - Added new tests for deep-path AOT/runtime parity with chaotic inputs in `fuzz-runtime-parity.test.ts`, addressing critical gaps in testing. - Implemented crash-proofing and bug fixes for `arr.filter` and `arr.find` in the stdlib, ensuring robust handling of non-array inputs. - Created fuzz tests for parser functions in `fuzz-parser.test.ts`, ensuring resilience against invalid inputs and verifying serializer round-trip integrity. - Enhanced array and string tools with additional guards against unexpected input types, returning `undefined` for non-conforming values. - Updated `package.json` files to include `fast-check` as a devDependency across relevant packages. --- docs/fuzz-expansion.md | 137 ++++++++ .../test/fuzz-regressions.todo.test.ts | 16 + .../test/fuzz-runtime-parity.test.ts | 228 +++++++++++++ packages/bridge-stdlib/package.json | 1 + packages/bridge-stdlib/src/tools/arrays.ts | 4 + packages/bridge-stdlib/src/tools/strings.ts | 8 +- .../bridge-stdlib/test/fuzz-stdlib.test.ts | 174 ++++++++++ packages/bridge/package.json | 1 + packages/bridge/test/fuzz-parser.test.ts | 312 ++++++++++++++++++ pnpm-lock.yaml | 77 +---- 10 files changed, 886 insertions(+), 72 deletions(-) create mode 100644 docs/fuzz-expansion.md create mode 100644 packages/bridge-compiler/test/fuzz-runtime-parity.test.ts create mode 100644 packages/bridge-stdlib/test/fuzz-stdlib.test.ts create mode 100644 packages/bridge/test/fuzz-parser.test.ts diff --git a/docs/fuzz-expansion.md b/docs/fuzz-expansion.md new file mode 100644 index 00000000..cd9628f7 --- /dev/null +++ b/docs/fuzz-expansion.md @@ -0,0 +1,137 @@ +# Fuzz Testing Expansion Plan + +> Tracking document for expanding fuzz/property-based testing coverage across the Bridge codebase. + +## Current State + +All fuzzing lives in `packages/bridge-compiler/test/fuzz-compile.test.ts` (6 tests, ~24,500 property runs). +`fast-check` is a devDependency of `bridge-compiler` only. + +**What's covered:** + +- JS syntax validity of AOT compiler output +- Compiler determinism +- AOT/runtime parity on flat single-segment paths with `fc.jsonValue()` inputs +- Parser round-trip (text spec → parse → serialize → reparse → execute parity) +- Fallback-heavy and logical-wire syntax validity + +**Key gaps:** + +- No deep-path parity testing with chaotic inputs (`NaN`, `undefined`, nested shapes) +- No purely textual parser fuzzing (random `.bridge` strings) +- No serializer round-trip fuzzing on full ASTs (only limited text specs) +- No stdlib tool fuzzing (known crash in `arr.filter`/`arr.find` on non-array input) +- No array mapping / shadow tree parity testing +- No simulated tool call parity testing + +--- + +## P0 — Critical (implement first) + +### [x] 1A. Deep-path AOT/runtime parity with chaotic inputs + +**File:** `packages/bridge-compiler/test/fuzz-runtime-parity.test.ts` + +- `chaosInputArb` that extends beyond `fc.jsonValue()`: includes `NaN`, `Infinity`, `-0`, `undefined`, empty strings, deeply nested objects (5+ levels), arrays where objects expected and vice versa +- `deepBridgeArb` using multi-segment paths (1–4 segments) instead of `flatPathArb`; all input refs use `rootSafe: true` + `pathSafe` for safe navigation +- Property: `executeRuntime` vs `executeAot` → `deepEqual` on `.data`, or both throw the same error class +- 3,000 runs + +**Regressions discovered during implementation (tracked in `fuzz-regressions.todo.test.ts`):** + +- Unsafe path traversal divergence: AOT silently returns `undefined` where runtime throws `TypeError` when `rootSafe` is not set (seeds 1798655022, -481664925) +- Fallback chain null/undefined divergence: AOT returns `null` where runtime returns `undefined` when a fallback constant resolves to `"null"` — see `deepFallbackBridgeArb` investigation needed + +**What this catches:** Type coercion divergence (verified working), NaN propagation, deep path access with missing keys. + +### [x] 1B. arr.filter / arr.find crash proof + bug fix + +**File:** `packages/bridge-stdlib/test/fuzz-stdlib.test.ts` +**Fixes applied in `packages/bridge-stdlib/src/tools/`:** + +- `arrays.ts`: Added `Array.isArray(arr)` guard in `filter` and `find` +- `arrays.ts`: Added `obj == null || typeof obj !== "object"` guard inside filter/find callback (null array elements) +- `strings.ts`: Replaced `?.` optional chaining with `typeof === "string"` guards in all four string tools (the `?.` only guards null/undefined, not non-string types like arrays whose method property is `undefined`, causing `TypeError: undefined is not a function`) + +- 2,000 runs per tool, across both array and string tools + +--- + +## P1 — Important (implement after P0) + +### [x] 2A. Parser textual fuzzing — `parseBridge` never panics + +**File:** `packages/bridge/test/fuzz-parser.test.ts` + +- `bridgeTokenSoupArb` — weighted mix of Bridge-like tokens (keywords, identifiers, `{`, `}`, `<-`, `=`, `.`, `||`, `??`) + random noise +- Property: `parseBridge(text)` either returns a `BridgeDocument` or throws a standard `Error`. Must never throw `TypeError`, `RangeError`, or crash with a stack overflow. +- 5,000 runs + +**What this catches:** Null dereferences in CST→AST visitor, unbounded recursion, Chevrotain edge cases. + +### [x] 2B. Parser textual fuzzing — `parseBridgeDiagnostics` never crashes + +**File:** same file, separate test + +- Same `bridgeTokenSoupArb` +- Property: `parseBridgeDiagnostics(text)` **always** returns `{ document, diagnostics }`, never throws +- 5,000 runs + +**What this catches:** Uncaught exceptions in recovery mode, diagnostic formatting crashes. + +### [x] 3A. Serializer round-trip (AST → text → AST) + +**File:** same file, separate test + +- Valid `.bridge` text → `parseBridge` → `serializeBridge` → `parseBridge` → instruction count and wire count must match +- Also: `prettyPrintToSource` idempotence — `format(format(text)) === format(text)` +- Also: `prettyPrintToSource` output is always parseable +- 2,000 runs each + +--- + +## P2 — Valuable (future work) + +### [ ] 1B-ext. Array mapping parity + +- Generate bridges with `arrayIterators` (the `[] as iter { ... }` pattern) +- Test shadow tree execution parity between AOT and runtime +- Complex arbitrary: needs `element: true` NodeRefs, iterator handles + +### [ ] 1C. Simulated tool call parity + +- Generate bridges referencing mock tools (`identity`, `fail`) +- Test tool error propagation + `catchFallback` + `onError` wires between engines + +### [ ] 3B. `prettyPrintToSource` advanced stability + +- Deeper formatter testing with all block types (tool, define, const, bridge) + +### [ ] 4C. `httpCall` input surface + +- URL construction edge cases, `JSON.stringify` on circular refs +- Requires fetch mocking + +--- + +## P3 — Nice to have + +### [ ] Additional string tool coverage + +- Confirm Symbol, BigInt, circular refs are handled gracefully across all stdlib tools + +### [ ] Parser diagnostics completeness + +- Valid text parsed via `parseBridgeDiagnostics` produces zero error-severity diagnostics +- Returned document matches strict `parseBridge` output + +--- + +## Implementation Notes + +- **Test framework:** `node:test` + `node:assert` (no Jest/Vitest) +- **Fuzz library:** `fast-check` ^4.5.3 +- **Regression workflow:** fuzz finding → `test.todo` entry with seed → tracking issue → deterministic reproducer → fix → cleanup (see `packages/bridge-compiler/test/README.md`) +- Parser fuzz tests go in `packages/bridge/test/` (that's where all parser/integration tests live) +- Stdlib fuzz tests go in `packages/bridge-stdlib/test/` +- Run a single test: `node --experimental-transform-types --conditions source --test ` diff --git a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts index 6e80ff36..5f9c02b7 100644 --- a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts +++ b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts @@ -4,4 +4,20 @@ describe("fuzz-discovered AOT/runtime divergence backlog", () => { // test.todo( // "parser round-trip: serializeBridge output can be unparsable for some valid parsed documents (seed=1864118703)", // ); + // AOT compiler uses `?.` safe-navigation everywhere in generated code; runtime + // throws TypeError for unsafe path traversal when `rootSafe` is not set. + // These seeds reproduce the divergence via fuzz-runtime-parity.test.ts. + // test.todo( + // "deep-path parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=1798655022)", + // ); + // test.todo( + // "deep-path fallback parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=-481664925)", + // ); + // AOT and runtime diverge on null-vs-undefined semantics when fallback constant + // wires (catchFallback / falsy fallbacks) resolve to "null". The AOT emits null + // while the runtime applies overdefinition coalescing and produces undefined. + // Repro: fuzz-runtime-parity deepFallbackBridgeArb + chaosInputArb, any random seed. + // test.todo( + // "fallback parity: AOT returns null where runtime returns undefined when fallback constant resolves to null (seed=random)", + // ); }); diff --git a/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts new file mode 100644 index 00000000..8751edee --- /dev/null +++ b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts @@ -0,0 +1,228 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; +import fc from "fast-check"; +import type { + Bridge, + BridgeDocument, + NodeRef, + Wire, +} from "@stackables/bridge-core"; +import { executeBridge as executeRuntime } from "@stackables/bridge-core"; +import { executeBridge as executeAot } from "../src/index.ts"; + +// ── Shared infrastructure ─────────────────────────────────────────────────── + +const forbiddenPathSegments = new Set([ + "__proto__", + "prototype", + "constructor", +]); + +const identifierArb = fc + .stringMatching(/^[a-zA-Z_][a-zA-Z0-9_]{0,20}$/) + .filter((segment) => !forbiddenPathSegments.has(segment)); + +const pathArb = fc.array(identifierArb, { minLength: 1, maxLength: 4 }); +const flatPathArb = fc.array(identifierArb, { minLength: 1, maxLength: 1 }); + +const constantValueArb = fc + .oneof( + fc.string({ maxLength: 64 }), + fc.double({ noNaN: true, noDefaultInfinity: true }), + fc.boolean(), + fc.constant(null), + ) + .map((value) => JSON.stringify(value)); + +// ── Chaotic input arbitrary ───────────────────────────────────────────────── +// Goes beyond fc.jsonValue() to exercise type-coercion and null-navigation +// edge cases the engine must handle without crashing. + +const chaosLeafArb = fc.oneof( + { weight: 4, arbitrary: fc.string({ maxLength: 64 }) }, + { weight: 3, arbitrary: fc.integer() }, + { weight: 2, arbitrary: fc.double({ noNaN: false }) }, // includes NaN + { weight: 2, arbitrary: fc.boolean() }, + { weight: 3, arbitrary: fc.constant(null) }, + { weight: 2, arbitrary: fc.constant(undefined) }, + { weight: 1, arbitrary: fc.constant("") }, + { weight: 1, arbitrary: fc.constant(0) }, + { weight: 1, arbitrary: fc.constant(-0) }, + { weight: 1, arbitrary: fc.constant(Infinity) }, + { weight: 1, arbitrary: fc.constant(-Infinity) }, +); + +const chaosValueArb: fc.Arbitrary = fc.letrec((tie) => ({ + tree: fc.oneof( + { weight: 6, arbitrary: chaosLeafArb }, + { + weight: 2, + arbitrary: fc.array(tie("tree"), { maxLength: 5 }), + }, + { + weight: 2, + arbitrary: fc.dictionary(identifierArb, tie("tree"), { maxKeys: 6 }), + }, + ), +})).tree; + +const chaosInputArb = fc.dictionary(identifierArb, chaosValueArb, { + maxKeys: 16, +}); + +// ── Ref helpers ───────────────────────────────────────────────────────────── +// rootSafe: true mirrors the parser's `?.` safe-navigation semantics on input +// refs: missing keys return undefined rather than throwing TypeError. This lets +// us test value-coercion parity (NaN, Infinity, etc.) without hitting the +// known unsafe-path-traversal divergence between AOT and runtime +// (tracked separately as fuzz-regressions seeds 1798655022 / -481664925). + +function inputRef(type: string, field: string, path: string[]): NodeRef { + // rootSafe + pathSafe mirror full `?.` safe-navigation on all segments so + // missing keys return undefined rather than throwing TypeError. This lets + // us test value-coercion parity (NaN, Infinity, etc.) without hitting the + // known unsafe-path-traversal divergence between AOT and runtime + // (tracked separately as fuzz-regressions seeds 1798655022 / -481664925). + return { + module: "_", + type, + field, + path, + rootSafe: true, + pathSafe: Array(path.length).fill(true), + }; +} + +function outputRef(type: string, field: string, path: string[]): NodeRef { + return { module: "_", type, field, path }; +} + +// ── Deep-path bridge arbitrary ────────────────────────────────────────────── +// Uses multi-segment paths (1–4 segments) to exercise deep property access. + +const deepWireArb = (type: string, field: string): fc.Arbitrary => { + const toArb = flatPathArb.map((path) => outputRef(type, field, path)); + const fromArb = pathArb.map((path) => inputRef(type, field, path)); + + return fc.oneof( + fc.record({ + value: constantValueArb, + to: toArb, + }), + fc.record({ + from: fromArb, + to: toArb, + }), + ); +}; + +const deepBridgeArb: fc.Arbitrary = fc + .record({ + type: identifierArb, + field: identifierArb, + }) + .chain(({ type, field }) => + fc.record({ + kind: fc.constant<"bridge">("bridge"), + type: fc.constant(type), + field: fc.constant(field), + handles: fc.constant([ + { kind: "input", handle: "i" } as const, + { kind: "output", handle: "o" } as const, + ]), + wires: fc.uniqueArray(deepWireArb(type, field), { + minLength: 1, + maxLength: 20, + selector: (wire) => wire.to.path.join("."), + }), + }), + ); + +// Note: deepFallbackBridgeArb / deepFallbackWireArb are intentionally omitted — +// parity testing of fallback chains with chaotic inputs exposes an AOT/runtime +// null-vs-undefined divergence tracked in fuzz-regressions.todo.test.ts. + +// ── Parity assertion helper ───────────────────────────────────────────────── + +async function assertParity(bridge: Bridge, input: Record) { + const document: BridgeDocument = { instructions: [bridge] }; + const operation = `${bridge.type}.${bridge.field}`; + + let runtimeResult: { data: any } | undefined; + let runtimeError: unknown; + let aotResult: { data: any } | undefined; + let aotError: unknown; + + try { + runtimeResult = await executeRuntime({ + document, + operation, + input, + tools: {}, + }); + } catch (err) { + runtimeError = err; + } + + try { + aotResult = await executeAot({ document, operation, input, tools: {} }); + } catch (err) { + aotError = err; + } + + // Both must succeed or both must fail. + if (runtimeError && !aotError) { + assert.fail( + `Runtime threw but AOT did not.\nRuntime error: ${runtimeError}\nAOT data: ${JSON.stringify(aotResult?.data)}`, + ); + } + if (!runtimeError && aotError) { + assert.fail( + `AOT threw but runtime did not.\nAOT error: ${aotError}\nRuntime data: ${JSON.stringify(runtimeResult?.data)}`, + ); + } + + if (runtimeError && aotError) { + // Both threw — they should be the same error class. + const rName = (runtimeError as Error)?.name ?? "unknown"; + const aName = (aotError as Error)?.name ?? "unknown"; + assert.equal( + aName, + rName, + `Error class mismatch: runtime=${rName}, aot=${aName}`, + ); + return; + } + + // Both succeeded — normalize NaN for comparison since JSON.stringify drops NaN. + // deepEqual handles NaN correctly (NaN === NaN in assert). + assert.deepEqual(aotResult!.data, runtimeResult!.data); +} + +// ── Tests ─────────────────────────────────────────────────────────────────── + +describe("runtime parity fuzzing — deep paths + chaotic inputs", () => { + test( + "AOT matches runtime on deep-path bridges with chaotic inputs", + { timeout: 180_000 }, + async () => { + await fc.assert( + fc.asyncProperty( + deepBridgeArb, + chaosInputArb, + async (bridge, input) => { + await assertParity(bridge, input); + }, + ), + { + numRuns: 3_000, + endOnFailure: true, + }, + ); + }, + ); + + // Note: parity testing of fallback chains with chaotic inputs is deferred — + // the AOT compiler and runtime diverge on null-vs-undefined semantics when + // fallback constants resolve to null. Tracked in fuzz-regressions.todo.test.ts. +}); diff --git a/packages/bridge-stdlib/package.json b/packages/bridge-stdlib/package.json index 66c746cf..5de6981d 100644 --- a/packages/bridge-stdlib/package.json +++ b/packages/bridge-stdlib/package.json @@ -34,6 +34,7 @@ }, "devDependencies": { "@types/node": "^25.3.3", + "fast-check": "^4.5.3", "typescript": "^5.9.3" }, "publishConfig": { diff --git a/packages/bridge-stdlib/src/tools/arrays.ts b/packages/bridge-stdlib/src/tools/arrays.ts index 6518e7b7..44ddcc03 100644 --- a/packages/bridge-stdlib/src/tools/arrays.ts +++ b/packages/bridge-stdlib/src/tools/arrays.ts @@ -7,7 +7,9 @@ const syncUtility = { export function filter(opts: { in: any[]; [key: string]: any }) { const { in: arr, ...criteria } = opts; + if (!Array.isArray(arr)) return undefined; return arr.filter((obj) => { + if (obj == null || typeof obj !== "object") return false; for (const [key, value] of Object.entries(criteria)) { if (obj[key] !== value) { return false; @@ -21,7 +23,9 @@ filter.bridge = syncUtility; export function find(opts: { in: any[]; [key: string]: any }) { const { in: arr, ...criteria } = opts; + if (!Array.isArray(arr)) return undefined; return arr.find((obj) => { + if (obj == null || typeof obj !== "object") return false; for (const [key, value] of Object.entries(criteria)) { if (obj[key] !== value) { return false; diff --git a/packages/bridge-stdlib/src/tools/strings.ts b/packages/bridge-stdlib/src/tools/strings.ts index a3bdffe5..126cb6d5 100644 --- a/packages/bridge-stdlib/src/tools/strings.ts +++ b/packages/bridge-stdlib/src/tools/strings.ts @@ -6,25 +6,25 @@ const syncUtility = { } satisfies ToolMetadata; export function toLowerCase(opts: { in: string }) { - return opts.in?.toLowerCase(); + return typeof opts.in === "string" ? opts.in.toLowerCase() : undefined; } toLowerCase.bridge = syncUtility; export function toUpperCase(opts: { in: string }) { - return opts.in?.toUpperCase(); + return typeof opts.in === "string" ? opts.in.toUpperCase() : undefined; } toUpperCase.bridge = syncUtility; export function trim(opts: { in: string }) { - return opts.in?.trim(); + return typeof opts.in === "string" ? opts.in.trim() : undefined; } trim.bridge = syncUtility; export function length(opts: { in: string }) { - return opts.in?.length; + return typeof opts.in === "string" ? opts.in.length : undefined; } length.bridge = syncUtility; diff --git a/packages/bridge-stdlib/test/fuzz-stdlib.test.ts b/packages/bridge-stdlib/test/fuzz-stdlib.test.ts new file mode 100644 index 00000000..ab15638e --- /dev/null +++ b/packages/bridge-stdlib/test/fuzz-stdlib.test.ts @@ -0,0 +1,174 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; +import fc from "fast-check"; +import { filter, find, first, toArray } from "../src/tools/arrays.ts"; +import { + toLowerCase, + toUpperCase, + trim, + length, +} from "../src/tools/strings.ts"; + +// ── Chaotic value arbitrary ───────────────────────────────────────────────── +// Exercises every type boundary the stdlib tools might encounter. + +const chaosValueArb: fc.Arbitrary = fc.oneof( + fc.string({ maxLength: 64 }), + fc.integer(), + fc.double({ noNaN: false }), + fc.boolean(), + fc.constant(null), + fc.constant(undefined), + fc.constant(""), + fc.constant(0), + fc.constant(-0), + fc.constant(Infinity), + fc.constant(-Infinity), + fc.array(fc.jsonValue(), { maxLength: 5 }), + fc.dictionary(fc.string({ maxLength: 8 }), fc.jsonValue(), { maxKeys: 4 }), +); + +// ── Array tool fuzzing ────────────────────────────────────────────────────── + +describe("stdlib fuzz — array tools", () => { + test("filter never throws on any input type", { timeout: 30_000 }, () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + // Must not throw — returns undefined for non-arrays + const result = filter({ in: value, key: "x" }); + if (!Array.isArray(value)) { + assert.equal(result, undefined); + } + }), + { numRuns: 2_000 }, + ); + }); + + test("find never throws on any input type", { timeout: 30_000 }, () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + const result = find({ in: value, key: "x" }); + if (!Array.isArray(value)) { + assert.equal(result, undefined); + } + }), + { numRuns: 2_000 }, + ); + }); + + test( + "filter produces correct results on valid array input with chaotic criteria", + { timeout: 30_000 }, + () => { + fc.assert( + fc.property( + fc.array( + fc.dictionary(fc.string({ maxLength: 8 }), fc.jsonValue(), { + maxKeys: 4, + }), + { maxLength: 10 }, + ), + fc.string({ maxLength: 8 }), + fc.jsonValue(), + (arr, key, value) => { + const result = filter({ in: arr, [key]: value }); + assert.ok(Array.isArray(result)); + // Every element in the result must match the criterion + for (const item of result) { + assert.equal(item[key], value); + } + }, + ), + { numRuns: 2_000 }, + ); + }, + ); + + test( + "first never throws on any input type (non-strict mode)", + { timeout: 30_000 }, + () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + // Non-strict mode must not throw + const result = first({ in: value }); + if (!Array.isArray(value)) { + assert.equal(result, undefined); + } + }), + { numRuns: 2_000 }, + ); + }, + ); + + test("toArray never throws on any input type", { timeout: 30_000 }, () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + const result = toArray({ in: value }); + assert.ok(Array.isArray(result)); + }), + { numRuns: 2_000 }, + ); + }); +}); + +// ── String tool fuzzing ───────────────────────────────────────────────────── + +describe("stdlib fuzz — string tools", () => { + test( + "toLowerCase never throws on any input type", + { timeout: 30_000 }, + () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + // Must not throw — returns undefined for non-strings via optional chaining + const result = toLowerCase({ in: value }); + if (typeof value === "string") { + assert.equal(result, value.toLowerCase()); + } + }), + { numRuns: 2_000 }, + ); + }, + ); + + test( + "toUpperCase never throws on any input type", + { timeout: 30_000 }, + () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + const result = toUpperCase({ in: value }); + if (typeof value === "string") { + assert.equal(result, value.toUpperCase()); + } + }), + { numRuns: 2_000 }, + ); + }, + ); + + test("trim never throws on any input type", { timeout: 30_000 }, () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + const result = trim({ in: value }); + if (typeof value === "string") { + assert.equal(result, value.trim()); + } + }), + { numRuns: 2_000 }, + ); + }); + + test("length never throws on any input type", { timeout: 30_000 }, () => { + fc.assert( + fc.property(chaosValueArb, (value) => { + const result = length({ in: value }); + if (typeof value === "string") { + assert.equal(result, value.length); + } + }), + { numRuns: 2_000 }, + ); + }); +}); diff --git a/packages/bridge/package.json b/packages/bridge/package.json index dfab4c61..494533f7 100644 --- a/packages/bridge/package.json +++ b/packages/bridge/package.json @@ -39,6 +39,7 @@ "@graphql-tools/executor-http": "^3.1.0", "@stackables/bridge-compiler": "workspace:*", "@types/node": "^25.3.3", + "fast-check": "^4.5.3", "graphql": "^16.13.1", "graphql-yoga": "^5.18.0", "typescript": "^5.9.3" diff --git a/packages/bridge/test/fuzz-parser.test.ts b/packages/bridge/test/fuzz-parser.test.ts new file mode 100644 index 00000000..b2318c61 --- /dev/null +++ b/packages/bridge/test/fuzz-parser.test.ts @@ -0,0 +1,312 @@ +import assert from "node:assert/strict"; +import { describe, test } from "node:test"; +import fc from "fast-check"; +import { + parseBridgeFormat as parseBridge, + parseBridgeDiagnostics, + serializeBridge, + prettyPrintToSource, +} from "../src/index.ts"; +import type { BridgeDocument } from "../src/index.ts"; + +// ── Token-soup arbitrary ──────────────────────────────────────────────────── +// Generates strings composed of a weighted mix of Bridge-like tokens and noise. +// The goal is to exercise the parser/lexer on inputs that are structurally +// plausible but semantically invalid — the space where crashes lurk. + +const bridgeKeywords = [ + "version", + "bridge", + "tool", + "define", + "const", + "with", + "input", + "output", + "context", + "as", + "from", + "force", + "catch", + "throw", + "panic", + "continue", + "break", + "on", + "error", + "null", + "true", + "false", +]; + +const bridgeOperators = [ + "<-", + "=", + "||", + "??", + "?.", + ".", + ",", + "&&", + "?", + ":", + "+", + "-", + "*", + "/", + "==", + "!=", + ">", + ">=", + "<", + "<=", + "!", +]; + +const bridgeStructural = ["{", "}", "(", ")", "[", "]", "\n", "\n\n", " "]; + +const bridgeTokenArb = fc.oneof( + { weight: 5, arbitrary: fc.constantFrom(...bridgeKeywords) }, + { weight: 3, arbitrary: fc.constantFrom(...bridgeOperators) }, + { weight: 4, arbitrary: fc.constantFrom(...bridgeStructural) }, + { weight: 3, arbitrary: fc.stringMatching(/^[a-zA-Z_]\w{0,12}$/) }, // identifiers + { weight: 2, arbitrary: fc.stringMatching(/^"[^"\\]{0,20}"$/) }, // string literals + { weight: 2, arbitrary: fc.integer({ min: -1000, max: 1000 }).map(String) }, // numbers + { weight: 1, arbitrary: fc.stringMatching(/^1\.\d$/) }, // version-like + { weight: 1, arbitrary: fc.string({ maxLength: 8 }) }, // random noise + { weight: 1, arbitrary: fc.constant("#") }, // comment start +); + +const bridgeTokenSoupArb = fc + .array(bridgeTokenArb, { minLength: 1, maxLength: 60 }) + .map((tokens) => tokens.join(" ")); + +// ── Valid bridge text arbitrary ───────────────────────────────────────────── +// Generates structurally valid .bridge text for round-trip testing. + +const canonicalIdArb = fc.constantFrom("a", "b", "c", "d", "e", "f", "g", "h"); + +const textConstantValueArb = fc.oneof( + fc.boolean().map((v) => (v ? "true" : "false")), + fc.constant("null"), + fc.integer({ min: -1_000_000, max: 1_000_000 }).map(String), + fc.string({ maxLength: 32 }).map((v) => JSON.stringify(v)), +); + +const wireSpecArb = fc.oneof( + fc.record({ + kind: fc.constant<"pull">("pull"), + to: canonicalIdArb, + from: canonicalIdArb, + }), + fc.record({ + kind: fc.constant<"constant">("constant"), + to: canonicalIdArb, + value: textConstantValueArb, + }), +); + +const bridgeTextSpecArb = fc.record({ + type: canonicalIdArb, + field: canonicalIdArb, + wires: fc.uniqueArray(wireSpecArb, { + minLength: 1, + maxLength: 8, + selector: (wire) => wire.to, + }), +}); + +function buildBridgeText(spec: { + type: string; + field: string; + wires: Array< + | { kind: "pull"; to: string; from: string } + | { kind: "constant"; to: string; value: string } + >; +}): string { + const lines = [ + "version 1.5", + `bridge ${spec.type}.${spec.field} {`, + " with input as i", + " with output as o", + "", + ]; + + for (const wire of spec.wires) { + if (wire.kind === "pull") { + lines.push(` o.${wire.to} <- i.${wire.from}`); + } else { + lines.push(` o.${wire.to} = ${wire.value}`); + } + } + + lines.push("}"); + return lines.join("\n"); +} + +// ── Tests ─────────────────────────────────────────────────────────────────── + +describe("parser fuzz — textual fuzzing", () => { + test( + "parseBridge never throws unstructured errors on random input", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(bridgeTokenSoupArb, (text) => { + try { + parseBridge(text); + } catch (err) { + // Structured Error is acceptable — the parser is allowed to reject + // invalid input by throwing an Error with a message. + if (err instanceof Error) { + assert.ok( + typeof err.message === "string" && err.message.length > 0, + `Error must have a non-empty message, got: ${String(err)}`, + ); + return; + } + // Non-Error throws are a parser bug. + assert.fail( + `parseBridge threw a non-Error value: ${typeof err} — ${String(err)}`, + ); + } + }), + { + numRuns: 5_000, + endOnFailure: true, + }, + ); + }, + ); + + test( + "parseBridgeDiagnostics never throws on random input", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(bridgeTokenSoupArb, (text) => { + // This function is used in the IDE/LSP — it must NEVER throw. + const result = parseBridgeDiagnostics(text); + + assert.ok( + result !== null && result !== undefined, + "must return a result", + ); + assert.ok("document" in result, "result must have a document"); + assert.ok("diagnostics" in result, "result must have diagnostics"); + assert.ok( + Array.isArray(result.diagnostics), + "diagnostics must be an array", + ); + }), + { + numRuns: 5_000, + endOnFailure: true, + }, + ); + }, + ); +}); + +describe("parser fuzz — serializer round-trip", () => { + test( + "serializeBridge output is always parseable for valid documents", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(bridgeTextSpecArb, (spec) => { + const sourceText = buildBridgeText(spec); + const parsed = parseBridge(sourceText); + const serialized = serializeBridge(parsed); + + // The serialized output must parse without errors. + let reparsed: BridgeDocument; + try { + reparsed = parseBridge(serialized); + } catch (error) { + assert.fail( + `serializeBridge produced unparsable output: ${String(error)}\n--- SOURCE ---\n${sourceText}\n--- SERIALIZED ---\n${serialized}`, + ); + } + + // The reparsed document must have the same number of instructions. + assert.equal( + reparsed.instructions.length, + parsed.instructions.length, + "instruction count must survive round-trip", + ); + + // Each bridge instruction must preserve its wires. + for (let i = 0; i < parsed.instructions.length; i++) { + const orig = parsed.instructions[i]!; + const rt = reparsed.instructions[i]!; + assert.equal(rt.kind, orig.kind, "instruction kind must match"); + if (orig.kind === "bridge" && rt.kind === "bridge") { + assert.equal( + rt.wires.length, + orig.wires.length, + "wire count must match", + ); + } + } + }), + { + numRuns: 2_000, + endOnFailure: true, + }, + ); + }, + ); + + test("prettyPrintToSource is idempotent", { timeout: 60_000 }, () => { + fc.assert( + fc.property(bridgeTextSpecArb, (spec) => { + const sourceText = buildBridgeText(spec); + + // First format pass + const formatted1 = prettyPrintToSource(sourceText); + // Second format pass + const formatted2 = prettyPrintToSource(formatted1); + + // Must be identical — formatting is idempotent. + assert.equal( + formatted2, + formatted1, + "prettyPrintToSource must be idempotent\n--- FIRST ---\n" + + formatted1 + + "\n--- SECOND ---\n" + + formatted2, + ); + }), + { + numRuns: 2_000, + endOnFailure: true, + }, + ); + }); + + test( + "prettyPrintToSource output is always parseable", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(bridgeTextSpecArb, (spec) => { + const sourceText = buildBridgeText(spec); + const formatted = prettyPrintToSource(sourceText); + + try { + parseBridge(formatted); + } catch (error) { + assert.fail( + `prettyPrintToSource produced unparsable output: ${String(error)}\n--- SOURCE ---\n${sourceText}\n--- FORMATTED ---\n${formatted}`, + ); + } + }), + { + numRuns: 2_000, + endOnFailure: true, + }, + ); + }, + ); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ff474e30..b3bf281d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -127,6 +127,9 @@ importers: '@types/node': specifier: ^25.3.3 version: 25.3.3 + fast-check: + specifier: ^4.5.3 + version: 4.5.3 graphql: specifier: 16.13.1 version: 16.13.1 @@ -240,6 +243,9 @@ importers: '@types/node': specifier: ^25.3.3 version: 25.3.3 + fast-check: + specifier: ^4.5.3 + version: 4.5.3 typescript: specifier: ^5.9.3 version: 5.9.3 @@ -412,7 +418,7 @@ importers: devDependencies: '@cloudflare/vite-plugin': specifier: ^1.26.0 - version: 1.26.0(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2))(workerd@1.20260305.0)(wrangler@4.70.0(@cloudflare/workers-types@4.20260305.1)) + version: 1.26.0(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2))(workerd@1.20260301.1)(wrangler@4.70.0(@cloudflare/workers-types@4.20260305.1)) '@cloudflare/workers-types': specifier: ^4.20260305.1 version: 4.20260305.1 @@ -710,60 +716,30 @@ packages: cpu: [x64] os: [darwin] - '@cloudflare/workerd-darwin-64@1.20260305.0': - resolution: {integrity: sha512-chhKOpymo0Eh9J3nymrauMqKGboCc4uz/j0gA1G4gioMnKsN2ZDKJ+qjRZDnCoVGy8u2C4pxlmyIfsXCAfIzhQ==} - engines: {node: '>=16'} - cpu: [x64] - os: [darwin] - '@cloudflare/workerd-darwin-arm64@1.20260301.1': resolution: {integrity: sha512-PPIetY3e67YBr9O4UhILK8nbm5TqUDl14qx4rwFNrRSBOvlzuczzbd4BqgpAtbGVFxKp1PWpjAnBvGU/OI/tLQ==} engines: {node: '>=16'} cpu: [arm64] os: [darwin] - '@cloudflare/workerd-darwin-arm64@1.20260305.0': - resolution: {integrity: sha512-K9aG2OQk5bBfOP+fyGPqLcqZ9OR3ra6uwnxJ8f2mveq2A2LsCI7ZeGxQiAj75Ti80ytH/gJffZIx4Np2JtU3aQ==} - engines: {node: '>=16'} - cpu: [arm64] - os: [darwin] - '@cloudflare/workerd-linux-64@1.20260301.1': resolution: {integrity: sha512-Gu5vaVTZuYl3cHa+u5CDzSVDBvSkfNyuAHi6Mdfut7TTUdcb3V5CIcR/mXRSyMXzEy9YxEWIfdKMxOMBjupvYQ==} engines: {node: '>=16'} cpu: [x64] os: [linux] - '@cloudflare/workerd-linux-64@1.20260305.0': - resolution: {integrity: sha512-tt7XUoIw/cYFeGbkPkcZ6XX1aZm26Aju/4ih+DXxOosbBeGshFSrNJDBfAKKOvkjsAZymJ+WWVDBU+hmNaGfwA==} - engines: {node: '>=16'} - cpu: [x64] - os: [linux] - '@cloudflare/workerd-linux-arm64@1.20260301.1': resolution: {integrity: sha512-igL1pkyCXW6GiGpjdOAvqMi87UW0LMc/+yIQe/CSzuZJm5GzXoAMrwVTkCFnikk6JVGELrM5x0tGYlxa0sk5Iw==} engines: {node: '>=16'} cpu: [arm64] os: [linux] - '@cloudflare/workerd-linux-arm64@1.20260305.0': - resolution: {integrity: sha512-72QTkY5EzylmvCZ8ZTrnJ9DctmQsfSof1OKyOWqu/pv/B2yACfuPMikq8RpPxvVu7hhS0ztGP6ZvXz72Htq4Zg==} - engines: {node: '>=16'} - cpu: [arm64] - os: [linux] - '@cloudflare/workerd-windows-64@1.20260301.1': resolution: {integrity: sha512-Q0wMJ4kcujXILwQKQFc1jaYamVsNvjuECzvRrTI8OxGFMx2yq9aOsswViE4X1gaS2YQQ5u0JGwuGi5WdT1Lt7A==} engines: {node: '>=16'} cpu: [x64] os: [win32] - '@cloudflare/workerd-windows-64@1.20260305.0': - resolution: {integrity: sha512-BA0uaQPOaI2F6mJtBDqplGnQQhpXCzwEMI33p/TnDxtSk9u8CGIfBFuI6uqo8mJ6ijIaPjeBLGOn2CiRMET4qg==} - engines: {node: '>=16'} - cpu: [x64] - os: [win32] - '@cloudflare/workers-types@4.20260305.1': resolution: {integrity: sha512-835BZaIcgjuYIUqgOWJSpwQxFSJ8g/X1OCZFLO7bmirM6TGmVgIGwiGItBgkjUXXCPrYzJEldsJkuFuK7ePuMw==} @@ -4564,11 +4540,6 @@ packages: engines: {node: '>=16'} hasBin: true - workerd@1.20260305.0: - resolution: {integrity: sha512-JkhfCLU+w+KbQmZ9k49IcDYc78GBo7eG8Mir8E2+KVjR7otQAmpcLlsous09YLh8WQ3Bt3Mi6/WMStvMAPukeA==} - engines: {node: '>=16'} - hasBin: true - wrangler@4.70.0: resolution: {integrity: sha512-PNDZ9o4e+B5x+1bUbz62Hmwz6G9lw+I9pnYe/AguLddJFjfIyt2cmFOUOb3eOZSoXsrhcEPUg2YidYIbVwUkfw==} engines: {node: '>=20.0.0'} @@ -5172,15 +5143,9 @@ snapshots: optionalDependencies: workerd: 1.20260301.1 - '@cloudflare/unenv-preset@2.14.0(unenv@2.0.0-rc.24)(workerd@1.20260305.0)': - dependencies: - unenv: 2.0.0-rc.24 - optionalDependencies: - workerd: 1.20260305.0 - - '@cloudflare/vite-plugin@1.26.0(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2))(workerd@1.20260305.0)(wrangler@4.70.0(@cloudflare/workers-types@4.20260305.1))': + '@cloudflare/vite-plugin@1.26.0(vite@7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2))(workerd@1.20260301.1)(wrangler@4.70.0(@cloudflare/workers-types@4.20260305.1))': dependencies: - '@cloudflare/unenv-preset': 2.14.0(unenv@2.0.0-rc.24)(workerd@1.20260305.0) + '@cloudflare/unenv-preset': 2.14.0(unenv@2.0.0-rc.24)(workerd@1.20260301.1) miniflare: 4.20260301.1 unenv: 2.0.0-rc.24 vite: 7.3.1(@types/node@25.3.3)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2) @@ -5194,33 +5159,18 @@ snapshots: '@cloudflare/workerd-darwin-64@1.20260301.1': optional: true - '@cloudflare/workerd-darwin-64@1.20260305.0': - optional: true - '@cloudflare/workerd-darwin-arm64@1.20260301.1': optional: true - '@cloudflare/workerd-darwin-arm64@1.20260305.0': - optional: true - '@cloudflare/workerd-linux-64@1.20260301.1': optional: true - '@cloudflare/workerd-linux-64@1.20260305.0': - optional: true - '@cloudflare/workerd-linux-arm64@1.20260301.1': optional: true - '@cloudflare/workerd-linux-arm64@1.20260305.0': - optional: true - '@cloudflare/workerd-windows-64@1.20260301.1': optional: true - '@cloudflare/workerd-windows-64@1.20260305.0': - optional: true - '@cloudflare/workers-types@4.20260305.1': {} '@codemirror/autocomplete@6.20.1': @@ -9353,15 +9303,6 @@ snapshots: '@cloudflare/workerd-linux-arm64': 1.20260301.1 '@cloudflare/workerd-windows-64': 1.20260301.1 - workerd@1.20260305.0: - optionalDependencies: - '@cloudflare/workerd-darwin-64': 1.20260305.0 - '@cloudflare/workerd-darwin-arm64': 1.20260305.0 - '@cloudflare/workerd-linux-64': 1.20260305.0 - '@cloudflare/workerd-linux-arm64': 1.20260305.0 - '@cloudflare/workerd-windows-64': 1.20260305.0 - optional: true - wrangler@4.70.0(@cloudflare/workers-types@4.20260305.1): dependencies: '@cloudflare/kv-asset-handler': 0.4.2 From 49f152eee81320027a251c6f1af65b74ef8ae552 Mon Sep 17 00:00:00 2001 From: Aarne Laur Date: Fri, 6 Mar 2026 11:44:08 +0100 Subject: [PATCH 2/5] feat: implement BridgeTimeoutError for tool timeout handling and enhance fuzz testing documentation --- .changeset/aot-timeout-error-class.md | 10 + docs/fuzz-expansion.md | 137 --------- docs/fuzz-testing.md | 120 ++++++++ packages/bridge-compiler/src/codegen.ts | 14 +- .../bridge-compiler/src/execute-bridge.ts | 8 +- packages/bridge-compiler/test/README.md | 58 ---- packages/bridge-compiler/test/codegen.test.ts | 2 +- .../test/fuzz-regressions.todo.test.ts | 46 ++- .../test/fuzz-runtime-parity.test.ts | 270 +++++++++++++++++- packages/bridge/test/fuzz-parser.test.ts | 126 +++++++- 10 files changed, 572 insertions(+), 219 deletions(-) create mode 100644 .changeset/aot-timeout-error-class.md delete mode 100644 docs/fuzz-expansion.md create mode 100644 docs/fuzz-testing.md delete mode 100644 packages/bridge-compiler/test/README.md diff --git a/.changeset/aot-timeout-error-class.md b/.changeset/aot-timeout-error-class.md new file mode 100644 index 00000000..9dd6fcb7 --- /dev/null +++ b/.changeset/aot-timeout-error-class.md @@ -0,0 +1,10 @@ +--- +"@stackables/bridge-compiler": patch +--- + +Fix AOT compiler to throw `BridgeTimeoutError` on tool timeout + +AOT-compiled bridges now throw `BridgeTimeoutError` (with the same name and +message format as the runtime) when a tool exceeds `toolTimeoutMs`. Previously +the generated code constructed a generic `Error`, causing a class mismatch when +callers caught and inspected the error type. diff --git a/docs/fuzz-expansion.md b/docs/fuzz-expansion.md deleted file mode 100644 index cd9628f7..00000000 --- a/docs/fuzz-expansion.md +++ /dev/null @@ -1,137 +0,0 @@ -# Fuzz Testing Expansion Plan - -> Tracking document for expanding fuzz/property-based testing coverage across the Bridge codebase. - -## Current State - -All fuzzing lives in `packages/bridge-compiler/test/fuzz-compile.test.ts` (6 tests, ~24,500 property runs). -`fast-check` is a devDependency of `bridge-compiler` only. - -**What's covered:** - -- JS syntax validity of AOT compiler output -- Compiler determinism -- AOT/runtime parity on flat single-segment paths with `fc.jsonValue()` inputs -- Parser round-trip (text spec → parse → serialize → reparse → execute parity) -- Fallback-heavy and logical-wire syntax validity - -**Key gaps:** - -- No deep-path parity testing with chaotic inputs (`NaN`, `undefined`, nested shapes) -- No purely textual parser fuzzing (random `.bridge` strings) -- No serializer round-trip fuzzing on full ASTs (only limited text specs) -- No stdlib tool fuzzing (known crash in `arr.filter`/`arr.find` on non-array input) -- No array mapping / shadow tree parity testing -- No simulated tool call parity testing - ---- - -## P0 — Critical (implement first) - -### [x] 1A. Deep-path AOT/runtime parity with chaotic inputs - -**File:** `packages/bridge-compiler/test/fuzz-runtime-parity.test.ts` - -- `chaosInputArb` that extends beyond `fc.jsonValue()`: includes `NaN`, `Infinity`, `-0`, `undefined`, empty strings, deeply nested objects (5+ levels), arrays where objects expected and vice versa -- `deepBridgeArb` using multi-segment paths (1–4 segments) instead of `flatPathArb`; all input refs use `rootSafe: true` + `pathSafe` for safe navigation -- Property: `executeRuntime` vs `executeAot` → `deepEqual` on `.data`, or both throw the same error class -- 3,000 runs - -**Regressions discovered during implementation (tracked in `fuzz-regressions.todo.test.ts`):** - -- Unsafe path traversal divergence: AOT silently returns `undefined` where runtime throws `TypeError` when `rootSafe` is not set (seeds 1798655022, -481664925) -- Fallback chain null/undefined divergence: AOT returns `null` where runtime returns `undefined` when a fallback constant resolves to `"null"` — see `deepFallbackBridgeArb` investigation needed - -**What this catches:** Type coercion divergence (verified working), NaN propagation, deep path access with missing keys. - -### [x] 1B. arr.filter / arr.find crash proof + bug fix - -**File:** `packages/bridge-stdlib/test/fuzz-stdlib.test.ts` -**Fixes applied in `packages/bridge-stdlib/src/tools/`:** - -- `arrays.ts`: Added `Array.isArray(arr)` guard in `filter` and `find` -- `arrays.ts`: Added `obj == null || typeof obj !== "object"` guard inside filter/find callback (null array elements) -- `strings.ts`: Replaced `?.` optional chaining with `typeof === "string"` guards in all four string tools (the `?.` only guards null/undefined, not non-string types like arrays whose method property is `undefined`, causing `TypeError: undefined is not a function`) - -- 2,000 runs per tool, across both array and string tools - ---- - -## P1 — Important (implement after P0) - -### [x] 2A. Parser textual fuzzing — `parseBridge` never panics - -**File:** `packages/bridge/test/fuzz-parser.test.ts` - -- `bridgeTokenSoupArb` — weighted mix of Bridge-like tokens (keywords, identifiers, `{`, `}`, `<-`, `=`, `.`, `||`, `??`) + random noise -- Property: `parseBridge(text)` either returns a `BridgeDocument` or throws a standard `Error`. Must never throw `TypeError`, `RangeError`, or crash with a stack overflow. -- 5,000 runs - -**What this catches:** Null dereferences in CST→AST visitor, unbounded recursion, Chevrotain edge cases. - -### [x] 2B. Parser textual fuzzing — `parseBridgeDiagnostics` never crashes - -**File:** same file, separate test - -- Same `bridgeTokenSoupArb` -- Property: `parseBridgeDiagnostics(text)` **always** returns `{ document, diagnostics }`, never throws -- 5,000 runs - -**What this catches:** Uncaught exceptions in recovery mode, diagnostic formatting crashes. - -### [x] 3A. Serializer round-trip (AST → text → AST) - -**File:** same file, separate test - -- Valid `.bridge` text → `parseBridge` → `serializeBridge` → `parseBridge` → instruction count and wire count must match -- Also: `prettyPrintToSource` idempotence — `format(format(text)) === format(text)` -- Also: `prettyPrintToSource` output is always parseable -- 2,000 runs each - ---- - -## P2 — Valuable (future work) - -### [ ] 1B-ext. Array mapping parity - -- Generate bridges with `arrayIterators` (the `[] as iter { ... }` pattern) -- Test shadow tree execution parity between AOT and runtime -- Complex arbitrary: needs `element: true` NodeRefs, iterator handles - -### [ ] 1C. Simulated tool call parity - -- Generate bridges referencing mock tools (`identity`, `fail`) -- Test tool error propagation + `catchFallback` + `onError` wires between engines - -### [ ] 3B. `prettyPrintToSource` advanced stability - -- Deeper formatter testing with all block types (tool, define, const, bridge) - -### [ ] 4C. `httpCall` input surface - -- URL construction edge cases, `JSON.stringify` on circular refs -- Requires fetch mocking - ---- - -## P3 — Nice to have - -### [ ] Additional string tool coverage - -- Confirm Symbol, BigInt, circular refs are handled gracefully across all stdlib tools - -### [ ] Parser diagnostics completeness - -- Valid text parsed via `parseBridgeDiagnostics` produces zero error-severity diagnostics -- Returned document matches strict `parseBridge` output - ---- - -## Implementation Notes - -- **Test framework:** `node:test` + `node:assert` (no Jest/Vitest) -- **Fuzz library:** `fast-check` ^4.5.3 -- **Regression workflow:** fuzz finding → `test.todo` entry with seed → tracking issue → deterministic reproducer → fix → cleanup (see `packages/bridge-compiler/test/README.md`) -- Parser fuzz tests go in `packages/bridge/test/` (that's where all parser/integration tests live) -- Stdlib fuzz tests go in `packages/bridge-stdlib/test/` -- Run a single test: `node --experimental-transform-types --conditions source --test ` diff --git a/docs/fuzz-testing.md b/docs/fuzz-testing.md new file mode 100644 index 00000000..c74db36d --- /dev/null +++ b/docs/fuzz-testing.md @@ -0,0 +1,120 @@ +# Fuzz Testing + +> Reference document for fuzz/property-based testing coverage and workflow in the Bridge codebase. + +## Overview + +Bridge uses [fast-check](https://github.com/dubzzz/fast-check) (^4.5.3) for property-based testing alongside the standard `node:test` + `node:assert` framework. Fuzz tests live co-located with each package and run as part of the normal `pnpm test` suite. + +### Test files + +| File | Package | Purpose | +| ------------------------------------ | ----------------- | ------------------------------------------------------------------- | +| `test/fuzz-compile.test.ts` | `bridge-compiler` | JS syntax validity, determinism, flat-path AOT/runtime parity | +| `test/fuzz-runtime-parity.test.ts` | `bridge-compiler` | Deep-path parity, array mapping parity, tool-call timeout parity | +| `test/fuzz-regressions.todo.test.ts` | `bridge-compiler` | Backlog of known fuzz-discovered divergences as `test.todo` entries | +| `test/fuzz-stdlib.test.ts` | `bridge-stdlib` | Array and string tool crash-safety | +| `test/fuzz-parser.test.ts` | `bridge` | Parser crash-safety, serializer round-trip, formatter stability | + +--- + +## Coverage + +### What's tested + +- JS syntax validity of AOT compiler output +- Compiler determinism +- AOT/runtime parity on flat single-segment paths (`fc.jsonValue()` inputs) +- AOT/runtime parity on deep multi-segment paths with chaotic inputs (`NaN`, `Infinity`, `-0`, `undefined`, deeply nested objects) +- AOT/runtime parity on array-mapping bridges (`[] as el { ... }`) with chaotic element data +- AOT/runtime parity on tool-call timeout (`BridgeTimeoutError` class and message match) +- Parser round-trip: text → parse → serialize → reparse → execute parity +- `parseBridge` never throws unstructured errors on random input +- `parseBridgeDiagnostics` never throws (LSP/IDE safety) +- `prettyPrintToSource` idempotence and output parseability (bridge, tool, const blocks) +- `arr.filter`, `arr.find`, `arr.first`, `arr.toArray` crash-safety on any input type +- `str.toLowerCase`, `str.toUpperCase`, `str.trim`, `str.length` crash-safety on any input type + +### Known gaps (P3) + +- `Symbol`, `BigInt`, circular-ref handling across all stdlib tools +- `parseBridgeDiagnostics` completeness: valid input should produce zero error-severity diagnostics + +--- + +## Property run counts + +| Test | Runs | +| ---------------------------------------------------- | ----- | +| Deep-path AOT/runtime parity | 3,000 | +| Array mapping parity | 1,000 | +| Tool-call timeout parity | 500 | +| `parseBridge` never panics | 5,000 | +| `parseBridgeDiagnostics` never throws | 5,000 | +| Serializer round-trip | 2,000 | +| `prettyPrintToSource` idempotence (basic) | 2,000 | +| `prettyPrintToSource` parseability (basic) | 2,000 | +| `prettyPrintToSource` idempotence (extended blocks) | 1,000 | +| `prettyPrintToSource` parseability (extended blocks) | 1,000 | +| stdlib tool crash-safety (per tool) | 2,000 | + +--- + +## Generator design principles + +**Text-first over AST-first.** Generating valid `.bridge` text strings and parsing them is preferred over building `Bridge` AST objects directly with `fc.letrec`. Text-first generation avoids exponential shrinking blowup: fast-check shrinks by removing tokens from a string, not by exploring recursive AST tree variants. This is especially important for array mapping and nested-block tests. + +**Depth limits with `fc.letrec`.** When recursive arbitraries are necessary (e.g. `chaosValueArb` for deep input objects), always pass `depthFactor` or cap with `maxLength`/`maxKeys` at every level. Without this, the shrinking phase can explore exponentially many candidates and halt CI. + +**Safety margins for timing tests.** Timer-based parity tests skip inputs in the "grey zone" where `|toolDelay - toolTimeout| < 20ms` to avoid flakiness on slow CI runners. + +**Forbidden path segments.** All generated identifier arbitraries filter out `__proto__`, `prototype`, and `constructor` to stay within the valid domain for path traversal. + +--- + +## Regression workflow + +When a fuzz run finds a new issue: + +1. **Capture evidence immediately** — seed, failure path, counterexample input, whether it is `AOT != runtime`, a parser crash, or a runtime panic. + +2. **Add a `test.todo` entry** in `packages/bridge-compiler/test/fuzz-regressions.todo.test.ts`: + + ```ts + test.todo("class label — short description (seed=123456)"); + ``` + +3. **Open a tracking note** — link to the todo, add impact, expected vs actual behaviour, suspected component. + +4. **Create a deterministic reproducer** — prefer a minimal hand-authored bridge + input over rerunning fuzz with a seed. Add it to `codegen.test.ts` or a dedicated regression file as a normal `test(...)`. + +5. **Fix at root cause** — keep fixes small and targeted. + +6. **Promote and clean up** — ensure reproducer passes, remove the `test.todo` entry, keep the fuzz property in place. + +--- + +## Running fuzz tests + +```bash +# All tests (includes fuzz) +pnpm test + +# Single fuzz file +node --experimental-transform-types --conditions source --test packages/bridge-compiler/test/fuzz-runtime-parity.test.ts +node --experimental-transform-types --conditions source --test packages/bridge/test/fuzz-parser.test.ts +node --experimental-transform-types --conditions source --test packages/bridge-stdlib/test/fuzz-stdlib.test.ts + +# Reproduce a specific failing seed +# Add { seed: -1234567, path: "0", endOnFailure: true } to fc.assert options +``` + +--- + +## Implementation notes + +- **Test framework:** `node:test` + `node:assert` (no Jest/Vitest) +- **Fuzz library:** `fast-check` ^4.5.3 — devDependency of `bridge-compiler`, `bridge-stdlib`, `bridge` +- Parser fuzz tests live in `packages/bridge/test/` +- Stdlib fuzz tests live in `packages/bridge-stdlib/test/` +- Compiler parity fuzz tests live in `packages/bridge-compiler/test/` diff --git a/packages/bridge-compiler/src/codegen.ts b/packages/bridge-compiler/src/codegen.ts index f4245c83..9dfc978c 100644 --- a/packages/bridge-compiler/src/codegen.ts +++ b/packages/bridge-compiler/src/codegen.ts @@ -146,7 +146,8 @@ function detectControlFlow( for (const w of wires) { if ("fallbacks" in w && w.fallbacks) { for (const fb of w.fallbacks) { - if (fb.control) return fb.control.kind as "break" | "continue" | "throw" | "panic"; + if (fb.control) + return fb.control.kind as "break" | "continue" | "throw" | "panic"; } } if ("catchControl" in w && w.catchControl) { @@ -638,6 +639,9 @@ class CodegenContext { lines.push( ` const __BridgeAbortError = __opts?.__BridgeAbortError ?? class extends Error { constructor(m) { super(m ?? "Execution aborted by external signal"); this.name = "BridgeAbortError"; } };`, ); + lines.push( + ` const __BridgeTimeoutError = __opts?.__BridgeTimeoutError ?? class extends Error { constructor(n, ms) { super('Tool "' + n + '" timed out after ' + ms + 'ms'); this.name = "BridgeTimeoutError"; } };`, + ); lines.push(` const __signal = __opts?.signal;`); lines.push(` const __timeoutMs = __opts?.toolTimeoutMs ?? 0;`); lines.push( @@ -652,7 +656,7 @@ class CodegenContext { lines.push(` let result;`); lines.push(` if (__timeoutMs > 0) {`); lines.push( - ` let t; const timeout = new Promise((_, rej) => { t = setTimeout(() => rej(new Error("Tool timeout")), __timeoutMs); });`, + ` let t; const timeout = new Promise((_, rej) => { t = setTimeout(() => rej(new __BridgeTimeoutError(toolName, __timeoutMs)), __timeoutMs); });`, ); lines.push( ` try { result = await Promise.race([p, timeout]); } finally { clearTimeout(t); }`, @@ -1810,7 +1814,7 @@ class CodegenContext { const controlWire = elemWires.find( (w) => w.to.path.length === 1 && - (("fallbacks" in w && w.fallbacks?.some(fb => fb.control != null)) || + (("fallbacks" in w && w.fallbacks?.some((fb) => fb.control != null)) || ("catchControl" in w && w.catchControl != null)), ); @@ -1833,7 +1837,9 @@ class CodegenContext { // Determine the check type const isNullish = - controlWire.fallbacks?.some(fb => fb.type === "nullish" && fb.control != null) ?? false; + controlWire.fallbacks?.some( + (fb) => fb.type === "nullish" && fb.control != null, + ) ?? false; if (mode === "continue") { if (isNullish) { diff --git a/packages/bridge-compiler/src/execute-bridge.ts b/packages/bridge-compiler/src/execute-bridge.ts index 8c81ace9..af29b206 100644 --- a/packages/bridge-compiler/src/execute-bridge.ts +++ b/packages/bridge-compiler/src/execute-bridge.ts @@ -17,6 +17,7 @@ import { TraceCollector, BridgePanicError, BridgeAbortError, + BridgeTimeoutError, } from "@stackables/bridge-core"; import { std as bundledStd } from "@stackables/bridge-stdlib"; import { compileBridge } from "./codegen.ts"; @@ -98,6 +99,7 @@ type BridgeFn = ( ) => void; __BridgePanicError?: new (...args: any[]) => Error; __BridgeAbortError?: new (...args: any[]) => Error; + __BridgeTimeoutError?: new (...args: any[]) => Error; }, ) => Promise; @@ -112,10 +114,7 @@ const AsyncFunction = Object.getPrototypeOf(async function () {}) const fnCache = new WeakMap>(); /** Build a cache key that includes the sorted requestedFields. */ -function cacheKey( - operation: string, - requestedFields?: string[], -): string { +function cacheKey(operation: string, requestedFields?: string[]): string { if (!requestedFields || requestedFields.length === 0) return operation; return `${operation}:${[...requestedFields].sort().join(",")}`; } @@ -253,6 +252,7 @@ export async function executeBridge( logger, __BridgePanicError: BridgePanicError, __BridgeAbortError: BridgeAbortError, + __BridgeTimeoutError: BridgeTimeoutError, __trace: tracer ? ( toolName: string, diff --git a/packages/bridge-compiler/test/README.md b/packages/bridge-compiler/test/README.md deleted file mode 100644 index 38c43bc0..00000000 --- a/packages/bridge-compiler/test/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# bridge-compiler test workflow - -This folder contains unit tests, fuzz/property tests, and a backlog of known fuzz-discovered issues. - -## Files and intent - -- `codegen.test.ts` — deterministic, scenario-based behavior tests. -- `fuzz-compile.test.ts` — property/fuzz tests (syntax safety, determinism, AOT/runtime parity). -- `fuzz-regressions.todo.test.ts` — **backlog** of known issues as `test.todo(...)` entries. - -## Why `test.todo` exists here - -Fuzzing can find valid issues faster than we can fix them. `test.todo` is used to: - -1. Preserve findings immediately (so they are not lost). -2. Keep CI green while investigation/fix work is queued. -3. Make known risk areas visible in test output. - -`test.todo` is not a permanent state. It is a staging area between discovery and a real executable regression test. - -## Preferred process when fuzz finds an issue - -1. **Capture evidence immediately** - - Seed, failure path, and minimized/counterexample input. - - Whether mismatch is `AOT != runtime`, parser/serializer, or runtime crash. - -2. **Add a `test.todo` entry** in `fuzz-regressions.todo.test.ts` - - Include a short class label plus seed (if available). - - Example format: `"nullish fallback parity ... (seed=123456)"`. - -3. **Open a tracking issue/PR note** - - Link to the todo label. - - Add impact, expected behavior, and suspected component (`bridge-core`, `bridge-compiler`, `bridge-parser`). - -4. **Create a deterministic reproducer test** - - Prefer a minimal hand-authored bridge/input over rerunning random fuzz. - - Add to `codegen.test.ts` (or a dedicated regression file) as a normal `test(...)`. - -5. **Fix at root cause** - - Update compiler/runtime/parser behavior. - - Keep fix small and targeted. - -6. **Promote and clean up** - - Ensure reproducer test passes. - - Remove corresponding `test.todo` entry. - - Keep fuzz property in place to guard against nearby regressions. - -## How we fix issues without losing them - -- Discovery path: fuzz failure -> `test.todo` backlog entry. -- Stabilization path: add deterministic reproducer -> implement fix -> remove todo. -- Verification path: run package tests (`pnpm --filter @stackables/bridge-compiler test`) and then broader repo checks as needed. - -## Practical tips - -- Keep generated identifiers/simple values parser-safe in text round-trip fuzzers. -- Constrain parity fuzz generators when an oracle (runtime/parser) has known unstable surfaces. -- Prefer multiple small targeted properties over one giant mixed generator for easier shrinking and diagnosis. diff --git a/packages/bridge-compiler/test/codegen.test.ts b/packages/bridge-compiler/test/codegen.test.ts index 7cc7ebe6..d8224223 100644 --- a/packages/bridge-compiler/test/codegen.test.ts +++ b/packages/bridge-compiler/test/codegen.test.ts @@ -1231,7 +1231,7 @@ bridge Query.test { }, toolTimeoutMs: 50, }), - /Tool timeout/, + /timed out/, ); }); }); diff --git a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts index 5f9c02b7..4bba5c27 100644 --- a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts +++ b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts @@ -1,23 +1,43 @@ -import { describe } from "node:test"; +import { describe, test } from "node:test"; describe("fuzz-discovered AOT/runtime divergence backlog", () => { - // test.todo( - // "parser round-trip: serializeBridge output can be unparsable for some valid parsed documents (seed=1864118703)", - // ); // AOT compiler uses `?.` safe-navigation everywhere in generated code; runtime // throws TypeError for unsafe path traversal when `rootSafe` is not set. // These seeds reproduce the divergence via fuzz-runtime-parity.test.ts. - // test.todo( - // "deep-path parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=1798655022)", - // ); - // test.todo( - // "deep-path fallback parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=-481664925)", - // ); + test.todo( + "deep-path parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=1798655022)", + ); + test.todo( + "deep-path fallback parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=-481664925)", + ); // AOT and runtime diverge on null-vs-undefined semantics when fallback constant // wires (catchFallback / falsy fallbacks) resolve to "null". The AOT emits null // while the runtime applies overdefinition coalescing and produces undefined. // Repro: fuzz-runtime-parity deepFallbackBridgeArb + chaosInputArb, any random seed. - // test.todo( - // "fallback parity: AOT returns null where runtime returns undefined when fallback constant resolves to null (seed=random)", - // ); + test.todo( + "fallback parity: AOT returns null where runtime returns undefined when fallback constant resolves to null (seed=random)", + ); + // Array mapping: when an array contains null elements, the runtime throws TypeError + // when accessing a field on the null element (element refs lack rootSafe), while + // the AOT generates element?.field and silently returns undefined. + // Repro: fuzz-runtime-parity arrayBridgeSpec + [null] element in source array. + test.todo( + "array mapping: runtime throws TypeError for null array elements; AOT returns undefined (null element divergence)", + ); + // Array mapping: when a bridge has `.elemField <- el.elemField` where elemField + // equals the source array path (e.g. `o.items <- i.data[] as el { .data <- el.data }`), + // the runtime conflates the shadow-tree element wire with the outer input-array source + // wire because they have the same trunk key (element flag not factored into trunkKey). + // AOT correctly handles this via separate code paths for element refs. + // Repro: arrayBridgeSpec where elemFields contains srcField. + test.todo( + "array mapping: element field with same name as source field causes trunk key collision in runtime", + ); + // Array mapping: when source value is a non-array non-null (e.g. a string), AOT + // throws TypeError (.map is not a function), while the runtime iterates iterable + // types (strings char-by-char) via createShadowArray. Numbers return []. + // Repro: arrayBridgeSpec + source = "hello" or 42. + test.todo( + "array mapping: non-array source diverges — AOT throws TypeError, runtime uses iterable semantics", + ); }); diff --git a/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts index 8751edee..1380a4fd 100644 --- a/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts +++ b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts @@ -7,7 +7,11 @@ import type { NodeRef, Wire, } from "@stackables/bridge-core"; -import { executeBridge as executeRuntime } from "@stackables/bridge-core"; +import { + BridgeTimeoutError, + executeBridge as executeRuntime, +} from "@stackables/bridge-core"; +import { parseBridgeFormat } from "@stackables/bridge-parser"; import { executeBridge as executeAot } from "../src/index.ts"; // ── Shared infrastructure ─────────────────────────────────────────────────── @@ -226,3 +230,267 @@ describe("runtime parity fuzzing — deep paths + chaotic inputs", () => { // the AOT compiler and runtime diverge on null-vs-undefined semantics when // fallback constants resolve to null. Tracked in fuzz-regressions.todo.test.ts. }); + +// ── P2-1B-ext: Array mapping parity ─────────────────────────────────────── +// +// Design note (re: Suggestion 2 / AST depth limits): +// We generate valid .bridge TEXT rather than Bridge AST objects directly. +// This avoids fc.letrec recursive-depth explosions during the shrinking phase: +// fast-check shrinks text by removing tokens, not by exploring AST tree variants, +// so there is no exponential blowup. Text is bounded by the token-count limit. + +const canonicalIdArb = fc.constantFrom("a", "b", "c", "d", "e", "f", "g", "h"); + +// A single "leaf" value — safe for array element fields. +const chaosLeafArb2 = fc.oneof( + { weight: 4, arbitrary: fc.string({ maxLength: 32 }) }, + { weight: 3, arbitrary: fc.integer() }, + { weight: 2, arbitrary: fc.boolean() }, + { weight: 2, arbitrary: fc.constant(null) }, + { weight: 1, arbitrary: fc.constant("") }, + { weight: 1, arbitrary: fc.double({ noNaN: false }) }, // includes NaN +); + +// A single array element object. +// Note: null or primitive elements trigger a known null/undefined divergence — +// the runtime throws TypeError when accessing `.field` on a null element (no +// rootSafe on element refs from the parser), while AOT silently returns +// undefined via `?.`. Tracked in fuzz-regressions.todo.test.ts. +// We restrict to objects here so the test covers value-type parity, not the +// null-element divergence. +const chaosElementArb = fc.dictionary(canonicalIdArb, chaosLeafArb2, { + maxKeys: 4, +}); + +// Source value for the array field: one of several chaotic shapes. +// undefined is excluded: AOT's `?.map(...) ?? null` returns null, runtime returns +// undefined — the same null/undefined divergence tracked in fuzz-regressions.todo.test.ts. +// Primitive (string/number) sources are excluded: AOT throws TypeError (.map is not +// a function), while the runtime iterates strings character-by-character (strings +// are iterable) or treats numbers as empty. Both tracked in regressions. +const chaosArraySourceArb = fc.oneof( + { weight: 5, arbitrary: fc.array(chaosElementArb, { maxLength: 8 }) }, + { weight: 2, arbitrary: fc.constant(null) }, + { weight: 1, arbitrary: fc.constant([]) }, +); + +const arrayBridgeSpecArb = fc + .record({ + type: canonicalIdArb, + field: canonicalIdArb, + // source field on the input (the array) + srcField: canonicalIdArb, + // output field for the mapped array + outField: canonicalIdArb, + // element fields to map inside the iterator block (max 3 to keep bridges concise) + elemFields: fc.uniqueArray(canonicalIdArb, { minLength: 1, maxLength: 3 }), + }) + // Filter out cases where element field names overlap with srcField or outField. + // When elemField == srcField, the runtime resolver conflates the shadow-tree + // element wire with the outer input-array source wire (same trunk key), + // producing wrong values. Tracked in fuzz-regressions.todo.test.ts. + .filter( + (spec) => + !spec.elemFields.includes(spec.srcField) && + !spec.elemFields.includes(spec.outField) && + spec.srcField !== spec.outField, + ); + +function buildArrayBridgeText(spec: { + type: string; + field: string; + srcField: string; + outField: string; + elemFields: string[]; +}): string { + const lines = [ + "version 1.5", + `bridge ${spec.type}.${spec.field} {`, + " with input as i", + " with output as o", + "", + ` o.${spec.outField} <- i.${spec.srcField}[] as el {`, + ]; + for (const f of spec.elemFields) { + lines.push(` .${f} <- el.${f}`); + } + lines.push(" }"); + lines.push("}"); + return lines.join("\n"); +} + +describe("runtime parity fuzzing — array mapping (P2-1B-ext)", () => { + test( + "AOT matches runtime on array-mapping bridges with chaotic inputs", + { timeout: 120_000 }, + async () => { + await fc.assert( + fc.asyncProperty( + arrayBridgeSpecArb, + chaosArraySourceArb, + async (spec, sourceValue) => { + const bridgeText = buildArrayBridgeText(spec); + const document = parseBridgeFormat(bridgeText); + const operation = `${spec.type}.${spec.field}`; + const input = { [spec.srcField]: sourceValue }; + + let runtimeResult: { data: any } | undefined; + let runtimeError: unknown; + let aotResult: { data: any } | undefined; + let aotError: unknown; + + try { + runtimeResult = await executeRuntime({ + document, + operation, + input, + tools: {}, + }); + } catch (err) { + runtimeError = err; + } + try { + aotResult = await executeAot({ + document, + operation, + input, + tools: {}, + }); + } catch (err) { + aotError = err; + } + + if (runtimeError && !aotError) { + assert.fail( + `Runtime threw but AOT did not.\nBridge:\n${bridgeText}\nInput: ${JSON.stringify(input)}\nRuntime error: ${runtimeError}\nAOT data: ${JSON.stringify(aotResult?.data)}`, + ); + } + if (!runtimeError && aotError) { + assert.fail( + `AOT threw but runtime did not.\nBridge:\n${bridgeText}\nInput: ${JSON.stringify(input)}\nAOT error: ${aotError}\nRuntime data: ${JSON.stringify(runtimeResult?.data)}`, + ); + } + if (runtimeError && aotError) { + // Both threw — acceptable regardless of error class. + // Array mapping error-class divergence from mismatched input + // types (e.g. non-array values) is a known engine behaviour + // difference tracked separately. + return; + } + + assert.deepEqual(aotResult!.data, runtimeResult!.data); + }, + ), + { numRuns: 1_000, endOnFailure: true }, + ); + }, + ); +}); + +// ── P2-1C: Simulated tool call parity with timeout fuzzing ──────────────── +// +// Tests that AOT and runtime agree on success/failure under varying tool delays +// and timeout settings. +// +// Design note (re: Suggestion 1 / timeout fuzzing): +// The original AOT preamble threw new Error("Tool timeout"), diverging from the +// runtime's BridgeTimeoutError. This was fixed before this test was added — both +// engines now throw BridgeTimeoutError with the same message format. +// +// We avoid flakiness by maintaining a 20ms safety margin around the timeout +// boundary. Tests in the "grey zone" (|delay - timeout| < 20ms) are skipped. +// +// Promise leak concern: both engines clear their timer in try/finally (AOT) or +// .finally() (runtime raceTimeout), so the timeout Promise itself never leaks. +// The underlying tool function may still be pending but that is inherent to +// JavaScript Promises with no native cancellation. + +const toolCallBridgeText = `version 1.5 +bridge Query.toolTest { + with mockTool as t + with output as o + o.value <- t.value +}`; + +const toolCallDocument = parseBridgeFormat(toolCallBridgeText); + +const timeoutParityArb = fc.record({ + toolDelayMs: fc.integer({ min: 0, max: 80 }), + toolTimeoutMs: fc.integer({ min: 10, max: 50 }), +}); + +describe("runtime parity fuzzing — tool call timeout (P2-1C)", () => { + test( + "AOT and runtime agree on success/failure for varying tool delays and timeouts", + { timeout: 120_000 }, + async () => { + await fc.assert( + fc.asyncProperty( + timeoutParityArb, + async ({ toolDelayMs, toolTimeoutMs }) => { + // Skip timing-sensitive grey zone to avoid flakiness on slow CI. + const margin = 20; + const clearlyTimedOut = toolDelayMs > toolTimeoutMs + margin; + const clearlySucceeds = toolDelayMs < toolTimeoutMs - margin; + if (!clearlyTimedOut && !clearlySucceeds) return; + + const mockTool = async () => { + await new Promise((r) => setTimeout(r, toolDelayMs)); + return { value: "ok" }; + }; + const opts = { + document: toolCallDocument, + operation: "Query.toolTest", + input: {}, + tools: { mockTool }, + toolTimeoutMs, + }; + + let runtimeResult: { data: any } | undefined; + let runtimeError: unknown; + let aotResult: { data: any } | undefined; + let aotError: unknown; + + try { + runtimeResult = await executeRuntime(opts); + } catch (err) { + runtimeError = err; + } + try { + aotResult = await executeAot(opts); + } catch (err) { + aotError = err; + } + + if (clearlyTimedOut) { + // Both must throw BridgeTimeoutError. + assert.ok( + runtimeError instanceof BridgeTimeoutError, + `Runtime should throw BridgeTimeoutError (delay=${toolDelayMs}ms, timeout=${toolTimeoutMs}ms), got: ${runtimeError}`, + ); + assert.equal( + (aotError as Error)?.name, + "BridgeTimeoutError", + `AOT should throw BridgeTimeoutError (delay=${toolDelayMs}ms, timeout=${toolTimeoutMs}ms), got: ${aotError}`, + ); + } else { + // Both must succeed with the same data. + assert.equal( + runtimeError, + undefined, + `Runtime should not throw (delay=${toolDelayMs}ms, timeout=${toolTimeoutMs}ms): ${runtimeError}`, + ); + assert.equal( + aotError, + undefined, + `AOT should not throw (delay=${toolDelayMs}ms, timeout=${toolTimeoutMs}ms): ${aotError}`, + ); + assert.deepEqual(aotResult!.data, runtimeResult!.data); + } + }, + ), + { numRuns: 500, endOnFailure: true }, + ); + }, + ); +}); diff --git a/packages/bridge/test/fuzz-parser.test.ts b/packages/bridge/test/fuzz-parser.test.ts index b2318c61..f7f46fe0 100644 --- a/packages/bridge/test/fuzz-parser.test.ts +++ b/packages/bridge/test/fuzz-parser.test.ts @@ -123,9 +123,19 @@ function buildBridgeText(spec: { | { kind: "pull"; to: string; from: string } | { kind: "constant"; to: string; value: string } >; +}): string { + return "version 1.5\n" + buildBridgeBlock(spec); +} + +function buildBridgeBlock(spec: { + type: string; + field: string; + wires: Array< + | { kind: "pull"; to: string; from: string } + | { kind: "constant"; to: string; value: string } + >; }): string { const lines = [ - "version 1.5", `bridge ${spec.type}.${spec.field} {`, " with input as i", " with output as o", @@ -310,3 +320,117 @@ describe("parser fuzz — serializer round-trip", () => { }, ); }); + +// ── P2-3B: prettyPrintToSource advanced stability — all block types ──────── +// +// Design note (re: Suggestion 2 / depth limits): +// We generate text directly using bounded arbitraries rather than recursive +// AST objects. This is the same "text-first" strategy used throughout this +// file. There is no fc.letrec here — every block type is generated with a +// fixed small output size, and fast-check shrinks by reducing the string, +// not by traversing a recursive data structure. + +const constBlockArb = fc + .record({ + name: canonicalIdArb.map((n) => n.toUpperCase()), + value: textConstantValueArb, + }) + .map(({ name, value }) => `const ${name} = ${value}`); + +const toolBlockArb = fc + .record({ + name: canonicalIdArb, + // Use a limited set of real stdlib tools available in the parser's namespace + fn: fc.constantFrom( + "std.httpCall", + "std.str.toUpperCase", + "std.arr.filter", + ), + wireCount: fc.integer({ min: 0, max: 3 }), + wireKey: canonicalIdArb, + wireValue: textConstantValueArb, + }) + .map(({ name, fn, wireCount, wireKey, wireValue }) => { + const lines = [`tool ${name} from ${fn} {`]; + for (let i = 0; i < wireCount; i++) { + lines.push(` .${wireKey}${i > 0 ? i : ""} = ${wireValue}`); + } + lines.push("}"); + return lines.join("\n"); + }); + +// Extended document spec: optional const + optional tool + required bridge block. +const extendedDocSpecArb = fc.record({ + includeConst: fc.boolean(), + includeTool: fc.boolean(), + constBlock: constBlockArb, + toolBlock: toolBlockArb, + bridge: bridgeTextSpecArb, +}); + +function buildExtendedDocText(spec: { + includeConst: boolean; + includeTool: boolean; + constBlock: string; + toolBlock: string; + bridge: { + type: string; + field: string; + wires: Array< + | { kind: "pull"; to: string; from: string } + | { kind: "constant"; to: string; value: string } + >; + }; +}): string { + const parts: string[] = ["version 1.5"]; + if (spec.includeConst) parts.push(spec.constBlock); + if (spec.includeTool) parts.push(spec.toolBlock); + parts.push(buildBridgeBlock(spec.bridge)); + return parts.join("\n\n"); +} + +describe("parser fuzz — advanced formatter stability (P2-3B)", () => { + test( + "prettyPrintToSource is idempotent on documents with tool and const blocks", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(extendedDocSpecArb, (spec) => { + const sourceText = buildExtendedDocText(spec); + const formatted1 = prettyPrintToSource(sourceText); + const formatted2 = prettyPrintToSource(formatted1); + assert.equal( + formatted2, + formatted1, + "prettyPrintToSource must be idempotent\n--- FIRST ---\n" + + formatted1 + + "\n--- SECOND ---\n" + + formatted2, + ); + }), + { numRuns: 1_000, endOnFailure: true }, + ); + }, + ); + + test( + "prettyPrintToSource output is always parseable for documents with tool and const blocks", + { timeout: 60_000 }, + () => { + fc.assert( + fc.property(extendedDocSpecArb, (spec) => { + const sourceText = buildExtendedDocText(spec); + const formatted = prettyPrintToSource(sourceText); + try { + parseBridge(formatted); + } catch (error) { + assert.fail( + `prettyPrintToSource produced unparsable output: ${String(error)}\n--- SOURCE ---\n${sourceText}\n--- FORMATTED ---\n${formatted}`, + ); + } + }), + { numRuns: 1_000, endOnFailure: true }, + ); + }, + ); +}); From a7181da09d757f3fb6e420d0e3e4c158a77e3b40 Mon Sep 17 00:00:00 2001 From: Aarne Laur Date: Fri, 6 Mar 2026 11:48:21 +0100 Subject: [PATCH 3/5] cleanup --- docs/execution-tree-refactor.md | 197 -------------------------------- 1 file changed, 197 deletions(-) delete mode 100644 docs/execution-tree-refactor.md diff --git a/docs/execution-tree-refactor.md b/docs/execution-tree-refactor.md deleted file mode 100644 index c8c8a35f..00000000 --- a/docs/execution-tree-refactor.md +++ /dev/null @@ -1,197 +0,0 @@ -# ExecutionTree Refactoring Plan - -> Track incremental extraction of `packages/bridge-core/src/ExecutionTree.ts` (~2000 lines) -> into focused modules with a thin coordinator class. - ---- - -## Goal - -Split the monolithic `ExecutionTree` class into **procedural modules + thin coordinator**. -Each module is a set of pure(ish) functions that receive a narrow context interface. -The class keeps one-line delegation methods so the public API is unchanged. - -### Principles - -- **Zero behaviour change** — every phase must pass `pnpm test && pnpm e2e` -- **No new allocations on hot paths** — extracted functions use the same patterns -- **Narrow dependency contracts** — modules depend on a `TreeContext` interface, not the full class -- **Incremental** — each phase is a standalone PR-sized change - ---- - -## Current method inventory - -| Concern | ~Lines | Methods | -| --------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Top-level helpers | ~300 | `trunkKey`, `sameTrunk`, `pathEquals`, `isFatalError`, `coerceConstant`, `setNested`, `getSimplePullRef`, `applyControlFlow`, `TraceCollector`, `roundMs`, `isPromise`, `MaybePromise`, sentinels, errors | -| Tool resolution | ~200 | `lookupToolFn`, `resolveToolDefByName`, `resolveToolWires`, `resolveToolSource`, `resolveToolDep` | -| Scheduling | ~200 | `schedule`, `scheduleFinish`, `scheduleToolDef` | -| Instrumented tool calling | ~130 | `callTool` | -| Wire resolution | ~250 | `resolveWires`, `resolveWiresAsync`, `evaluateWireSource`, `pullSingle`, `pullSafe` | -| Shadow trees / materializer | ~250 | `shadow`, `createShadowArray`, `planShadowOutput`, `materializeShadows` | -| Output / response | ~400 | `pullOutputField`, `collectOutput`, `run`, `response`, `findDefineFieldWires`, `applyPath` | -| Lifecycle / state | ~150 | constructor, `push`, `executeForced`, `resolvePreGrouped`, `getTraces` | - ---- - -## Target file structure - -``` -bridge-core/src/ - ExecutionTree.ts ← thin coordinator (constructor, shadow, run, response, lifecycle) - resolveWires.ts ← wire resolution + modifier layers - scheduleTools.ts ← schedule, scheduleFinish, scheduleToolDef, callTool - materializeShadows.ts ← planShadowOutput + materializeShadows - toolLookup.ts ← lookupToolFn, resolveToolDefByName, resolveToolDep - tracing.ts ← TraceCollector, ToolTrace, TraceLevel, OTel helpers - tree-types.ts ← Trunk, MaybePromise, TreeContext interface, sentinels, errors - tree-utils.ts ← trunkKey, sameTrunk, pathEquals, isFatalError, coerceConstant, setNested, etc. - types.ts ← existing (Wire, Bridge, NodeRef, …) - utils.ts ← existing (parsePath) - … -``` - ---- - -## Phases - -### Phase 1 — Extract utility helpers ✅ - -Move **zero-class-dependency** helpers out of `ExecutionTree.ts`: - -**New file: `tree-types.ts`** - -- `BridgePanicError`, `BridgeAbortError` (error classes) -- `CONTINUE_SYM`, `BREAK_SYM` (sentinels) -- `MAX_EXECUTION_DEPTH` constant -- `MaybePromise` type alias -- `Trunk` type -- `Logger` interface -- `Path` interface (GraphQL path) -- `isPromise()` helper -- `isFatalError()` helper -- `applyControlFlow()` helper - -**New file: `tree-utils.ts`** - -- `trunkKey()` -- `sameTrunk()` -- `pathEquals()` -- `coerceConstant()` + `constantCache` -- `setNested()` + `UNSAFE_KEYS` -- `getSimplePullRef()` -- `roundMs()` - -**New file: `tracing.ts`** - -- `TraceCollector` class -- `ToolTrace` type -- `TraceLevel` type -- OTel meter/tracer setup (`otelTracer`, `otelMeter`, counters, histogram) -- `isOtelActive()` helper - -`ExecutionTree.ts` re-imports everything and the public API (`index.ts`) stays unchanged. - -**Status:** Done - ---- - -### Phase 2 — Define `TreeContext` interface + Extract wire resolution - -Define the narrow `TreeContext` interface that extracted modules depend on. -This enables mock-based unit testing of individual modules and establishes -the pattern for all subsequent phases. - -**New: `TreeContext` in `tree-types.ts`** - -```ts -export interface TreeContext { - pullSingle(ref: NodeRef, pullChain?: Set): MaybePromise; -} -``` - -Move wire evaluation into `resolveWires.ts` — functions take `TreeContext`: - -- `resolveWires(ctx, wires, pullChain)` (fast path + delegation) -- `resolveWiresAsync(ctx, wires, pullChain)` (full loop with modifier layers) -- `evaluateWireSource(ctx, w, pullChain)` (Layer 1) -- `pullSafe(ctx, ref, safe, pullChain)` (safe-navigation wrapper) - -`ExecutionTree` implements `TreeContext` and keeps a one-line: - -```ts -private resolveWires(wires: Wire[], pullChain?: Set): MaybePromise { - return resolveWires(this, wires, pullChain); -} -``` - -**Status:** Done - ---- - -### Phase 3 — Extract tool lookup ✅ - -Move tool resolution into `toolLookup.ts`: - -- `ToolLookupContext` interface (narrow contract for tool resolution) -- `lookupToolFn()` -- `resolveToolDefByName()` + cache -- `resolveToolWires()` -- `resolveToolSource()` -- `resolveToolDep()` - -`ExecutionTree` implements `ToolLookupContext` alongside `TreeContext`. -Exposed `toolFns`, `toolDefCache`, `toolDepCache`, `context`, `parent`, -`instructions` getter, and `callTool` (public) to satisfy the interface. - -**Status:** Done - ---- - -### Phase 4 — Extract materializer ✅ - -Move shadow output assembly into `materializeShadows.ts`: - -- `MaterializerHost` interface (narrow view into bridge metadata) -- `MaterializableShadow` interface (duck type for shadow trees) -- `planShadowOutput()` -- `materializeShadows()` - -`ExecutionTree` delegates via a single one-line wrapper. - -**Status:** Done - ---- - -### Phase 5 — Extract scheduler ✅ - -Move scheduling into `scheduleTools.ts`: - -- `SchedulerContext` interface (central dispatch contract) -- `schedule()` -- `scheduleFinish()` -- `scheduleToolDef()` - -`callTool` stays in `ExecutionTree` — it's a standalone instrumentation -wrapper already public for `ToolLookupContext`, and extracting it would -create unnecessary indirection without narrowing the dependency surface. - -Removed 5 private delegation methods from ExecutionTree (`getToolName`, -`lookupToolFn`, `resolveToolDefByName`, `resolveToolWires`, -`resolveToolSource`) — the scheduler calls `toolLookup.ts` functions directly. -Made `pipeHandleMap`, `handleVersionMap`, `resolveWires` public. - -**Status:** Done - ---- - -## Progress log - -| Date | Phase | Notes | -| ---------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 2026-03-01 | Phase 1 | Extracted tree-types.ts (101 L), tree-utils.ts (135 L), tracing.ts (130 L). ExecutionTree.ts 1997→1768 lines. 621 unit + all e2e pass. | -| 2026-03-01 | Phase 2 | Added TreeContext interface to tree-types.ts. Extracted resolveWires.ts (206 L). ExecutionTree 1768→1599 lines. pullSingle now public (satisfies TreeContext). 621 unit + 35 e2e pass. | -| 2026-03-01 | Phase 3 | Extracted toolLookup.ts (310 L) with ToolLookupContext interface. ExecutionTree 1599→1448 lines. callTool now public. 621 unit + 35 e2e pass. | -| 2026-03-01 | Phase 4 | Extracted materializeShadows.ts (247 L) with MaterializerHost/MaterializableShadow interfaces. ExecutionTree 1446→1265 lines. 621 unit + 35 e2e pass. | -| 2026-03-01 | Phase 5 | Extracted scheduleTools.ts (324 L) with SchedulerContext interface. Removed 5 delegation methods. ExecutionTree 1265→1018 lines. 621 unit + 35 e2e pass. | From 3c2fa67aeb0e1a76fec6b3c2dbefe3fe90bf9b8e Mon Sep 17 00:00:00 2001 From: Aarne Laur Date: Fri, 6 Mar 2026 12:16:11 +0100 Subject: [PATCH 4/5] fix: improve AOT/runtime parity for null element traversal and non-array source handling --- .changeset/runtime-parity-fixes.md | 36 +++++++++++++++++++ .changeset/stdlib-type-guard-fixes.md | 8 +++++ packages/bridge-compiler/src/codegen.ts | 28 ++++++++++----- .../test/fuzz-regressions.todo.test.ts | 30 ---------------- .../test/fuzz-runtime-parity.test.ts | 20 +---------- packages/bridge-core/src/ExecutionTree.ts | 6 ++-- packages/bridge-core/src/resolveWires.ts | 9 +++-- 7 files changed, 75 insertions(+), 62 deletions(-) create mode 100644 .changeset/runtime-parity-fixes.md create mode 100644 .changeset/stdlib-type-guard-fixes.md diff --git a/.changeset/runtime-parity-fixes.md b/.changeset/runtime-parity-fixes.md new file mode 100644 index 00000000..79cf0677 --- /dev/null +++ b/.changeset/runtime-parity-fixes.md @@ -0,0 +1,36 @@ +--- +"@stackables/bridge-core": patch +"@stackables/bridge-compiler": patch +--- + +Fix AOT/runtime parity for null element traversal, catch-null recovery, and non-array source handling + +**bridge-core:** + +- `catch` gate now correctly recovers with an explicit `null` fallback value. + Previously, `if (recoveredValue != null)` caused the catch gate to rethrow + the original error when the fallback resolved to `null`; changed to + `!== undefined` so `null` is treated as a valid recovered value. + +- Element refs (array-mapping `el.field` references) are now null-safe during + path traversal. When an array element is `null` or `undefined`, the runtime + returns `undefined` instead of throwing `TypeError`, matching AOT-generated + code which uses optional chaining on element accesses. + +- Array-mapping fields (`resolveNestedField`) now return `null` when the + resolved source value is not an array, instead of returning the raw value + unchanged. This aligns with AOT behavior and makes non-array source handling + consistent. + +**bridge-compiler:** + +- AOT-generated code now respects `rootSafe` / `pathSafe` flags on input refs, + using strict property access (`["key"]`) instead of optional chaining + (`?.["key"]`) for non-safe segments. Previously all input-ref segments used + optional chaining regardless of flags, silently swallowing TypeErrors that + the runtime would throw. + +- Array-mapping expressions now guard the source with `Array.isArray` before + calling `.map` / `.flatMap`. Previously, a non-array non-null source + (e.g. a string) would cause a `TypeError` in the generated code while the + runtime returned `null`. diff --git a/.changeset/stdlib-type-guard-fixes.md b/.changeset/stdlib-type-guard-fixes.md new file mode 100644 index 00000000..afac207b --- /dev/null +++ b/.changeset/stdlib-type-guard-fixes.md @@ -0,0 +1,8 @@ +--- +"@stackables/bridge-stdlib": patch +--- + +Fix `filter`, `find`, `toLowerCase`, `toUpperCase`, `trim`, and `length` crashing on unexpected input types + +- `filter` and `find` now return `undefined` (instead of throwing `TypeError`) when passed a non-array `in` value, and silently skip null/non-object elements rather than crashing +- `toLowerCase`, `toUpperCase`, `trim`, and `length` now return `undefined` (instead of throwing `TypeError`) when passed a non-string value diff --git a/packages/bridge-compiler/src/codegen.ts b/packages/bridge-compiler/src/codegen.ts index 9dfc978c..363a1e03 100644 --- a/packages/bridge-compiler/src/codegen.ts +++ b/packages/bridge-compiler/src/codegen.ts @@ -1590,7 +1590,7 @@ class CodegenContext { 6, "continue", ); - mapExpr = `(${arrayExpr})?.flatMap((_el0) => {\n${cfBody}\n }) ?? null`; + mapExpr = `((__s) => Array.isArray(__s) ? __s.flatMap((_el0) => {\n${cfBody}\n }) ?? null : null)(${arrayExpr})`; } else if (cf === "break") { const cfBody = this.buildElementBodyWithControlFlow( shifted, @@ -1599,10 +1599,10 @@ class CodegenContext { 8, "break", ); - mapExpr = `(() => { const _src = ${arrayExpr}; if (_src == null) return null; const _result = []; for (const _el0 of _src) {\n${cfBody}\n } return _result; })()`; + mapExpr = `(() => { const _src = ${arrayExpr}; if (!Array.isArray(_src)) return null; const _result = []; for (const _el0 of _src) {\n${cfBody}\n } return _result; })()`; } else { const body = this.buildElementBody(shifted, arrayIterators, 0, 6); - mapExpr = `(${arrayExpr})?.map((_el0) => (${body})) ?? null`; + mapExpr = `((__s) => Array.isArray(__s) ? __s.map((_el0) => (${body})) ?? null : null)(${arrayExpr})`; } if (!tree.children.has(arrayField)) { @@ -1755,7 +1755,7 @@ class CodegenContext { innerCf === "continue" ? "for-continue" : "break", ) : `${" ".repeat(indent + 4)}_result.push(${this.buildElementBody(shifted, arrayIterators, depth + 1, indent + 4)});`; - mapExpr = `await (async () => { const _src = ${srcExpr}; if (_src == null) return null; const _result = []; for (const ${innerElVar} of _src) {\n${innerBody}\n${" ".repeat(indent + 2)}} return _result; })()`; + mapExpr = `await (async () => { const _src = ${srcExpr}; if (!Array.isArray(_src)) return null; const _result = []; for (const ${innerElVar} of _src) {\n${innerBody}\n${" ".repeat(indent + 2)}} return _result; })()`; } else if (innerCf === "continue") { const cfBody = this.buildElementBodyWithControlFlow( shifted, @@ -1764,7 +1764,7 @@ class CodegenContext { indent + 2, "continue", ); - mapExpr = `(${srcExpr})?.flatMap((${innerElVar}) => {\n${cfBody}\n${" ".repeat(indent + 2)}}) ?? null`; + mapExpr = `((__s) => Array.isArray(__s) ? __s.flatMap((${innerElVar}) => {\n${cfBody}\n${" ".repeat(indent + 2)}}) ?? null : null)(${srcExpr})`; } else if (innerCf === "break") { const cfBody = this.buildElementBodyWithControlFlow( shifted, @@ -1773,7 +1773,7 @@ class CodegenContext { indent + 4, "break", ); - mapExpr = `(() => { const _src = ${srcExpr}; if (_src == null) return null; const _result = []; for (const ${innerElVar} of _src) {\n${cfBody}\n${" ".repeat(indent + 2)}} return _result; })()`; + mapExpr = `(() => { const _src = ${srcExpr}; if (!Array.isArray(_src)) return null; const _result = []; for (const ${innerElVar} of _src) {\n${cfBody}\n${" ".repeat(indent + 2)}} return _result; })()`; } else { const innerBody = this.buildElementBody( shifted, @@ -1781,7 +1781,7 @@ class CodegenContext { depth + 1, indent + 2, ); - mapExpr = `(${srcExpr})?.map((${innerElVar}) => (${innerBody})) ?? null`; + mapExpr = `((__s) => Array.isArray(__s) ? __s.map((${innerElVar}) => (${innerBody})) ?? null : null)(${srcExpr})`; } if (!tree.children.has(field)) { @@ -2463,7 +2463,19 @@ class CodegenContext { !ref.element ) { if (ref.path.length === 0) return "input"; - return "input" + ref.path.map((p) => `?.[${JSON.stringify(p)}]`).join(""); + // Respect rootSafe / pathSafe flags, same as tool-result refs. + // A bare `.` access (no `?.`) on a null intermediate throws TypeError, + // matching the runtime's applyPath strict-null behaviour. + return ( + "input" + + ref.path + .map((p, i) => { + const safe = + ref.pathSafe?.[i] ?? (i === 0 ? (ref.rootSafe ?? false) : false); + return safe ? `?.[${JSON.stringify(p)}]` : `[${JSON.stringify(p)}]`; + }) + .join("") + ); } // Tool result reference diff --git a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts index 4bba5c27..5f55e588 100644 --- a/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts +++ b/packages/bridge-compiler/test/fuzz-regressions.todo.test.ts @@ -1,29 +1,6 @@ import { describe, test } from "node:test"; describe("fuzz-discovered AOT/runtime divergence backlog", () => { - // AOT compiler uses `?.` safe-navigation everywhere in generated code; runtime - // throws TypeError for unsafe path traversal when `rootSafe` is not set. - // These seeds reproduce the divergence via fuzz-runtime-parity.test.ts. - test.todo( - "deep-path parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=1798655022)", - ); - test.todo( - "deep-path fallback parity: AOT silently returns undefined where runtime throws TypeError for unsafe traversal (seed=-481664925)", - ); - // AOT and runtime diverge on null-vs-undefined semantics when fallback constant - // wires (catchFallback / falsy fallbacks) resolve to "null". The AOT emits null - // while the runtime applies overdefinition coalescing and produces undefined. - // Repro: fuzz-runtime-parity deepFallbackBridgeArb + chaosInputArb, any random seed. - test.todo( - "fallback parity: AOT returns null where runtime returns undefined when fallback constant resolves to null (seed=random)", - ); - // Array mapping: when an array contains null elements, the runtime throws TypeError - // when accessing a field on the null element (element refs lack rootSafe), while - // the AOT generates element?.field and silently returns undefined. - // Repro: fuzz-runtime-parity arrayBridgeSpec + [null] element in source array. - test.todo( - "array mapping: runtime throws TypeError for null array elements; AOT returns undefined (null element divergence)", - ); // Array mapping: when a bridge has `.elemField <- el.elemField` where elemField // equals the source array path (e.g. `o.items <- i.data[] as el { .data <- el.data }`), // the runtime conflates the shadow-tree element wire with the outer input-array source @@ -33,11 +10,4 @@ describe("fuzz-discovered AOT/runtime divergence backlog", () => { test.todo( "array mapping: element field with same name as source field causes trunk key collision in runtime", ); - // Array mapping: when source value is a non-array non-null (e.g. a string), AOT - // throws TypeError (.map is not a function), while the runtime iterates iterable - // types (strings char-by-char) via createShadowArray. Numbers return []. - // Repro: arrayBridgeSpec + source = "hello" or 42. - test.todo( - "array mapping: non-array source diverges — AOT throws TypeError, runtime uses iterable semantics", - ); }); diff --git a/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts index 1380a4fd..9d045b2a 100644 --- a/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts +++ b/packages/bridge-compiler/test/fuzz-runtime-parity.test.ts @@ -75,26 +75,8 @@ const chaosInputArb = fc.dictionary(identifierArb, chaosValueArb, { }); // ── Ref helpers ───────────────────────────────────────────────────────────── -// rootSafe: true mirrors the parser's `?.` safe-navigation semantics on input -// refs: missing keys return undefined rather than throwing TypeError. This lets -// us test value-coercion parity (NaN, Infinity, etc.) without hitting the -// known unsafe-path-traversal divergence between AOT and runtime -// (tracked separately as fuzz-regressions seeds 1798655022 / -481664925). - function inputRef(type: string, field: string, path: string[]): NodeRef { - // rootSafe + pathSafe mirror full `?.` safe-navigation on all segments so - // missing keys return undefined rather than throwing TypeError. This lets - // us test value-coercion parity (NaN, Infinity, etc.) without hitting the - // known unsafe-path-traversal divergence between AOT and runtime - // (tracked separately as fuzz-regressions seeds 1798655022 / -481664925). - return { - module: "_", - type, - field, - path, - rootSafe: true, - pathSafe: Array(path.length).fill(true), - }; + return { module: "_", type, field, path }; } function outputRef(type: string, field: string, path: string[]): NodeRef { diff --git a/packages/bridge-core/src/ExecutionTree.ts b/packages/bridge-core/src/ExecutionTree.ts index 95bb6607..2f37d157 100644 --- a/packages/bridge-core/src/ExecutionTree.ts +++ b/packages/bridge-core/src/ExecutionTree.ts @@ -396,7 +396,7 @@ export class ExecutionTree implements TreeContext { // Root-level null check if (result == null) { - if (ref.rootSafe) return undefined; + if (ref.rootSafe || ref.element) return undefined; throw new TypeError( `Cannot read properties of ${result} (reading '${ref.path[0]}')`, ); @@ -413,7 +413,7 @@ export class ExecutionTree implements TreeContext { } result = result[segment]; if (result == null && i < ref.path.length - 1) { - const nextSafe = ref.pathSafe?.[i + 1] ?? false; + const nextSafe = (ref.pathSafe?.[i + 1] ?? false) || !!ref.element; if (nextSafe) return undefined; throw new TypeError( `Cannot read properties of ${result} (reading '${ref.path[i + 1]}')`, @@ -641,7 +641,7 @@ export class ExecutionTree implements TreeContext { // Array mapping on a sub-field: resolve the array source, // create shadow trees, and materialise with field mappings. const resolved = await this.resolveWires(regularWires); - if (!Array.isArray(resolved)) return resolved; + if (!Array.isArray(resolved)) return null; const shadows = this.createShadowArray(resolved); return this.materializeShadows(shadows, prefix); } diff --git a/packages/bridge-core/src/resolveWires.ts b/packages/bridge-core/src/resolveWires.ts index 89175ccb..55d2aff9 100644 --- a/packages/bridge-core/src/resolveWires.ts +++ b/packages/bridge-core/src/resolveWires.ts @@ -11,7 +11,12 @@ import type { NodeRef, Wire } from "./types.ts"; import type { MaybePromise, TreeContext } from "./tree-types.ts"; -import { isFatalError, isPromise, applyControlFlow, BridgeAbortError } from "./tree-types.ts"; +import { + isFatalError, + isPromise, + applyControlFlow, + BridgeAbortError, +} from "./tree-types.ts"; import { coerceConstant, getSimplePullRef } from "./tree-utils.ts"; // ── Wire type helpers ──────────────────────────────────────────────────────── @@ -101,7 +106,7 @@ async function resolveWiresAsync( if (isFatalError(err)) throw err; const recoveredValue = await applyCatchGate(ctx, w, pullChain); - if (recoveredValue != null) return recoveredValue; + if (recoveredValue !== undefined) return recoveredValue; lastError = err; } From 1020ff33a6721b6081273c4a94ce98a62d956400 Mon Sep 17 00:00:00 2001 From: Aarne Laur Date: Fri, 6 Mar 2026 12:33:51 +0100 Subject: [PATCH 5/5] fix: exclude "in" from string generation in array filter tests --- packages/bridge-stdlib/test/fuzz-stdlib.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/bridge-stdlib/test/fuzz-stdlib.test.ts b/packages/bridge-stdlib/test/fuzz-stdlib.test.ts index ab15638e..b3a69835 100644 --- a/packages/bridge-stdlib/test/fuzz-stdlib.test.ts +++ b/packages/bridge-stdlib/test/fuzz-stdlib.test.ts @@ -68,7 +68,7 @@ describe("stdlib fuzz — array tools", () => { }), { maxLength: 10 }, ), - fc.string({ maxLength: 8 }), + fc.string({ maxLength: 8 }).filter((k) => k !== "in"), fc.jsonValue(), (arr, key, value) => { const result = filter({ in: arr, [key]: value });