Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changeset/aot-timeout-error-class.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"@stackables/bridge-compiler": patch
---

Fix AOT compiler to throw `BridgeTimeoutError` on tool timeout

AOT-compiled bridges now throw `BridgeTimeoutError` (with the same name and
message format as the runtime) when a tool exceeds `toolTimeoutMs`. Previously
the generated code constructed a generic `Error`, causing a class mismatch when
callers caught and inspected the error type.
36 changes: 36 additions & 0 deletions .changeset/runtime-parity-fixes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
"@stackables/bridge-core": patch
"@stackables/bridge-compiler": patch
---

Fix AOT/runtime parity for null element traversal, catch-null recovery, and non-array source handling

**bridge-core:**

- `catch` gate now correctly recovers with an explicit `null` fallback value.
Previously, `if (recoveredValue != null)` caused the catch gate to rethrow
the original error when the fallback resolved to `null`; changed to
`!== undefined` so `null` is treated as a valid recovered value.

- Element refs (array-mapping `el.field` references) are now null-safe during
path traversal. When an array element is `null` or `undefined`, the runtime
returns `undefined` instead of throwing `TypeError`, matching AOT-generated
code which uses optional chaining on element accesses.

- Array-mapping fields (`resolveNestedField`) now return `null` when the
resolved source value is not an array, instead of returning the raw value
unchanged. This aligns with AOT behavior and makes non-array source handling
consistent.

**bridge-compiler:**

- AOT-generated code now respects `rootSafe` / `pathSafe` flags on input refs,
using strict property access (`["key"]`) instead of optional chaining
(`?.["key"]`) for non-safe segments. Previously all input-ref segments used
optional chaining regardless of flags, silently swallowing TypeErrors that
the runtime would throw.

- Array-mapping expressions now guard the source with `Array.isArray` before
calling `.map` / `.flatMap`. Previously, a non-array non-null source
(e.g. a string) would cause a `TypeError` in the generated code while the
runtime returned `null`.
8 changes: 8 additions & 0 deletions .changeset/stdlib-type-guard-fixes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
"@stackables/bridge-stdlib": patch
---

Fix `filter`, `find`, `toLowerCase`, `toUpperCase`, `trim`, and `length` crashing on unexpected input types

- `filter` and `find` now return `undefined` (instead of throwing `TypeError`) when passed a non-array `in` value, and silently skip null/non-object elements rather than crashing
- `toLowerCase`, `toUpperCase`, `trim`, and `length` now return `undefined` (instead of throwing `TypeError`) when passed a non-string value
197 changes: 0 additions & 197 deletions docs/execution-tree-refactor.md

This file was deleted.

120 changes: 120 additions & 0 deletions docs/fuzz-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# Fuzz Testing

> Reference document for fuzz/property-based testing coverage and workflow in the Bridge codebase.

## Overview

Bridge uses [fast-check](https://github.com/dubzzz/fast-check) (^4.5.3) for property-based testing alongside the standard `node:test` + `node:assert` framework. Fuzz tests live co-located with each package and run as part of the normal `pnpm test` suite.

### Test files

| File | Package | Purpose |
| ------------------------------------ | ----------------- | ------------------------------------------------------------------- |
| `test/fuzz-compile.test.ts` | `bridge-compiler` | JS syntax validity, determinism, flat-path AOT/runtime parity |
| `test/fuzz-runtime-parity.test.ts` | `bridge-compiler` | Deep-path parity, array mapping parity, tool-call timeout parity |
| `test/fuzz-regressions.todo.test.ts` | `bridge-compiler` | Backlog of known fuzz-discovered divergences as `test.todo` entries |
| `test/fuzz-stdlib.test.ts` | `bridge-stdlib` | Array and string tool crash-safety |
| `test/fuzz-parser.test.ts` | `bridge` | Parser crash-safety, serializer round-trip, formatter stability |

---

## Coverage

### What's tested

- JS syntax validity of AOT compiler output
- Compiler determinism
- AOT/runtime parity on flat single-segment paths (`fc.jsonValue()` inputs)
- AOT/runtime parity on deep multi-segment paths with chaotic inputs (`NaN`, `Infinity`, `-0`, `undefined`, deeply nested objects)
- AOT/runtime parity on array-mapping bridges (`[] as el { ... }`) with chaotic element data
- AOT/runtime parity on tool-call timeout (`BridgeTimeoutError` class and message match)
- Parser round-trip: text → parse → serialize → reparse → execute parity
- `parseBridge` never throws unstructured errors on random input
- `parseBridgeDiagnostics` never throws (LSP/IDE safety)
- `prettyPrintToSource` idempotence and output parseability (bridge, tool, const blocks)
- `arr.filter`, `arr.find`, `arr.first`, `arr.toArray` crash-safety on any input type
- `str.toLowerCase`, `str.toUpperCase`, `str.trim`, `str.length` crash-safety on any input type

### Known gaps (P3)

- `Symbol`, `BigInt`, circular-ref handling across all stdlib tools
- `parseBridgeDiagnostics` completeness: valid input should produce zero error-severity diagnostics

---

## Property run counts

| Test | Runs |
| ---------------------------------------------------- | ----- |
| Deep-path AOT/runtime parity | 3,000 |
| Array mapping parity | 1,000 |
| Tool-call timeout parity | 500 |
| `parseBridge` never panics | 5,000 |
| `parseBridgeDiagnostics` never throws | 5,000 |
| Serializer round-trip | 2,000 |
| `prettyPrintToSource` idempotence (basic) | 2,000 |
| `prettyPrintToSource` parseability (basic) | 2,000 |
| `prettyPrintToSource` idempotence (extended blocks) | 1,000 |
| `prettyPrintToSource` parseability (extended blocks) | 1,000 |
| stdlib tool crash-safety (per tool) | 2,000 |

---

## Generator design principles

**Text-first over AST-first.** Generating valid `.bridge` text strings and parsing them is preferred over building `Bridge` AST objects directly with `fc.letrec`. Text-first generation avoids exponential shrinking blowup: fast-check shrinks by removing tokens from a string, not by exploring recursive AST tree variants. This is especially important for array mapping and nested-block tests.

**Depth limits with `fc.letrec`.** When recursive arbitraries are necessary (e.g. `chaosValueArb` for deep input objects), always pass `depthFactor` or cap with `maxLength`/`maxKeys` at every level. Without this, the shrinking phase can explore exponentially many candidates and halt CI.

**Safety margins for timing tests.** Timer-based parity tests skip inputs in the "grey zone" where `|toolDelay - toolTimeout| < 20ms` to avoid flakiness on slow CI runners.

**Forbidden path segments.** All generated identifier arbitraries filter out `__proto__`, `prototype`, and `constructor` to stay within the valid domain for path traversal.

---

## Regression workflow

When a fuzz run finds a new issue:

1. **Capture evidence immediately** — seed, failure path, counterexample input, whether it is `AOT != runtime`, a parser crash, or a runtime panic.

2. **Add a `test.todo` entry** in `packages/bridge-compiler/test/fuzz-regressions.todo.test.ts`:

```ts
test.todo("class label — short description (seed=123456)");
```

3. **Open a tracking note** — link to the todo, add impact, expected vs actual behaviour, suspected component.

4. **Create a deterministic reproducer** — prefer a minimal hand-authored bridge + input over rerunning fuzz with a seed. Add it to `codegen.test.ts` or a dedicated regression file as a normal `test(...)`.

5. **Fix at root cause** — keep fixes small and targeted.

6. **Promote and clean up** — ensure reproducer passes, remove the `test.todo` entry, keep the fuzz property in place.

---

## Running fuzz tests

```bash
# All tests (includes fuzz)
pnpm test

# Single fuzz file
node --experimental-transform-types --conditions source --test packages/bridge-compiler/test/fuzz-runtime-parity.test.ts
node --experimental-transform-types --conditions source --test packages/bridge/test/fuzz-parser.test.ts
node --experimental-transform-types --conditions source --test packages/bridge-stdlib/test/fuzz-stdlib.test.ts

# Reproduce a specific failing seed
# Add { seed: -1234567, path: "0", endOnFailure: true } to fc.assert options
```

---

## Implementation notes

- **Test framework:** `node:test` + `node:assert` (no Jest/Vitest)
- **Fuzz library:** `fast-check` ^4.5.3 — devDependency of `bridge-compiler`, `bridge-stdlib`, `bridge`
- Parser fuzz tests live in `packages/bridge/test/`
- Stdlib fuzz tests live in `packages/bridge-stdlib/test/`
- Compiler parity fuzz tests live in `packages/bridge-compiler/test/`
Loading