Skip to content

testing.md

Latest commit

 

History

History
791 lines (593 loc) · 52 KB

testing.md

File metadata and controls

791 lines (593 loc) · 52 KB
Raw

Testing

For contributors writing, running, or debugging tests.

Executive Summary

  • Three testing layers — JavaScript end-to-end tests (primary), CLI behavior tests (secondary), Pascal unit tests (tertiary)
  • Valid code → tests/; rejected code → scripts/test-cli-*.ts — JavaScript tests assert valid code runs and returns the right result; parser/lexer rejection (SyntaxError, caret, error envelope) belongs in scripts/test-cli-parser.ts / test-cli-lexer.ts, because with no eval/Function a parse error cannot be asserted from inside a JS test
  • Built-in test framework — describe/test/expect with async support, mock functions, lifecycle hooks, and Vitest-compatible matchers
  • One method per file — Each test file focuses on a single method; edge cases are co-located with happy-path tests
  • Cover the contract, not only the example — Include boundaries, invalid inputs and receivers, coercion/order, state transitions, descriptors, and both execution modes when those perspectives apply
  • Structure is checked in CI — Run bun run scripts/check-test-structure.ts before submitting test-suite organization changes
  • Run with: ./build.pas testrunner, ./build/GocciaTestRunner tests, and ./build/GocciaTestRunner tests --mode=bytecode

GocciaScript uses three testing layers in priority order:

  1. JavaScript end-to-end tests (primary) -- .js tests in tests/ that exercise the full pipeline through the same public surface that users call. CI runs the full suite in both interpreter mode and bytecode mode. Every new feature or bug fix should include tests at this layer.
  2. CLI behavior tests (CI integration) -- Standalone bun scripts under scripts/test-cli-*.ts (test-cli.ts, test-cli-lexer.ts, test-cli-parser.ts, test-cli-config.ts, test-cli-apps.ts, test-cli-embedded-resources.ts) that the PR and main workflows run via bun run in the cli job. They invoke GocciaScriptLoader, GocciaTestRunner, and GocciaBenchmarkRunner as subprocesses and assert on exit codes, output structure, and error envelopes — above all parser/lexer rejection that a JS test cannot express (malformed source must fail with a SyntaxError, caret, suggestion, and JSON error envelope, in both modes), plus JSON output structure, coverage CLI, source maps, numeric separator rejection, timeout handling, global injection, and config loading.
  3. Pascal unit tests (tertiary) -- Native *.Test.pas coverage for low-level runtime and value system internals that are not reachable through a stable public API.

When choosing where to add coverage, prefer the most public entry point — and match the kind of check to the layer that can actually express it:

What you are verifying Where it goes Why there
Valid code runs and produces the correct result — semantics, conformance, edge cases JavaScript tests under tests/ (run by GocciaTestRunner in both interpreter and bytecode mode) Exercises the full lexer → parser → interpreter/VM pipeline through the public surface
Malformed source is rejected — a SyntaxError/early error, and its message, caret, suggestion, exit code, and JSON error envelope scripts/test-cli-parser.ts (parser) or scripts/test-cli-lexer.ts (lexer), run by the CI cli job A JS test cannot assert rejection: with no eval/Function, a parse or lex error simply fails to load the whole test file. These scripts run GocciaScriptLoader as a subprocess and assert on its exit code and error output, in both modes
CLI tool contract — output formats (--output=json, --format), --coverage, --source-map, --timeout, global injection, config loading the matching scripts/test-cli-*.ts (test-cli.ts, test-cli-apps.ts, test-cli-config.ts, test-cli-embedded-resources.ts), run by the CI cli job Command-level behaviour over the real binaries
Low-level runtime/value internals not reachable from JavaScript Pascal *.Test.pas Internal-only behaviour

A JavaScript test that merely parses and runs a construct — e.g. (a / b) evaluating to 1, or (/x/).test(...) returning true — is a positive execution/conformance test. It confirms the construct works, but it does not prove that a different, malformed construct is rejected, and it is not the lexer/parser gate. When a change is about what the parser or lexer accepts or rejects (disambiguation, early errors, token classification), put the rejection cases in scripts/test-cli-parser.ts / test-cli-lexer.ts.

Do not use dynamic source execution (Function(...), eval(...), or equivalent runtime compilation) inside the JavaScript suite merely to smuggle parser failures into a runtime test. Keep dynamic-code assertions in tests/ only when the dynamic-code API itself is the feature under test, such as Function constructor behavior. Direct eval is host-gated, so its private test262-host coverage belongs in scripts/test-cli-apps.ts, outside the JavaScript suite.

Avoid tests that lock onto private helper functions or transient implementation structure when the same behavior can be validated through a documented user-facing command or script entry point. Where native Pascal tests are still appropriate, keep them as stateless and repeatable as possible so they behave like pure input/output checks rather than process-sensitive probes.

Keep suite titles, test names, and failure messages aligned with the layer under test. End-to-end JavaScript and CLI tests should use JavaScript/runtime terminology and describe behavior at that layer, while lower-level Pascal tests may use implementation terms when that is the behavior being exercised.

Test Organization

tests/
├── built-ins/              # Built-in object tests
│   ├── Array/              # Array constructor and prototype methods
│   │   ├── constructor.js
│   │   ├── from.js         # Array.from
│   │   ├── of.js           # Array.of
│   │   └── prototype/
│   │       ├── map.js, filter.js, reduce.js, forEach.js
│   │       ├── find.js, findIndex.js, indexOf.js, lastIndexOf.js
│   │       ├── sort.js, splice.js, shift.js, unshift.js
│   │       ├── fill.js, at.js, includes.js, concat.js, reverse.js
│   │       └── ...
│   ├── ArrayBuffer/        # ArrayBuffer constructor, static/prototype methods
│   │   ├── constructor.js, isView.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── Error/
│   ├── JSON/
│   ├── Map/
│   ├── Math/
│   ├── Number/             # Number methods and constants
│   │   ├── parseInt.js, parseFloat.js, isNaN.js, isFinite.js, isInteger.js
│   │   ├── isSafeInteger.js, epsilon.js, maxSafeInteger.js
│   │   └── maxValue.js, minValue.js, nan.js
│   ├── Object/             # Object static and prototype methods
│   │   ├── keys.js, values.js, entries.js, assign.js, create.js, is.js
│   │   ├── defineProperty.js, defineProperties.js, getOwnPropertyDescriptor.js
│   │   ├── freeze.js, isFrozen.js
│   │   ├── getPrototypeOf.js, setPrototypeOf.js
│   │   ├── fromEntries.js, groupBy.js
│   │   ├── prototype/      # Object.prototype instance methods
│   │   │   ├── toString.js, hasOwnProperty.js, isPrototypeOf.js
│   │   │   ├── propertyIsEnumerable.js, toLocaleString.js, valueOf.js
│   │   └── ...
│   ├── Promise/             # Promise constructor, static methods, microtask ordering
│   │   ├── constructor.js, resolve.js, reject.js
│   │   ├── all.js, all-settled.js, race.js, any.js
│   │   ├── microtask-ordering.js, thenable-adoption.js
│   │   └── prototype/
│   │       ├── then.js, catch.js, finally.js
│   ├── Set/
│   ├── SharedArrayBuffer/  # SharedArrayBuffer constructor, prototype methods
│   │   ├── constructor.js, toString-tag.js
│   │   └── prototype/
│   │       └── slice.js
│   ├── String/
│   │   └── prototype/
│   ├── structuredClone/    # structuredClone tests (primitives, objects, collections, errors, arraybuffer)
│   ├── Symbol/
│   └── TypedArray/         # TypedArray constructors, prototype methods
│       ├── constructors.js, element-access.js, buffer-sharing.js
│       ├── from.js, of.js  # Static methods at top level
│       └── prototype/      # Instance methods (one file per method, edge cases included)
│           ├── fill.js, map.js, filter.js, reduce.js, reduceRight.js
│           ├── indexOf.js, lastIndexOf.js, includes.js
│           ├── find.js, findIndex.js, findLast.js, findLastIndex.js
│           ├── sort.js, reverse.js, slice.js, subarray.js
│           ├── set.js, copyWithin.js, at.js, with.js
│           ├── every.js, some.js, forEach.js, join.js
│           ├── toReversed.js, toSorted.js
│           └── values.js, keys.js, entries.js
│
└── language/               # Core language feature tests
    ├── classes/            # Class declarations, inheritance, private fields/methods/getters/setters
    ├── declarations/       # let, const
    ├── decorators/         # TC39 Stage 3 decorators and decorator metadata
    ├── expressions/        # Arithmetic, comparison, logical, destructuring, trailing commas, etc.
    │   ├── addition/       # Addition with ToPrimitive
    │   ├── arithmetic/     # Division (IEEE-754 signed zeros, Infinity), exponentiation (Infinity edge cases)
    │   ├── bitwise/        # Bitwise OR, AND, XOR, NOT, shifts
    │   ├── modulo/         # Floating-point modulo
    │   ├── conditional/    # Ternary precedence
    │   ├── optional-chaining/  # Optional chaining (?.) edge cases
    │   ├── trailing-commas.js  # Trailing commas in calls, parameters, constructors
    │   └── ...
    ├── functions/          # Arrow functions, closures, higher-order, recursion
    │   └── function-length-name.js  # Function.length and Function.name
    ├── identifiers/        # Unicode support, reserved words
    ├── modules/            # Import/export
    ├── objects/            # Object literals, methods, computed properties
    ├── statements/         # if/else, switch/case/break, try/catch/finally
    │   ├── try-catch/      # Try-catch-finally edge cases
    │   └── unsupported-features/    # Warning-mode recovery tests for disabled compatibility syntax
    └── unary-operators.js

Constructor requirements belong with the constructor they constrain. For example, the Map()-without-new rejection belongs in tests/built-ins/Map/constructor.js, not in a shared tests/built-ins/constructors/require-new.js file.

File Naming and Layout Conventions

Follow these rules when creating or organizing test files:

1. One method per file — Each built-in method or static function gets its own test file. Never bundle multiple methods into a single file like prototype-methods.js or static-methods.js.

# Correct — each method is its own file
tests/built-ins/TypedArray/prototype/fill.js
tests/built-ins/TypedArray/prototype/map.js
tests/built-ins/TypedArray/prototype/indexOf.js

# Wrong — multiple methods in one file
tests/built-ins/TypedArray/prototype-methods.js

2. Prototype methods go in a prototype/ subfolder — Instance methods (e.g., Array.prototype.map) live in BuiltIn/prototype/methodName.js. Static methods and constructor tests live directly in the BuiltIn/ folder (no separate static/ subfolder).

tests/built-ins/TypedArray/
├── constructors.js              # new TypedArray(...) constructor variants
├── buffer-sharing.js            # Buffer sharing behavior across views
├── from.js, of.js               # Static methods at top level
└── prototype/                   # Instance methods
    ├── at.js
    ├── fill.js
    ├── indexOf.js
    ├── map.js
    └── ...

3. Edge cases are co-located, not separate — Edge cases (NaN handling, Infinity, negative indices, empty arrays, clamping, etc.) belong in the same file as the happy-path tests for that method. Do not create standalone edge-cases.js files — they become disconnected from the feature they test and make it unclear which method the edge case validates.

// tests/built-ins/TypedArray/prototype/fill.js — CORRECT
describe("TypedArray.prototype.fill", () => {
  test("fills entire array", () => { ... });
  test("fills with start and end", () => { ... });

  // Edge cases are part of the same file
  test("negative start index counts from end", () => { ... });
  test("NaN fill value becomes 0 for integer types", () => { ... });
  test("Infinity clamps to 255 for Uint8ClampedArray", () => { ... });
});
# Wrong — edge cases in a separate catch-all file
tests/built-ins/TypedArray/edge-cases.js

4. Object prototype methods use prototype/ too — Object.prototype.toString, Object.prototype.hasOwnProperty, etc. follow the same convention as Array. Use Object/prototype/toString.js, not Object/prototype-toString.js.

tests/built-ins/Object/
├── keys.js                      # Object.keys (static method)
├── assign.js                    # Object.assign (static method)
├── prototype/                   # Instance methods on Object.prototype
│   ├── toString.js
│   ├── hasOwnProperty.js
│   ├── isPrototypeOf.js
│   ├── propertyIsEnumerable.js
│   ├── toLocaleString.js
│   └── valueOf.js

5. Test titles should match the test layer — describe(...) and test(...) strings should use the vocabulary of the layer they exercise. End-to-end JavaScript tests should avoid leaking Pascal implementation terms; lower-level Pascal tests do not need that restriction.

// Correct — end-to-end JavaScript test uses JavaScript/runtime terminology
test("JSON.stringify preserves round-trip precision for large fractional floating-point numbers", () => {
  // ...
});

// Wrong — end-to-end JavaScript test leaks Pascal implementation terms
test("JSON.stringify preserves round-trip precision for large fractional doubles", () => {
  // ...
});

6. Cross-surface files are narrow exceptions — A file may span several methods only when the behavior itself is one indivisible cross-surface contract, not merely because the methods are related or share an implementation. Examples include a proposal-version absence contract such as Temporal/removed-methods.js, or a constructor-family invariant such as Error/error-prototype-constructor.js. A shared receiver-branding matrix such as Date/prototype/receiver-branding.js is also appropriate when it verifies the same internal-slot requirement across methods without duplicating that matrix in every method file.

These exceptions must be:

  • named after the single contract rather than methods.js, misc.js, or error-cases.js;
  • explicitly allowlisted in scripts/check-test-structure.ts with a concrete reason;
  • kept small enough that a failure still identifies the contract that broke.

Shared setup or table data is not by itself a reason to combine operations. Prefer a helper imported by focused files when several operations need the same fixtures.

Coverage Perspectives

A passing happy-path example proves only that one route works. For every method or operation, inspect its public contract and implementation branches, then add the perspectives that can change its observable result:

  • ordinary behavior, including more than one representative input where the operation has distinct modes;
  • boundaries and sentinels such as empty input, zero and signed zero, NaN, infinities, minimum/maximum values, and first/last valid indexes;
  • invalid arguments and incompatible receivers, including the exact error type;
  • argument coercion, throwing coercions, and observable evaluation order;
  • repeated calls and state transitions for mutable or stateful APIs;
  • property metadata and descriptors for exposed built-in properties;
  • independent expected values for encoders/decoders and getters/setters rather than round trips that can let two matching defects cancel out;
  • interpreter and bytecode parity, which the full JavaScript suite gate verifies.

Not every bullet applies to every operation. Each added test must protect an observable contract or a meaningful implementation branch; do not add combinatorial duplicates merely to satisfy a checklist.

Structural Check

Run the test-suite structure check after adding or reorganizing JavaScript tests:

bun run scripts/check-test-structure.ts

The check rejects catch-all filenames, known multi-operation files, reliably detectable prototype-method suites outside prototype/, and parser-rejection assertions hidden behind Function(...) or eval(...). Dynamic-code rejection tests are allowed only where the dynamic-code API itself is under test.

Cross-Platform Newline Rules for Data Format Parsers

When a parser implements a format with spec-defined newline semantics, test the format semantics directly instead of inheriting the host OS newline convention.

  • Do not treat LineEnding / sLineBreak as the expected runtime result for parsed data just because the test is running on Windows.
  • Prefer explicit \r\n or #13#10 fixtures when adding regression tests for multiline parsing, folding, or block-scalar behavior.
  • Assert the format-defined canonical result. Example: TOML multiline strings normalize recognized newlines to LF (\n) even when the source text uses CRLF.
  • For parser inputs that come from files, use Goccia.TextFiles.ReadUTF8FileText for file I/O and TextSemantics helpers (CreateUTF8FileTextLines, StringListToLFText, newline normalization) for parser-facing text. Avoid TStringList.LoadFromFile and string(UTF8String(...)). Keep file text as UTF8String until parser entry, canonicalize parser-facing source text to LF through the shared text semantics, and add at least one regression that hits the real file-loading path with non-ASCII data.
  • When code needs a "10 characters" or "first identifier code point" style rule on UTF-8 text, do not use raw Length, Copy, or byte indexing on string/UTF8String; those operate on bytes under our FPC settings.
  • Keep one public-surface regression in tests/ and, when the parser exposes a reusable native utility like TGocciaTOMLParser, add a focused Pascal regression alongside it as well.

Test Metadata (Optional)

Test files can include JSDoc-style metadata inspired by test262:

/*---
description: Tests for basic addition operations
features: [addition, arithmetic]
---*/

Running Tests

Build the GocciaTestRunner

./build.pas testrunner

This also builds the native FFI fixture library from fixtures/ffi/fixture.c, so the full JavaScript suite can run the folder-configured FFI tests locally.

Run All Tests

# Interpreter mode (default)
./build/GocciaTestRunner tests

# Bytecode mode
./build/GocciaTestRunner tests --mode=bytecode

Both execution modes must pass. Test subtrees that require opt-in parser or runtime behavior declare it with a local goccia.json (for example, tests/language/asi/goccia.json enables ASI and tests/built-ins/FFI/goccia.json enables FFI). CI runs the full suite in interpreter mode and bytecode mode as separate matrix jobs.

Run a Specific Test File

./build/GocciaTestRunner tests/language/expressions/addition/basic-addition.js

Run a Test Category

./build/GocciaTestRunner tests/built-ins/Math/

GocciaTestRunner Options

Flag Description
--no-progress Suppress per-file progress output
--no-results Suppress test results summary
--exit-on-first-failure Stop on first test failure
--silent Suppress all console output from test scripts
--jobs=N / -j N Number of parallel worker threads (default: CPU count)
--update-snapshots / -u Create, update, and prune snapshots
--update Vitest-compatible alias for --update-snapshots 
# CI-friendly: no progress, stop on first failure
./build/GocciaTestRunner tests --no-progress --exit-on-first-failure

# Silent mode: only show results, suppress script console output
./build/GocciaTestRunner tests --silent

# Run tests with 4 parallel workers; --jobs=1 forces sequential execution
./build/GocciaTestRunner tests --jobs=4

Snapshot Testing

toMatchSnapshot() writes external snapshots to __snapshots__/<test-file>.snap. toMatchInlineSnapshot() inserts or updates the snapshot in the test source. Both use Vitest-compatible formatting and are available in interpreter and bytecode modes.

test("user", () => {
  const user = { id: 42, name: "Ada" };

  expect(user).toMatchSnapshot("public user");
  expect(user).toMatchInlineSnapshot({
    id: expect.any(Number),
  });
});

The first object argument is a snapshot property shape. Matching happens before any write, and the stored value replaces shaped fields with their asymmetric matcher descriptions. Snapshot shapes support expect.anything, expect.any, expect.closeTo, expect.arrayContaining, expect.objectContaining, expect.stringContaining, expect.stringMatching, and expect.schemaMatching, plus expect.toBeOneOf, expect.toSatisfy, expect.toBeFasterThan, and expect.toBeSlowerThan, including their supported expect.not forms. Snapshot assertions themselves do not support .not.

Local runs create missing snapshots. Existing mismatches fail until the runner is invoked with -u, --update, or --update-snapshots; update mode also prunes obsolete entries. Unless an explicit update flag is supplied, environments detected as CI by Vitest's std-env provider conventions—including GitHub Actions, GitLab, TeamCity, Buildkite, CircleCI, CI, and CONTINUOUS_INTEGRATION—do not write snapshots and fail on missing, mismatched, or obsolete entries. External snapshots are unavailable for stdin input. Existing inline snapshots can be compared from stdin, but they cannot be created or updated there.

Register a Vitest/pretty-format-compatible serializer when a test value needs a domain-specific representation. The most recently registered serializer runs first and must provide test plus either serialize or legacy print:

expect.addSnapshotSerializer({
  test(value) {
    return value?.kind === "point";
  },
  serialize(value) {
    return `Point(${value.x}, ${value.y})`;
  },
});

Native embedders can replace snapshot text formatting through IGocciaSnapshotFormatter and persistence/source editing through IGocciaSnapshotHost. See ADR 0096 for the boundary.

Official YAML Suite

For YAML parse-validity checks against the official yaml-test-suite (pinned to a specific SHA in the script), run:

python3 scripts/run_yaml_test_suite.py
python3 scripts/run_yaml_test_suite.py --output=tmp/yaml-suite-results.json
python3 scripts/run_yaml_test_suite.py --suite-dir=/path/to/yaml-test-suite

This check is intentionally parse-only: it compares whether each suite case should parse or fail, not whether Goccia's runtime values exactly match the suite's JSON fixtures. That distinction matters because Goccia intentionally canonicalizes complex YAML mapping keys into strings.

Official TOML Suite

For TOML 1.1.0 checks against a prepared checkout of the official toml-test corpus, run:

./build.pas tomlcompliancerunner
./build/GocciaTOMLComplianceRunner --suite-dir=/path/to/toml-test --output=tmp/toml-suite-results.json

The runner verifies the checkout against tests/compliance/toml-test.pin without fetching or cloning. It launches its private worker mode once per case, applies a bounded --jobs limit, and classifies mismatches, false accepts, false rejects, timeouts, crashes, and infrastructure failures. Valid cases are compared with the official tagged JSON fixtures through TGocciaTOMLParser.ParseDocument(...), preserving scalar distinctions such as integer, float, datetime, datetime-local, date-local, and time-local.

The runner prints a human summary, optionally writes the shared compliance JSON envelope with --output, and exits non-zero for compliance or infrastructure failures.

The harness and any file-backed parser regression must read source text as UTF-8 bytes first and decode it strictly before handing UTF-16 text to the parser. Do not place encoded bytes in a Pascal string or rely on compiler code pages. The shared TextEncoding and FileUtils boundaries keep behavior identical on Windows and non-Windows hosts and prevent mojibake such as José -> José.

Official JSON5 Suite

The upstream JSON5 parser cases are committed as an ordinary generated JavaScript test suite. Run parser and stringify compliance directly through TestRunner:

./build.pas testrunner
./build/GocciaTestRunner tests/built-ins/JSON5/upstream-parse.js tests/built-ins/JSON5/stringify.js --output=tmp/json5-suite-results.json

The generated suite records the revision in tests/compliance/json5.pin. TestRunner owns concurrency, per-file timeouts, aggregation, exit status, and the JSON report. Invalid JSON5 remains input to JSON5.parse(...) inside valid outer JavaScript. Normal compliance execution requires neither an upstream checkout, Python, nor Node.js.

Maintainers regenerate the JavaScript suite from an already-prepared checkout:

node scripts/regenerate-json5-tests.js /path/to/json5 <pinned-sha> tests/built-ins/JSON5/upstream-parse.js

Regeneration uses Node.js to execute the upstream reference tests, verifies and records the checkout commit, and is intentionally separate from normal compliance runs.

Pinned es-toolkit library probes

The bounded es-toolkit lane runs named public-package probes in both execution modes and emits classified JSON without making a whole-suite percentage claim. See es-toolkit Validation for the pinned artifact, command, failure taxonomy, and sandbox/host-surface boundaries.

Run Pascal Unit Tests

./build.pas tests
for t in build/Goccia.*.Test; do "$t"; done

Writing Tests

See Test Framework API for the complete test authoring reference including assertions, mocks, lifecycle hooks, async patterns, and cross-runtime compatibility.

Test Principles

  1. JavaScript tests are the source of truth — The JavaScript test suite is the primary mechanism for verifying correctness and ECMAScript compatibility. If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

  2. Isolation — Each test file is independent. No shared state between files. The GocciaTestRunner executes each file in a fresh TGocciaEngine, and each engine owns its own realm. Engine tear-down between files unpins the per-engine intrinsic prototypes (Array.prototype, Object.prototype, every error and Temporal prototype, …), so userland mutations in one file — including non-configurable property additions — cannot leak into the next file even when the same worker thread runs both. When multiple explicit input paths are passed, the runner also keeps folder configs per-file; use --config for a deliberately shared root config.

  3. Grouped by feature — Tests are organized by the feature they validate, mirroring the structure of the language specification. Prototype methods live in prototype/ subfolders; static methods live in the built-in's root folder.

  4. One method per file — Each test file focuses on a single method, operation, or language feature. fill.js tests TypedArray.prototype.fill; map.js tests Array.prototype.map. Never bundle multiple methods into a single file.

  5. Edge cases are co-located — Edge cases are co-located with happy-path tests (see File Naming and Layout Conventions).

  6. One scenario per test — Keep each test(...) focused on a single behavior branch or input class. If undefined, null, and missing input go through different validation paths, they should be three tests, not one combined test. Combining many assertions into one block makes failures harder to localize and silently reduces test-count granularity during refactors.

  7. Readable as specification — Test names should describe the expected behavior. A passing test suite serves as living documentation of what GocciaScript supports.

  8. Always pass an error constructor to .toThrow() — When testing that code throws, pass the expected error constructor (TypeError, RangeError, SyntaxError, Error, etc.) to .toThrow() rather than calling it bare. This ensures the test verifies the correct error type, not just that something throws. For async code, prefer await expect(promise).rejects.toThrow(TypeError) over manually checking err.name.

// Preferred — verifies the error type
expect(() => null.foo).toThrow(TypeError);
expect(() => new Array(-1)).toThrow(RangeError);
await expect(Promise.reject(new TypeError("boom"))).rejects.toThrow(TypeError);

// Acceptable for cases where the error type is implementation-defined
expect(() => riskyOp()).toThrow();

// Avoid — doesn't verify the error type
expect(() => null.foo).toThrow();  // passes even if the wrong error is thrown

// Avoid — bypasses the matcher and only inspects a stringly-typed property
promise.catch((err) => {
  expect(err.name).toBe("TypeError");
});

Testing Strategy Design Decision

JavaScript end-to-end tests are the primary testing mechanism. Every new feature or bug fix must include tests that validate the behavior through the full pipeline (lexer → parser → interpreter/VM execution) using the most public surface available. CI runs all tests in interpreter mode and bytecode mode.

  • Specification by example -- Each test file is a runnable specification of expected behavior.
  • End-to-end validation -- Tests exercise the full pipeline, catching integration issues that unit tests would miss.
  • Readable specifications -- JavaScript test files are readable by anyone familiar with Jest/Vitest conventions.
  • Source of truth -- If a behavior isn't covered by a JavaScript test, it isn't guaranteed.

CLI behavior tests — the bun scripts under scripts/test-cli-*.ts, run by the workflows' cli job — form the second layer, verifying that the command-line tools produce correct output structure, reject malformed source with the right error, and that options like --coverage, --output=json, --source-map, and --timeout work end-to-end. These tests run in CI on every pull request.

Pascal unit tests (*.Test.pas) exist as a tertiary layer for behavior that cannot be reached through script code or other documented user-facing entry points. Even there, prefer stateless, repeatable input/output checks over tests that are tightly coupled to incidental implementation structure.

How the GocciaTestRunner Works

The GocciaTestRunner program:

  1. Scans the provided path for .js, .jsx, .ts, .tsx, and .mjs files.
  2. For each file, creates a fresh TGocciaEngine, applies source type from CLI/config or .mjs inference, attaches TGocciaRuntimeCore, applies the test-runner runtime profile, and installs the FFI runtime extension when --unsafe-ffi or the file's goccia.json enables it.
  3. Loads the source and appends a runTests() call.
  4. Executes the script — describe/test blocks register themselves during execution. Nested describe blocks are supported; suite names are composed with > separators (e.g., "Outer > Inner"). Skip state is inherited by nested describes.
  5. runTests() executes all registered tests, reconciles snapshots through the installed host, and collects results.
  6. When running multiple files, GC.Collect runs after each file to reclaim memory between script executions.
  7. Aggregates pass/fail/skip counts across all files.
  8. Prints a summary with total statistics.

CLI Behavior Tests (CI Integration)

The CLI behaviour tests are standalone bun scripts under scripts/test-cli-*.ts that the PR and main workflows run (via bun run) in the cli job on every pull request. They assert on command output, exit codes, and error envelopes rather than internal state, catching regressions in the user-facing interface. test-cli-parser.ts and test-cli-lexer.ts specifically own the parser/lexer rejection paths (malformed source must fail with the correct SyntaxError and envelope) that the JavaScript suite cannot express.

What the CLI tests cover:

Area What is checked
GocciaTestRunner JSON output --output=<file> writes valid JSON with mode, totalFiles fields to a file; --output=json emits the same envelope to stdout and suppresses the human-readable summary; --output=compact-json emits the same stdout envelope without build, memory, stdout, or stderr
GocciaTestRunner coverage --coverage prints summary; --coverage-format=lcov and --coverage-format=json write valid output files; branch coverage includes BRDA/BRF entries
GocciaTestRunner snapshots External and inline creation, comparison and update in both execution modes; exact Vitest formatting; local/CI obsolete behavior; update aliases; stdin restrictions
GocciaScriptLoader JSON output --output=json envelope includes aggregate ok, output, stdout, stderr, build, timing, memory, workers, and per-input files[] entries with fileName and result; --output=compact-json emits the same envelope without build, memory, stdout, or stderr (the normalized output array and structured error are preserved)
GocciaBenchmarkRunner JSON output --format=json writes the same JSON envelope as the loader and test runner; --format=compact-json emits the same envelope without build, memory, stdout, or stderr at the top level or per-file
GocciaScriptLoader error display Syntax errors show source context, caret, and suggestions
GocciaScriptLoader coverage --coverage summary, lcov/json file output, bytecode mode coverage, JSX source map translation for branch positions
GocciaScriptLoader source maps --source-map writes valid source map JSON; rejects stdin without explicit path
Numeric separator rejection Trailing, leading, consecutive separators and invalid positions produce errors
Timeout handling --timeout produces TimeoutError in JSON output
Global injection --global, --globals file/module injection, collision detection
Stdin smoke tests Piped input executes correctly in interpreter mode and bytecode mode
GocciaBenchmarkRunner output --format=json produces valid JSON with benchmark structure

This table is non-exhaustive — the scripts/test-cli-*.ts scripts that the cli job runs are the source of truth. The intent is to check all CLI options, all parser/lexer error paths, and all output-format correctness. To add a new check, add a case to the matching scripts/test-cli-*.ts (parser/lexer rejection → test-cli-parser.ts / test-cli-lexer.ts; CLI flags/output → test-cli.ts / test-cli-apps.ts); the cli job already invokes these scripts, so no workflow change is needed.

Pascal Unit Tests (Tertiary)

Pascal unit tests are the tertiary testing layer for low-level internals that are difficult to validate through JavaScript alone (e.g., value representation, function/scope internals, and test-framework state behavior).

Test File Tests
Goccia.Values.Primitives.Test.pas Primitive value creation, type conversion (ToStringLiteral, ToBooleanLiteral, ToNumberLiteral, TypeName, TypeOf)
Goccia.Values.FunctionValue.Test.pas Function/method creation, closure capture, parameter handling, scope resolution, AST evaluation
Goccia.Values.ObjectValue.Test.pas Object property operations (AssignProperty, GetProperty, DeleteProperty), prototype chain resolution
Goccia.Builtins.TestingLibrary.Test.pas All TGocciaExpectationValue matchers (toBe, toEqual, toStrictEqual, toContainEqual, toMatchObject, toMatch, toBeNull, toBeNaN, toBeUndefined, toBeDefined, toBeTruthy, toBeFalsy, toBeGreaterThan, toBeLessThan, toContain, toHaveLength, toHaveProperty, toBeCloseTo, not); skip and conditional APIs (describe.skip, describe.skipIf, describe.runIf, test.skipIf, test.runIf)

These compile as standalone executables and run directly:

./build.pas tests
./build/Goccia.Values.Primitives.Test
./build/Goccia.Values.FunctionValue.Test
./build/Goccia.Values.ObjectValue.Test
./build/Goccia.Builtins.TestingLibrary.Test

Pascal Test Framework

Pascal tests use TestingPascalLibrary.pas which provides a TTestSuite base class and a generic Expect<T> fluent assertion API:

type
  TMyTests = class(TTestSuite)
    procedure TestSomething;
    procedure SetupTests; override;
  end;

procedure TMyTests.SetupTests;
begin
  Test('should do something', TestSomething);
end;

procedure TMyTests.TestSomething;
begin
  Expect<string>(Value.ToStringLiteral.Value).ToBe('hello');
  Expect<Double>(Value.ToNumberLiteral.Value).ToBe(42);
  Expect<Integer>(Length(Params)).ToBe(2);
  Expect<Boolean>(Value.ToBooleanLiteral.Value).ToBe(True);
  Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(False);
end;

Generic Assertions

The primary assertion API uses a standalone generic Expect<T> function that returns a TExpect<T> record:

Usage Description
Expect<string>(actual).ToBe(expected) String equality
Expect<Double>(actual).ToBe(expected) Double equality (exact)
Expect<Integer>(actual).ToBe(expected) Integer equality
Expect<Boolean>(actual).ToBe(True) Assert value is True
Expect<Boolean>(actual).ToBe(False) Assert value is False

On failure, ToBe produces descriptive messages like Expected "hello" to be "world" or Expected 3.14 to be 2.71.

TTestSuite also provides a Fail(Message) method to force a test failure with a custom message.

FPC 3.2.2 AArch64 Compiler Bug

Expect<T> is intentionally a standalone generic function (not a method on TTestSuite) and TExpect<T> is a record (not a class). This design works around a known FPC 3.2.2 AArch64 compiler crash: when a class with a generic method is inherited across compilation units, the compiler raises an internal EAccessViolation. The standalone function + record pattern avoids this entirely while preserving the fluent Expect<T>(...).ToBe(...) syntax.

NaN Checks in Pascal Tests

When testing for NaN, use the IsNaN property on TGocciaNumberLiteralValue:

// Correct — uses the property accessor
Expect<Boolean>(Value.ToNumberLiteral.IsNaN).ToBe(True);

// Also correct — IsNaN delegates to Math.IsNaN(FValue) internally
Math.IsNaN(Value.ToNumberLiteral.Value)

TGocciaNumberLiteralValue stores a single Double in FValue using standard IEEE 754 bit patterns for NaN, Infinity, and -0. The IsNaN, IsInfinity, and IsNegativeZero property accessors delegate to Math.IsNaN, Math.IsInfinite, and an endian-neutral sign-bit check respectively. Prefer the property accessors for readability.

Testing the Test Framework Itself

Goccia.Builtins.TestingLibrary.Test.pas validates the JS-level assertion matchers from Pascal. Since the matchers call AssertionPassed/AssertionFailed (which update TGocciaTestAssertions internal state) rather than raising Pascal exceptions, the native tests use two helpers to verify both the return value and the recorded pass/fail outcome:

  • ExpectPass(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is False
  • ExpectFail(result) — Asserts the matcher returned UndefinedValue and CurrentTestHasFailures is True

Each test starts with a BeforeEach override that calls FAssertions.ResetCurrentTestState to clear the failure flag. The CurrentTestHasFailures property and ResetCurrentTestState method are public on TGocciaTestAssertions for this purpose.

When to write Pascal unit tests vs JavaScript tests:

  • JavaScript tests (preferred) — For any behavior observable from script code. This is almost everything.
  • Pascal unit tests — Only for internal implementation details not reachable from script code (e.g., the test framework's own pass/fail recording logic).

CI Integration

GitHub Actions CI (.github/workflows/ci.yml) runs on push to main and tags, with a post-build job fan-out plus release packaging:

build → test             → artifacts
      → toml-compliance  →
      → json5-compliance →
      → test262          →
      → awfy             →
      → benchmark        →
      → cli              →

build — Installs FPC once per platform, compiles all binaries, uploads them as intermediate artifacts.

test (needs build, all platforms) — Downloads pre-built binaries, runs all JavaScript tests and Pascal unit tests. Outputs JSON files via --output=<file> for CI timing comparison.

toml-compliance (all platforms) — Downloads the prebuilt GocciaTOMLComplianceRunner, prepares the exact pinned toml-test checkout, and relies on the runner exit status. CI performs only lightweight validation of the report envelope before uploading it.

json5-compliance (all platforms) — Downloads GocciaTestRunner, verifies that the committed generated parser suite names the pinned JSON5 revision, and runs it together with the local stringify suite. CI relies on TestRunner's exit status and performs only lightweight validation of its JSON report before upload.

test262 (needs build, ubuntu-latest x64 only, non-blocking) — Runs the official conformance suite in bytecode mode from the shared pin in scripts/test262-suite-sha.txt, uploads the JSON report, and saves a main baseline cache for PR deltas. The run step is continue-on-error: true so known steady-state conformance failures do not block unrelated work; the downstream PR comment still gates regressions against the cached main baseline. See test262.md for the harness contract and Build System for the workflow wiring.

awfy (needs build, ubuntu-latest x64 only) — Runs the pinned AWFY JavaScript report set from perf/awfy/manifest.json under Goccia bytecode, QuickJS, and the latest Node Current release resolved at workflow time with five interleaved samples per engine. Uploads the normalized awfy-report artifact; on main, publishes the compressed report plus daily pointer to the awfy/ Vercel Blob namespace when BLOB_READ_WRITE_TOKEN is configured.

benchmark (needs build, all platforms) — Downloads pre-built binaries, runs all benchmarks.

cli (needs build, all platforms) — Downloads pre-built binaries and runs CLI behavior smoke tests via Bun: test-cli.ts (options across all apps), test-cli-lexer.ts (numeric-separator rejection), test-cli-parser.ts (error display), test-cli-config.ts (config-file loading and per-file inheritance), and test-cli-apps.ts (app-specific features, including GocciaScriptLoaderBare stdin/file checks, CLI-local print, and runtime-global absence). The x86-64 Linux leg also runs the pinned es-toolkit compatibility probes and uploads their classified JSON report. Windows runs additionally assert that the loader binary does not link against OpenSSL DLLs.

artifacts (needs test + toml-compliance + json5-compliance + awfy + benchmark + cli, main only) — Uploads release binaries after all checks pass. test262 is not a gating dependency — failing tests there cannot block a release.

release (needs test + toml-compliance + json5-compliance + awfy + benchmark + cli, tags only) — Packages and publishes release archives after the same gates pass.

The test, awfy, benchmark, cli, toml-compliance, json5-compliance, and test262 jobs run in parallel after build. The AWFY, TOML, JSON5, and test262 lanes all reuse the already-built binaries from the matrix build artifacts instead of installing FPC again, so platform-specific issues are caught on the same Linux, macOS, and Windows targets as the main test lanes (AWFY and test262 run on Linux only since those outcomes are platform-independent for this engine).

PR Workflow (.github/workflows/pr.yml)

Runs on pull requests targeting main, on ubuntu-latest x64 only:

build → test
      → benchmark → benchmark / timing comments
      → test262   → test262 comment
      → awfy      → AWFY Results comment
      → cli

The PR workflow also runs the pinned es-toolkit compatibility probes in the Linux cli job and retains the classified JSON report. It posts a Suite Timing comment with expandable test-runner and benchmark summaries. Each summary shows timing, top-level GocciaScript GC metrics, and selected FreePascal heap allocation metrics for interpreter mode and bytecode mode. GC memory rows aggregate the main thread plus all worker thread-local GCs. The test runner does not count worker shutdown reclamation as GC collections, while the benchmark runner explicitly collects between benchmark files, so collection counts are expected to differ. The comment hides negative FreePascal heap free-space deltas because they are valid allocator diagnostics but are noisy in a PR summary. See benchmarks.md for details on the benchmark comparison format.

The AWFY Results comment posts median timings and geomean ratios for the pinned AWFY report set under Goccia bytecode, QuickJS, and the latest Node Current release. The underlying awfy-report artifact keeps the five raw interleaved samples per engine, min/max/CV, environment metadata, and first-class failure outcomes.

The test262 conformance comment posts a non-blocking summary using the actions/github-script@v7 pattern, marker <!-- test262-results -->. The markdown is generated by bun scripts/run_test262_suite.ts --comment <results.json> <baseline.json|->. It contains:

  • A per-category breakdown table (run / passed / Δ pass / failed / timeouts / Δ timeout / wrapper-infra / pass-rate / Δ rate) for each of built-ins, harness, intl402, language, staging, plus a totals row.
  • A wrapper-infra-failure top-line warning when non-zero (gates the run as untrustworthy).
  • An Areas closest to 100% table listing the three test262 directories (keyed by the first two path components, e.g. built-ins/Math, language/expressions) with the highest pass rate, filtered to areas with at least 25 attempted tests and below 100%.
  • When a main baseline is cached, Δ vs main columns show absolute count delta and percentage-point delta. Deltas below ±0.05pp render as ±0pp to keep the table readable.
  • A collapsible Per-test deltas section listing newly-passing, newly-failing, newly-timed-out, and no-longer-timed-out test IDs.

The website compatibility dashboard at /compatibility reads main-branch test262 reports from Vercel Blob at request time with CDN caching. Main CI publishes future dashboard points directly, and cd website && bun run backfill-test262 can seed historical days from retained artifacts or fresh historical reruns. Website builds do not download artifact ZIPs or bake test262 data into the deployment. This keeps the public pass-rate timeline and group coverage tables tied to generated CI data rather than hand-maintained percentages.

Weekly crons open (or update) a single PR every Monday with the latest upstream SHA so pins don't go stale:

Suite Workflow Schedule Pin location
test262 test262-bump.yml 06:00 UTC scripts/test262-suite-sha.txt
toml-test toml-test-bump.yml 06:30 UTC tests/compliance/toml-test.pin
json5 json5-test-bump.yml 06:45 UTC tests/compliance/json5.pin and tests/built-ins/JSON5/upstream-parse.js
yaml-test-suite yaml-test-bump.yml 07:00 UTC scripts/run_yaml_test_suite.py

Each workflow reuses a fixed branch (chore/<suite>-bump), so an unmerged PR is updated in place rather than replaced. See test262.md for details on the test262 harness contract.

Running test262 locally

./build.pas loaderbare  # build the binary the runner needs
bun scripts/run_test262_suite.ts --filter "built-ins/Array/*" --output=results.json

The runner clones test262 into a tempdir on first use, or accepts an existing checkout via --suite-dir. See bun scripts/run_test262_suite.ts --help for the full option set, including --categories, --max-tests, --mode={interpreted,bytecode}, --jobs, --timeout-ms, and --max-memory.

Coverage

The GocciaTestRunner and GocciaScriptLoader support JavaScript source-level coverage reporting via the --coverage flag. Coverage tracks which lines and branches of JavaScript source code are executed at runtime.

Usage

# Console summary (printed after test results)
./build/GocciaTestRunner tests --coverage

# lcov output (for Codecov, Coveralls, or genhtml)
./build/GocciaTestRunner tests --coverage --coverage-format=lcov --coverage-output=coverage.lcov

# JSON output (Istanbul-compatible)
./build/GocciaTestRunner tests --coverage --coverage-format=json --coverage-output=coverage.json

# GocciaScriptLoader also supports coverage
./build/GocciaScriptLoader example.js --coverage

What is Tracked

Line coverage: Which source lines were executed and how many times. Instrumented in both the tree-walk interpreter (EvaluateStatement/EvaluateExpression) and the bytecode VM (main dispatch loop with line-change deduplication).

Branch coverage: Which branch arms were taken at:

  • if/else statements (branch 0 = consequent, branch 1 = alternate)
  • Ternary expressions (? :)
  • Short-circuit operators (&&, ||, ??)
  • switch statement case clauses

Output Formats

Format Flag Description
Console --coverage Summary table printed to stdout after test results
lcov --coverage-format=lcov --coverage-output=<file> Standard lcov tracefile with DA: and BRDA: entries
JSON --coverage-format=json --coverage-output=<file> Istanbul-compatible JSON for tooling integration

JSX Source Map Integration

When coverage is used with .jsx/.tsx files, the JSX transformer produces a source map that maps transformed (post-JSX) coordinates back to the original source. The coverage system integrates this source map so that lcov and JSON reports reference original JSX source positions, not transformed positions. This ensures downstream tools (Codecov, genhtml, Istanbul) display accurate line and branch locations.

The source map is registered with TGocciaCoverageTracker during file registration and applied at report generation time — the recording hot path is unaffected.

Architecture

Coverage uses a runtime boolean check (CoverageEnabled on TGocciaEvaluationContext for the interpreter, FCoverageEnabled on TGocciaVM for bytecode). When --coverage is not passed, the boolean is False and branch prediction makes the check effectively free — no separate build is needed.

Data is collected by TGocciaCoverageTracker (Goccia.Coverage.pas), a per-thread tracker that follows the same Initialize/Shutdown pattern as TGarbageCollector and TGocciaCallStack. During parallel test runs each worker thread initializes its own thread-local instance; after all workers complete, TGocciaThreadPool.MergeCoverageInto merges their data into the main thread's tracker via TGocciaCoverageTracker.MergeFrom. Output formatting is in Goccia.Coverage.Report.pas. When a JSX source map is available for a file, BuildTranslatedLineHits translates transformed line hits back to original coordinates, and branch positions are translated via TGocciaSourceMap.Translate during report emission.