Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `generateScaffoldFiles(source, flags, options): Record<string, string>` — new programmatic API that returns a map of filename → content; used by the CLI when `--outdir` is given ([#25](https://github.com/ConfiguredThings/RDP.js/issues/25))
- `InterpreterMixin<TTree, TResult>` and `WalkerMixin<TTree>` exported from the main package — TypeScript mixin interfaces for interpreter and tree-walker traversal patterns

### Fixed
- `--transformer json --facade --pipeline` now produces three separate scaffold files (`{base}-transformer.ts`, `{base}-pipeline.ts`, `{base}-facade.ts`) so the facade is a genuine module boundary; consumers import only from the facade ([#25](https://github.com/ConfiguredThings/RDP.js/issues/25))

### Changed

- `--output <file>` renamed to `--outdir <dir>` — the flag now accepts a directory; artifact filenames are derived from the parser base name (e.g. `date-facade.ts`, `date-transformer.ts`) ([#25](https://github.com/ConfiguredThings/RDP.js/issues/25))
- `--traversal` alone now adds mixin stubs directly to the generated parser file (can be re-generated freely) rather than producing a separate scaffold file. Using `--traversal` together with `--facade`, `--pipeline`, or `--transformer` still produces a scaffold file

## [0.6.0] - 2026-04-20

### Added
Expand Down
104 changes: 64 additions & 40 deletions docs-site/content/guide/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ rdp-gen <grammar> [options]
| Option | Default | Description |
|--------|---------|-------------|
| `<grammar>` | (required) | Path to a `.ebnf` or `.abnf` grammar file |
| `-o, --output <file>` | stdout | Write output to a file instead of stdout |
| `-o, --outdir <dir>` | stdout | Write output files to a directory; each artifact gets a derived filename |
| `--format <fmt>` | inferred from extension | Force `ebnf` or `abnf` |
| `--parser-name <name>` | `GeneratedParser` | Class name for the generated parser |
| `--tree-name <name>` | `ParseTree` | Type name for the generated parse tree |
Expand All @@ -56,16 +56,16 @@ rdp-gen <grammar> [options]

#### Examples

Generate a parser to a file:
Generate a parser to a directory (writes `src/JsonParser.ts`):

```bash
rdp-gen grammar.ebnf --parser-name JsonParser --output src/JsonParser.ts
rdp-gen grammar.ebnf --parser-name JsonParser --outdir src/
```

Generate with tracing support:

```bash
rdp-gen grammar.ebnf --parser-name MyParser --observable --output src/MyParser.ts
rdp-gen grammar.ebnf --parser-name MyParser --observable --outdir src/
```

Parse an ABNF grammar and write to stdout:
Expand All @@ -76,9 +76,31 @@ rdp-gen protocol.abnf --format abnf --parser-name FrameParser

---

### Traversal stubs (`--traversal`)

When used alone (without `--facade`, `--pipeline`, or `--transformer`), `--traversal`
adds evaluation scaffolding **directly to the generated parser file** — the output is
still a parser, not a separate scaffold. Because the stubs follow the grammar
mechanically, the file can be **regenerated** as the grammar evolves.

| Flags | What it adds to the parser |
|---|---|
| `--traversal interpreter` | Parser class implements `InterpreterMixin<ParseTree, unknown>` — one `eval{Rule}(node): unknown` stub per rule and a `static evaluate()` entry point |
| `--traversal tree-walker` | Exports a `walk(root, fn)` function alongside the existing `childNodes()` helper; includes a commented-out `Visitor` template covering every rule |

```bash
# Generate (or regenerate) a parser with interpreter stubs baked in (writes src/DateParser.ts)
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --outdir src/
```

When `--traversal` is combined with `--facade`, `--pipeline`, or `--transformer`, the
traversal strategy moves _inside_ the scaffold — see [Scaffolding](#scaffolding) below.

---

### Scaffolding

When any of `--traversal`, `--transformer`, `--facade`, `--pipeline`, or `--lexer` is
When any of `--transformer`, `--facade`, `--pipeline`, or `--lexer span` is
present, `rdp-gen` emits a **scaffold** rather than a parser. A scaffold is a
**one-time starter file** — unlike the generated parser it is not designed to be
regenerated. It imports the parser by a relative path, wires up the chosen pattern,
Expand All @@ -89,21 +111,17 @@ a real grammar.

| Flags | What it emits |
|---|---|
| `--traversal interpreter` | One typed `eval{Rule}()` function per rule, `evaluate()` entry, `RDParserException` wrapping |
| `--traversal tree-walker` | `walk()` utility using `childNodes`, commented `Visitor` stub for every rule |
| `--traversal interpreter --facade` | Module-as-facade: `parse{Base}()`, `{Base}Result` class with `from()`, private eval functions |
| `--traversal tree-walker --facade` | Module-as-facade: `parse{Base}()`, `{Base}Result` class, private walk utility and visitor stubs |
| `--traversal tree-walker --pipeline` | Exported `parse`/`validate`/`transform` + `load{Base}()` combinator; tree-walker inside `transform` |
| `--traversal interpreter --pipeline --facade` | Module-as-facade wrapping a pipeline; eval inside `#transform` |
| `--traversal tree-walker --pipeline --facade` | As above but `#transform` uses a tree walker |
| `--traversal tree-walker --pipeline --facade` | Module-as-facade wrapping a pipeline with a tree walker inside `#transform` |
| `--transformer` | Exported `Transformer<ParseTree, unknown>` object with a stub per rule + entry function |
| `--transformer json` | Two-way stubs: `{Base}ToJSON: Transformer<ParseTree, JSONAST>` and `jsonTo{Base}: Transformer<JSONAST, string>` plus round-trip helpers |
| `--lexer span` | Span tokeniser + classifier + `{Base}TokenParser` stubs (see [Tokenising for performance](/docs/tokenising/)) |
| `--lexer span --traversal interpreter` | As above but `{Base}TokenParser` evaluates directly during descent — no intermediate tree |

#### Flag compatibility

The scaffold flags are designed to compose. The table below shows which combinations are valid (`✓`), invalid (`✗`), or not applicable (`—`). The one constraint is that `--traversal interpreter` cannot be combined with `--pipeline` unless `--facade` is also set (the facade wraps a private pipeline whose `#transform` uses the interpreter, so an intermediate tree is still produced).
The scaffold flags are designed to compose. The table below shows which combinations are valid (`✓`), invalid (`✗`), or not applicable (`—`). The one constraint is that `--traversal interpreter` cannot be combined with `--pipeline` the interpreter evaluates directly during traversal, leaving no intermediate tree for the validate stage.

| | `--facade` | `--pipeline` | `--transformer [json]` |
|---|:---:|:---:|:---:|
Expand All @@ -113,37 +131,46 @@ The scaffold flags are designed to compose. The table below shows which combinat
| `--facade` | — | ✓ | ✓ |
| `--transformer [json]` | ✓ | — | — |

`--lexer span` is compatible with any of the above combinations (pass it alongside `--traversal interpreter` to wire evaluation directly into the token parser).
`--lexer span` is compatible with any of the above combinations.

When `--transformer json` is combined with `--facade` (with or without `--pipeline`),
`--outdir` produces **separate files** for each concern so the facade is a genuine module boundary:

```bash
# Writes: src/DateParser.ts, src/date-transformer.ts, src/date-pipeline.ts, src/date-facade.ts
rdp-gen date.abnf --parser-name DateParser --transformer json --facade --pipeline --lexer span --outdir src/
```

The typical workflow for a new grammar:

```bash
# 1. Generate the parser (regenerate freely as the grammar evolves)
rdp-gen date.ebnf --parser-name DateParser --output src/DateParser.ts
# 1. Generate the parser (regenerate freely as the grammar evolves) → src/DateParser.ts
rdp-gen date.ebnf --parser-name DateParser --outdir src/

# 2. Generate a scaffold once (edit this file; do not regenerate)
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --facade --output src/date.ts
# 2. Generate a scaffold once (edit; do not regenerate) → src/date-facade.ts
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --facade --outdir src/
```

The scaffold imports the parser by a relative path (`'./DateParser.js'`), so keep
both files in the same directory.
Both files land in the same directory, so the scaffold's import of `'./DateParser.js'` resolves correctly.

#### Interpreter scaffold
#### Interpreter traversal stubs

Emits one `eval{Rule}(node: {Rule}Node): unknown` function per rule, an `evaluate()`
entry point that calls the first rule, and wraps `RDParserException` in a plain `Error`:
Adds `InterpreterMixin<ParseTree, unknown>` to the parser class, emitting one
`eval{Rule}(node: {Rule}Node): unknown` stub per rule and a `static evaluate()` entry
point. The file header notes that stubs are present and safe to edit; the parser
structure itself can still be regenerated:

```bash
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --output src/date-eval.ts
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --outdir src/
```

#### Tree-walker scaffold
#### Tree-walker stubs

Emits a `walk(root, fn)` utility using `childNodes` (always present in the generated
parser), plus a commented-out `Visitor` stub covering every rule in the grammar:
Exports a `walk(root, fn)` function alongside the existing `childNodes()` helper,
plus a commented-out `Visitor` template covering every rule in the grammar:

```bash
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --output src/date-walker.ts
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --outdir src/
```

#### Facade scaffold
Expand All @@ -153,17 +180,14 @@ class with a `static from(tree)` factory, and a `{Base}Error` class. Combine wit
`--traversal` to specify the strategy used inside the facade:

```bash
# Facade wrapping recursive interpreter functions
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --facade --output src/date.ts

# Facade wrapping a tree walker
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --facade --output src/date.ts
# Facade wrapping recursive interpreter functions → src/date-facade.ts
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --facade --outdir src/

# Facade wrapping a full pipeline (interpreter inside #transform)
rdp-gen date.ebnf --parser-name DateParser --traversal interpreter --pipeline --facade --output src/date.ts
# Facade wrapping a tree walker → src/date-facade.ts
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --facade --outdir src/

# Facade wrapping a full pipeline (tree walker inside #transform)
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --pipeline --facade --output src/date.ts
# Facade wrapping a full pipeline (tree walker inside #transform) → src/date-facade.ts
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --pipeline --facade --outdir src/
```

#### Pipeline scaffold
Expand All @@ -175,8 +199,8 @@ inside `transform`. Note that `--traversal interpreter` is not compatible with `
(see [flag compatibility](#flag-compatibility) above):

```bash
# Pipeline with tree walker inside transform
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --pipeline --output src/date-pipeline.ts
# Pipeline with tree walker inside transform → src/date-pipeline.ts
rdp-gen date.ebnf --parser-name DateParser --traversal tree-walker --pipeline --outdir src/
```

#### Transformer scaffold
Expand All @@ -187,7 +211,7 @@ concrete return type, then fill in the handlers. Because `Transformer` requires
adding a new rule to the grammar will immediately cause a compile error in the scaffold:

```bash
rdp-gen date.ebnf --parser-name DateParser --transformer --output src/date-transformer.ts
rdp-gen date.ebnf --parser-name DateParser --transformer --outdir src/
```

See [`Transformer<ParseTree, T>` and `transform()`](/docs/parse-tree/#transformer-parsetree-t-and-transform)
Expand All @@ -200,10 +224,10 @@ functions. This is the recommended starting point when JSON is one endpoint of y
translation:

```bash
rdp-gen date.ebnf --parser-name DateParser --transformer json --output src/date-json.ts
rdp-gen date.ebnf --parser-name DateParser --transformer json --outdir src/
```

The generated file contains:
Writes `src/date-transformer.ts` containing:
- `dateToJSON: Transformer<ParseTree, JSONAST>` — one stub per grammar rule
- `jsonToDate: Transformer<JSONAST, string>` — one stub per JSON kind (`string`, `number`, `boolean`, `null`, `array`, `object`)
- `dateToJSONString(input: string): string` — parses the input and calls `fromJSONAST`
Expand Down
31 changes: 31 additions & 0 deletions docs-site/src/components/CopyCodeBlock.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import React, { useRef, useState } from 'react'

type Props = React.HTMLAttributes<HTMLPreElement>

export function CopyCodeBlock({ children, ...props }: Props) {
const preRef = useRef<HTMLPreElement>(null)
const [copied, setCopied] = useState(false)

const handleCopy = () => {
const text = preRef.current?.textContent ?? ''
void navigator.clipboard.writeText(text).then(() => {
setCopied(true)
setTimeout(() => setCopied(false), 2000)
})
}

return (
<div className="code-block-wrapper">
<pre ref={preRef} {...props}>
{children}
</pre>
<button
className={`code-copy-btn${copied ? ' code-copy-btn--copied' : ''}`}
onClick={handleCopy}
aria-label="Copy code"
>
{copied ? 'Copied!' : 'Copy'}
</button>
</div>
)
}
34 changes: 34 additions & 0 deletions docs-site/src/styles/global.css
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,40 @@ a:hover { color: var(--color-link-hover); text-decoration: underline; }
}

/* ── Shiki code blocks ────────────────────────────────────────────────────── */
.doc-content .code-block-wrapper {
position: relative;
}

.doc-content .code-block-wrapper:not(:hover) .code-copy-btn:not(.code-copy-btn--copied) {
opacity: 0;
}

.code-copy-btn {
position: absolute;
top: 0.5rem;
right: 0.5rem;
padding: 0.2rem 0.55rem;
font-size: 0.75rem;
font-family: var(--font-sans);
line-height: 1.5;
border-radius: 6px;
border: 1px solid var(--color-border);
background: var(--color-bg);
color: var(--color-text-muted);
cursor: pointer;
transition: opacity 0.15s, background 0.15s, color 0.15s;
}

.code-copy-btn:hover {
background: var(--color-border);
color: var(--color-text);
}

.code-copy-btn--copied {
color: var(--color-accent);
border-color: var(--color-accent);
}

.doc-content pre {
margin: 1.25rem 0;
border-radius: 10px;
Expand Down
2 changes: 2 additions & 0 deletions docs-site/src/templates/doc-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { SiteHeader } from '../components/SiteHeader'
import { DocRailroadDiagram } from '../components/DocRailroadDiagram'
import { BenchmarkChart } from '../components/BenchmarkChart'
import { TableOfContents } from '../components/TableOfContents'
import { CopyCodeBlock } from '../components/CopyCodeBlock'

interface PageContext {
id: string
Expand All @@ -23,6 +24,7 @@ const mdxComponents = {
img: ({ src, alt, ...props }: React.ImgHTMLAttributes<HTMLImageElement>) => (
<img src={src?.startsWith('/') ? withPrefix(src) : src} alt={alt} {...props} />
),
pre: CopyCodeBlock,
RailroadDiagram: DocRailroadDiagram,
BenchmarkChart,
}
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,10 @@
"npm": ">=10"
},
"scripts": {
"clean": "rm -rf dist",
"generate:grammars": "node scripts/generate-grammar-index.mjs",
"pretest": "npm run generate:grammars",
"prebuild": "npm run generate:grammars",
"prebuild": "npm run clean && npm run generate:grammars",
"build": "tsc -p tsconfig.cjs.json && tsc -p tsconfig.esm.json",
"test": "node --experimental-vm-modules --no-warnings=ExperimentalWarning node_modules/.bin/jest",
"lint": "eslint src docs-site/src",
Expand Down
70 changes: 70 additions & 0 deletions src/__tests__/generator/codegen.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -444,3 +444,73 @@ describe('generated walker — runtime correctness', () => {
])
})
})

// ── traversal: interpreter ────────────────────────────────────────────────────

const DATE_GRAMMAR = `Date = Year, '-', Month, '-', Day;\nYear = Digit, Digit, Digit, Digit;\nMonth = Digit, Digit;\nDay = Digit, Digit;\nDigit = '0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9';`
const PARSER_NAME = 'DateParser'

describe('generateParser — traversal: interpreter', () => {
const src = generateParser(DATE_GRAMMAR, { parserName: PARSER_NAME, traversal: 'interpreter' })

it('emits scaffold header (not regenerated)', () => {
expect(src).toContain('not regenerated; edit freely')
})

it('imports InterpreterMixin from @configuredthings/rdp.js', () => {
expect(src).toMatch(/import \{[^}]*InterpreterMixin[^}]*\} from '@configuredthings\/rdp\.js'/)
})

it('class implements InterpreterMixin<ParseTree, unknown>', () => {
expect(src).toContain('implements InterpreterMixin<ParseTree, unknown>')
})

it('emits one eval stub per grammar rule', () => {
expect(src).toContain('evalDate(node: DateNode): unknown')
expect(src).toContain('evalYear(node: YearNode): unknown')
expect(src).toContain('evalMonth(node: MonthNode): unknown')
expect(src).toContain('evalDay(node: DayNode): unknown')
expect(src).toContain('evalDigit(node: DigitNode): unknown')
})

it('emits static evaluate() entry point', () => {
expect(src).toContain('static evaluate(input: string): unknown')
expect(src).toContain('parser.evalDate(parser.parse())')
})

it('type-checks without errors', () => {
expect(typeCheck(src)).toEqual([])
})
})

// ── traversal: tree-walker ────────────────────────────────────────────────────

describe('generateParser — traversal: tree-walker', () => {
const src = generateParser(DATE_GRAMMAR, { parserName: PARSER_NAME, traversal: 'tree-walker' })

it('emits scaffold header (not regenerated)', () => {
expect(src).toContain('not regenerated; edit freely')
})

it('imports visit and Visitor from @configuredthings/rdp.js', () => {
expect(src).toMatch(/import \{[^}]*visit[^}]*\} from '@configuredthings\/rdp\.js'/)
expect(src).toMatch(/import \{[^}]*Visitor[^}]*\} from '@configuredthings\/rdp\.js'/)
})

it('exports walk()', () => {
expect(src).toContain('export function walk(root: ParseTree')
})

it('walk() uses childNodes for recursion', () => {
expect(src).toContain('for (const child of childNodes(root)) walk(child, fn)')
})

it('includes visitor template comment', () => {
expect(src).toContain(`// const visitor: Visitor<ParseTree> = {`)
expect(src).toContain(`// 'Date': (node) => { /* ... */ },`)
})

it('type-checks without errors', () => {
expect(typeCheck(src)).toEqual([])
})
})
Loading