wlp4-compiler

A WLP4 compiler implemented as a classic multi-pass pipeline in C++17: lexical analysis, parsing, semantic analysis, and ARM64 code generation. WLP4 is a small teaching language (C++-like subset with long / long*, procedures, and wain as entry point).

Headline result: on a 65-program benchmark, our .com output is −79.6% smaller than the reference course compiler wlp4c — see docs/benchmark.md and the optimization journey writeup.

Pipeline

flowchart LR
    SRC[/"program.wlp4"/] --> SCAN[wlp4scan]
    SCAN -->|tokens| PARSE[wlp4parse]
    PARSE -->|.wlp4i tree| TYPE[wlp4type]
    TYPE -->|.wlp4ti annotated| GEN[wlp4gen]
    GEN -->|ARM64 .asm| LINK[linkasm]
    LINK --> COM[/"program.com"/]
    COM --> EMU[arm64emu]

    classDef ours fill:#d4f4dd,stroke:#2a7d3c,color:#000;
    classDef ref  fill:#f4e6d4,stroke:#7d5a2a,color:#000;
    class SCAN,PARSE,TYPE,GEN ours;
    class LINK,EMU ref;

Loading

Green = code in this repo (src/). Tan = reference toolchain from bin_ref/ (course-provided binaries; out of scope to reimplement).

Stage	Binary	Source	Role
Scan	wlp4scan	src/wlp4scan.cc	Source → token stream
Parse	wlp4parse	src/wlp4parse.cc	LR parse → .wlp4i preorder tree
Type check	wlp4type	src/wlp4type.cc	.wlp4i → .wlp4ti (annotated)
Codegen	wlp4gen	src/wlp4gen.cc	.wlp4ti → ARM64 assembly

Quick start

# Build (either path produces bin/wlp4{scan,parse,type,gen}):
./build-toolchain.sh
# or:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Compile a program end-to-end (asm or .com):
bin/wlp4 -S -o prog.asm test/procedures/proc.wlp4
bin/wlp4 -c -o prog.com test/procedures/proc.wlp4   # uses bin_ref/linkasm

# Run the regression suite (25 programs, byte-equal vs reference):
bash scripts/run-tests.sh

# Run the benchmark vs wlp4c (65 programs, emits CSV):
bash tools/bench.sh > docs/benchmark.csv

On macOS, bin/wlp4 -c, scripts/run-tests.sh, and tools/bench.sh transparently route bin_ref/* (Linux x86-64 ELFs) through colima ssh when colima is installed.

Optimization workflow

flowchart TD
    A[Edit src/] --> B[bash scripts/run-tests.sh]
    B -->|25/25 pass| C[bash tools/bench.sh]
    B -.->|fail| A
    C --> D{Delta vs baseline?}
    D -->|better/neutral| E[git commit + push]
    D -->|worse| A
    E --> F[GitHub Actions]
    F --> G[CMake smoke + regression + benchmark artifact]

Loading

CI: .github/workflows/ci.yml runs the regression suite + a CMake smoke build + tools/bench.sh, attaches benchmark.csv as a downloadable artifact and posts the totals to the run's step summary.

Documentation

Path	Contents
docs/wlp4.txt	Language definition (lexical, grammar, semantics)
docs/wlp4i.txt, docs/wlp4ti.txt	Intermediate text formats
docs/bin.txt	Tool invocations, external wlp4c / emulator notes
docs/armcom.txt, docs/arm64_ref_sheet.md	Object format and ARM64 ISA reference
docs/benchmark.md, docs/benchmark.csv	Size benchmark vs wlp4c
docs/blog-optimization-journey.md	Long-form writeup: why we ended up 5× smaller than the reference

Repository layout

src/           # wlp4scan.cc, wlp4parse.cc, wlp4type.cc, wlp4gen.cc
bin/           # build output + wlp4 driver script
bin_ref/       # vendored reference toolchain (linkasm, arm64emu, wlp4c, …)
docs/          # language reference, benchmark, blog
scripts/       # run-tests.sh, count_instructions.py
test/          # hand-written *.wlp4 by category (procedures/, pointers/, …)
tools/         # bench.sh, gen_random_wlp4.py

Contributing

Issues and pull requests are welcome. Please keep changes focused; match existing style in src/ and run scripts/run-tests.sh (and ideally tools/bench.sh) when you touch the compiler.