Skip to content

Extend bench-ab.rkt with --refs for multi-way A/B/C+ comparison #63

Description

@hierophantos

Summary

Extend racket/prologos/tools/bench-ab.rkt with a --refs flag (accepting multiple commit refs) so a single invocation can produce a multi-way A/B/C+ comparison report (vs the current --ref which supports only A/B).

Origin

Surfaced during PPN 4C Tropical Quantale Addendum 1C-vi A/B/C report work (commit TBD, 2026-05-16). Per the addendum's §13.7 "Bench infrastructure" cross-track note, this was flagged as a "small tool enhancement" deferred to a future track:

A/B/C comparison tool: tools/bench-ab.rkt currently supports A/B (--ref HEAD~1). For A/B/C: run 3 separate bench-ab.rkt invocations against tagged baseline commits (e.g., --ref pre-1c-baseline for A; current HEAD for B/C). Or extend bench-ab.rkt with --refs for multi-way comparison (Phase 1C-vi requirement; small tool enhancement).

Per 1C-vi β3 resolution: a one-off markdown A/B/C report sufficed for THIS addendum because A baseline data file already existed (live A re-measurement was structurally impossible — OLD struct-field counter retired at 1C-iv-b). Future tracks with all 3+ variants LIVE will benefit from tool-level support.

Consumed by (forward-references)

  • OE Series Track 1 (weighted parsing) — multi-variant cost comparison (Goodman semiring framework on PPN). See docs/tracking/MASTER_ROADMAP.org § OE Series.
  • PReduce Track 4 (cost-guided extraction with tropical-quantale residuation) — multiple cost-extraction strategies to compare. See docs/tracking/MASTER_ROADMAP.org § PReduce Series + docs/tracking/2026-05-02_PREDUCE_MASTER.md.
  • Future PAR tracks (parallel scheduler variants comparison) — Variant A / Variant B / future scheduler patterns.

Design sketch

racket tools/bench-ab.rkt --refs HEAD~5,HEAD~2,HEAD benchmarks/comparative/
  • Multiple --ref flags OR comma-separated --refs argument
  • Each ref produces a sample set (stash + checkout + run + restore; existing pattern)
  • Output: per-ref Mann-Whitney U vs the others (currently bench-ab.rkt does U for 2 samples; extend to pairwise or vs-baseline matrix)
  • Markdown report option (currently bench-ab.rkt outputs JSON; add markdown table generation)

Cross-references

  • Design doc: docs/tracking/2026-04-26_PPN_4C_TROPICAL_QUANTALE_ADDENDUM_DESIGN.md §13.7 (per-phase measurement plan; "small tool enhancement" cross-track note)
  • 1C-vi A/B/C report (consumer; one-off workaround): docs/tracking/2026-05-16_TROPICAL_1C_VI_ABC_REPORT.md
  • Tracking: docs/tracking/DEFERRED.md (this Issue cross-referenced from there)
  • Tracking: docs/tracking/MASTER_ROADMAP.org § OE Series + § PReduce Series (forward-references)

Multi-surface tracking discipline

This issue is part of a dual-surface tracking pattern (Issue + MASTER_ROADMAP forward-refs + DEFERRED.md entry, all cross-referenced) — established as a stronger discipline than DEFERRED.md alone, per the user direction at the 1C-vi β resolution conversation (2026-05-16): "capturing in deferred is not meaningful follow up and likely will be work lost." Each surface explicitly cites the others so any one getting updated/closed propagates via the cross-references.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions