Interop / plug-fest: mechanical conformance testing against OSATE + Ocarina to prevent silent AADL dialect drift

[avrabe]

Problem

spar today has no mechanical check that an OSATE-produced AADL model parses in spar, or that a spar-produced model parses in OSATE / Ocarina. The current "interop story" is purely internal — we hand-write fixtures, parse them, run analyses, assert on golden output. As we grow we will silently drift into a spar-specific dialect of AADL, and we won't notice until an external user can't load their model.

We need an interop matrix. Treating it as a continuous CI artefact ("the matrix is the plug-fest") rather than a physical event is the right shape — DICOM Connectathon-style multi-vendor week is overkill for year one.

Reference tools

• OSATE (SEI/CMU, EPL, quarterly cadence) — the reference. Eclipse-based; the established CI invocation is a headless Tycho plugin, used in practice by RAMSES and ESA TASTE.
• Ocarina (OpenAADL/ocarina, ESA mirror) — the independent second opinion. Different codebase from OSATE, still releasing.
• *Not_ RAMSES — it reuses OSATE's front-end so it's not actually a second parser.

Exchange format

• .aadl text catches lexer/parser/grammar drift. Cheap.
• AAXL2 XMI (AS5506 Annex C — the only standardised interchange format) catches semantic drift after instantiation. spar would grow spar emit-aaxl2 and spar import-aaxl2.

There is no public AS5506 conformance test suite to lift. Corpus comes from osate/examples + osate/alisa-examples (the Aircraft Tier1/2/3 reference models) + a small set of hand-typed snippets from the AS5506 annexes.

Pattern to copy

CycloneDX ↔ SPDX interop: shared corpus, cross-tool CI matrix, with an explicit upfront allow-list of fields that don't survive the round-trip. For spar, the Spar_* property sets (Spar_TSN, Spar_Network, Spar_Identity, Spar_Migration, Spar_Power) are exactly that allow-list — tolerated as opaque on OSATE's side, asserted-equal on spar's side.

Proposed design

Tiered CI matrix

Tier	What	When	Cost
A	osate/examples corpus parses cleanly in spar	every PR	seconds, pure Rust
B	OSATE in Docker round-trips text → aaxl2 → spar and spar → aaxl2 → OSATE	nightly	minutes
C	Add Ocarina as third opinion on Tier-A corpus	quarterly	one CI day

Round-trip tests (cumulative)

• T1 parse-text(spar) — corpus parses without error
• T2 text → spar HIR → text — AST-equivalent (canonicalised: sorted modifiers, normalised whitespace, expanded inherited properties)
• T3 text → spar instance → aaxl2 → reparse(spar) — instance-equivalent
• T4 cross-tool: OSATE → aaxl2 → spar parses
• T5 cross-tool: spar → aaxl2 → OSATE re-instantiates

What "pass" means

Not byte-identical. Compare on the canonical instance: for each instance object, equal category, classifier, set of feature instances, set of connection instances, and property association bag with declared-vs-inherited collapsed. Implement once in spar as instance_eq() and emit a structural diff on failure.

Disagreement handling

When T4/T5 diverges between spar and OSATE:

1. Minimise to ≤30-line fixture
2. Check AS5506C clause + Annex C XMI schema — spec is arbiter
3. File mirrored issues on osate/osate2 and spar with the clause cite
4. Record the agreed interpretation under test-data/interop/adjudicated/<id>/ with a RULING.md
5. If ambiguous, defer to OSATE and open an AS-2C clarification

Quick win (proposed as a first PR, ~half a day's work)

Smallest signal-bearing increment, no Docker / no OSATE-in-CI yet:

1. git submodule add https://github.com/osate/examples test-data/interop/osate-examples (EPL — record in LICENSES/)
2. New tests/osate_corpus.rs that walks every .aadl file and asserts spar parse returns Ok, with a SKIP.toml allow-list for known-broken upstream files (upstream's own README flags some)
3. Print a single number per CI run: N parsed / M total, broken into core / EMV2 / BA buckets

That number, tracked over time, is the cheapest possible canary against silent dialect drift. No commitment to the heavier Tier-B work; proves the corpus + license + submodule mechanics first.

Out of scope

• Physical multi-vendor plug-fest event (year one — the CI matrix IS the plug-fest)
• AGREE / Resolute / Cheddar interop (annex-only or narrow scope; not parser-level)
• AADL V3 (AS-2C is still drafting; we track but don't implement)

References

• Research transcript: in-conversation report, citations include OSATE releases, SEI headless-plugin wiki, Ocarina, cdx2spdx, IHE-Tools/dicom-connectathon
• AS5506C (SAE)
• Memory: feedback_rigor_and_honesty, pulseengine-claude:pulseengine-feature-loop

[avrabe]

Status: Tier-A delivered in v0.16.0; Tier-B/C tracked for v0.21.0 — keeping open, re-milestoning

Reconciled against the code at v0.16.0 HEAD:

✅ Tier-A — DELIVERED (v0.12.0 + v0.16.0):

• OSATE corpus parse-ratchet live in CI (.github/workflows/aadl-interop.yml, crates/spar-cli/tests/osate_corpus.rs): 532/548 files parse (core 297/310, emv2 187/190, behavior 5/5, annex-other 43/43), with committed baselines under test-data/interop/baseline/. → REQ-PLUGFEST-001 (implemented)
• Four core AADL parser gaps closed in v0.16.0 (PR feat(parser): close four core AADL parser gaps (REQ-PLUGFEST-003) #271): array subscripts on reference paths, named-constant property-type ranges, parenthesised classifier reference lists, in-mode property associations — corpus advanced 519→532. → REQ-PLUGFEST-003 (implemented)

◻️ Tier-B / Tier-C — NOT STARTED, already scheduled v0.21.0:

• OSATE-in-Docker oracle (headless Tycho), .aadl↔AAXL2 round-trip, instance-equivalence, adjudication framework (adjudicated/RULING.md) → REQ-PLUGFEST-002
• Ocarina independent second-opinion parser → REQ-PLUGFEST-004
• Remaining EMV2 annex-path parser gaps (4 files: applies to ({EMV2}**error detection)) → REQ-PLUGFEST-005 (prerequisite that unblocks the Tier-B oracle)

The three-tier design intentionally defers B/C — this is scoping, not a gap. The v0.16.0 milestone was only ever Tier-A, which is done; re-milestoning the issue to v0.21.0 where Tier-B/C land. Keeping open as the umbrella for that work.