Skip to content

test(testreport): pin the SLFO packages envelope against a real metadata record - #444

Merged
mimi1vx merged 2 commits into
openSUSE:mainfrom
plusky:test/397-real-slfo-metadata-fixture
Aug 12, 2026
Merged

test(testreport): pin the SLFO packages envelope against a real metadata record#444
mimi1vx merged 2 commits into
openSUSE:mainfrom
plusky:test/397-real-slfo-metadata-fixture

Conversation

@plusky

@plusky plusky commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Closes #397.

What the evidence says

#397 asked for a real SLFO metadata.json and left two questions open. Both are
now answered from observed records, fetched from TeReGen for two independent
updates — SUSE:SLFO:1.1:418286 (the one #396 named) and a current
SUSE:SLFO:1.2:6766:

"packages": {
    "standard": [
        "afterburn = 5.10.0.git73.b97f772-99999_stage.1.1",
        "afterburn-dracut = 5.10.0.git73.b97f772-99999_stage.1.1"
    ]
}
  1. Entries are exactly three whitespace tokens, <name> = <version> — the
    parser's strict form is right, with = as the discarded middle token.
  2. The map is keyed standard, and nothing else — so packages_for_map's
    single-key branch is the live path for SLFO, and seeding does reach hosts.

Both assumptions hold, so this is not a hidden #396 root cause and no
production behaviour changes
. It is the coverage the issue asked for, plus the
correction of a belief the tree was encoding.

To be precise about what was and was not already covered: the = form and
version-is-token[2] were already pinned against real data by the Maintenance
golden, which asserts whole-map equality on a captured record. The genuinely
unpinned things were the standard key, the map.len() == 1 half of the
guard, and any assertion at all on base().packages after TestReport::read.

The belief that was wrong

slfo_populated_envelope_parses landed with #396 as an explicitly synthetic
probe (its own doc said real-fixture coverage remained #397) and guessed both
halves wrong: key 6.1, separator _. So the tree held two irreconcilable
claims — packages_for_map's doc saying SLFO ⇒ standard, and that test saying
SLFO ⇒ 6.1. It is retargeted onto the observed shape.

The _ came from the envelope rustdoc writing "pkg _ version", which is
Rust's ignored-binding notation for the discarded token; read as a wire format
it says SLFO ships underscores. Second commit fixes the notation.

What is pinned, and what is not

test pins
json_parser_parses_slfo_real_fixture the real key set (whole set, not contains_key) and both real versions
slfo_real_fixture_seeds_packages_for_any_base_version real fixture → packages_for_map with a base_version deliberately absent from the map; asserts required(), not just names
packages_for_map_second_product_key_disables_standard_branch the map.len() == 1 half of the guard
make_testreport_slfo_real_metadata_populates_packages the on-disk loader keeps the real values through TestReport::read

Two deliberate restraints:

  • get_package_list() is only a labelled secondary assertion. The issue
    suggests asserting it is non-empty; it flattens every product and never
    consults base_version, so it stays green under a total failure of the
    standard assumption. Resting the case on it would have been another check
    that cannot fail.
  • Host seeding is not claimed. Session::connect_one is private and
    Target::connect hard-codes SshConnection::connect with no injectable seam,
    so no test can reach the production seeding call site. The packages_for_map
    half is re-implemented explicitly instead, and the test doc says so rather
    than implying coverage that does not exist.

Mutation evidence

Every new assertion has a mutation applied to production code, observed red, and
reverse-restored. Two worth naming:

  • Dropping map.len() == 1 from packages_for_map's guard leaves the entire
    264-test lib suite green
    — the existing single-key test cannot distinguish it,
    because its map has one key either way. The new probe is the only thing in the
    workspace that catches it.
  • Clearing base().packages right after the loader's parse_str reddens the new
    lifecycle test alone, which is what isolates its added coverage from the
    eleven pre-existing SLFO lifecycle tests.

Also fixed along the way: a pre-existing probe in the same file used the same
value for an entry's second and third token, so it could not distinguish
version-from-second-token from version-from-third.

Fixture provenance

crates/mtui-testreport/tests/fixtures/metadata/slfo_metadata.json is
field-for-field the captured record with one substitution: packager
someone@suse.com, the address the lifecycle fixtures already use, so a named
individual's address is not published. Nothing under test depends on it. Say the
word if you would rather have it verbatim — the existing golden fixture already
ships a real address and a real IBS URL, so either is consistent with precedent.

Field-for-field, not byte-for-byte: the record arrives from TeReGen as a single
~950-byte line, and the fixture is re-serialized with sorted keys and 4-space
indent to match its golden sibling. Every value is byte-identical; the file is
not. So a diff against a fresh re-fetch is reformatting, not tampering. The
in-tree provenance note on the const says the same.

Every other value — RRID, products, packages, testplatform, repository, bugs,
SRCRPMs, product_composer — is the real record, unedited.

plusky added 2 commits August 11, 2026 18:14
openSUSE#397)

The `packages` map being keyed `standard` was load-bearing and never checked
against a real SLFO record. `packages_for_map` routes SLFO through a branch
that fires only for a single `standard` key, and if real metadata were keyed
anything else — a product version, a product name — seeding would resolve to
nothing. The only SLFO end-to-end fixture shipped `"packages": {}`, and the
one populated SLFO probe was synthetic, guessing the key as `6.1` and the
separator as `_`.

Capture the real metadata.json for SUSE:SLFO:1.1:418286 as a fixture and test
against it. The map is keyed `standard` and nothing else, and entries are
`<name> = <version>` — the same three-token form the Maintenance golden
already pins. The synthetic probe, now redundant against a real record, is
removed.

Two further gaps this closes. The `map.len() == 1` half of the `standard`
guard was invisible to the suite: the existing single-key test cannot
distinguish it from a bare `contains_key`, and dropping it leaves all 264 lib
tests green. And nothing anywhere asserted `base().packages` after
`TestReport::read` — no unit test drives `read()` at all — so the on-disk
loader could stop parsing envelopes for SL reports without a package-level
failure.

`get_package_list()` is deliberately only a labelled secondary assertion: it
flattens every product and never consults `base_version`, so it stays green
under a total failure of the `standard` assumption.

Also fixes a probe that could not distinguish version-from-second-token from
version-from-third, having used the same value for both.

No production behaviour changes. The fixture is field-for-field the captured
record apart from `packager`, replaced with the address the lifecycle fixtures
already use.
…penSUSE#397)

The envelope docs rendered a package entry as `"pkg _ version"`, using
Rust's ignored-binding notation for the token the parser throws away. Read
as a wire format instead, it says SLFO ships underscore-separated entries —
which is how the synthetic test fixtures came to use `_`, while every real
record uses `=`.

Write the shape as `<name> <op> <version>` with an observed example, and say
outright that the middle token is not significant.

Comments only.
@plusky plusky added bug Something isn't working ai-assisted labels Aug 11, 2026

@mimi1vx mimi1vx left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test/docs-only; the sole src edit is rustdoc prose, so no production behaviour changes.

Verified on the branch: cargo fmt --all --check, cargo clippy -p mtui-testreport --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc -p mtui-testreport,
and cargo test -p mtui-testreport (154 integration + 264 lib) all green.

Both new assertions were observed red against broken code, so they pin something:

  • Dropping map.len() == 1 && from the packages_for_map guard (src/testreport.rs:254)
    reddens only packages_for_map_second_product_key_disables_standard_branch — no
    pre-existing test caught that mutation.
  • The fixture fix "test_pkg 1.0 1.0" -> "1.0 2.0" makes a version/token swap in the
    parser detectable; before it, that assertion could not fail. A real fix, not cosmetic.

Also checked: both sides sort, so no HashMap-iteration flake; no fixture-directory
globbing that the new slfo_metadata.json could perturb; no secrets in the fixture.

Non-blocking notes, all wording/placement:

  • tests/metadata_parsers.rs:586-598packages_for_map_second_product_key_disables_standard_branch
    is a pure unit test (hand-built HashMap, no I/O) sitting in an integration module.
    Every sibling packages_for_map test is colocated in src/testreport.rs. Moving it
    would put the probe next to the guard, reuse the existing base_with_packages helper,
    and drop the public-API import for something reachable in-crate. The two fixture-driven
    tests in the same file are correctly placed.
  • The #397 T1/T2/T2b/T3 prefixes are task-breakdown scaffolding that goes stale once the
    issue closes; the bare issue number carries the same provenance.
  • The 15-20 line doc blocks are largely review argument. The durable parts are the "what
    breaks if this changes" lines — that base_version must not be a map key or the
    standard branch isn't under test, and that get_package_list() is the weaker
    secondary assertion. The rest reads better in the PR description, where it already is.
  • The fixture is include_str!'d into two consts in the one tests/it.rs binary, each
    with its own provenance doc to keep in sync by hand; one pub(crate) const would do.
  • The PR body says "byte-for-byte ... one substitution", while
    tests/metadata_parsers.rs:29 says it is re-serialized with sorted keys and re-indented.
    The in-tree doc is the accurate one — worth fixing the body, since fixture provenance is
    the point of the change.

@mimi1vx
mimi1vx merged commit ff061cd into openSUSE:main Aug 12, 2026
16 checks passed
@plusky
plusky deleted the test/397-real-slfo-metadata-fixture branch August 12, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No fixture exercises a populated SLFO packages envelope, so the SLFO parse path is untested

2 participants