Make dims usable for a real pytest-benchmark consumer (UNSERIALIZABLE params + function-as-dim)

[FBumann]

Found dogfooding the plotting against linopy's (lean) pytest-benchmark suite — out of the box the dims aren't usable, for two separate reasons.

Context

linopy parametrizes its benchmarks over a spec object and an int, one test function per phase:

@pytest.mark.parametrize(("spec", "n"), params, ids=[spec_param_id(...) for ...])
def test_to_lp(benchmark, spec, n, tmp_path): ...

A real --benchmark-json then contains:

fullname:   benchmarks/test_to_lp.py::test_to_lp[basic-n=10]
params:     {'spec': "UNSERIALIZABLE[BenchSpec('basic', axis='n', sweep=(10, 250))]", 'n': 10}
extra_info: {}

Since _dims reads only params + scalar extra_info (no id parsing — correct by design), the dims come out as {spec: "UNSERIALIZABLE[…]", n: 10}: a garbage spec, and no phase (it's the function name). A plot_scaling is then one indistinguishable blob. Two concerns:

1. UNSERIALIZABLE[...] param values poison dims

pytest-benchmark stringifies non-JSON params as "UNSERIALIZABLE[<repr>]". As a dim that's worse than useless — it's a long, unreadable, per-object-identical string. Proposal: in _dims, detect values matching UNSERIALIZABLE[...] and drop them (a param that couldn't be serialized isn't a usable analysis axis). Cheap, defensive, clearly right.

2. The test function / module isn't available as a dim

The single most important grouping axis for a multi-operation suite — which operation (build vs to_lp vs to_solver) — lives in the node id's function name, which the dims model deliberately never parses. So there's no way to facet/scale per phase from a single JSON without the producer manually mirroring the function name into extra_info.

Proposal: expose the node's function (and/or module) as a built-in dim — e.g. func / module — derived from fullname. That's not "id parsing for structure"; it's surfacing pytest's own grouping that every suite already has. With it, phase comes for free.

Notes

• A consumer can already work around both via an autouse fixture that writes clean scalars into extra_info (linopy will do this) — but these make pytest-benchmem usable out of the box for any pytest-benchmark suite, which is the point.
• measure() silently swallows failing cases — surface them #1 is a small defensive fix; Case has no channel for consumer per-case state (selection ergonomics) #2 is a design call (does the function name belong in the dims model?). Filed together since they surfaced from the same "can I just plot this?" test.

🤖 Issue drafted by Claude from the linopy integration.

[FBumann]

Version observed on: pytest-benchmem v0.1.0 (latest release) — behavior confirmed on main @ 0b6fad6 (dogfooded build 0.1.1.dev11+gea666b5d3). The _dims contract (dims = params + scalar extra_info, minus the benchmem blob; no id parsing) is unchanged across these.

[FBumann]

Third producer-side ask: ship the dims bridge, don't make every consumer write it

Beyond the two fixes above (drop UNSERIALIZABLE[...]; function-as-dim), the deeper ergonomic point from dogfooding: a real suite has to hand-write an autouse fixture to get clean dims out at all. In linopy that fixture is ~8 lines:

@pytest.fixture(autouse=True)
def _benchmem_dims(request, benchmark):
    cs = getattr(request.node, "callspec", None)
    info = getattr(benchmark, "extra_info", None)
    if cs is None or info is None:
        return
    # mirror clean scalar dims (the spec param is an unserialisable object;
    # the phase lives in the function name)
    info.update(spec=cs.params["spec"].name,
                phase=request.node.function.__name__.removeprefix("test_"),
                axis=cs.params["spec"].axis)

That bridge belongs in the tool, not in every consumer's conftest. Options:

• a shipped benchmem_dims autouse fixture (opt-in via a flag/marker) that mirrors a caller-supplied extra_info mapping;
• a @pytest.mark.benchmem(dims=lambda **params: {...}) arg that records dims from the parametrize values;
• or just: function-as-dim (above) + UNSERIALIZABLE handling, which together cover the common case with zero producer code.

The combination of those two built-ins would make "point pytest-benchmem at any pytest-benchmark suite and plot it" actually true out of the box — which is the promise. Filed the surrounding ergonomics/versatility findings as #46–#49.

🤖 From the linopy integration.