`merge` of ragged expressions is the peak allocation in PyPSA model builds

[FBumann]

Note: This issue was drafted by an AI agent (Claude Code) during a profiling/investigation session with @FBumann. The measurements (memray C-level profiling, benchmarks, the 284× / 132 MB figures) and the git-history archaeology are real and reproducible, but the framing and proposed directions are a starting point for discussion — please sanity-check before acting on them.

merge (expressions.py, merge(...)) concatenates several LinearExpressions along a new/shared dimension by aligning their _term axes. Because the _term axis is a dense rectangle, alignment pads every block to the global maximum term count before concatenation, and the concatenation itself allocates the full padded result.

Evidence

Memray on a full SciGRID-DE create_model() (585 buses, 24 snapshots, 59,640 vars, 142,968 cons) — peak C-level memory 351 MB, and the single largest live allocation at the high-water mark:

132.6 MB  concatenate (xarray duck_array_ops)
          <- concat (xarray)
          <- merge (linopy/expressions.py)
          <- define_kirchhoff_voltage_constraints (pypsa/.../constraints.py)

So merge is the peak operation of the whole build — i.e. the allocation that actually sets the OOM ceiling for large networks, more than groupby (#745) on realistic group-size distributions.

Two contributing factors:

1. The inputs are already densified (see the KVL @ issue) — merge inherits that bloat.
2. merge pads each block to the global-max _term before concat; blocks with few terms waste the rest.

Why it matters

PyPSA works around exactly this by splitting the nodal-balance constraint into two (strongly- vs weakly-meshed buses) so each merge/groupby operates on a bucket of similar term counts — see #745 for that evidence. A merge that handled ragged _term (or operated in long format) would remove the need for that manual bucketing.

Possible directions

• Concatenate in long/COO form and only rectangularize once, padding to the per-result max rather than per-input-then-global.
• Or the long-format/sparse _term kernel (Umbrella: long-format / sparse _term kernel (dense-_term memory cluster) #756).

Sibling of #745 and the KVL @ issue — all three are the dense-_term representation surfacing at different ops.

[FBumann]

Note: Same session (AI agent + @FBumann); env as in the #745 comment (linopy 0.7.0 / pypsa 1.2.2.post1.dev2+g30e4ed0ef / xarray 2026.4.0, build-only).

Clarification on which merges the "pad to global-max _term" cost applies to — two distinct merge shapes in PyPSA with different fix locations:

• Concat along _term (nodal balance: merge(exprs, join="outer"), constraints.py:1139): measured output _term = sum of inputs (8000+119+1 = 8120), i.e. no extra padding. Merge faithfully materializes inputs that groupby().sum() already padded (groupby-sum pads every group to the largest group size #745). Fix is upstream.
• Concat along another dim (KVL: merge(lhs, dim="snapshot"), constraints.py:1246): blocks must align their _term axes → each pads to the global max. This is where this issue's padding bites, on inputs already densified by the sparse @ dot (@/dot against a sparse matrix densifies the result to full _term #748).

So merge is the common materialization site, but "merge pads to global-max" is specific to the cross-dim concat (KVL); the nodal-balance merge just concatenates _term. A ragged/long-format merge helps the former — the latter is fixed upstream in #745/#748.