fix: gate grid_first on export materiality instead of intent classification

[pookey]

Summary

• Decouples hardware-mode selection from strategic-intent classification (bug: low-rate BATTERY_EXPORT writes grid_first at a forced sub-load discharge rate, causing avoidable grid import on any load spike #352, direction 1): a BATTERY_EXPORT period keeps its label for accounting but only commits the inverter to grid_first when the planned export is material.
• Materiality = planned battery_to_grid > 0.1 kWh (the flow-resolution floor from fix: cross-sensor noise misattributed to battery flows corrupts observed intent (BATTERY_EXPORT/GRID_CHARGING) #350/fix: sub-resolution battery_to_grid residual corrupts observed_intent #351) AND (export > planned battery_to_home OR export > the load-following headroom the commitment forfeits, max_discharge × 15 min − planned discharge).
• Demoted periods write load_first + discharge_rate=100 (LOAD_SUPPORT semantics): spike-immune, forgoing the small export. Applies only where discharge_rate_is_load_following (Vpp battery dump b17 #324) — VPP-style platforms untouched. Consumes the per-period planned flows added as preparatory plumbing in bug: DP discharge-candidate search bakes in inverter-specific percent-of-max resolution instead of exact kWh, causing frequent unnecessary Growatt MIN mode flips #320 (previously no consumer).

Root cause

From the issue (confirmed by @johanzander against source): classify_strategic_intent's 0.01 kWh threshold (#253) routes periods planning a few Wh of export into INTENT_TO_MODE's BATTERY_EXPORT → grid_first, with _map_intent_to_rates scaling the discharge rate from the forecast action. grid_first has no load-following, so any load spike above the written rate imports at buy price to protect pennies of export — measured live as 17/184 periods over 5 days with zero export delivered and 3.2 kWh imported with battery energy available (13–23 % of realized savings given back). #253 was a legitimate accounting fix; the gap is that classification fidelity and hardware-mode materiality were one decision.

One finding made during implementation, worth knowing: #351's sub-0.1 kWh fold also runs on DP-planned flows (EnergyData._calculate_detailed_flows is shared between planned and observed data), so since 75a9f0e the sub-0.1-with-home-deficit case already classifies LOAD_SUPPORT before this PR. This PR still matters because (a) the 0.1–0.5 kWh home-dominant band remained exposed, (b) the fold's counter-noise rationale doesn't apply to predictions — if it is ever scoped to actual data (relevant to the separate "pointless flips" investigation), #352 would silently reopen without an explicit gate, and (c) a sub-0.1 export with no home deficit still classified BATTERY_EXPORT.

Fix

InverterController gains _export_demoted() / mode_for_period(); compute_rates_for_period, get_period_settings, get_detailed_period_groups, the Growatt MIN TOU grouping, and both solax-modbus TOU write paths route through them. backend/api.py's two period-group calls pass the matching per-period PeriodData so the dashboard shows the real hardware mode/rate for demoted periods (intent label unchanged). The simulator's derive_control_command intentionally keeps the plain mapping — like the #187/#318 intra-period discharge gate, demotion is a sub-period hardware-robustness behavior invisible to the 15-minute plan/sim (noted in both design docs).

Why the two-sided materiality test: the harm of grid_first is bounded by the load-following headroom given up. A near-full-rate spike export (e.g. 1.24 of 1.25 kWh max into a 0.95 kWh/period house) has ~no headroom left to protect, so demoting it would only forgo revenue — observed live in mock-HA E2E with a 6 SEK/kWh sell spike, where a home-dominance-only rule would have permanently demoted every export on any high-consumption system. Conversely, low-rate smeared exports (0.02 kWh export vs ~1 kWh headroom) are exactly the measured failure mode and demote.

Docs: updated docs/agents/bess-knowledge.md (hardware mapping + new gate section) and docs/SOFTWARE_DESIGN.md (intent→mode table + gate paragraph; also corrected the stale BATTERY_EXPORT classification threshold claim, 0.1 → 0.01 per #253, and noted the #350 fold there).

Test plan

• 10 new behavioral tests in core/bess/tests/unit/test_export_mode_materiality.py (demotion, materiality via dominance and via headroom, legacy no-period-data fallback, non-load-following platform exclusion, intent-label/hardware decoupling, TOU segment behavior)
• ./scripts/quality-check.sh passes locally; full slow suite passes (380 passed)
• Observed live via the mock-HA E2E stack (docker-compose.ci.yml, synthetic Growatt MIN + Nordpool): a home-dominant partial-rate export period (planned b2g 0.288 vs b2h 0.95 at low rate) rendered BATTERY_EXPORT/load_first/dis 100 with no grid_first TOU segment; an export-dominant period (b2g 0.588 vs b2h 0.15) kept grid_first/dis 59 with a real TOU segment; full-rate spike exports (b2g 0.29, rate 99–100 %) kept grid_first with a grid_first TOU segment 10:30–11:29 under the final predicate.

Closes #352

🤖 Generated with Claude Code

[johanzander]

Hi @pookey , how is the testing going? do you feel comfortable merging this so others can test it as part of next beta?

[johanzander]

Hi @pookey — an update on this one, and an apology for how long it has sat.

Your diagnosis holds and the materiality test is being kept. What changes is the layer it lives at.

Since this PR opened, the optimizer has been through three refactor phases (#521, #525, #534) that consolidated the planning side: one action selector, one preference table for tie resolution, one flow record per candidate. Phase 4 is next, and it changes what a candidate is — a candidate stops being an abstract kW figure and becomes an executable inverter command (mode plus rate on the platform's real lattice), scored by simulating that command against the forecast.

That is what makes this PR's layer the wrong one now. Gating grid_first on export materiality in the controller is a post-hoc demotion: the planner still proposes an action the hardware won't deliver, and the controller catches it afterwards. Under Phase 4 the unexecutable action is never a candidate in the first place, so a controller-side gate would be a second place where the same rule lives — and if both exist they can disagree, which is precisely the mirrored-implementation class the refactor exists to remove.

So the plan for #352 now records your two-sided test — dominance OR forfeited headroom — as candidate scoring inside Phase 4 rather than as a demotion after the fact. That is your analysis, in the place where it can also prevent the plan rather than only suppress the command.

Two things worth saying plainly:

• This PR will not be rebased, and that is not a judgement on the work. It touches inverter_controller.py, solax_modbus_growatt_controller.py and growatt_min_controller.py, all three of which fix: apply the intra-period discharge gate to LOAD_SUPPORT on TOU (#520) #524, shadow_price == 0.0 is ambiguous between "worth nothing" and "never computed", and the discharge gate opens unconditionally there #526 and feat: simulate Growatt VPP and pin it against v10.0.2 (#539) #541 have since rewritten. Porting it would mean carrying a design onto a layer we are about to move it off.
• bug: low-rate BATTERY_EXPORT writes grid_first at a forced sub-load discharge rate, causing avoidable grid import on any load spike #352 is still open and still real. Measured on the current corpus: 22 sub-load grid_first periods, 16 of them in the 0.1–0.5 kWh band that fix: stop the DP proposing discharges the inverter cannot execute (#497) #511's executability filter does not reach. Frank hit it again on 29 Jul during an EV charging session. It has not been quietly closed by the refactor.

I will close this PR when Phase 4 covers the band, and link the commit that does, so the trail from your work to the fix stays visible. If you would rather pick up the Phase 4 candidate-scoring piece yourself, say so and I will hold it for you — you have the most context on the materiality question of anyone.

Thanks for the patience, and for the original analysis.

[bess-agent]

Update from the #352 work, since this PR's rule was measured directly.

Your two-sided test was independently re-derived and re-confirmed. A session working #352 arrived at the same predicate from the other direction — "is the period mostly about exporting, or covering the house with a spill?" — and then measured what happens without your second clause. Bare dominance costs +15.24 SEK across the fixture corpus; with the "nothing left to protect" exemption it costs +3.67 SEK. The entire difference is 8 full-rate export periods carrying 31.5 SEK that have no load-following headroom to defend. That is exactly the case you learned from live E2E and wrote the second clause for, and it holds up as load-bearing rather than defensive.

What changed is the diagnosis underneath it. At the reproduction period the optimizer's action space offers 2.70 kW or 3.30 kW and nothing between, while the house needs 2.80 kW — exact cover is off the percent lattice, the near-cover steps are removed by #497's band, and the existing exact-cover candidate only fires below the smallest lattice step. So the DP chooses between under-covering (importing at the buy price) and over-covering (exporting at the sell price). Over-covering is the cheaper error, and it is what turns the period into a BATTERY_EXPORT and puts the inverter into grid_first.

The tiny export you identified as not worth defending was never being defended. It is a by-product of an action the planner could not express.

Adding the missing candidate measures −3.12 SEK (i.e. cheaper than today), takes committing exports from 52 to 19, and fixes the reproduction — against +3.67 SEK for the materiality rule. So the fix lands as an addition rather than a refusal, and this PR's rule stays on the shelf as optional cleanup for the residual.

One measurement worth passing on, because it bears on your original numbers: every materiality figure — yours and the re-derivation's — was taken on that impoverished action space, where each non-exporting alternative carries a forced sub-lattice import at the buy price. That is the very buy≫sell asymmetry your report is about, and it systematically overstates how much the exports are worth. With exact cover present, the reproduction period's export margin collapses from 0.0388 SEK to 0.0067 — inside the optimizer's own tie threshold. Most of these periods are near-ties, not decisions.

Full detail on #352. This PR stays open and unmerged; it will be closed by the commit that lands the fix, with the trail intact.