docs: normative optimizer target architecture + migration plan

[johanzander]

Summary

• Adds docs/agents/optimizer-architecture.md — normative: every PR touching the optimizer core must uphold principles P1–P7 or amend the document in the same PR. Codifies the 2026-08-08 architecture assessment (ten correction layers, three defect classes), the PWL-everywhere benchmark results (Grid DP discretization leaves 0.01-0.36 SEK/day unrealized on 20/22 benchmark fixtures #512, PWL window solver mis-ranks plans: picks 0.584 SEK worse schedule than grid DP under its own objective #513), and ridax67's Question: How is IDLE used? #466 principle ("IDLE must express that the optimizer WANTS to hold, never that it EXPECTS balance") as the intent-semantics rule.
• Adds the phased migration plan: Phase 0 drain WIP (fix: charge-early tie-break under the export-curtailment price floor #510, fix: stop the DP proposing discharges the inverter cannot execute (#497) #511, fix: reported cost no longer charges the honest price for curtailed periods #507, fix: distinguish planned PV curtailment from Solar Export in the UI #508, fold-scoping) → Phase 1 single selector (bit-parity, mechanical) → Phase 2 lexicographic tie-policy table (closes the Question: How is IDLE used? #466 sunrise-crossover residual, subsumes the Question: How is IDLE used? #466/fix: charge-early tie-break under the export-curtailment price floor #510 tie-breaks, unblocks Hysteresis: don't rewrite inverter schedule when the new plan is a near-tie with the applied one #485) → Phase 3 single flow record (closes LOAD_SUPPORT overshoot in the 0.01-0.1 kWh band produces inconsistent flows and phantom export revenue #497) → Phase 4 executable-command candidates (closes bug: DP discharge-candidate search bakes in inverter-specific percent-of-max resolution instead of exact kWh, causing frequent unnecessary Growatt MIN mode flips #320/bug: low-rate BATTERY_EXPORT writes grid_first at a forced sub-load discharge rate, causing avoidable grid import on any load spike #352, supersedes fix: gate grid_first on export materiality instead of intent classification #354). Parallel tracks: consumption statistics feedback loop: inverter self-consumption unmeasured while battery sleeps, night estimate degrades progressively #487, terminal-value extraction, PWL window solver mis-ranks plans: picks 0.584 SEK worse schedule than grid DP under its own objective #513, Grid DP discretization leaves 0.01-0.36 SEK/day unrealized on 20/22 benchmark fixtures #512 (sweep-gated).
• Indexes the architecture doc in CLAUDE.md's agent documentation table.

Notes

• Docs only — no code changes.
• Phases 1–2 are specified to implementation level; Phases 3–4 to acceptance level and start with their own detailed plan/design doc per repo convention.

🤖 Generated with Claude Code

[johanzander]

Thorough review of the proposed architecture (claims fact-checked against main, tie-policy economics stress-tested via bess-analyst).

Verdict: the architecture is sound and worth adopting — the three defect classes are real, the issue lineage checks out against the code, and the compliance checklist is enforceable rather than aspirational. But two findings should be fixed in this PR since the document is normative from the moment it lands:

1. Row 2 silently narrows the #466 guard from "non-idle winner" to "discharge winner" — an undeclared third semantic change (inline comment).
2. P6 is aspirational but reads as descriptive — today's splice_schedule relies on PWL exactness unconditionally, which is exactly what P6 forbids, and nothing polices it (inline comment).

Plus three smaller fixes (P1 wording vs the Phase 1 vectorized escape hatch, two cited artifacts that don't exist on main yet, and a wrong "branch for branch" citation) and a few docstring-level notes — all inline.

What checked out clean: all named symbols exist where claimed; the four tie-break call sites and the epsilon <= 0 early-return; the terminal-value method split (battery_system_manager.py:1920, #422 day-scoping in the caller at :2124); the PLAN_EXECUTION_GAP_SEK +0.73/+0.56/+0.45 pins; the #350 fold still in EnergyData._calculate_detailed_flows (models.py:157); no stray hardcoded epsilons contradicting P5; row 1 subsumes (not contradicts) the #510 tie-break — _prefer_curtailed_charge_absorb contains the identical grid-import guard internally; and P7's structural-over-stochastic stance is economically bounded (worst observed forfeiture +0.032 SEK vs the 0.05 SEK/fixture budget).

[johanzander]

Should-fix: row 2 silently narrows the #466 guard from "non-idle winner" to "discharge winner".

Today both _prefer_* functions bail on any non-idle argmax winner — including a charge winner (_prefer_load_covering_discharge bails at dp_battery_algorithm.py:1430 on abs(best_power) > POWER_TOLERANCE_KW; _prefer_curtailed_charge_absorb at :1494). Row 2 as written protects only discharge winners, so under the new table a within-epsilon charge winner during positive net load falls through to row 3 and can be flipped to a load-covering discharge — a charge→discharge action flip that passes the row-1 guard (discharge imports less grid) and is not one of the plan's "two deliberate semantic changes".

Either extend row 2 to "non-idle winner" (true parity with today's ordering), or declare this as the third deliberate change with its own acceptance test.

[johanzander]

Should-fix: P6 is aspirational but reads as descriptive.

splice_schedule (dp_battery_algorithm.py:2290) splices unconditionally — there is no compare-against-grid-segment, accept-only-if-better gate. The only guard is the exactness certification raise (PWLWindowUnderRefinedError), and #513 is the proof that certification doesn't hold. So the existing reliance on PWL exactness is exactly what P6 forbids, and nothing polices it — "treated as heuristic" is currently a doc claim, not a property of the code.

Cheap fix that makes P6 true: accept a spliced window only if its replayed cost is ≤ the grid segment's (a _replay_accounting_pass comparison). Suggest adding that as a P6 compliance item or a small Phase 1/2 rider.

[johanzander]

Should-fix: P1's "exactly one implementation" collides with Phase 1's own escape hatch.

The backward induction (_run_dynamic_programming:949) is fully numpy-vectorized — on the order of tens of thousands of candidates per period — and the migration plan sensibly allows keeping the vectorized fast path if bit-parity holds. But that outcome (vectorized path + select_action sharing enumeration functions, pinned by a parity test) violates P1 as literally worded, forcing a doc amendment during Phase 1 itself.

Suggest rewording P1 to what's actually meant: one candidate-enumeration + one tie policy, with any vectorized evaluator parity-pinned against the selector.

[johanzander]

Should-fix: wrong citation. The "branch for branch" mirror comment is at dp_battery_algorithm.py:447, on the _compute_reward/_compute_reward_grid pair — which this doc's own "What this document does NOT change" section explicitly blesses. The duplication P1 actually targets (_best_action_at_continuous_state 213 lines vs the PWL clone ~221 lines) has "mirroring" comments (pwl_window_dp.py:164,442,458,912) but not that phrase — a reviewer grepping the quote lands on the protected pair.

Also worth tightening: the _prefer_*/_discharge_candidates/_charge_candidate/_tie_margin helpers are not hand-cloned in PWL — pwl_window_dp.py:21-28 imports and calls them directly. Only the two enumerate/select functions are true clones. That makes P1's claim more precise, not weaker.

[johanzander]

Should-fix: cited artifact doesn't exist on main. test_flow_coherence.py (and its 182/1875 pin) is not on main — it lives in an unmerged worktree. Since it's Phase 3's acceptance anchor, either add landing it to Phase 0's drain list or mark it "pending WIP" here. Same for scripts/bench_pwl_everywhere.py (the #512 gate, referenced at line ~345) — also not on main yet.

[johanzander]

Docstring note (no change needed): row 1 is consistent with today — _prefer_curtailed_charge_absorb contains the identical grid-import guard internally (dp_battery_algorithm.py:1504), so row 1 subsumes rather than contradicts the #510 rule; the #510 winner holds SOE with ~zero import, and the only candidate blocked is the grid top-up, blocked today too.

The one foreclosure worth recording in row 1's rationale: in negative buy-price windows (exactly when curtailment floors bite), a within-epsilon grid top-up is arguably the safer pick against a solar shortfall — and row 1 permanently bans it. Codifies status quo, so fine, but a future amendment will want this context.

[johanzander]

Note: the acceptance hinges on whether the 06:00 crossover is a bit-exact tie. Dropping the epsilon <= 0 early-return is economically sound — cycle cost is already inside _compute_reward, so on an exact tie tracking is weakly dominant under forecast error (IDLE imports the whole consumption miss at buy price; tracking absorbs it at ~breakeven). But at epsilon 0, "within epsilon" catches only bit-exact ties. If the ridax67 crossover period actually has a tiny nonzero gap, the test can only pass by introducing an epsilon floor — and a floor would trade real model value and needs its own stated bound in the row docstring. Worth checking the bundle before writing the RED test.

[johanzander]

Note: state the golden-fixture lifecycle across phases. Phase 2 deliberately changes behavior, which breaks Phase 1's bit-parity goldens by design. Presumably the answer is "regenerate goldens as part of Phase 2's measured-delta step" — say so explicitly, so nobody "fixes" the parity test by deleting it.

[johanzander]

Already done — #510 merged 2026-08-08 (742d5906, current main HEAD). Can be checked off.

[johanzander]

All five should-fixes and the notes are in (single commit):

1. Row 2 → non-idle parity. Extended to guard any non-idle winner (dp:1430/dp:1494 bail conditions reproduced exactly); the plan's deliberate semantic changes stay at two. Chose parity over declaring a third change — a charge→discharge flip under positive net load has no field evidence motivating it.
2. P6 made honest + enforceable. Now explicitly aspirational with the existing splice path grandfathered, and a Phase 2 rider added: splice accepted only if its replayed cost ≤ the grid segment, worse windows discarded with a WARNING naming both costs (visible decision, not a silent fallback), RED test on the PWL window solver mis-ranks plans: picks 0.584 SEK worse schedule than grid DP under its own objective #513 shape.
3. P1 reworded to one enumeration + one tie policy; the vectorized backward evaluator is allowed iff it consumes the same enumeration functions and is parity-pinned. No doc amendment needed mid-Phase-1.
4. Citations fixed. "branch for branch" removed from the P1 rationale (it's the protected reward pair at dp:447); the actual targets are the two enumerate/select clones with mirror comments at pwl:164/442/458/912; helpers noted as imported, not cloned.
5. Missing artifacts → Phase 0 drain items: land test_flow_coherence.py (pre-Phase-3 gate) and scripts/bench_pwl_everywhere.py (from bench/pwl-everywhere). fix: charge-early tie-break under the export-curtailment price floor #510 checked off (742d590).

Notes folded in: row 1 records the negative-buy-price foreclosure for future amendments; Phase 2's crossover acceptance now starts by measuring the bundle's actual margin (an epsilon floor, if needed, requires a stated forfeiture bound inside the 0.05 budget); golden lifecycle stated — Phase 2+ regenerates goldens in the measured-delta step, the parity test is never deleted.