feat: mode=release advisory tier — gate on correctness, not the perf-flake tail

[Jammy2211]

Overview

workspace-validation.yml mode=release currently conflates correctness and perf-flakiness into a single RED. After the 2026-07-13 tail burn-down (PyAutoHeart#72) fixed every real bug, the integrate stage still can't reach a clean verdict because a population of genuinely-slow real-search scripts (real Nautilus fits + finite-difference JAX gradients) flakes around the per-script timeout cap — the failing set shifts every run (shared_preloads literally flip-flopped between re-val #3 and #4). Raising the cap and adding no_run SLOW-skips one script at a time never converges and erodes coverage.

This introduces an advisory tier: a timeout on a declared-slow script is downgraded to YELLOW (loud, tracked), while a timeout on any undeclared script stays RED. This is the path that lets mode=release reach its normal shippable GREEN/YELLOW once the real bugs are fixed, instead of being held hostage by the perf tail.

Plan

• Add a companion config/build/advisory.yaml per workspace — same flat-list + inline-comment format as no_run.yaml, but the listed scripts still run and report; only their timeouts are advisory. Growing this list never erodes coverage, so it converges where SLOW-skips whack-a-mole. Absent file → today's behaviour exactly.
• Build (runner mechanism): a new TIMEOUT_ADVISORY result status; the runner tags a timeout as advisory when the script matches advisory.yaml; has_failures / aggregate ready exclude advisory timeouts; a non-advisory timeout still fails the run.
• Heart (policy): the stage report and validation_report carry an advisory_timeouts count; readiness maps advisory_timeouts > 0 to a YELLOW reason (not RED), keeping the release GREEN→YELLOW rather than RED-blocked.
• Keep it loud: advisory timeouts surface in the aggregate report, the CI step summary, and Heart's YELLOW reasons — a tracked backlog, with durable speedups owned by the Profiling Agent.
• Ship the mechanism in PyAutoHeart + PyAutoBuild first (self-contained, unit-tested, graceful fallback); seed advisory.yaml in the *_test workspaces from the re-val PyAutoPulse v1.3 — stop --all (recover orphan daemon) #3/PyAutoPulse v1.4 — classify generated json/png as dirty-noise #4 flip-flop population as a ship_workspace follow-up.

Detailed implementation plan

Affected Repositories

• PyAutoHeart (primary) — policy: heart/validate.py, heart/readiness.py, .github/workflows/workspace-validation.yml, docs/release_validation.md
• PyAutoBuild — runner mechanism: autobuild/result_collector.py, autobuild/run_python.py, autobuild/build_util.py, autobuild/aggregate_results.py

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoHeart	main	clean
./PyAutoBuild	main	clean

No worktree conflict (active claims are all on the science libraries). Suggested branch: feature/release-advisory-tier-slow-scripts

The failure path being severed

build_util timeout → Status.TIMEOUT → RunReport.has_failures → aggregate_results.ready=false → workflow "Fail if not ready" + emit_release_report status=fail → validate.py stage integrate=fail → release_ready=false → readiness.py hit("validation_failed") → RED.

Implementation Steps

PyAutoBuild (runner mechanism):

1. autobuild/result_collector.py — add Status.TIMEOUT_ADVISORY = "timeout_advisory"; RunReport.has_failures excludes it; summary/to_dict carry it.
2. autobuild/run_python.py — load config/build/advisory.yaml (workspace override → empty default, mirroring the no_run.yaml resolution), parse its reasons, thread advisory_list/advisory_reasons into execute_scripts_in_folder.
3. autobuild/build_util.py — in execute_script (and execute_notebook for symmetry), on subprocess.TimeoutExpired, if the file matches advisory_list (reuse should_skip path semantics) record TIMEOUT_ADVISORY with the advisory reason instead of TIMEOUT.
4. autobuild/aggregate_results.py — timeout_advisory is not in the failure set (so ready is unaffected); add an advisory_timeouts list + a loud "## Advisory-Tier Timeouts (slow real-search; not release-blocking)" markdown section. A non-advisory timeout still flips ready=false.

PyAutoHeart (policy):
5. heart/validate.py — to_stage_report carries advisory_timeouts (count + entries) from the aggregate into the stage report (NOT folded into failures); _Accumulator + report schema sum it into the top-level validation_report; status still derives from aggregate.ready. Bump the schema docstring.
6. heart/readiness.py — in the release-validation gate, when the report has advisory_timeouts > 0, append a YELLOW reason ("N advisory-tier script timeout(s) in release validation (slow real-search; not release-blocking)") + hit("validation_advisory_timeout") with a _SCORES entry. This turns a correctness-clean run GREEN→YELLOW instead of RED.
7. .github/workflows/workspace-validation.yml — comment the tier; echo the advisory-timeout count into $GITHUB_STEP_SUMMARY. No gate-logic change (already keys off ready).
8. docs/release_validation.md — document the advisory tier and the # ADVISORY <date> - reason marker convention.

Tests

• PyAutoBuild/tests/test_result_collector.py (+ new aggregate test): advisory-matched timeout → TIMEOUT_ADVISORY, has_failures false, ready true, advisory_timeouts populated; non-advisory timeout → ready false.
• PyAutoHeart/tests/test_validate.py: to_stage_report carries the advisory count and stage status stays pass; readiness maps advisory_timeouts > 0 → YELLOW (not RED).

Key Files

• PyAutoBuild/autobuild/result_collector.py — Status, RunReport.has_failures
• PyAutoBuild/autobuild/build_util.py — execute_script/execute_notebook timeout branch, should_skip
• PyAutoBuild/autobuild/run_python.py — advisory.yaml load + thread-through
• PyAutoBuild/autobuild/aggregate_results.py — aggregate() failure set + report section
• PyAutoHeart/heart/validate.py — to_stage_report, _Accumulator, schema
• PyAutoHeart/heart/readiness.py — release-validation gate YELLOW reason
• PyAutoHeart/.github/workflows/workspace-validation.yml — step summary
• config/build/advisory.yaml (new, per *_test workspace) — the declared-slow registry

Design notes

• advisory.yaml vs a duration threshold: an explicit list is reviewable and stable; a duration threshold would re-capture exactly the borderline near-cap scripts that flip-flop. Chosen: explicit list.
• Advisory downgrades timeouts only — a genuine FAILED (exception) on an advisory script stays RED, preserving the correctness signal.
• Cross-ref: PyAutoHeart#72 (release tail epic), draft/feature/profiling/profiling_agent_jax_compile_time_scope.md (durable speedups).

Original Prompt

Click to expand starting prompt

mode=release should tier slow real-search scripts as advisory, not RED-block on the perf tail

Type: feature
Target: PyAutoHeart
Repos:

• PyAutoHeart
• PyAutoBuild
  Difficulty: medium
  Autonomy: supervised
  Priority: high
  Status: formalised

The 2026-07-13 release-validation burn-down (PyAutoHeart#72) fixed every real
correctness bug in the tail (A–H + the jax_grad config gap). But the
workspace-validation.yml mode=release integrate stage still cannot reach a clean
verdict, because it runs the FULL script set — including a population of genuinely
slow real-search scripts (PYAUTO_TEST_MODE=0/1 real Nautilus fits +
finite-difference JAX gradients) that flake around the per-script timeout cap. The
failure set shifts run to run:

• re-val PyAutoPulse v1.3 — stop --all (recover orphan daemon) #3 → shapelets, jax_grad×2, multi/delaunay_mge
• re-val PyAutoPulse v1.4 — classify generated json/png as dirty-noise #4 (after fixing those) → cpu_fast_modeling, interferometer datacube/delaunay,
  interferometer mge/slam, multi/shared_preloads — all of which passed in PyAutoPulse v1.3 — stop --all (recover orphan daemon) #3
  (shared_preloads literally flip-flopped).

Raising BUILD_SCRIPT_TIMEOUT (300→1800) and adding no_run SLOW-skips one script at
a time does not converge — each run surfaces a different handful of borderline
scripts, and de-gating them individually is whack-a-mole that also erodes coverage.

The structural problem: mode=release gates a RELEASE on the full-set run, but a
big fraction of that set is genuinely-slow real work whose pass/fail near a per-script
cap is nondeterministic (CI runner load). Correctness and perf-flakiness are conflated
into one RED. (This is the "mode=release full-tail RED policy" question first raised on
#72 comment-4966797130.)

Proposal — tier the run so integrate gates on CORRECTNESS, not the perf tail:

• Introduce a slow/advisory tier for known-slow real-search scripts (a
  slow: true marker in config/build/no_run.yaml or a companion slow.yaml, or a
  duration threshold): these still RUN and are reported, but a timeout on an
  advisory-tier script is advisory (yellow), not an integrate-RED.
• Alternatively/additionally, run the slow tier as a separate off-critical-path job
  with a large budget (e.g. 3600s+), so a slow-tier flake never blocks the release
  verdict while still surfacing regressions.
• A timeout on a non-advisory script is still RED (a script that should be fast
  timing out IS a signal). Only the declared-slow population is de-gated.
• Keep it loud: advisory failures surface in the report + Heart's YELLOW reasons
  (same spirit as the existing SLOW/NEEDS_FIX banners), so they're a tracked backlog,
  not silently hidden — the durable speedups are the Profiling Agent's job
  (draft/feature/profiling/profiling_agent_jax_compile_time_scope.md).

Outcome: after the real bugs are fixed, mode=release reaches GREEN/YELLOW on
correctness and the release-decision gate stops being held hostage by real-search
perf-flakes. Design in PyAutoHeart (workspace-validation.yml + heart/validate.py
tiering) with the runner support in PyAutoBuild (run_python.py/build_util.py).
Cross-ref project_release_2026_07_13_blocked_3bugs and PyAutoHeart#72.

[Jammy2211]

Library PRs open (mechanism) — corrective-red, PR-open only

The advisory-tier mechanism is implemented, verified, and shipped to pending-release PRs:

• PyAutoHeart#75 — policy: advisory_timeouts carried through validate.py; readiness.py raises validation_advisory_timeout (YELLOW, not RED); workspace-validation.yml + docs/release_validation.md updated.
• PyAutoBuild#153 — runner: Status.TIMEOUT_ADVISORY, config/build/advisory.yaml load, aggregate_results keeps advisory timeouts out of ready + surfaces them.

Gate

• Tests: Heart 268 passed, Build 118 passed (1 pre-existing unrelated deselect: test_pre_build_skill, an admin_jammy/PyAutoBrain manifest drift that fails on clean main).
• Smoke: n/a (Heart+Build tooling; no workspace scripts changed).
• Review: self-CLEAN (both diffs scope-clean).
• End-to-end (drove run_python.py on a temp workspace): an undeclared slow-script timeout → run exits 1 (still RED); an advisory-declared timeout → exits 0 (de-gated, timeout_advisory); aggregate ready: True with the "Advisory-Tier Timeouts" section surfaced.
• Heart: RED — shipped under the human-authorized corrective-PR exception for the exact reason release validation FAILED (stage integrate) (the structural RED this change fixes). The other live RED reason (PyAutoArray on a feature branch) is unrelated sibling work. No merge, no release — merge stays a human act.

Next

• Workspace advisory.yaml seeds — the re-val PyAutoPulse v1.3 — stop --all (recover orphan daemon) #3/PyAutoPulse v1.4 — classify generated json/png as dirty-noise #4 flip-flop population (shapelets, jax_grad×2, multi/delaunay_mge, interferometer datacube/delaunay/mge/slam, multi/shared_preloads, cpu_fast_modeling) — in progress this session as the ship_workspace follow-up. The mechanism is a graceful no-op until Build#153 merges and the workspaces are seeded.
• After merge: build fresh wheels, rerun mode=release integration on main, obtain a new Heart verdict — the advisory tier should let a correctness-clean run reach GREEN/YELLOW.

[Jammy2211]

Workspace advisory.yaml seeds open — full set at PR-open

The workspace half is now seeded (behind the library-first gate — inert until PyAutoBuild#153's run_python.py loader merges):

• autolens_workspace_test#168 — 9 entries (multi/shared_preloads the literal flip-flopper; datacube delaunay/shared_preloads; interferometer mge/mge_group/delaunay/delaunay_mge/rectangular_mge; jax_grad/interferometer).
• autogalaxy_workspace_test#72 — 8 entries (interferometer jax_likelihood MGE/Delaunay family; imaging delaunay_mge; jax_grad interferometer/multi MGE).
• autolens_workspace#276 — 2 entries (group cpu_fast_modeling; group shapelets modeling).

All 19 patterns are .py-anchored and verified to match exactly one real script (no typos, no delaunay↔delaunay_mge over-match). Scripts already no_run-skipped (imaging jax_grad/delaunay_mge/mge_group, multi/delaunay_mge, database slam_*, agw#131 shapelets) are not repeated — no_run skips them before they can flake.

Note: autolens_workspace_test / autolens_workspace carried worktree claims from in-flight sibling tasks (aggregator PR#275 already merged; the tail-burndown epic closing out); the human waived the conflict, so those seeds proceeded.

Full PR set (all pending-release, PR-open — merge stays human)

Repo	PR
PyAutoHeart (policy)	#75
PyAutoBuild (runner)	#153
autolens_workspace_test (seed)	#168
autogalaxy_workspace_test (seed)	#72
autolens_workspace (seed)	#276

Merge order (library-first)

1. PyAutoBuild#153 (the loader) → 2. PyAutoHeart#75 (the policy) → 3. the 3 workspace seeds → 4. build fresh wheels, rerun mode=release on main for a new Heart verdict.

Follow-ups

• Consider migrating recently-SLOW-no_run'd real-search entries → advisory.yaml to restore coverage (a deliberate decision tied to the Profiling Agent's durable speedups).
• Pre-existing, unrelated: PyAutoBuild/tests/test_pre_build_skill.py manifest drift (admin_jammy vs PyAutoBrain) — fails on clean main; separate hygiene fix.

[Jammy2211]

Reverted the advisory tier → going with no_run + hygiene

On reflection (maintainer call), the advisory-tier mechanism was more machinery than warranted. Its core premise — that no_run SLOW-skips erode coverage forever and never converge — is substantially handled by the periodic hygiene sweep of no_run; and for scripts that reliably exceed the cap, a skip and an advisory-timeout are equivalent (a timed-out script asserts nothing).

Fully reverted: all 5 advisory PRs (Heart#75, Build#153, alwt#168, agwt#72, alw#276) closed, their branches deleted, the mechanism removed. Nothing of the advisory tier remains.

Replacement — the leaner no_run approach (2 PRs):

• autolens_workspace_test#169 — 9 real-search JAX likelihood/gradient flakers → SLOW.
• autogalaxy_workspace_test#73 — 8 → SLOW.

All entries .py-anchored and verified 1:1 against real scripts. Excluded deliberately: cpu_fast_modeling (its release-tail failure is a singular FAIL being fixed in PyAutoArray#388, not a timeout) and autolens group/shapelets (uncertain; autogalaxy shapelets already handled by agw#131).

Accepted trade-off: the flaky set shifts run to run, so a future mode=release may surface a different handful — those get added via the same hygiene pass rather than an automated tier. no_run PRs are pending-release, PR-open only (merge stays human). Heart stays RED until the unrelated PyAutoArray branch reason also clears.

Closing this issue's advisory-tier framing; the no_run PRs carry the actual unblock.

[Jammy2211]

Resolved via no_run — merged

Both no_run PRs are squash-merged to main:

• autolens_workspace_test#169 (9 SLOW entries)
• autogalaxy_workspace_test#73 (8 SLOW entries)

The flaky real-search scripts that were RED-blocking mode=release on the perf tail are now SLOW-skipped, and are owned by the hygiene sweep going forward. The advisory-tier mechanism this issue originally proposed was built, reviewed, and deliberately reverted in favour of this leaner approach.

Closing. Remaining is operational, not code: rerun mode=release for a fresh Heart verdict (the flaky set may shift → next handful added via hygiene), and note Heart stays RED until the unrelated PyAutoArray feature-branch reason clears. cpu_fast_modeling was intentionally left out (singular-FAIL fix tracked in PyAutoArray#388, not a timeout).