fix: load database latent variables in wheel environments

[Jammy2211]

Overview

Release-wheel validation fails all six PyAutoFit database scripts with a bare AssertionError, while the same source passes in the development environment. Reproduce the failure in the exact Python 3.12 TestPyPI wheel stack, isolate the dependency that changes behaviour, and fix the latent-variable loading producer in PyAutoFit without masking the warning or weakening workspace assertions.

Plan

• Reproduce scripts/database/scrape/general.py in an isolated wheel-only Python 3.12 environment and capture the complete traceback and package manifest.
• Pin SQLAlchemy, NumPy, and dill independently to isolate the dependency that flips the result, then repeat the winning forward/backward transition to establish causality.
• Trace the failure through latent-sample file loading and database serialization, fixing the producer in PyAutoFit rather than adding a workspace workaround or silent guard.
• Add focused regression coverage for latent-variable scraping/loading under the triggering dependency behaviour.
• Validate focused and full PyAutoFit tests, the exact reproduction, and all six database integration scripts.

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary; source and unit-test changes)
• autofit_workspace_test (validation only; no script changes expected)

Work Classification

Library

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autofit_workspace_test	main	clean

No active worktree claims PyAutoFit. The workspace is a read-only reproduction and validation surface.

Suggested branch: feature/database-latent-wheel-load

Worktree root: ~/Code/PyAutoLabs-wt/database-latent-wheel-load/

Implementation Steps

1. Create an isolated Python 3.12 environment with TestPyPI autolens[optional]==2026.7.13.1.dev65601, jax, jaxnnls, and jax_zero_contour. Record pip freeze, verify the imported autofit location is inside the venv, and ensure no local source checkout is on PYTHONPATH.
2. Remove prior generated output/database files, run autofit_workspace_test/scripts/database/scrape/general.py under the release profile, and preserve the full Python traceback plus the produced files/latent/ artifacts.
3. Hold the wheel environment fixed and vary one dependency at a time. Compare SQLAlchemy 2.0.51 with the dev version 2.0.32, NumPy 2.4.6 with the dev version 2.2.6, and dill 0.4.1 with a preceding compatible release as a control. Use --no-deps, pip check, and a clean output directory for every run. Reinstall the triggering version after the passing pin to prove the failure returns.
4. Trace the actual exception through SearchOutput._load_samples, _add_files_fit, and the database Fit/Object serialization path. Patch the component that produces or reconstructs the invalid latent samples. Do not broaden the warning catch to suppress the exception, silently omit latent data, or change workspace assertions to tolerate corrupt/missing data.
5. Add a regression fixture that contains real latent samples.csv and samples_info.json output and exercises the scrape-to-database round trip. Assert the latent samples retain their class, parameter values, model, and usable summary/instance after loading.
6. Run the focused aggregator/database tests, the relevant PyAutoFit test subtrees, then the full test_autofit/ suite. Re-run the exact wheel-stack script and all six scripts/database/{directory,scrape}/ scripts to confirm the release failure is removed.

Key Files

• autofit/aggregator/search_output.py — latent sample CSV/metadata reconstruction and likely exception origin.
• autofit/database/aggregator/scrape.py — boundary that assigns latent samples and currently emits the warning.
• autofit/database/model/fit.py — database representation of latent samples.
• autofit/database/model/ — object reconstruction if dependency behaviour changes serialization.
• test_autofit/aggregator/test_scrape.py — focused scraper regression coverage.
• test_autofit/database/test_serialize.py — database round-trip coverage if the traceback reaches object reconstruction.
• autofit_workspace_test/scripts/database/scrape/general.py — exact reproduction and validation script; no masking edits expected.

Original Prompt

Click to expand starting prompt

Database scripts fail on release wheels: "Failed to load latent variables" → AssertionError

Type: bug
Target: autofit
Repos:

• PyAutoFit
• autofit_workspace_test
  Difficulty: medium
  Autonomy: supervised
  Priority: normal
  Status: formalised

Surfaced burning down the 2026-07-13 release-validation tail (PyAutoHeart#72). On the release
wheels (mode=release), all six autofit_workspace_test/scripts/database/** scripts fail
FAST (4–13 s) with a bare AssertionError:

scripts/database/directory/general.py       ... FAIL (5.4s) AssertionError
scripts/database/directory/multi_analysis.py... FAIL (4.4s) AssertionError
scripts/database/scrape/general.py          ... FAIL (4.3s) AssertionError
scripts/database/scrape/grid_search.py      ... FAIL (8.8s) AssertionError
scripts/database/scrape/multi_analysis.py   ... FAIL (4.4s) AssertionError
scripts/database/scrape/sensitivity.py      ... FAIL (12.7s) AssertionError

with a recurring warning:

autofit.database.aggregator.scrape - WARNING - Failed to load latent variables for <hash>

Not jax, not perf, not the release profile. These PASS on the current dev venv
(jax 0.10.2) even under the release profile (PYAUTO_TEST_MODE=0) — the latent-variable
load emits only the WARNING there and the assertion passes. The wheel code is identical to
main HEAD, so the delta is the wheel-env dependency set: the release stack resolves
SQLAlchemy 2.0.51, dill 0.4.1, numpy 2.4.6 (full list in the rehearsal install log of
run 29279095224). The most likely culprit is a serialization/deserialization
incompatibility in the latent-variable load path (autofit.database.aggregator.scrape →
whatever loads latent samples from the SQLAlchemy-backed database) under one of those
dependency versions.

Reproduce with the wheel dependency set (the dev venv won't show it): build a venv with
the release stack — pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ "autolens[optional]==2026.7.13.1.dev65601" jax jaxnnls jax_zero_contour
(a scratch rel-venv approximating this already exists from the 2026-07-14 session) — then run
scripts/database/scrape/general.py from autofit_workspace_test/ and capture the full
traceback (the one-line CI summary hides the assertion detail). Isolate which dependency flips
it (SQLAlchemy vs dill vs numpy) by pinning them one at a time.

Fix the producer of the bad load (don't silence the warning — [[feedback_no_silent_guards]]):
if latent variables genuinely fail to deserialize under the newer dep, fix the
load/serialization in autofit.database; if the scripts' assertion is over-strict about a
tolerable-missing latent, fix the assertion. Release run: PyAutoHeart workspace-validation
29279095224, TestPyPI 2026.7.13.1.dev65601. See [[project_release_2026_07_13_blocked_3bugs]].

[Jammy2211]

Reproduction complete on the exact Python 3.12 TestPyPI wheel stack.

Real traceback

PYAUTO_TEST_MODE=0 python scripts/database/scrape/general.py writes the fit under output/test_mode/database/..., but the script scrapes output/database/.... The aggregator therefore finds zero searches and fails at:

File \"scripts/database/scrape/general.py\", line 127
    assert len(agg) > 0
AssertionError

The clean failing run does not reach latent-variable scraping, so the recurring Failed to load latent variables warning is incidental rather than causal. With PYAUTO_TEST_MODE unset, the same wheel stack finds one search, emits the known latent warning, and exits successfully.

Dependency controls

Fresh detached workspace for every run, with all other packages held fixed:

• SQLAlchemy 2.0.51 → 2.0.32: still fails identically.
• NumPy 2.4.6 → 2.2.6: still fails identically.
• dill 0.4.1 → 0.4.0: still fails identically. (The dev environment already uses dill 0.4.1.)

No listed dependency flips the result.

Root cause

autofit.non_linear.paths.abstract._test_mode_segment() uses raw environment-string truthiness:

return \"test_mode\" if os.environ.get(\"PYAUTO_TEST_MODE\") else None

The release profile intentionally sets PYAUTO_TEST_MODE: \"0\" for real searches. PyAutoConf's canonical is_test_mode() correctly treats level 0 as disabled, but this PyAutoFit path helper treats the non-empty string \"0\" as enabled. The fix is to make output path selection consume the canonical numeric test-mode semantics and add explicit level-0 path regressions.

[Jammy2211]

Implementation complete locally — ship blocked by Heart RED

The source fix and regression tests are complete in ~/Code/PyAutoLabs-wt/database-latent-wheel-load/PyAutoFit on feature/database-latent-wheel-load. Nothing has been committed or pushed.

Change

• autofit/non_linear/paths/abstract.py: _test_mode_segment() now uses PyAutoConf's canonical is_test_mode() numeric semantics. PYAUTO_TEST_MODE=0 is a real run and no longer adds output/test_mode/; active levels still do.
• test_autofit/non_linear/paths/test_paths.py: explicit level-zero regressions for both DirectoryPaths.output_path and _make_path().
• The latent-variable warning and workspace scripts are unchanged.

Validation

• Focused path tests: 14 passed.
• Full PyAutoFit suite: 1475 passed, 1 skipped.
• Exact Python 3.12 branch-source + wheel-dependency stack (SQLAlchemy 2.0.51, dill 0.4.1, numpy 2.4.6): all 6 database scripts pass under the release profile.
• Dependency controls remained failing before the patch: SQLAlchemy 2.0.32, NumPy 2.2.6, and dill 0.4.0 each reproduced the same wrong-output-root assertion. No dependency is causal.
• git diff --check: clean.

Proposed commit

fix: respect disabled test mode in output paths

Proposed PR body

## Summary

Use PyAutoConf's canonical numeric test-mode state when selecting AutoFit's output namespace. This prevents the release profile's `PYAUTO_TEST_MODE=0` from writing real-search results under `output/test_mode/`, where database scripts cannot find them, while preserving isolation for active test-mode levels.

## API Changes

Changed behavior only: `DirectoryPaths.output_path` now treats `PYAUTO_TEST_MODE=0` as disabled, consistently with the rest of the test-mode API. Active levels continue to use `output/test_mode/`. No symbols or signatures changed.
See full details below.

## Test Plan

- [x] `python -m pytest test_autofit/non_linear/paths/test_paths.py -q` — 14 passed
- [x] `python -m pytest test_autofit/ -x` — 1475 passed, 1 skipped
- [x] Release-profile Python 3.12 wheel-stack validation — all 6 database scripts passed

<details>
<summary>Full API Changes (for automation & release notes)</summary>

### Removed
- None.

### Added
- None.

### Renamed
- None.

### Changed Signature
- None.

### Changed Behaviour
- `autofit.non_linear.paths.directory.DirectoryPaths.output_path` — `PYAUTO_TEST_MODE=0` now writes to the normal output root; values greater than zero continue to write under `output/test_mode/`.

### Migration
- None required. Release-profile callers already set the documented value `PYAUTO_TEST_MODE=0`.

</details>

Generated by the PyAutoLabs agent workflow.

Ship blocker

The refreshed authoritative Heart verdict is RED: release validation FAILED (stage integrate). This is the release run the patch is intended to repair, but the autonomy contract forbids commit/push/PR while Heart is RED. Resume with /ship_library after Heart is no longer RED; it must re-run the gate before shipping.

[Jammy2211]

Human authorization — corrective-PR exception

I authorize the corrective-PR exception for PyAutoFit#1367 against Heart RED reason release validation FAILED (stage integrate). Proceed to commit, push, and open one pending-release PR for the existing two-file fix.

Authorization received live on 2026-07-14.

• Exact current Heart RED reason: release validation FAILED (stage integrate)
• Corrective issue: fix: load database latent variables in wheel environments #1367
• Permitted scope: commit and push the existing two-file producer fix, then open one pending-release PR
• Not authorized: automatic merge, issue closure, release, rehearsal, or unrelated changes
• Required recovery: after a separate human merge decision, build fresh wheels, rerun release integration validation, and obtain a new Heart verdict

[Jammy2211]

Corrective PR opened

The human-authorized corrective-PR exception has been exercised for the exact Heart RED reason release validation FAILED (stage integrate).

• PR: fix: respect disabled test mode in output paths #1368
• Commit: 2ff67195cfd479082888655d8a9c321e44fbd94c
• Scope: exactly the existing two-file producer fix and regression tests
• Label: pending-release
• Review: CLEAN — one causal commit, no mixed or unrelated scope
• Tests: 14 focused passed; full PyAutoFit 1475 passed, 1 skipped; all 6 release-profile database scripts passed on the exact wheel dependencies

No merge, issue close, release, or rehearsal action was taken. Merge remains a separate human decision. After merge, the required recovery is to build fresh wheels, rerun release integration validation, and obtain a new Heart verdict.

[Jammy2211]

Corrective fix merged

PR #1368 was squash-merged by explicit human direction after all required checks passed.

• Merge commit: 4724e20a103edd5b2745b8ad39abc98a0529054a
• CI: docs passed; unit tests passed on Python 3.12 and 3.13
• Scope: one causal commit, two files, unchanged from the CLEAN review
• Workspace verification: all 6 release-profile database scripts passed on the exact wheel dependency stack before merge

This issue remains open. No release or rehearsal was triggered. The remaining corrective recovery path is to build fresh wheels containing the merge, rerun release integration validation, and obtain a new Heart verdict for release validation FAILED (stage integrate).

[Jammy2211]

Corrective validation GREEN — closing

The fresh mode=release validation run on the current main HEADs passed end-to-end:

• Stage 2 TestPyPI rehearsal v2026.7.14.1.dev66001 — all 5 wheels built + uploaded.
• Stage 3 wheel-based real-search integration — 542 passed, 0 failed, 0 timeout, 87 skipped.
• Ingested → release_ready: true; Heart is now YELLOW (no RED reasons remain).

This fix is on the validated main HEADs (commit_shas matched), so its corrective validation is satisfied. Closing.