research: pre_build.sh git-add failure-class audit — what is the local commit for?

[Jammy2211]

Overview

Phase 1 of the build-chain integrity campaign (#155). An audit with a design question attached: on 2026-07-15 the manual release hit two bugs in pre_build.sh, both the same shape — a git add whose failure is handled wrongly in one of two opposite directions (fatal under set -e, or swallowed by || true). A third instance (:55, the README bump sed) was found while writing the brief: all three failure-tolerant sites in the file are implicated. The job is to enumerate the whole class empirically, and answer the design question that outranks the bug list: if release.yml regenerates and commits these artifacts on the runner anyway, what is pre_build's local commit for? If redundant, the best deliverable is a smaller file (the #47 shape), not a more defensive one — but that must be found, not assumed.

Plan

• Enumerate the failure class across pre_build.sh end to end + callees (check_dataset_allowlist.py, generate.py): every command whose failure is unhandled under set -e or swallowed by || true/2>/dev/null, both directions. Deliverable: a per-repo × per-line exit-code matrix, measured with git add -n (safe, no staging) against the real repos — both .gitignore dialect groups.
• Settle the unresolved claim: does the :55 sed edit README.md and merely go uncommitted (staging no-op'd by :88), or not edit at all?
• Answer redundant-vs-load-bearing for the local commit, with evidence from the pre_build vs release.yml division of labour; include the unstaged-black-reformat collateral question.
• Produce the target error-handling design + what was rejected and why + open decisions for the human.

Research task (Model: Fable, supervised): judgment and a design doc are the deliverable; any pre_build.sh edits are a follow-up PR behind plan approval.

Detailed implementation plan

Work Classification

Research (PyAutoBuild primary; reads workspaces read-only)

Affected Repositories

• PyAutoBuild (primary)
• autofit/autogalaxy/autolens_workspace, _workspace_test, HowTo (read-only measurement targets)

Branch Survey (2026-07-16)

Repository	Current Branch	Dirty?
./PyAutoBuild	main	clean

Suggested branch: feature/pre-build-audit
Worktree root: ~/Code/PyAutoLabs-wt/pre-build-audit/

Method (from the brief — binding)

• Run things: git add -n per repo per pattern; bash -n; primitives against throwaway clones. NEVER run pre_build.sh itself (it dispatches a live PyPI release; release.yml rehearsal defaults to false).
• Reproduce in the REAL repos, never synthetic ones (the synthetic-repo shortcut already failed once — "Trust nothing here").
• Exercise both .gitignore dialect groups (dataset/ whole-dir vs dataset/**+allowlist).
• Adversarial pass before concluding; prefer deleting the trap to hardening it.
• Never reintroduce git add -f for dataset/ (bug: release pipeline force-commits simulated workspace datasets (git add -f) #126/fix: honor .gitignore for dataset/ staging in pre_build (drop -f) [#126 leg 1] #150).

Key Files

• PyAutoBuild/pre_build.sh — :13 set -e; :55/:79/:88 failure-tolerant sites; 13 run_workspace calls; final git add -A + dispatch
• PyAutoBuild/.github/workflows/release.yml — the runner-side regeneration that masks bug 2
• draft/bug/pyautobuild/root_level_git_add_stages_nothing_on_unmatched_glob.md — bug 2, held pending this audit's verdict

Original Prompt

Click to expand starting prompt

Audit pre_build.sh for the silent/fatal git-add failure class — and ask what its local commit is even for

Type: research
Target: PyAutoBuild
Repos:

• PyAutoBuild
• autofit_workspace
• autolens_workspace
• autogalaxy_workspace
• HowToFit
  Difficulty: medium
  Autonomy: supervised
  Priority: normal
  Status: formalised
  Model: Fable

What this is

An audit task with a design question attached. On 2026-07-15 the first manual live release
since 2026-07-09 hit two bugs in pre_build.sh, both the same shape, one after the other.
Each was ~5 lines to fix. Neither was hard. The point of this task is not to fix them — one is
already merged and the other is already filed — it is to find out how many more of that shape
are in this file, and whether the file's design is what keeps producing them.

The shape: a git add whose failure is handled wrongly in one of two opposite directions.

1. Fatal. [ -d dataset ] && git add dataset/ under set -e. git add exits non-zero when
   every path it matches is ignored, so this killed the entire release on the first workspace
   processed. Merged as PyAutoBuild#154.
2. Swallowed. git add -- *.py *.md *.txt *.cfg … 2>/dev/null || true. Bash leaves unmatched
   globs literal, git rejects the whole pathspec list (fatal: pathspec '*.cfg' did not match any files, exit 128), and || true eats it — so nothing from that line is ever staged in
   any repo missing one of those extensions. Filed as
   draft/bug/pyautobuild/root_level_git_add_stages_nothing_on_unmatched_glob.md, unfixed.

One over-fails, one under-fails. Both come from the same underlying assumption: that git add
either works or is worth dying over, when in this script it is routinely and correctly a no-op.

There is already a confirmed third instance, and I found it while writing this prompt — which
is the whole argument for the audit. pre_build.sh has exactly three || true / 2>/dev/null
sites (:55, :79, :88), and :55 is the README version bump:

echo "  Bumping README version → $readme_pkg v$VERSION..."   # prints unconditionally
sed -i "s/…/…/g" README.rst README.md 2>/dev/null || true

No workspace has a README.rst (checked autofit_workspace, autolens_workspace, HowToFit).
So this sed exits 2, every time, in every repo — the || true is not a safety net for a rare
case, it is permanently load-bearing, and a genuine sed failure is indistinguishable from the
expected missing-file one. The banner prints either way. Three sites, three instances of the class.
Assume there are more; that is the job.

Take the time this needs. This file commits and pushes to ~14 repos and then dispatches a live
PyPI release. It is the least-exercised, highest-consequence code in the organism.

Why this went undetected (this is the interesting part)

Both bugs were latent for days in a file that runs constantly — because the live path barely
runs:

• The 2026.7.9.1 release (2026-07-09) predates PyAutoBuild#150 (merged 2026-07-13), which
  introduced bug 1 by dropping a git add -f (correctly — the -f was force-committing simulated
  datasets, bug: release pipeline force-commits simulated workspace datasets (git add -f) #126).
• Every night since, nightly-release died at Stage 4 (validation) on an unrelated bug, so it
  never reached step 6 (the live release) where pre_build runs.
• So between 07-13 and 07-15, pre_build's staging path executed zero times. Merged is not
  the same as ever ran.

And bug 2 is still invisible in production, because release.yml's release_workspaces job
regenerates and commits the same artifacts on the runner. Verified 2026-07-15: origin/main's
READMEs correctly pin 2026.7.15.1 and llms-full.txt/workspace_index.json match freshly
generated local output — even though pre_build staged none of it locally. The runner covers
for the script.

The design question worth more than the audit

If release.yml regenerates and commits these artifacts on the runner anyway, what is
pre_build's local commit actually for?

Bug 2 has been silently no-op'ing the root-level staging for an unknown length of time and
nobody noticed, because the outcome was correct anyway. That is either:

• evidence the local staging is redundant and should be deleted (the lean-lever, delete-the-trap
  outcome — cf. PyAutoBuild#47's shape: it deleted 31 lines of config and a whole bug class by
  noticing the dangerous setting bought nothing); or
• evidence it is load-bearing in a case not exercised on 2026-07-15, and got lucky.

Find out which. Do not assume the first because it is tidier. If it is redundant, the audit's
best deliverable is a smaller file, not a more defensive one. If it is load-bearing, say exactly
where and why.

Measured facts (2026-07-15 — re-measure; do not inherit)

• pre_build.sh:13 — set -e. This is what makes bug 1 fatal.

• pre_build.sh processes 13 run_workspace calls, then commits/pushes PyAutoBuild itself
  with git add -A on whatever branch is checked out, then dispatches release.yml with no
  rehearsal field — which defaults to false, i.e. a real PyPI release.

• run_workspace stages only: dataset/, config/, notebooks/, scripts/, optional
  slam_pipeline/, and the root-level glob line. Anything else black touches is never staged.

• Two .gitignore dialects, different git behaviour — measured with git add -n dataset/:

  pattern	repos	git add dataset/
  dataset/ (whole dir)	autofit_workspace, autofit/autogalaxy/autolens_workspace_test, HowToFit/Galaxy/Lens = 7 of 11	exit 1
  dataset/** + ! allowlist	autolens_workspace (13 entries), autogalaxy_workspace (5)	exit 0, stages real data

• autofit_workspace has no .cfg/.ini/.toml → the root glob line (:88) exits 128 there.

• The only three failure-tolerant sites in the file are :55, :79, :88 — and on
  2026-07-15, two of the three were masking a bug (:88) or were themselves the bug (:79,
  pre-fix(pre_build): a fully-ignored dataset/ must not abort the release #154). :55 masks unconditionally (no repo has README.rst; sed exits 2 every run). That
  is a 3-for-3 hit rate on a very small sample — treat the idiom itself as the suspect.

• Observed collateral, unexamined: black reformats files that run_workspace never stages and
  therefore never commits — e.g. .github/scripts/check_tutorials_complete.py in all three HowTo
  repos, and ~11 autoassistant/*.py files in autolens_assistant. These are reformatted on every
  release and left dirty locally, forever. Is that a third instance of this class, or intended?
  I did not determine this. It is a good place to start.

What to sweep

Not just the two known lines. The whole file, and what it calls:

• pre_build.sh end to end — every command whose failure is either unhandled under set -e or
  swallowed by || true / 2>/dev/null. Both directions are bugs; enumerate both.
• The interaction between set -e and && lists. Note [ -d x ] && cmd under set -e is subtle:
  a failing [ -d x ] is ignored (non-final in an && list) but a failing cmd is fatal.
  The idiom appears more than once. Is it right anywhere?
• Bash glob behaviour: unmatched globs stay literal without nullglob. Where else does this bite?
• check_dataset_allowlist.py, generate.py — same lens: what do they do on failure, and does
  pre_build notice?
• The pre_build vs release.yml division of labour — who is responsible for producing which
  artifact, where they overlap, and where the overlap is masking a defect.
• Anything that prints a claim it did not do. "Bumping README version → …" prints before a
  sed that always exits non-zero into a || true, and the result is then not staged by :88. Log
  lines that lie are how this stayed hidden — the 2026-07-15 log printed that banner for all 14
  repos and I believed it. (Precisely which half fails is unresolved: the sed does still edit
  README.md despite erroring on the absent README.rst, so the bump likely happens and merely
  goes uncommitted. I did not confirm this — a version-like grep of the local README returned
  a pre-release string, but that grep matched other version references in the file and proves
  nothing. Settle it properly; do not inherit either reading.)

Trust nothing here

This document was written by the agent that made the mistakes on 2026-07-15. Two of its
conclusions were wrong that day and were caught only by re-checking:

• "I reproduced bug 1." — false. The first repro used a synthetic repo with dataset/**
  and exited 0; it did not reproduce the bug at all. The real .gitignore is dataset/, which
  git treats differently. I was one step from "verifying" a fix against a bug I had never
  triggered. Reproduce in the real repo.
• "The tags are missing on PyAutoFit and PyAutoLens." — false. gh api repos/X/tags --jq '.[0].name' is not date-sorted; it returned pull and v1.15.2. All five tags
  existed. Use git/ref/tags/<version>.

Both had the same shape as the day's other errors: checking the path already assumed and calling
it proof. Treat this document as a lead sheet, not a source of truth.

What to produce

1. A complete enumeration of the failure class in pre_build.sh and its callees — each with
   the concrete repo/condition that triggers it, measured, not reasoned. A per-repo exit-code
   matrix (git add -n is safe and does this without staging) beats prose.
2. An answer to the design question above: is the local staging redundant, load-bearing, or
   partly each? With evidence.
3. A target design for error handling in this file. Hypotheses to argue with, not requirements:
   • a no-op and a failure should not be the same event;
   • anything that may legitimately match nothing should say so explicitly, not rely on || true;
   • a log line should not be able to claim work that did not happen;
   • if the runner is the real producer, the local script should stop pretending to be.
4. What you rejected and why. Including: is shopt -s nullglob the right fix for bug 2, or does
   it just make a redundant line silently succeed?
5. Open decisions for the human — separated from what you're confident about. Specifically: the
   unstaged-black-reformat question, and whether pre_build's local commits should exist at all.

Method

• Run things. git add -n per repo per pattern; bash -n; bash -x pre_build.sh against
  throwaway clones. Nothing here is expensive; being wrong is.
• Reproduce in the real repo, never a synthetic one you built from your own mental model of the
  .gitignore. See "Trust nothing here" — this exact shortcut failed on 2026-07-15.
• Exercise both dialect groups. Any claim about dataset/ staging must be tested against a
  dataset/-ignored repo and a dataset/**+allowlist repo. The allowlist group is the one that
  agrees with you.
• Prove no-ops empirically. "Provably inert" preceded being wrong that day.
• Adversarial pass before concluding: for each load-bearing claim, what would have to be true
  for this to be false, and have I looked?
• Prefer deleting the trap to hardening it. If the answer is "add defensive error handling to
  every line", ask first whether the line should exist.

Constraints

• This script dispatches live PyPI releases. release.yml's rehearsal input defaults to
  false. Never test by running pre_build.sh; test the primitives against throwaway clones.
  To exercise the pipeline, dispatch release.yml with rehearsal=true explicitly.
• Never reintroduce git add -f for dataset/ — that was the simulated-dataset leak (bug: release pipeline force-commits simulated workspace datasets (git add -f) #126),
  and fix: honor .gitignore for dataset/ staging in pre_build (drop -f) [#126 leg 1] #150 removed it deliberately. Bug 1 is the consequence of that correct fix, not an argument
  against it.
• pre_build commits with git add -A on the current branch — any fix lands on main before a
  release runs, or the release commits onto your feature branch.
• Behaviour-preserving where the behaviour is correct: main ends up right today, and it must still
  end up right after.
• Design first. Plan approval before implementation, per the usual workflow.

[Jammy2211]

Measured findings — session 1 (2026-07-16, all read-only: git add -n, sed-to-stdout, origin-verified)

Per-repo matrix across all 13 run_workspace targets:

site	measured	verdict
:55 README sed	exit 2 in 13/13 (no repo has README.rst)	|| true permanently load-bearing, as briefed
:79 git add dataset/	exit 1 in 5 repos (whole-dir ignore), exit 0 in 4 (allowlist dialect), nodir in 4	post-#154 swallow is the correct direction today
:82 git add config/ notebooks/ scripts/	exit 0 wherever the dir exists	latent bug-1-class fatal (present-but-fully-ignored dir would kill the release under set -e); benign today, 13/13
:88 root glob	exit 128 in 13/13	global no-op — worse than briefed (brief measured 1 repo). Six patterns (*.cfg *.ini *.toml *.yml *.yaml setup*) match nothing in ANY repo; their unmatched-glob literals poison the whole pathspec list everywhere

The unresolved claim from the brief is settled, in the worst direction. The sed DOES edit README.md where a pin exists; :88 then never stages it; and release.yml does NOT cover for it — the runner-side README step was deliberately removed (release.yml comment: "The former 'Update version in README' and 'Write workspace version' steps", #120/#121). Verified against origin/main, not local checkouts:

• autofit_workspace README.md pins PyAutoFit v2026.7.9.1 (release is 2026.7.15.1 — the 07-13 hand-bump value, untouched by the 07-15 release)
• autofit_workspace_test + autolens_workspace_test pin v2026.5.14.1 (~2 months stale)
• the other 7 repos with readme_pkg set have no pin pattern in README.md at all — for them the banner + sed are semantically empty every run

So the \"Bumping README version → X v$VERSION\" banner has printed a false claim on every release for months: the bump is edited-then-dropped in 3 repos and a no-op in 10. The brief's premise "main ends up right today" is half-wrong — right for notebooks/Colab URLs (runner-committed), wrong for README pins (orphaned artifact, stale on main now).

Division of labour (partial map, to finish): runner commits *.ipynb, api_audit_baseline.json (assistant), Colab URLs. pre_build's local staging is load-bearing for dataset/ (allowlisted real data), config/, scripts/, and black formatting in staged dirs — and broken/orphaned for everything on the root-glob line. Still to map: who (if anyone) commits root-level llms-full.txt / workspace_index.json — the brief says origin matched fresh output on 07-15; the commit path for that is not yet identified.

Feeds Phase 4: stale README pins are one of the frozen version references in release_version_sync_back_to_main.md; this audit confirms they are currently orphaned (neither side owns them).

Remaining legs: llms-full/index commit-path, black-collateral enumeration (unstaged reformats in HowTo*/assistant), callee failure analysis (generate.py, check_dataset_allowlist.py), the target design + adversarial pass, and the redundant-vs-load-bearing verdict per artifact.

[Jammy2211]

Audit complete — design doc drafted (docs/pre_build_failure_audit.md on feature/pre-build-audit)

Closing findings beyond the session-1 comment:

llms-full.txt / workspace_index.json commit path identified: they reach main only as accidental git add -A sweep inside the runner's "bump Colab URL tag refs" commit (verified: commit 7e3c007 in autofit_workspace carries llms-full.txt + workspace_index.json + README Colab URLs under that message). Unowned, mislabeled, and only for bump_colab_urls: true repos.

Black collateral measured: 13 files reformatted every release and staged by no rule (3× HowTo .github/scripts/check_tutorials_complete.py, 10× autoassistant/*.py) — perpetual dirty-local churn; a third instance of the class.

Callee findings: generate.py:107 runs os.system(\"git add -f <notebook>\") — ignored return AND a git add -f, a hidden fourth failure-tolerant site; pre_build invokes generate.py with report=None so any single conversion error is fatal to the whole release.

Design-question verdict (per artifact): the local commit is mostly redundant, one genuine product, one broken orphan — genuine only for black's scripts/ reformat; redundant for notebooks (runner rebuilds them); unowned for the root-glob artifacts; orphaned for the README pin (nobody owns it; stale on origin now); vestigial for dataset/config (sweeps human dirt — #126's mechanism).

Recommended shape (3 landable steps, full doc has the argument): (1) delete the dead :88 glob line and the :55 README bump (both measured no-ops/orphans; README-pin ownership decided in Phase 4); (2) replace the runner's git add -A with explicit paths + honest commit message; (3) shrink pre_build to what it genuinely produces, fix generate.py:107, and give the 13 collateral files a staged home or stop black'ing them. shopt -s nullglob explicitly rejected — it makes a redundant line silently succeed.

Doc ends with open decisions for the human (README-pin fate → Phase 4 fork, the dataset/config sweep vestige, atomicity). Implementation is deliberately NOT in this PR — design first, per the brief.

[Jammy2211]

PR open: #157 (feature/pre-build-audit, pending-release, docs-only + one 1-line pre-existing drift fix its own gate caught — test_pre_build_skill failed on clean main, skill-doc preflight list named admin_jammy instead of PyAutoBrain; synced in commit abbd01f).

113/113 tests pass on the branch. Shipped under the human-authorized corrective-PR exception for Heart RED (second authorization, 2026-07-16). Merge stays human. Implementation of the doc's 3-step design follows as separate PRs behind plan approval — step 1 (delete the dead :88/:55 lines) is ready to plan whenever the audit doc is accepted.

[Jammy2211]

Step-1 implementation PR open: #158 (feature/pre-build-dead-lines, pending-release). Deletes the two measured-dead sites (root-glob staging, README bump + readme_pkg param across 13 call sites) and syncs the three stale doc claims (skill step description, the "README bump handled inside pre_build.sh" note, internals.md's pre-#150 git add -f dataset claim + root-files row). bash -n clean; 113/113 tests incl. the drift guard. Behaviour on main unchanged by construction — neither site produced anything that ever landed.

Steps 2 (own the runner's git add -A sweep in release.yml) and 3 (shrink pre_build to its genuine product) remain, each behind its own plan. Awaiting human merge of #158.

[Jammy2211]

Step-2 PR open: #159 (feature/release-yml-owned-staging, pending-release). Notebooks step stages notebooks/ + the catalogue files explicitly with an honest message; both Colab bump steps move git add -A → git add -u. Activity-gate Release <version>: prefix preserved (checked against its test before editing). Workflow branches can't be CI-run pre-merge — PR carries a post-merge verification item (inspect the next release/rehearsal's commit contents).

Remaining: step 3 (shrink pre_build to its genuine product) — deliberately unstarted; it changes real behaviour and hinges on the open human decision about the dataset/config sweep vestige (docs/pre_build_failure_audit.md §6).

[Jammy2211]

Step-3 PR open: #160 (feature/pre-build-slim, pending-release) — the §6 sweep-vestige decision came back DELETE (human, 2026-07-16). black on staged dirs only; dataset/config adds removed (allowlist check retained); the latent [ -d ] && fatal made explicit; generate.py's os.system(git add -f) → checked subprocess.run without -f (measured safe). With #158+#159+#160, every failure-tolerant site the audit enumerated is resolved. 113/113 tests.