Skip to content

ci: bound Windows build parallelism (intermittent OOM/SIGTERM) - #9

Merged
Hornfisk merged 1 commit into
masterfrom
fix/ci-bound-build-parallelism
Jul 25, 2026
Merged

ci: bound Windows build parallelism (intermittent OOM/SIGTERM)#9
Hornfisk merged 1 commit into
masterfrom
fix/ci-bound-build-parallelism

Conversation

@Hornfisk

Copy link
Copy Markdown
Collaborator

Found while auditing the fleet after Caja's CI died to this exact pattern.

JOBS defaulted to a bare -j and was overridden with an explicit cap only for
Linux (-j 2) and macOS (-j 3). The windows-2022 leg fell through to the
unbounded default.

A bare -j means make/cmake spawn one compiler per ready translation unit. With
JUCE's large TUs that exhausts the runner and the job is killed with SIGTERM —
exit 143, Terminated, and crucially no compiler error above it, so it does
not read as a build failure. It is intermittent by nature: the same commit can
pass one run and fail the next.

Windows is the leg where it matters most, because the Configure step
deliberately keeps LTO on for Windows only (if [ "$RUNNER_OS" != "Windows" ]; then EXTRA="-DBOMBO_LTO=OFF"; fi), which makes each link heavier than the
LTO-off Linux/macOS builds.

Now every OS is explicitly bounded and the fallback default is -j 2 rather than
unbounded, so a future runner added to the matrix is safe by default.

CI-only. No product code touched, so this does not intrude on the scope freeze.

Background: this cost about a day on Caja before it was traced — the tell is that
the same commit passed on its pull_request run and failed on its push run.

🤖 Generated with Claude Code

The Build step set JOBS="-j" as its default and only overrode it with
an explicit cap for Linux (-j 2) and macOS (-j 3), leaving the
windows-2022 matrix leg with a bare, unbounded -j. A bare -j/--parallel
means make/cmake spawn one compiler or linker per ready translation
unit; with JUCE's large TUs, plus LTO staying enabled specifically on
Windows in the Configure step, that exhausts the runner's RAM and the
job gets killed with SIGTERM (exit 143, "Terminated", no compiler
error above it). It's intermittent - the same commit can pass one run
and fail the next depending on scheduling.

Pin Windows to -j 2 (matching Linux's conservative cap, since Windows
keeps LTO on and each link is heavier) and change the fallback default
from bare -j to -j 2 so any future unlisted matrix OS is safe too.
@Hornfisk

Copy link
Copy Markdown
Collaborator Author

Validated

build.yml only triggers on v* tags and workflow_dispatch, not on pull
requests — so this PR shows no checks and cannot validate itself. Dispatched it
against the branch instead:

gh workflow run build.yml --ref fix/ci-bound-build-parallelism

Run 30156870758all three legs green:

leg result
linux-x86_64 success
macos-universal success
win64 success ← the leg this PR changes

Suggestive side note, not proof: the whole run took 11m48s, against 21m21s
for the last successful build (the v1.0.0 tag on 2026-07-05, unbounded -j on
Windows). Different commit and runner variance both apply, so treat it as
directional — but it points the way the thrashing hypothesis predicts. Bounding
parallelism did not cost build time here; it looks like it saved some.

@Hornfisk
Hornfisk merged commit 6f6155f into master Jul 25, 2026
3 checks passed
@Hornfisk
Hornfisk deleted the fix/ci-bound-build-parallelism branch July 25, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant