Skip to content

Commit e68feb2

Browse files
mydeaclaude
andauthored
test(node-integration-tests): Free up CI disk space to prevent ENOSPC failures (#22132)
Some Node integration runs fail with `ENOSPC: no space left on device`. The suite pulls several database docker images (mssql alone ~1.5GB) and installs per-suite `node_modules` concurrently, and on a small-disk runner this fills `/`. When it happens many unrelated suites fail at once — and because the flaky-test detector files one issue per failing test, a single disk-full run spawns a batch of unrelated "flaky" issues that are really infra noise. This adds a disk-reclaim step to `job_node_integration_tests`, but **gated behind a `df` check** so it only runs when free space is actually low (< 40GB). On a healthy runner it's a sub-second no-op ("Sufficient disk space — skipping"); on a constrained one it reclaims the large preinstalled toolchains this job never uses (Android SDK, .NET, GHC/ghcup, CodeQL — a slow ~30-60s `rm`), freeing ~20GB. `df` is logged either way so a future squeeze stays visible. ## Root cause Two runs hit ENOSPC (jobs `85837176962` and `85929899963`); every issue linked below traces back to one of them, so none are genuine test flakes. It was a genuine byte shortage, not inodes — alongside the `mkdir`/`mkdtemp` failures, one run failed with `write /opt/mssql/lib/system.common.sfp: no space left on device`, i.e. a file write running out of bytes. Why runs differ so much on the same label: `ubuntu-24.04` is a *label*, not a fixed disk size. GitHub-hosted runners are ephemeral VMs from a fleet that isn't uniform, and the free space on `/` has been enlarged over time and rolled out gradually without changing the label. Nothing in the workflow pins a disk size, so one job lands on an old ~14-30GB-free VM (→ ENOSPC) while another lands on a ~145GB one (89GB free). The failing runs and a recently measured healthy run all use `ubuntu-24.04`. The `df` gate keys on exactly the resource that ran out, so it fires precisely on the small-disk VMs and no-ops on the large ones. If GitHub has since fully migrated the fleet to larger disks it simply becomes a cheap permanent no-op; if the fleet is still mixed it protects the constrained runs. Either way it costs healthy runners nothing. Fixes #22076 Fixes #22071 Fixes #22070 Fixes #22106 Fixes #22073 Fixes #22072 Fixes #22077 Fixes #22074 Fixes #22075 Fixes #22108 Fixes #22107 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 047e6c8 commit e68feb2

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

.github/workflows/build.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -816,6 +816,19 @@ jobs:
816816
uses: actions/checkout@v7
817817
with:
818818
ref: ${{ env.HEAD_COMMIT }}
819+
- name: Free up disk space if low
820+
# The full suite pulls several DB docker images (mssql alone is ~1.5GB)
821+
# Available disk space is not consistent, if we detect low space this cleans up some unused toolchains
822+
run: |
823+
df -h /
824+
avail_kb=$(df -k --output=avail / | tail -1)
825+
if [ "$avail_kb" -lt $((40 * 1024 * 1024)) ]; then
826+
echo "Low disk space (<40GB free) — reclaiming unused toolchains"
827+
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup /opt/hostedtoolcache/CodeQL
828+
df -h /
829+
else
830+
echo "Sufficient disk space — skipping cleanup"
831+
fi
819832
- name: Set up Node
820833
uses: actions/setup-node@v6
821834
with:

0 commit comments

Comments
 (0)