Commit e68feb2
test(node-integration-tests): Free up CI disk space to prevent ENOSPC failures (#22132)
Some Node integration runs fail with `ENOSPC: no space left on device`.
The suite pulls several database docker images (mssql alone ~1.5GB) and
installs per-suite `node_modules` concurrently, and on a small-disk
runner this fills `/`. When it happens many unrelated suites fail at
once — and because the flaky-test detector files one issue per failing
test, a single disk-full run spawns a batch of unrelated "flaky" issues
that are really infra noise.
This adds a disk-reclaim step to `job_node_integration_tests`, but
**gated behind a `df` check** so it only runs when free space is
actually low (< 40GB). On a healthy runner it's a sub-second no-op
("Sufficient disk space — skipping"); on a constrained one it reclaims
the large preinstalled toolchains this job never uses (Android SDK,
.NET, GHC/ghcup, CodeQL — a slow ~30-60s `rm`), freeing ~20GB. `df` is
logged either way so a future squeeze stays visible.
## Root cause
Two runs hit ENOSPC (jobs `85837176962` and `85929899963`); every issue
linked below traces back to one of them, so none are genuine test
flakes.
It was a genuine byte shortage, not inodes — alongside the
`mkdir`/`mkdtemp` failures, one run failed with `write
/opt/mssql/lib/system.common.sfp: no space left on device`, i.e. a file
write running out of bytes.
Why runs differ so much on the same label: `ubuntu-24.04` is a *label*,
not a fixed disk size. GitHub-hosted runners are ephemeral VMs from a
fleet that isn't uniform, and the free space on `/` has been enlarged
over time and rolled out gradually without changing the label. Nothing
in the workflow pins a disk size, so one job lands on an old
~14-30GB-free VM (→ ENOSPC) while another lands on a ~145GB one (89GB
free). The failing runs and a recently measured healthy run all use
`ubuntu-24.04`.
The `df` gate keys on exactly the resource that ran out, so it fires
precisely on the small-disk VMs and no-ops on the large ones. If GitHub
has since fully migrated the fleet to larger disks it simply becomes a
cheap permanent no-op; if the fleet is still mixed it protects the
constrained runs. Either way it costs healthy runners nothing.
Fixes #22076
Fixes #22071
Fixes #22070
Fixes #22106
Fixes #22073
Fixes #22072
Fixes #22077
Fixes #22074
Fixes #22075
Fixes #22108
Fixes #22107
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 047e6c8 commit e68feb2
1 file changed
Lines changed: 13 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
816 | 816 | | |
817 | 817 | | |
818 | 818 | | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
819 | 832 | | |
820 | 833 | | |
821 | 834 | | |
| |||
0 commit comments