chore(ci): lower self-hosted earthly cache_size_pct 50 → 20#1867
chore(ci): lower self-hosted earthly cache_size_pct 50 → 20#1867scottbuckel wants to merge 3 commits into
Conversation
At 50 the shared earthly-buildkitd cache on the self-hosted runner host is allowed ~850 GB of the 1.7 TB disk and was observed at 889 GB with the disk at 99%, failing jobs with "No space left on device". 20 targets ~340 GB. Lockstep with shieldedtech/shielded-iac#1818 — cache_size_pct is an earthly settings-hash key and must stay byte-identical between this file and the host-level config on the runner, or CI jobs restart the shared buildkitd and cancel concurrent builds. Assisted-by: Claude:claude-fable-5 Signed-off-by: Scott Buckel <scott.buckel@shielded.io>
scottbuckel
added
skip-changes-check-all
skip-changes-check-issue
ai-assisted
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Jina ReviewJina has completed this review. Review: https://app.usejina.com/reviews/88002e9a-ea26-496f-b528-926758b3e12a
|
![jina-simulation[bot]](https://avatars.githubusercontent.com/u/97225700?s=60&v=4){kind=small}
There was a problem hiding this comment.
Runtime Review - Merge Readiness 5/5
0 issues found - Ready to merge
Ready to merge.
No accepted issues remain.
Configuration selection, self-hosted scoping, and BuildKit garbage-collection semantics were substantiated; the unavailable Earthly execution environment limits validation but does not establish a defect.
Final review: No publishable runtime issues found.
The 20% self-hosted cache setting is correctly scoped; the remaining filesystem-validation concern is low-likelihood and non-blocking.
Review Comments
-
Consider monitoring the filesystem used by Earthly cache
Earthly derives cache_size_pct from the total filesystem containing EARTHLY_TMP_DIR, not from available free space.
The repository and CI do not verify that this is the documented 1.7 TB pool or monitor non-cache disk consumers.
This is a useful operational follow-up, but the available evidence does not establish a production failure in this PR.
Evidence:
- Earthly v0.8.16 source computes the cache limit from EARTHLY_TMP_DIR filesystem size.
- The documented 1.7 TB host leaves approximately 1.36 TB outside the nominal 340 GB cache.
- CI includes cleanup for local-environment resources but no general disk-capacity preflight.
Related files:
.earthly/config.selfhosted.yml,.envrc,.github/workflows/continuous-integration.yml
Issues
No medium/high-confidence runtime issues were found.
Summary
- Status: warned
- Areas investigated: 3
- Tasks/probes performed: 4
- Issues reported: 0
- Publishable issues: 0
- Inline comments: 0
- File-level comments: 0
- Review comments: 1
- Unanchored publishable findings posted as file-level comments: 0
- Blocked validations: 1
- Dashboard: full runtime review
- Commit: f3524a5
- Changed files: 1
- Tool calls: none
Signed-off-by: Giles Cope <gilescope@gmail.com>
Jina ReviewJina has completed this review. Review: https://app.usejina.com/reviews/1ae8a929-6585-48c1-9d0c-42a9bc33a867
|
There was a problem hiding this comment.
Runtime Review - Merge Readiness 5/5
0 issues found - Ready to merge
Ready to merge.
No accepted candidate issues remain; the investigation outputs were incomplete, but direct source and diff review found no production-relevant defect.
Final review: No publishable runtime issues were identified.
Source review confirms cache scoping, argument routing, bounded retries, and exit-status propagation.
Issues
No medium/high-confidence runtime issues were found.
Summary
- Status: warned
- Areas investigated: 4
- Tasks/probes performed: 0
- Issues reported: 0
- Publishable issues: 0
- Inline comments: 0
- File-level comments: 0
- Review comments: 0
- Unanchored publishable findings posted as file-level comments: 0
- Blocked validations: 0
- Dashboard: full runtime review
- Commit: 5f15886
- Changed files: 5
- Tool calls: none
3 participants
Overview
Lowers
cache_size_pctin.earthly/config.selfhosted.ymlfrom 50 to 20. At 50, the shared earthly-buildkitd on the self-hosted runner host (fsn1-runner-01, 1.7 TB disk) is allowed ~850 GB of cache and was observed at 889 GB with the disk at 99% (2026-07-13), failing CI jobs with "No space left on device". 20 targets ~340 GB, leaving headroom for CI images and the local GHA cache server on the shared disk.cache_size_pctis an earthly settings-hash key shared with the host-level config the SRE ansible role writes on the runner. The twoglobalblocks must stay byte-identical or every earthly invocation restarts the shared buildkitd and cancels concurrent builds mid-run. SRE will converge the host right after both PRs merge; the hash change restarts the shared buildkitd once.🗹 TODO before merging
📌 Submission Checklist
git commit -s) for the DCO🧪 Testing Evidence
Config-only change to the self-hosted earthly config. Verified the resulting
globalblock stays byte-identical to the host-level config in shieldedtech/shielded-iac#1815 (settings-hash parity). Buildkit cache behaviour at the new cap is enforced lazily by the daemon after the host converges.🔱 Fork Strategy