Skip to content

CNTRLPLANE-3329: Add nightly CronJob to warm EFS-backed Go build cache #8496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 4 commits into from
May 13, 2026
+78 −0
Merged

CNTRLPLANE-3329: Add nightly CronJob to warm EFS-backed Go build cache #8496

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1631c3c
ci(arc-runners): add nightly CronJob to warm EFS-backed Go build cache
vismishr May 12, 2026
9dad273
ci(arc-runners): replace destructive cache wipe with age-based cleanup
vismishr May 12, 2026
c73ce25
ci(arc-runners): add security context to cache warmer pod
vismishr May 12, 2026
ca1f7ee
ci(arc-runners): skip mount root when pruning empty directories
vismishr May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions hack/github-actions-runner/cache-warming-cronjob.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
apiVersion: batch/v1
kind: CronJob
metadata:
name: go-cache-warmer
namespace: arc-runners
labels:
app.kubernetes.io/component: cache-warmer
app.kubernetes.io/part-of: arc-runner-set
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
activeDeadlineSeconds: 3600
backoffLimit: 1
template:
spec:
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: cache-warmer
image: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-gh-actions-runner:latest
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 12, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

# Check if the file exists and examine its structure
find . -name "cache-warming-cronjob.yaml" -type f

Repository: openshift/hypershift

Length of output: 120

🏁 Script executed:

# If found, examine the file around line 23
if [ -f "hack/github-actions-runner/cache-warming-cronjob.yaml" ]; then
  echo "=== File exists. Examining contents ===" && \
  cat -n "hack/github-actions-runner/cache-warming-cronjob.yaml" | head -40
else
  echo "File not found at specified path"
fi

Repository: openshift/hypershift

Length of output: 1659

Avoid mutable :latest for cron execution reproducibility.

Pin this image to a digest to prevent unreviewed runtime drift in nightly jobs. Resolve the actual digest value and update the image reference.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/github-actions-runner/cache-warming-cronjob.yaml` at line 23, The
cronjob is using a mutable image tag
"quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-gh-actions-runner:latest";
resolve the image digest (for example via `skopeo inspect
docker://quay.io/...:latest` or `podman pull` + `podman inspect --format
'{{index .Digest}}'`), then replace the tag with the immutable digest form
`quay.io/redhat-user-workloads/crt-redhat-acm-tenant@sha256:<DIGEST>` in the
image field so the cron execution is reproducible.

Copy link
Copy Markdown
Contributor Author

@vismishr vismishr May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally using :latest here — the CronJob needs to stay in sync with the ARC runner image so the Go build cache is compiled with the same Go version and toolchain the runners use. Pinning to a digest would require a manual update every time the runner image is rebuilt.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vismishr, that's a fair point — since the build cache must be ABI-compatible with the exact toolchain the runners use, keeping both on :latest is a sensible way to guarantee that alignment without manual digest chasing on every runner rebuild. Understood, I'll stand down on this one.

✏️ Learnings added 
Learnt from: vismishr
Repo: openshift/hypershift PR: 8496
File: hack/github-actions-runner/cache-warming-cronjob.yaml:23-23
Timestamp: 2026-05-12T19:47:18.581Z
Learning: In `hack/github-actions-runner/cache-warming-cronjob.yaml` (openshift/hypershift), the CronJob `go-cache-warmer` intentionally uses the mutable `:latest` tag for `quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-gh-actions-runner:latest`. This is by design so the Go build cache is always compiled with the same Go version and toolchain as the ARC runner pods, which also track `:latest`. Pinning to a digest is not desired because it would require a manual update on every runner image rebuild.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
command:
- /bin/bash
- -c
- |
set -euo pipefail

echo "=== Cloning openshift/hypershift main branch ==="
git clone --depth 1 --branch main https://github.com/openshift/hypershift.git /tmp/hypershift
cd /tmp/hypershift

echo "=== Compiling all packages ==="
go build ./...

echo "=== Compiling all test binaries ==="
go test -c -o /dev/null ./... 2>/dev/null || true

echo "=== Cleaning stale cache entries (older than 7 days) ==="
find /cache/go-build -type f -mtime +7 -delete
find /cache/go-build -mindepth 1 -type d -empty -delete

echo "=== Syncing build cache to PV ==="
cp -a "${GOCACHE}"/* /cache/go-build/

Comment thread
coderabbitai[bot] marked this conversation as resolved.
echo "=== Cache warming complete ==="
du -sh /cache/go-build/
env:
- name: GOCACHE
value: /tmp/go-build-cache
- name: GOMODCACHE
value: /tmp/go-mod-cache
- name: HOME
value: /tmp
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
volumeMounts:
- name: go-cache
mountPath: /cache/go-build
volumes:
- name: go-cache
persistentVolumeClaim:
claimName: go-cache-pvc