Migrate integration test to action-nebari-sandbox (fixes Docker Hub timeout)

[oren-openteams]

Summary

Migrate test-integration.yaml from its bespoke, hand-rolled platform bootstrap to the shared nebari-dev/action-nebari-sandbox action (platform profile) + the add-software-pack sub-action — the same pattern already used by nebari-nebi-pack and nebari-data-science-pack.

This repo is currently the outlier: it manually installs MetalLB, Envoy Gateway, cert-manager, Keycloak (via nebari-operator/dev/scripts/services/install.sh) and builds the nebari-operator from source, where the sibling packs get all of that from the shared action.

Motivation

The integration test is broken repo-wide and has been since ~2026-06-12:

Date	Branch	Duration	Result
May 14 – Jun 11	main & PR branches	~7–8 min	✅
Jun 12 onward	every branch (incl. unrelated PRs)	30m	❌ timeout

Root cause: the Install foundational services step (install.sh) helm installs Envoy Gateway, whose OCI chart pull from Docker Hub now blocks ~20 minutes (Docker Hub anonymous-pull throttling on shared GitHub-runner IPs), consuming the entire 30-min job budget before cert-manager or the chart ever install. Re-running does not help — it's deterministic, not flaky.

The shared action sidesteps this entirely: it provisions the foundational stack via NIC (median wall-clock is tracked in the action's README and is in the single-digit minutes), and pulls the operator image from quay.io, not Docker Hub.

Research / findings

The platform profile ships everything we currently hand-build. Confirmed in nebari-infrastructure-core:

• pkg/argocd/templates/apps/ includes cert-manager, envoy-gateway, keycloak, metallb, and nebari-operator.
• nebari-operator deploys from github.com/nebari-dev/nebari-operator.git/config/default?ref=v0.1.0-alpha.20, image quay.io/nebari/nebari-operator:0.1.0-alpha.20, into namespace nebari-operator-system — exactly the namespace/deployment our current kubectl wait targets. So the NebariApp CRD + controller are present out of the box.

What the migration deletes (≈90 lines of bootstrap):

• Create kind cluster → replaced by k3d via the action
• Install MetalLB + Configure IP pool → deleted; k3d's klipper servicelb provides the gateway-ip output
• Install foundational services (the 20-min hang) → provided by NIC
• Setup Keycloak realm → provided by NIC
• Build & load nebari-operator from source + deploy + wait → platform ships it

What carries over unchanged: all the verification steps (NebariApp Ready, HTTPRoutes ≥ 2, SecurityPolicies ≥ 2, dashboard port-forward health check) and the Debug-on-failure block.

Prerequisite: deterministic KubeRay CRD ordering (chart change)

The chart defaults kuberay-operator.enabled: true, so the KubeRay subchart's CRDs (crds/) and the RayService CR install in the same ArgoCD Application. Today's CI sidesteps ordering by helm install-ing KubeRay separately first; the GitOps path can't do that as cleanly.

Add a sync-wave to the RayService so it applies one wave after the CRDs + operator (wave 0). ArgoCD waits for a wave to be Healthy before the next, so the rayservices.ray.io CRD is Established and the controller is running before the CR is applied — deterministic, instead of leaning on SkipDryRunOnMissingResource + sync retries to absorb a first-sync race.

# chart/templates/rayservice.yaml
   annotations:
     argocd.argoproj.io/compare-options: IgnoreExtraneous
+    argocd.argoproj.io/sync-wave: "1"

Prototyped and verified: RayService renders with sync-wave: "1", the 3 KubeRay CRDs render at wave 0 (via --include-crds), helm lint passes. This is a standalone improvement that also hardens real production GitOps deploys.

Implementation sketch

1. chart/templates/rayservice.yaml — add the sync-wave annotation (prerequisite above).
2. New .github/ci/rayserve-application.yaml — ArgoCD Application modeled on nebi-pack-application.yaml: repoURL: "file://${GITOPS_DIR}", path: rayserve, helm values: with nebariapp.enabled=true, hostnames, and downsized resources.
3. Rewrite test-integration.yaml:
   • nebari-dev/action-nebari-sandbox@<sha> (pin to a v2 commit) with profile: platform
   • surface KUBECONFIG from the kubeconfig output
   • add-software-pack with app-name: rayserve, chart-source: ./chart (or an rsync-staged copy excluding .git, as nebi-pack does), application-manifest: ./.github/ci/rayserve-application.yaml
   • keep the existing readiness + verification + debug steps
   • cleanup: k3d cluster delete ${{ steps.sandbox.outputs.cluster-name }}

Decisions / risks to resolve during implementation

• Version-under-test shift: today we test nebari-operator HEAD; the platform pins v0.1.0-alpha.20. Arguably more correct (validate against the released platform), but a real change. If HEAD coverage is needed, override via nic-config / a custom app.
• helm template vs ArgoCD CRDs: bare helm template does not emit crds/ (needs --include-crds). ArgoCD passes --include-crds, so CRDs are managed there — but document this so future debugging doesn't conclude they're "missing."
• Application Healthy ≠ pods Ready: ArgoCD has no health check for ray.io/RayService, so it reports Healthy on sync, before Ray pods are up. The add-software-pack wait is necessary but not sufficient — keep the explicit kubectl wait pod -l ray.io/node-type=head --for=condition=ready gate.
• Auth-upgrade test: today it's a helm upgrade --set nebariapp.auth.enabled=true. Under GitOps decide between editing the Application's helm values + re-sync, or deploying auth-on from the start.
• Pin the action by commit SHA (siblings do), targeting v2.

Acceptance criteria

• test-integration.yaml uses action-nebari-sandbox (platform) + add-software-pack; no MetalLB/install.sh/operator-build steps remain.
• RayService sync-wave annotation landed; CRDs install before the CR deterministically.
• NebariApp / HTTPRoute / SecurityPolicy / dashboard verifications retained and green.
• Explicit Ray pod-readiness wait retained.
• Wall-clock back in the single-digit-minutes range; no Docker Hub timeout.