README: ArgoCD example silently drops Helm pod-spec changes (orgCABundle, sidecars, env, volumes)

[oren-openteams]

Summary

The README's On a Nebari cluster (via ArgoCD) section recommends an Argo Application configuration that combines two settings whose interaction silently drops Helm-rendered changes to head/worker pod specs:

• syncOptions: [..., RespectIgnoreDifferences=true]
• ignoreDifferences: [{ group: ray.io, kind: RayService, jsonPointers: [/spec/rayClusterConfig, /status] }]

When both are set, Argo's server-side apply skips every field under /spec/rayClusterConfig on the patch path. That includes the head and worker pod templates, where:

• orgCABundle initContainers / volumes / env / volumeMounts (introduced in feat(ca-bundle): inject org CA into Ray head + worker pods via ConfigMap (#15) #16) live
• Any head.containerEnv / worker.containerEnv the user sets via values lives
• Any sidecar / runtimeClassName / nodeSelector / tolerations / resources override lives

Net effect: Argo reports Synced, the rendered chart looks right under helm template, but the live RayService head template has only containers — none of the Helm-set additions reach the cluster.

How we hit it

Just integrated #16 into an Argo gitops deployment using the README's recommended ignoreDifferences block verbatim. The CA ConfigMap mounted fine, the chart re-rendered cleanly with orgCABundle.configMapName: org-ca-bundle, but the head/worker pods never picked up SSL_CERT_FILE. Running:

kubectl get rayservice -o json | jq '.items[0].spec.rayClusterConfig.headGroupSpec.template.spec | keys'

returned ["containers"] — no initContainers, no volumes — despite Argo being on the correct chart SHA and showing Synced.

Took an hour of bisecting (Argo App status, chart re-render verification, live spec inspection) to land on the syncOptions × ignoreDifferences interaction. Dropping RespectIgnoreDifferences=true immediately fixed it — Argo now applies the full rayClusterConfig, drift detection still honors the ignore rules, no permanent OutOfSync.

Background on the trap

ArgoCD docs make this explicit but easy to miss: with SSA, RespectIgnoreDifferences=true extends ignoreDifferences from a comparison-time concept (don't flag drift on this field) to an apply-time concept (don't actually push this field to the server). The README's example sets both — which is the foot-gun.

Proposed fix

Cheapest possible: add a warning to the Important: bullet list immediately below the example, around README.md line ~120:

• Do not combine RespectIgnoreDifferences=true (in syncOptions) with the broad /spec/rayClusterConfig entry in ignoreDifferences. That combination tells Argo's server-side apply to skip every field under rayClusterConfig on the patch path, silently dropping Helm-rendered changes to head/worker pod templates (env vars, volumes, initContainers, sidecars, orgCABundle, runtimeClassName, etc.). Argo will report Synced and the chart will render correctly under helm template, but the live RayService will not contain the changes. The example above omits RespectIgnoreDifferences=true for this reason — keep it omitted unless you have scoped the ignoreDifferences jsonPointers to specific sub-paths that KubeRay actually mutates (typically only /status and child resources).

…and remove - RespectIgnoreDifferences=true from the recommended syncOptions block in the YAML example.

Stretch (separate issue/PR — not blocking the warning)

The current ignoreDifferences entry of /spec/rayClusterConfig is broader than necessary. KubeRay's controller doesn't typically mutate fields under /spec/rayClusterConfig — its mutations land on /status (already ignored) and on child resources (Service, Pod, RayCluster), some of which are covered by the Service rule. If someone can enumerate the actual mutation set, the rule can be tightened to specific paths — and RespectIgnoreDifferences=true becomes safe to add back if needed for stricter SSA semantics.

Acceptance

• README's ArgoCD example no longer sets RespectIgnoreDifferences=true
• Important: bullet list includes the warning above (or equivalent)
• Optional: link from the warning to ArgoCD's diffing docs section