[Bug] Migration guide step order causes controller crash when upgrading across versions that introduce new CRDs

[jinjiaKarl]

What are you really trying to do?

Upgrading the temporal-worker-controller Helm chart from v0.14.0 to v0.24.0, following the Migration Guide for Existing Users.

Describe the bug

The migration guide instructs users to upgrade the controller chart before installing the CRDs chart (Step 1). This causes the controller to crash-loop when the target version introduces new CRDs that didn't exist in the source version.

  {"level":"error","ts":1775543722.2208555,"logger":"setup","msg":"unable to create
  controller","controller":"TemporalWorkerDeployment","error":"no matches for kind \"WorkerResourceTemplate\" in version
  \"temporal.io/v1alpha1\"","stacktrace":"main.main\n\t/home/runner/work/temporal-worker-controller/temporal-worker-controller/cmd/main.go:9
  7\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.8/x64/src/runtime/proc.go:285"}

Probably reorder the migration steps to: stamp labels → install CRDs chart → upgrade controller chart.

Minimal Reproduction

Environment/Versions

• OS and processor: [e.g. M1 Mac, x86 Windows, Linux]
• Temporal Version: [e.g. 1.14.0?] and/or SDK version
• Are you using Docker or Kubernetes or building Temporal from source?

Additional context

[carlydf]

Ah yeah, we only tested this going from chart versions that mix the CRDs into the main chart (temporal-worker-controller <=v0.21.0) to the first chart version with separate CRDs (v0.22.0). Because v0.22.0 does not introduce any new CRDs, this works. My recommendation is for people to migrate first from the earlier integrated-crds-chart to v0.22.0, and then upgrade one minor version at a time from there onward.

In general, we do upgrade-downgrade testing across each minor version boundary, so our recommendation in production is to upgrade or downgrade one minor version at a time instead of jumping forward many versions at once.

[carlydf]

Did the migration steps you proposed here work for you?

Probably reorder the migration steps to: stamp labels → install CRDs chart → upgrade controller chart.

[jinjiaKarl]

Ah yeah, we only tested this going from chart versions that mix the CRDs into the main chart (temporal-worker-controller <=v0.21.0) to the first chart version with separate CRDs (v0.22.0). Because v0.22.0 does not introduce any new CRDs, this works. My recommendation is for people to migrate first from the earlier integrated-crds-chart to v0.22.0, and then upgrade one minor version at a time from there onward.

In general, we do upgrade-downgrade testing across each minor version boundary, so our recommendation in production is to upgrade or downgrade one minor version at a time instead of jumping forward many versions at once.

Hi, thanks for replying. The reason I decided to jump to the latest is that we want to try the new WorkerResourceTemplate resources and I see there is no breaking changes.

[jinjiaKarl]

Did the migration steps you proposed here work for you?

Probably reorder the migration steps to: stamp labels → install CRDs chart → upgrade controller chart.

Yes, it worked. Controller starts successfully.