[Bug] Worker Deployment Read API rate limit still exceeded in v1.4.0

[gabriel-yahav]

Description

v1.4.0 release notes mention:

"Omit DescribeVersion calls for drained versions to avoid hitting RPS limits"

However, I am still hitting Worker Deployment Read API rate limit errors in v1.4.0.
The error is coming from DescribeWorkerDeployment (the top-level deployment describe),
not from DescribeVersion on individual versions.

Error

"error":"unable to get Temporal worker deployment state: unable to describe worker
deployment X/y: Worker Deployment Read API rate limit exceeded for namespace "ABCD""

The stacktrace points to the standard reconciler loop in controller-runtime,

Root Cause (suspected)

The v1.4.0 optimization skips DescribeVersion for drained versions, which reduces
per-version calls. However, DescribeWorkerDeployment is called unconditionally on
every reconcile iteration regardless of the deployment's state. With multiple
TemporalWorkerDeployment CRs in the same Temporal namespace and a short reconcile
interval, these calls aggregate and exceed the APS (actions-per-second) limit on
Temporal Cloud.

[carlydf]

Hi @gabriel-yahav, thank you for reporting this! Right now, when the reconcile loop gets an error, it retries immediately instead of waiting for the usual delay between successful reconciles. I think that we should detect rate limit / resource exhausted errors and handle them differently than other errors -- retrying with a backoff instead of immediately.