Description
Starting in v1.4.0, the controller attempts to patch Kubernetes events for
TemporalWorkerDeployment objects. However, the ClusterRole provisioned by
the Helm chart does not include create/patch verbs on the events resource,
causing the controller to continuously log permission errors and fail to record
events.
Setup
- Controller chart version: 1.4.0
- Controller namespace:
X
- TemporalWorkerDeployment namespace:
Y
The controller is deployed in a different namespace than the TemporalWorkerDeployment
CRs it manages. This is a common and supported pattern (deploying the controller
once cluster-wide and managing CRs in application namespaces).
Error
Server rejected event (will not retry!)" err="events \"dbtworker.18a6745739983ba1\" is forbidden: User \"system:serviceaccount:X:temporal-worker-controller-service-account\" cannot patch resource \"events\" in │
│ API group \"\" in the namespace \"Y\""
Root Cause
The ClusterRole created by the Helm chart is missing events permissions.
The controller-runtime library used by the controller (sigs.k8s.io/controller-runtime@v0.21.0)
records events against the namespace where the watched resource lives (in this case X),
not where the controller itself is deployed.
Since a ClusterRole + ClusterRoleBinding is used (as expected for a cluster-wide
controller), the fix is straightforward — add events to the ClusterRole rules.
Fix
Add to the controller's ClusterRole:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
Workaround
Manually patching the ClusterRole to add the above rule resolves the issue
until a chart fix is released.
Description
Starting in v1.4.0, the controller attempts to patch Kubernetes
eventsforTemporalWorkerDeploymentobjects. However, theClusterRoleprovisioned bythe Helm chart does not include
create/patchverbs on theeventsresource,causing the controller to continuously log permission errors and fail to record
events.
Setup
XYThe controller is deployed in a different namespace than the
TemporalWorkerDeploymentCRs it manages. This is a common and supported pattern (deploying the controller
once cluster-wide and managing CRs in application namespaces).
Error
Root Cause
The
ClusterRolecreated by the Helm chart is missingeventspermissions.The controller-runtime library used by the controller (
sigs.k8s.io/controller-runtime@v0.21.0)records events against the namespace where the watched resource lives (in this case
X),not where the controller itself is deployed.
Since a
ClusterRole+ClusterRoleBindingis used (as expected for a cluster-widecontroller), the fix is straightforward — add
eventsto the ClusterRole rules.Fix
Add to the controller's
ClusterRole:Workaround
Manually patching the
ClusterRoleto add the above rule resolves the issueuntil a chart fix is released.