Fix scheduler crash pickling in-cluster pod_override onto executor queue - #71344
Open
roshanprabu wants to merge 1 commit into
Open
Fix scheduler crash pickling in-cluster pod_override onto executor queue#71344roshanprabu wants to merge 1 commit into
roshanprabu wants to merge 1 commit into
Conversation
In-cluster, kubernetes-client 36.x's default Configuration (set by InClusterConfigLoader) carries a refresh_api_key_hook local closure. Any V1Pod/nested model built without an explicit local_vars_configuration picks up that closure via Configuration.get_default_copy() -- which is exactly what happens to a task's pod_override, since user DAG code never passes local_vars_configuration explicitly. execute_async then embeds that pod_override in a KubernetesJob and puts it on task_queue, a multiprocessing.Manager().JoinableQueue(). The manager's put() pickles the object to send it over IPC, and pickle can't serialize the closure -- crashing the scheduler process itself (not just failing the task) on every task with a pod_override, in a loop, since the same task gets re-queued after the scheduler restarts. PodGenerator.deserialize_model_dict already builds V1Pod objects against a fresh Configuration() specifically to avoid capturing the process-global in-cluster default (its docstring says as much), but execute_async never routed pod_override through it. Round-tripping kube_executor_config through PodGenerator.serialize_pod + deserialize_model_dict wires up that existing protection at the one call site that was missing it.
roshanprabu
requested review from
hussein-awala,
jedcunningham and
jscheffl
as code owners
August 9, 2026 08:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #68827.
In-cluster, kubernetes-client 36.x's default
Configuration(set byInClusterConfigLoader) carries arefresh_api_key_hooklocal closure. Every openapi-generated model's__init__callsConfiguration.get_default_copy()whenlocal_vars_configurationisn't explicitly passed:A task's
pod_override(aV1Podbuilt by user DAG code, which never passeslocal_vars_configuration) therefore inherits that closure in-cluster.execute_asyncembeds it in aKubernetesJoband puts it onself.task_queue, amultiprocessing.Manager().JoinableQueue()-- the manager pickles the object to send it over IPC, andpicklecan't serialize a closure. This crashes the scheduler process itself, not just the task, and since the task gets re-queued on restart, it loops.The fix wires up protection that already exists in this codebase but wasn't used at this call site:
PodGenerator.deserialize_model_dictbuildsV1Podobjects against a fresh, emptyConfiguration()specifically so neither the pod nor any nested model captures the process-global in-cluster default -- its docstring says as much verbatim.execute_asyncnever routedpod_overridethrough it before constructingKubernetesJob. This PR rounds it throughPodGenerator.serialize_pod+deserialize_model_dictright afterPodGenerator.from_obj, so the queued object is always built from a freshConfiguration.Scope note: the linked issue also mentions the same fix would help
OAuth2.py-style curl callers elsewhere and suggests a broaderCURLOPT_PROTOCOLSpass -- not applicable here, that's from a different (Appwrite) advisory I worked on separately. This PR only touches theexecute_async→task_queue.put()path described in #68827.Test plan
Configurationwith an unpicklablerefresh_api_key_hook(mirroringInClusterConfigLoader), set it as the process default, built aV1Podthe way user code would, and confirmedpickle.dumps()fails withAttributeError: Can't get local object ...-- then confirmedPodGenerator.serialize_pod+deserialize_model_dictproduces a pod that pickles/unpickles cleanly with the pod spec intact.test_execute_async_pod_override_is_picklable_in_cluster, which reproduces the same in-clusterConfigurationand calls the realexecute_async→task_queue.put()path (the actual productionmultiprocessing.ManagerIPC pickling, not a syntheticpickle.dumps()check).self.task_queue.put(job)with the exact sameAttributeError; re-applied the fix and confirmed it passes.test_kubernetes_executor.pyfile: 177 passed, 1 skipped (pre-existing, version-gated skip unrelated to this change) -- no regressions.ruff checkandruff format --checkpass on both changed files.