Skip to content

Fix scheduler crash pickling in-cluster pod_override onto executor queue - #71344

Open
roshanprabu wants to merge 1 commit into
apache:mainfrom
roshanprabu:fix-k8s-executor-config-pickling
Open

Fix scheduler crash pickling in-cluster pod_override onto executor queue#71344
roshanprabu wants to merge 1 commit into
apache:mainfrom
roshanprabu:fix-k8s-executor-config-pickling

Conversation

@roshanprabu

Copy link
Copy Markdown

Summary

Fixes #68827.

In-cluster, kubernetes-client 36.x's default Configuration (set by InClusterConfigLoader) carries a refresh_api_key_hook local closure. Every openapi-generated model's __init__ calls Configuration.get_default_copy() when local_vars_configuration isn't explicitly passed:

def __init__(self, ..., local_vars_configuration=None):
    if local_vars_configuration is None:
        local_vars_configuration = Configuration.get_default_copy()
    self.local_vars_configuration = local_vars_configuration

A task's pod_override (a V1Pod built by user DAG code, which never passes local_vars_configuration) therefore inherits that closure in-cluster. execute_async embeds it in a KubernetesJob and puts it on self.task_queue, a multiprocessing.Manager().JoinableQueue() -- the manager pickles the object to send it over IPC, and pickle can't serialize a closure. This crashes the scheduler process itself, not just the task, and since the task gets re-queued on restart, it loops.

The fix wires up protection that already exists in this codebase but wasn't used at this call site: PodGenerator.deserialize_model_dict builds V1Pod objects against a fresh, empty Configuration() specifically so neither the pod nor any nested model captures the process-global in-cluster default -- its docstring says as much verbatim. execute_async never routed pod_override through it before constructing KubernetesJob. This PR rounds it through PodGenerator.serialize_pod + deserialize_model_dict right after PodGenerator.from_obj, so the queued object is always built from a fresh Configuration.

Scope note: the linked issue also mentions the same fix would help OAuth2.py-style curl callers elsewhere and suggests a broader CURLOPT_PROTOCOLS pass -- not applicable here, that's from a different (Appwrite) advisory I worked on separately. This PR only touches the execute_asynctask_queue.put() path described in #68827.

Test plan

  • Reproduced the exact crash locally: constructed a Configuration with an unpicklable refresh_api_key_hook (mirroring InClusterConfigLoader), set it as the process default, built a V1Pod the way user code would, and confirmed pickle.dumps() fails with AttributeError: Can't get local object ... -- then confirmed PodGenerator.serialize_pod + deserialize_model_dict produces a pod that pickles/unpickles cleanly with the pod spec intact.
  • Added test_execute_async_pod_override_is_picklable_in_cluster, which reproduces the same in-cluster Configuration and calls the real execute_asynctask_queue.put() path (the actual production multiprocessing.Manager IPC pickling, not a synthetic pickle.dumps() check).
  • Verified the new test actually catches the bug: reverted the source fix and confirmed the test fails at self.task_queue.put(job) with the exact same AttributeError; re-applied the fix and confirmed it passes.
  • Ran the full test_kubernetes_executor.py file: 177 passed, 1 skipped (pre-existing, version-gated skip unrelated to this change) -- no regressions.
  • ruff check and ruff format --check pass on both changed files.

In-cluster, kubernetes-client 36.x's default Configuration (set by
InClusterConfigLoader) carries a refresh_api_key_hook local closure.
Any V1Pod/nested model built without an explicit local_vars_configuration
picks up that closure via Configuration.get_default_copy() -- which is
exactly what happens to a task's pod_override, since user DAG code never
passes local_vars_configuration explicitly.

execute_async then embeds that pod_override in a KubernetesJob and puts
it on task_queue, a multiprocessing.Manager().JoinableQueue(). The
manager's put() pickles the object to send it over IPC, and pickle can't
serialize the closure -- crashing the scheduler process itself (not just
failing the task) on every task with a pod_override, in a loop, since
the same task gets re-queued after the scheduler restarts.

PodGenerator.deserialize_model_dict already builds V1Pod objects against
a fresh Configuration() specifically to avoid capturing the process-global
in-cluster default (its docstring says as much), but execute_async never
routed pod_override through it. Round-tripping kube_executor_config
through PodGenerator.serialize_pod + deserialize_model_dict wires up
that existing protection at the one call site that was missing it.
@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesExecutor scheduler crashes with PicklingError on pod_override with kubernetes client 36.x

1 participant