[BUG]: asyncio create_task wrapping breaks anyio cancel scope handling (CancelledError leak)

[fgeracitano]

Tracer Version(s)

4.5.4 (also tested with 4.6.1 — same issue)

Python Version(s)

3.14.2

Pip Version(s)

uv 0.7.x (using uv instead of pip)

Bug Report

ddtrace's asyncio integration wraps coroutines passed to asyncio.BaseEventLoop.create_task() with a traced_coro wrapper in ddtrace/contrib/internal/asyncio/patch.py. This extra coroutine frame breaks anyio's CancelScope handling, causing CancelledError to leak out of task groups during normal operation.

The most visible symptom: first HTTP request via httpx returns 500 after every deployment, because httpx → httpcore → anyio connect_tcp() uses the Happy Eyeballs algorithm (RFC 6555) which races connections in a task group and cancels losers via cancel scope.

How it happens

1. httpx makes a request → httpcore calls anyio.connect_tcp()
2. connect_tcp() resolves DNS, gets multiple addresses (IPv4/IPv6 or multiple A records)
3. Happy eyeballs spawns connection attempts via tg.start_soon() → internally calls asyncio.create_task()
4. ddtrace's _wrapped_create_task wraps each coroutine in traced_coro (adds extra coroutine frame)
5. When the first connection wins, tg.cancel_scope.cancel() cancels the losers
6. The CancelledError does not propagate cleanly through the traced_coro wrapper — it leaks out of the task group
7. Error propagates up through middleware → 500

Subsequent requests work because httpx reuses pooled connections, bypassing connect_tcp().

The wrapping code causing the issue

# ddtrace/contrib/internal/asyncio/patch.py, line 62-66
async def traced_coro(*args_c, **kwargs_c):
    if dd_active != tracer.current_trace_context():
        tracer.context_provider.activate(dd_active)
    core.dispatch("asyncio.execute_task", (task_data,))
    return await coro

This wrapper adds an extra coroutine frame between the original coroutine and the task scheduler. When anyio cancels a task via cancel scope, the CancelledError doesn't propagate correctly through this extra frame.

Reproduction

Reliably reproduces on ECS (Linux, dual-stack VPC networking where DNS returns multiple addresses). Hard to reproduce locally due to Docker Desktop not providing real dual-stack DNS.

Confirmed fixes:

• Removing ddtrace-run entirely → fixed (no create_task wrapping)
• Removing the traced_coro wrapper from _wrapped_create_task → fixed (keeping ddtrace patch active but not wrapping coroutines)
• Replacing httpx with aiohttp → fixed (aiohttp uses its own TCP stack, no anyio connect_tcp)
• Disabling happy eyeballs in anyio connect_tcp() → fixed (no task group, no cancel scopes)

Did not fix:

• DD_PATCH_MODULES=httpx:false — the httpx patch is not the culprit
• Upgrading ddtrace to 4.6.1
• Vendoring anyio with the CancelScope.__exit__ fix from anyio PR #1092

Impact

This affects any code path where asyncio.create_task() is called internally by a library that then cancels tasks as part of normal operation. Beyond anyio/httpx, this could affect asyncio.TaskGroup with cancellation, timeout patterns, and connection pools.

Suggested Fix

Consider one of:

1. Skip wrapping internal library coroutines — detect coroutines from anyio/httpcore and pass through
2. Use contextvars for trace propagation instead of wrapping the coroutine in an extra async def
3. Ensure CancelledError is fully transparent through traced_coro

Related Issues

• Unexpected Cancellation propagation when using sse-starlette, fastapi, uvloop, and ...gevent? #10550 — Similar CancelledError issue but gevent-specific
• anyio #1091 — Related CancelScope filtering issue (fixing it did NOT resolve this bug)

[Mkashiii]

if i solve this problem for you will you pay me

[fgeracitano]

if i solve this problem for you will you pay me

sure here you go 💸

[fgeracitano]

Any help on this?

[Mkashiii]

Yes why not let me check *Muhammad Kashif* Python Automation & Web Infrastructure Engineer Web Scraping • SEO Automation • Server Management 📧 ***@***.*** 📱 +92 317 133 8662 GitHub: https://github.com/Mkashii LinkedIn: https://www.linkedin.com/in/kashif-fazal/

…

On Thu, 26 Mar 2026, 6:42 pm rgg-fgeracitano, ***@***.***> wrote: *fgeracitano* left a comment (DataDog/dd-trace-py#17035) <#17035 (comment)> Any help on this? — Reply to this email directly, view it on GitHub <#17035?email_source=notifications&email_token=BMTNGF23CUVV4BRCVYVE2ID4SUXUPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJTGQ4TEOBZGYZ2M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4134928963>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BMTNGFZM2JFVUXOAJVNQBU34SUXUPAVCNFSM6AAAAACWY5XZDWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMZUHEZDQOJWGM> . You are receiving this because you commented.Message ID: ***@***.***>

[Mkashiii]

The fix needs to ensure CancelledError is fully transparent through the wrapper. Here's the approach:The core issue here is that traced_coro catches CancelledError but doesn't re-raise it with the proper asyncio semantics that anyio's cancel scopes expect. Let me diagram the failure path and then walk through the fix.
::view-transition-group(),
::view-transition-old(),
::view-transition-new(*) {
animation-duration: 0.25s;
animation-timing-function: cubic-bezier(0.19, 1, 0.22, 1);
}
VvisualizeVvisualize show_widget
Root cause
The problem is subtle. When anyio's cancel scope cancels a task, it expects CancelledError to propagate with the cancellation request's scope identity preserved. The traced_coro wrapper inserts a frame that can inadvertently swallow or corrupt the CancelledError in a few ways:

Implicit swallowing — if an exception occurs before return await coro (e.g., inside core.dispatch()), Python 3.11+ ExceptionGroup handling may transform the cancellation path.
Missing shield semantics — traced_coro doesn't explicitly re-raise BaseException subclasses (which CancelledError is, as of Python 3.8), so any non-Exception exception hitting the frame boundary may be absorbed differently depending on the event loop implementation.
Frame boundary under asyncio.Task — Python's Task.__step calls coro.throw(CancelledError(...)) on the outermost frame. With traced_coro as the outermost frame, anyio's CancelScope.exit never gets the chance to inspect the cancellation token — it exits too late.

The fix
The key principle: CancelledError (and all BaseException subclasses) must be transparently re-raised through any wrapper frame. There are three tiers to this fix depending on what ddtrace is willing to change.
Option 1 — Minimal, targeted fix (preferred)
Make traced_coro explicitly transparent to BaseException:
pythonasync def traced_coro(*args_c, **kwargs_c):
if dd_active != tracer.current_trace_context():
tracer.context_provider.activate(dd_active)
# Dispatch before awaiting — but guard so cancellation pre-dispatch still propagates
try:
core.dispatch("asyncio.execute_task", (task_data,))
except BaseException:
raise # never swallow BaseException from dispatch itself
try:
return await coro
except BaseException as exc:
# CancelledError (and any BaseException) must propagate without modification.
# Do NOT wrap, log, or transform it — anyio cancel scopes depend on identity.
raise
This is essentially a no-op structurally, but it makes the intent explicit and prevents any future change from adding a bare except Exception that would accidentally stop CancelledError from propagating as BaseException.
However, this alone may not be enough if the issue is the frame count rather than exception handling. See Option 2.

Option 2 — The real fix: propagate via contextvars instead of frame wrapping
The traced_coro wrapper exists to propagate the trace context into the new task. This is the wrong tool for the job. asyncio.Task already copies the current contextvars.Context at creation time — so the right approach is to put the trace context into a contextvar before scheduling, not wrap the coroutine:
python# ddtrace/contrib/internal/asyncio/patch.py

_dd_trace_ctx: contextvars.ContextVar = contextvars.ContextVar('_dd_trace_ctx', default=None)

def _wrapped_create_task(wrapped, instance, args, kwargs):
task = wrapped(*args, **kwargs) # create the task FIRST, unwrapped

# Inject trace context via the task's context snapshot
# asyncio.Task copies the current Context at creation — we set our contextvar
# before the call so it's captured automatically.
# If we need to set it after the fact, use task.get_context().run():
dd_active = tracer.current_trace_context()
if dd_active is not None:
    task_data = ...  # whatever span data is needed
    # Schedule trace activation as a done callback instead of a wrapper frame
    def _activate_on_start(_):
        pass  # trace context already in contextvars; nothing needed here
    # Or: set the contextvar in the task's own context:
    task.get_context().run(_dd_trace_ctx.set, dd_active)

return task

And inside the tracer's context provider, read from _dd_trace_ctx instead of thread-local or wrapper-frame state.
This eliminates traced_coro entirely. No extra frame, no exception path to worry about. The contextvars.Context snapshot that asyncio.Task takes at creation time carries the trace context into the task natively.

Option 3 — Targeted shield for the CancelledError case (stop-gap)
If refactoring to contextvars isn't immediately feasible, add explicit CancelledError transparency using asyncio.current_task():
pythonimport asyncio

async def traced_coro(*args_c, **kwargs_c):
if dd_active != tracer.current_trace_context():
tracer.context_provider.activate(dd_active)
core.dispatch("asyncio.execute_task", (task_data,))
try:
return await coro
except asyncio.CancelledError:
# Re-raise immediately without any processing.
# Do not log, do not finish spans here — the task was cancelled,
# not completed. Span finishing should happen in a done_callback instead.
raise
except Exception:
# Normal exception path for span error recording
raise
And move span finishing to a Task.add_done_callback rather than the except block — done callbacks fire after the task fully unwinds, so anyio's cancel scope has already had its chance to run exit by then:
pythondef _wrapped_create_task(wrapped, instance, args, kwargs):
dd_active = tracer.current_trace_context()
coro = args[0] if args else kwargs.get('coro')
task_data = ...

async def traced_coro():
    if dd_active != tracer.current_trace_context():
        tracer.context_provider.activate(dd_active)
    core.dispatch("asyncio.execute_task", (task_data,))
    return await coro  # BaseException propagates freely

new_args = (traced_coro(),) + args[1:]
task = wrapped(*new_args, **kwargs)

def _on_done(fut):
    # Safe place to finish spans — cancel scope has already exited
    core.dispatch("asyncio.finish_task", (task_data, fut))

task.add_done_callback(_on_done)
return task