feat: cooperative scheduling — close the four CPython divergences#59
Merged
Conversation
Phase 1 shipped a synchronous trampoline that ran coroutines to
completion in declared order — correct value, divergent timing.
This converts coroutines to tagged generators and rebuilds the
trampoline cooperatively, so observable behavior matches CPython
for the patterns LLMs emit.
The discovery: Pyex's existing :lazy_iter generator continuation
machinery was the right substrate. Coroutines aren't a new
runtime construct — they're sync generators with a kind flag,
PEP 380 yield-from semantics over the inner iterator, and a
trampoline that interprets known sentinels.
## What changed mechanically
- `Ctx` grows two iter-pool entries:
* `{:gen_unstarted, body, env}` — coroutine staged but not
run. CPython parity: calling an async def does NOT execute
the body; the body runs when something drives the iterator.
* `{:gen_done, value}` (alongside the existing `:gen_done`
atom) — captures the body's return value at exhaustion so
`await` can surface it as the expression's result (PEP 380
`StopIteration(value)`).
- `resume_generator` returns `{:done_with_value, v}` instead of
dropping the value when a body falls through `{:returned, v}`.
Plumbed through `step_via_continuation`, `step_via_send`,
`cont_yield_from_iter`, `cont_for_gen_iter`, and the drain
helpers. Existing `:done` matchers keep working — added clauses
catch `:done_with_value`.
- `Invocation.build_coroutine` no longer eval-runs the body — it
builds a `:gen_unstarted` entry. The first `await` (or
`asyncio.run`) drives it via `run_unstarted`.
- `eval({:await, ...})` now calls `Invocation.initiate_await`,
which advances one step, propagates yields up with a
`:cont_await_iter` frame, and surfaces the inner's return value
as the result. On resume, the frame advances the same iterator
and uses `resume_generator_with_send` to land the return value
via `:cont_bind_sent` if the await is inside an assign
(`r = await coro`).
- `asyncio.sleep(t)` builds a single-yield coroutine that surfaces
an `{:asyncio_sleep, ms}` sentinel. The trampoline interprets
the sentinel; coroutines yield it up through their await chain.
- `asyncio.gather` is a round-robin trampoline implemented in
Elixir: advance each pending child one step per round, sleep
inline on sleep sentinels, collect results in declared order.
Returns a Task so `await asyncio.gather(...)` works against
strict await.
- `asyncio.create_task` becomes lazy — wraps the coroutine in
`{:asyncio_task_pending, coro}` instead of driving immediately.
`await pending_task` drives the wrapped coroutine. Pending
Tasks expose `.done() -> False`, `.result()` raises
`InvalidStateError`, `.cancel() -> True`, etc.
- `asyncio.run` tracks a `ctx.asyncio_running` flag. Nested
`asyncio.run()` raises `RuntimeError` matching CPython's
"asyncio.run() cannot be called from a running event loop".
- Parser accepts `async for` inside list/set/dict comprehensions
and generator expressions. AST shape is identical to sync
comprehensions (Pyex iterates async generators via the same
lazy_iter path).
## Divergences closed
The four `@tag :phase1_divergence` tests are flipped to assert
CPython parity:
- gather interleaves children at await points (ABABAB)
- create_task defers driving until awaited
- nested asyncio.run raises RuntimeError
- `[x async for x in g()]` parses and runs
5346 tests, 0 failures. Dialyzer clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
32cd5e8 to
73415e9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on top of #58. Closes the four documented divergences from CPython by converting coroutines to tagged generators driven by a cooperative trampoline.
The discovery
Pyex already had what cooperative scheduling needs: the `:lazy_iter` generator continuation infrastructure (used by sync generators, the islice fix, FastAPI streaming). The Phase 1 "sync trampoline" was actually a regression from the existing primitives — it ran coroutines to completion instead of letting them suspend.
The Phase 1.5 model is the natural one PEP 380 + 492 specify: coroutines are sync generators with a `kind` flag, `await EXPR` is yield-from over the inner iterator, and the trampoline interprets known sentinels. Same machinery for sync generators and coroutines, just driven by different surfaces.
What ships
```python
import asyncio
trace = []
async def step(label):
for _ in range(3):
trace.append(label)
await asyncio.sleep(0)
async def main():
await asyncio.gather(step("A"), step("B"))
return "".join(trace)
asyncio.run(main())
'ABABAB' — matches CPython, was 'AAABBB' in #58
```
The four divergences from #58, all closed:
The four `@tag :phase1_divergence` tests in the conformance suite are flipped to assert CPython parity instead of pinning divergence. When you read those tests, they're now positive specifications — what Pyex does — not regrets.
Mechanically
Test plan
Why I'm proud of this one
The cooperative trampoline is ~100 lines of Elixir. It works because Pyex's design is right — generators-as-continuations is the substrate, and async/await sits on top of it in the way PEP 492 originally specified. Building a real event loop on the BEAM normally means processes and message passing; this one is pure functional code that the `BannedCallTracer` accepts unchanged.
🤖 Generated with Claude Code