feat(async): add AsyncClient(max_concurrency=...)

[thomasht86]

⚠️ 🤖
This code was written by Claude (Fable/Opus5) - but understood, tested and vouched for by me.

Problem

AsyncClient dispatches every request with loop.run_in_executor(None, ...). Passing None selects asyncio's default executor, which has two consequences:

1. Concurrency is silently capped at min(32, cpu_count + 4) — 16 on a 12-core machine — no matter how many requests the caller has in flight. This was found from the other direction: in pyvespa, async feeding was measurably slower than a plain thread pool over the sync Client, which should never happen.

2. It monopolises a pool the library doesn't own. asyncio.to_thread resolves to that same executor, so httpr requests contend with every to_thread call in the host application — and conversely, a user's slow blocking work stalls HTTP requests.

Change

Each AsyncClient gets its own ThreadPoolExecutor, sized by a new max_concurrency (default 64).

client = httpr.AsyncClient(max_concurrency=128)   # 128 requests in flight
client = httpr.AsyncClient(max_concurrency=None)  # previous behaviour

Threads are created on demand and ThreadPoolExecutor retires them via a weakref callback once the client is collected, so an idle client costs nothing and there is nothing new to release. close, aclose and __aexit__ keep their existing behaviour, and a client stays reusable across several async with blocks — verified by a test, because downstream users (pyvespa) rely on it.

The diff on the client is three things: strip the new kwarg in __new__ before it reaches the Rust RClient.__new__, build the pool in __init__, and pass it to run_in_executor.

Measurements

Concurrency only pays off when there is latency to hide. Local server, 50 ms latency, 64 requests:

	wall clock	peak in flight
max_concurrency=None (previous behaviour)	316 ms	16
max_concurrency=64	143 ms	64

2.2×. On localhost the same comparison is ~1.0× — there is no latency for concurrency to hide. That is why the regression guard asserts on observed peak concurrency, not wall clock.

Tests

test_concurrency_is_not_capped_by_the_default_executor computes min(32, cpu_count + 4), asks for 16 more than that, and uses a threading.Barrier — so an undersized pool deadlocks and times out rather than merely running slower. Confirmed it fails against the old run_in_executor(None) implementation (5 of 10 tests in the file fail when reverted).

Also adds an async concurrency benchmark group. It runs against the async benchmark server rather than the base_url httpbin fixture: that fixture is a single-threaded WSGI server, so 64 concurrent requests queue on it and the numbers measure the server serialising — my first attempt showed higher concurrency as 20× slower with a standard deviation ~200× the signal, which would have flapped permanently against the 150% alert threshold. Retargeted, it's stable (StdDev ~1 ms) and honest: more threads cost slightly more dispatch overhead and buy nothing on localhost. It tracks overhead; it is not the regression guard.

Compatibility

Checked and safe: subclassing AsyncClient, positional constructor args, thread growth (lazy — 30 clients × 4 concurrent = 61 threads, back to baseline after GC). deepcopy/pickle fail, but they already did on the Rust object — pre-existing, not a regression.

One real behaviour change, so this wants a minor bump and a release note. An app that throttled httpr by shrinking asyncio's default executor loses that throttle. With set_default_executor(ThreadPoolExecutor(4)) and 32 requests:

• before / max_concurrency=None: peak 4 in flight
• after, default: peak 32

Such an app would silently hit its server ~8× harder. max_concurrency=None restores the old behaviour exactly; an explicit value is the better fix. Relatedly, the old default executor was also an accidental global cap shared by every client in a process; the ceiling is now per-client, so worst-case thread use scales with client count.

Not in scope

This does not make AsyncClient natively async — it makes the thread-pool ceiling explicit and tunable. Since reqwest is already async internally (built without the blocking feature; the sync API is py.detach(|| RUNTIME.block_on(future))), the longer-term fix is pyo3-async-runtimes::tokio::future_into_py plus swapping new_current_thread() for new_multi_thread(). That's a bigger change — it alters cancellation semantics — and worth doing separately. For reference, the current single-threaded runtime plateaus around 19k req/s against nginx, so it is a real but distant ceiling, not what caused this issue.

---

Draft: the pyvespa side needs a released version to pin against before it can merge.

[oystein-dev]

Looks good! Having native async like it is described in "Not in scope" would probably be the best solution to have many concurrent requests without the cost of threads, but I don't know how that works with the rust to python bridge. Could be something worth looking at later.

[thomasht86]

Looks good! Having native async like it is described in "Not in scope" would probably be the best solution to have many concurrent requests without the cost of threads, but I don't know how that works with the rust to python bridge. Could be something worth looking at later.

Agree that would be the best. When I looked into that about a year ago, there were some compatibility issues that made it seem difficult to work across all versions, but I'll create an issue to investigate current state and feasibility.