feat: parallel async probe runner (run_probes) (#2)#29
Merged
Conversation
Adds pytest_wardenbot.runners.run_probes: fan a corpus out against an AsyncChatbotAdapter concurrently (semaphore-bounded asyncio.gather), returning per-prompt ProbeResults in input order. Infra errors are captured per-probe (.error / .ok) so one unreachable bot doesn't sink the batch; real bugs propagate. This is the native-async counterpart to the serial sync tests, delivered as a runner + recipe rather than auto-collected @pytest.mark.asyncio tests (which would force pytest-asyncio on all users under --strict-markers and risk silent false-passes). Grade results with the existing deterministic helpers. - [async] extra (pytest-asyncio); how-to/parallel-async.md recipe. - runners/__init__ exports run_probes + ProbeResult; api.md documents them. - base.py async notes now point at run_probes. - 8 unit tests via stub async adapters (concurrency cap, order, error capture, propagation) -- 100% covered, no API. Closes #2.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #29 +/- ##
==========================================
+ Coverage 93.82% 93.97% +0.14%
==========================================
Files 41 42 +1
Lines 1329 1361 +32
Branches 127 128 +1
==========================================
+ Hits 1247 1279 +32
Misses 65 65
Partials 17 17
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2.
Lets async adapters fan probes out concurrently instead of the serial
to_syncpath.Design call
Rather than bundle auto-collected
@pytest.mark.asynciotests (which would forcepytest-asyncioon every user under--strict-markersand risk silent false-passes when no async plugin is active, plus double the test surface), this ships a native-async runner + recipe:pytest_wardenbot.runners.run_probes(adapter, prompts, *, concurrency=5)— semaphore-boundedasyncio.gatheroversend_message, returns per-promptProbeResults in input order.WardenBotInfraErroris captured per-probe (.error/.ok) so one unreachable bot doesn't sink the batch; non-infra exceptions propagate.[async]extra (pytest-asyncio) + a how-to recipe;base.py's "v0.2 native-async" notes now point here. (Also finally puts the reservedrunners/package to use.)Tests
tests/test_async_probe.py— 8 tests with stub async adapters: concurrency cap honored (tracks peak in-flight),concurrency=1is serial, input order preserved, infra error captured without sinking the batch, non-infra exception propagates, empty input,concurrency<1→ValueError. 100% coverage of the new module, no API.Verification
ruffclean ·pytest -q→ 382 passed ·mkdocs build --strictclean.