0.22.0: Batched inference - #50
Merged
Merged
Conversation
`compile_dylib` shelled out through `std::process::Command::output()`, a multi-second blocking call sitting on a tokio worker inside an async fn. Today nothing else shares the runtime — no host or example spawns — so it is inert. Under concurrent evolution lanes it is not: tokio's multi-thread runtime has no dedicated I/O thread (a worker parks on epoll when idle), so workers stuck in `waitpid` stop readiness events from being processed and the sibling lanes' inference responses stall behind the build they were supposed to overlap with. On a `current_thread` runtime — what bare `#[tokio::test]` gives you — one blocked worker is the only worker. Switch to `tokio::process::Command`, matching what `doc_string.rs` already does for its `cargo rustdoc` invocation. No thread is consumed at all, which is strictly better than `spawn_blocking` and needs no ownership changes to the arguments. `syn` trees are `!Send` (a `proc_macro2` token may wrap a `proc_macro` one), so introducing the await required scoping the ASTs in both `compile_dylib` and `evolve_no_backpressure` such that they are dropped before it; otherwise `evolve`'s `impl Future + Send` no longer holds.
Registration picked its revision id with `next_revision_id()` before
compiling, then asserted at push time that the registry length still
matched — an assertion whose message ("concurrent evolve() calls are not
supported") names exactly the thing this series is about to support. Two
lanes reaching that read together would pick the same id, and therefore
the same versioned `.so` path, and clobber each other's dylib.
Introduce `build_slot`, a tokio mutex covering the whole compile -> copy ->
load -> push sequence. Everything inside it is process-wide shared state:
the generated `crate_dir/src/lib.rs`, the unversioned `so_path` that cargo
overwrites each build, and the dense id. Cargo already takes an exclusive
lock on its build directory, so same-crate-dir builds serialize either
way; this makes the boundary explicit and makes id assignment correct by
construction — the id is read from the registry while holding the permit,
so nobody else can claim it. One build slot is the right starting point:
inference dominates a lane by an order of magnitude, so the contention is
noise.
Split the pipeline tail into two operations with different safety
requirements:
- `build_and_register` compiles, loads and retains a revision without
touching the dispatch pointers. A retained-but-unpublished revision is
invisible to running calls, so it is exempt from the feedback-loop
contract.
- `publish_revision` performs the pointer swap. This is the only thing
that can tear a multi-function revision apart mid-use, so it is now
where `assert_no_calls_in_flight` lives. `activate_revision` becomes a
thin wrapper over it, dropping a duplicated registry lookup.
`evolve` keeps a fail-fast copy of the assertion at entry so a contract
violation surfaces before minutes of inference rather than after.
Pure refactor: serial `evolve` takes the uncontended mutex and behaves
exactly as before. `evolve_no_backpressure` also shrank below the
`too_many_lines` threshold, so its exemption is removed.
`EvolveFailure` records carry a 1-based attempt index but no way to say *which* evolution they belong to. With a single `evolve` that is fine — there is only one. A batch drains all its lanes into the same buffer, so without attribution a host cannot tell which prompt variant produced which compiler diagnostics, which is exactly the per-variant view a batch is run to get. Add a `lane` field, defaulting to 0 and re-attributed with `with_lane`. `from_error` keeps its signature, so this is additive for existing callers.
Add `Runtime::evolve_batch` and `evolve_batch_limited`: one lane per prompt,
run concurrently, results positionally aligned with the input.
Against an OpenAI-compatible endpoint there is no batch API to call —
batching *is* issuing the requests concurrently and letting the server's
continuous batcher merge them into one forward pass. Decode is
memory-bandwidth-bound, so a batch of n reads the model weights once per
step instead of n times; and because all lanes share the symbiont system
preamble (which embeds the rustdoc-derived API surface, by far the largest
part of the request), prefix caching lets every lane after the first skip
that prefill. The existing `LLM_TOKENS{kind="cached_input"}` counter already
reports whether the server is really reusing it.
Two design decisions worth stating:
Lanes do not publish. Every successful lane is compiled, loaded and
retained, but the dispatch pointers keep pointing wherever they did. The
host evaluates the candidates through `<name>_fn` handles and commits to one
with `activate_revision`. That is what a batch is *for* — eight candidates
with no fitness signal between them is not a hill to climb — and it has a
second consequence: since nothing is published, `evolve_batch` is exempt
from the feedback-loop contract, so generation of round n+1 can overlap
evaluation of round n.
Each lane owns its retry ladder. Its own chat history, its own attempt
budget, its own repeat detection. A lane that answers with prose ten times
fails alone; its siblings neither see the corrections nor lose budget to
them. The failure buffer is cleared once per batch rather than once per
lane, and records are tagged with their lane.
Ordered bounded concurrency comes from `futures_util`'s `buffered`, which
was already in the dependency tree via rig-core. Lane futures are built
eagerly into a Vec: constructing one does no work, and it avoids a
higher-ranked lifetime error that a lazy `.map()` closure returning
`impl Future` cannot express.
`evolve` is now `evolve_lane(.., Publish::Yes)` with the failure-buffer
clear hoisted into it, so the single-prompt path shares one retry ladder
with the batch rather than duplicating it.
Tests add a `RoutedAgent` double that answers on prompt content rather than
call order, since concurrent lanes interleave nondeterministically.
Eight near-identical prompts at low sampling temperature do not reliably produce eight distinct implementations. When two lanes converge on the same source, building both spends a multi-second `cargo build` on an artifact that already exists — and hands the host two revision ids that are, in every observable respect, the same program. Before compiling, compare the rendered candidate against the sources of the already-registered revisions and reuse the match. The check sits inside the build permit, which is what makes it airtight under concurrency: two lanes holding identical code cannot both miss the check and then both build. A linear scan with full string comparison, not a hash index. The registry holds tens to hundreds of entries of a few KB, so a miss costs microseconds against a build that costs seconds — and there is no collision case to get wrong. This is visible behaviour, not just an optimization, so both entry points document it and `REVISION_DEDUP_HITS` counts it. `evolve` may now return an id it returned before; `evolve_batch` may return the same id in several slots. The latter is a genuinely useful signal — repeated ids mean the prompt variants are not diversifying the output, which is a prompt problem the host should see rather than a cost it should silently pay.
`EVOLVE_DURATION` is recorded per lane, so after a batch its distribution
describes lane latency and says nothing about the batch. But lane latency is
not the number that decides whether batching is working — overlap is. Four
new series make that measurable:
- `EVOLVE_BATCH_SIZE` and `EVOLVE_BATCH_DURATION`: divide one by the other
for seconds per candidate, which is what should fall as the batch grows.
Their ratio against the summed per-lane durations *is* the speedup.
- `EVOLVE_BATCH_LANES{outcome}`: lanes fail independently, so a rising
error share against a steady batch size localizes the problem to specific
prompt variants rather than to the batch.
- `BUILD_SLOT_WAIT`: the cost of serializing builds against one shared crate
directory. Near zero means inference dominates and one build slot was the
right call; if it climbs, the batch is bottlenecked on cargo and the crate
dir needs to be split per lane.
The new integration test pins the distinction the first metric exists to
make: batch size is sampled once per batch while lane duration is sampled
once per lane — including for the lane that exhausted its retry budget,
which is pre-existing behaviour the test now documents.
890fb8b ("include video of \`fractal-studio-example\` in README") added a bare `README.md` pattern to .gitignore — twice — while adding an 8 MB mp4. The duplication and the commit's subject both point at a slip rather than an intent: a bare pattern matches at every depth, and the repo tracks nine README.md files including one per example. It is inert for those, since ignore rules do not untrack. But it silently swallows every *new* README: the next one added just never appears in `jj status`, with no error anywhere. The following commit adds one.
A worked example of the population-search shape `evolve_batch` exists for:
eight prompts that differ only in a trailing strategy hint, evolved
concurrently, benchmarked against each other, and only then does one get
activated.
Counting primes below n is the task because *strategy* dominates it. Trial
division, a sieve, a bit-packed sieve and wheel factorization are all a few
dozen lines and all obviously correct to a model, but they span three orders
of magnitude in runtime. So eight near-identical prompts produce eight
genuinely different programs rather than eight paraphrases of one, which is
the premise the batch API is built on.
The example is written to demonstrate the parts that are easy to get wrong:
- The per-lane hint goes *last*. Prefix reuse stops at the first differing
token, so a hint spliced into the middle would throw away the cached
prefill of everything before it.
- Candidates are measured through `count_primes_below_fn(rev)` handles, not
by activating each in turn. Nothing is hot-swapped between measurements,
so all eight are timed against the same baseline.
- Panics are read with `handle.take_panic()`, not `runtime.take_panic()`.
A handle call's panic lands in *its* revision's buffer; the runtime
method reads whichever revision is active, which during evaluation is
still the initial one.
- Failures are grouped by `EvolveFailure::lane`, so an unproductive prompt
variant is identifiable rather than just "something failed".
- Revisions are deduplicated before being counted as distinct, since lanes
that converge on identical source share one.
`devenv.nix` gains `--parallel 8`. Without it `llama-server` decodes one
sequence at a time and the lanes would queue rather than batch, which would
make the example both slow and pointless as a demonstration. `--ctx-size` is
the total KV budget split across slots, so it is scaled with the slot count
to keep 16k per slot — comfortably above the largest preamble in the tree.
A round where every lane fails is reported rather than fatal, matching the
sort example: whether a small model invents a sieve is a property of the
model, not of the harness. `STRICT=1` demands a correct implementation.
A benchmark for the claim the previous commits are built on: that eight
concurrent lanes cost far less than eight sequential ones. Runs the same
eight-lane batch at in-flight limits 1, 2, 4 and 8 — lane count fixed so
every level does identical inference work and only the overlap differs —
against a containerized vLLM.
The sweep does not just report wall clock, because a wall-clock number alone
cannot distinguish batching from queueing. Each level installs its own
metrics recorder and reports the decomposition:
- summed per-lane inference time, which should stay flat across levels
(batching overlaps requests, it does not shorten them). If it grows with
the limit, the server is queueing — `--max-num-seqs` too low, or
`--parallel` unset on llama-server.
- compile time and build-slot wait, which is the local pipeline the one
build slot serializes. Near-zero wait is the regime that design assumes;
if it rivals inference time, the shared crate dir is the next bottleneck.
- prefix-cache hit rate from the provider's `cached_input_tokens`, which
reads zero the moment prefix caching is off or the prompts diverge too
early to share a prefix.
Level prompts are tagged with the level, so a level cannot silently ride on
revisions an earlier level registered — dedup would skip those builds and
make the later level look faster than it is.
Serves `prism-ml/Bonsai-8B-unpacked`, the safetensors base of the GGUF that
devenv.nix and CI already use, so the benchmark and the example smoke tests
exercise the same weights.
Two environment details worth recording, both of which cost an hour to
diagnose:
- The GPU is requested as a CDI device rather than via `--gpus all`. On
hosts where nvidia-container-toolkit only writes a CDI spec and does not
register a runtime with the daemon — the common NixOS setup — `--gpus all`
fails with a misleading "AMD CDI spec not found" while the CDI device
resolves fine.
- `ipc: host`, because vLLM's workers use shared memory and the 64 MB
default surfaces as an opaque worker crash.
The bench probes the endpoint with a plain TCP connect and exits 0 with a
message when there is no server, so `cargo bench` stays green on machines
without a GPU and CI's `cargo bench --no-run` only ever compiles it. Both
skip paths are verified.
README gains `evolve_batch` under core highlights and population search under
use cases. Two things worth stating outside the API docs:
CAVEATS records that builds are serialized and why — the crate directory, the
`.so` cargo writes and the dense revision id are all process-wide, and cargo
locks its build directory regardless — along with the metric
(`symbiont_build_slot_wait_seconds`) that says when that trade has stopped
paying.
TODO records that prefix-cache visibility is provider-dependent. This is not a
guess: sending an identical 923-token prompt to vLLM twice returns
`prompt_tokens_details: null` both times, so `LLM_TOKENS{kind="cached_input"}`
reads zero there even though `vllm:prefix_cache_hits_total` shows the cache
working. The throughput bench scrapes `/metrics` to work around it; a general
fix needs a hook for hosts to feed a server-side metric back in.
Minor bump: `evolve_batch`/`evolve_batch_limited`, `EvolveFailure::lane` and
`with_lane`, and four new metric constants are additive.
Eight lanes at in-flight limits 1/2/4/8 against `prism-ml/Bonsai-8B-unpacked`
under vLLM v0.26.0 on an RTX PRO 6000 Blackwell, using the compose.yaml in
this directory. 74.0s -> 21.3s wall for the same eight candidates.
The wall-clock speedup flattens between 4 and 8 lanes (3.43x -> 3.47x), which
would read as the batcher giving up if that were the only column. It is not:
output tokens per second of wall clock keeps climbing 76 -> 102 -> 212 -> 237,
and `llm (sum)` stays in a 50-71s band instead of growing with the limit,
which is what distinguishes batching from queueing. The flattening is the
fixed serial tail — every lane still parses, validates, compiles and loads on
its own, and a lane that exhausts its retry budget runs all ten attempts no
matter how many siblings are in flight.
Two things the run confirms about earlier design decisions:
- `slot wait` grows with concurrency exactly as predicted, 0ms -> 873ms, but
stays under 4% of wall clock while total compile never exceeds 1.6s. The
batch is inference-bound throughout, so one build slot was the right call
and splitting the crate directory per lane would buy nothing yet.
- Prefix caching serves 72-78% of prompt tokens — while the provider-reported
`cached_input_tokens` was 0 for every single request. Without the
`/metrics` scrape the column would have read a flat, wrong zero.
Both caveats are recorded alongside the numbers rather than smoothed over:
levels do unequal work because a failing lane spends ten requests (Bonsai-8B
lands 5-7 of 8 lanes, with the sieve prompts dying on u32-vs-usize indexing),
and `built` falls below `lanes ok` at the higher limits because dedup keys on
generated source, which the per-level prompt tag cannot reach.
The 8-lane sweep flattened between limits 4 and 8 (3.43x -> 3.47x), which read
as the batcher saturating. It was not: with only 8 lanes, limit 8 already has
every lane in flight and there is no further overlap to buy. Raise LANES to 32
and the curve runs to 9.97x wall / 9.10x decode throughput with no plateau —
32 candidates in 41s against 407s sequentially.
Making that measurable needed four changes:
- LEVELS to [1,2,4,8,16,32] and LANES to 32, with startup assertions that
every level is <= LANES and that there is a distinct strategy per lane.
`evolve_batch_limited` clamps its limit to the lane count, so a level above
LANES would silently duplicate the LANES row rather than measure anything.
- 32 distinct strategy hints instead of 8, of deliberately mixed difficulty.
- `--max-num-seqs=32` in compose.yaml. vLLM queues beyond that, so the top
rows would have measured the queue, not the batcher.
- A `dec tok/s` / `tok/s gain` column, which is the work-normalized view and
the one to trust when levels burn different numbers of retries.
Two fixes fell out of running it:
`scrape_prefix_cache` treated a zero counter as "server does not expose this"
and returned None. On a cold server that discarded the *baseline* of the first
level's diff, so level 1 fell back to the provider-reported figure — always
zero on vLLM — and reported a confident 0% next to 73-75% everywhere else.
Presence of the series, not a non-zero value, is the right test.
The runtime is now multi-threaded, with one global recorder snapshotted per
level in place of a per-level thread-local one (a thread-local recorder would
miss everything recorded off the installing worker). This was chased as the
suspected cause of some wild outliers and turned out not to be: limit 16 came
in at 56.4s, 58.4s and 57.2s across two single-threaded runs and this
multi-threaded one. It is kept because it is the honest default at these
widths, not because it fixed anything.
The actual cause of the outliers is retry variance, now documented at the top
of the file and beside the numbers. A level that draws unlucky lanes lands at
double the `output tok` and runs 4-8x slower — one came out slower than serial.
Retries inside a lane are sequential, so they extend that lane's critical path,
and a level's wall clock is its slowest lane: concurrency recovers none of it,
which is why 2x the tokens costs far more than 2x the time.
Adds a "Showcase: Batched Evolution" section between the fractal-studio showcase and the core highlights, carrying the measured numbers from `symbiont/benches/vllm/README.md`: 32 candidates in 41s against 407s sequentially, 9.1x decode throughput, 74-76% prefix-cache hit rate. The centrepiece is a column chart of decode throughput against concurrency. Its colour is not a taste call: the site's `$teal` (#4ECDC4) sits at OKLCH L 0.776, outside the dark-mode band of 0.48-0.67, so it is used only for the hover state. The bars use `$dark-teal` (#00897B), which passes the lightness band and clears 3:1 contrast against the page surface. One series means one hue and no legend — length encodes magnitude, colour encodes nothing — and only the two endpoints are directly labelled, with the y-axis carrying the rest. The 2-lane bar is a visible dip rather than a smooth curve, because that is what was measured: that level drew lanes that ground through their full retry budgets and generated twice the tokens of every other level. It is kept, with the explanation in the collapsible data table underneath, rather than dropped to make the curve look tidy. `.bench-grid` / `.bench-card` move out of `.benchmarks` to the top level. They were scoped to that one section, so reusing the markup here rendered the stat tiles as bare stacked text. Also adds "Batched evolution" and "Revision registry" cards to the core highlights (the registry has been shipped for a while and was never listed), and "Batched Population Search" to the hero list. No gallery tile yet: every existing one is a screenshot of real output, and running the example needs the GPU, so that entry is left until there is a genuine screenshot to put in it. Verified with `zola build` and screenshotted at 1100px and 390px.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes:
cargoinstead of blocking runtime worker.build_and_registerandpublish_revisionmethods to split up the evolve contract into two.evolve_batchandevolve_batch_limitedfor concurrent multi-prompt evolutionbatched-evolutionto showcase these new features.README.mdandCAVEATS.mdwith the appropriate docs.Benchmarks:
Concurrency benchmarks of Bonsai-8B against a
vllmbackend, showing up to 10x more inference throughput at 32 concurrency.So for the right workloads this architecture improvement it huge.