adding a blog for rl time-slicing#393
Conversation
✅ Deploy Preview for llm-d ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
On mobile the header Slack icon overlapped the llm-d logo. Hide it from the header at <=700px (mirroring the existing GitHub-stars rule), scoped to .navbar__inner so the Slack link in the collapsed hamburger menu (.navbar-sidebar) is unaffected. Applied to both the root site and docs (preview) stylesheets. Signed-off-by: IBRAHIM IBRAHIM <66755652+Ibrahim2595@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
Signed-off-by: Poonam Lamba <poonamlamba83@gmail.com>
|
Please resolve the conflicts and sign your commits. |
There was a problem hiding this comment.
Thanks for putting this together, @poonam-lamba time-slicing is a genuinely exciting addition to the project and this blog does a good job explaining the architecture and the end-to-end flow.
A few suggestions to bring it in line with the style and tone of the other llm-d blogs (heterogeneous inference, BLIS, networking, hybrid models, etc.):
Tone a neutrality: The existing blogs are project-centric. So in general the subject is always "llm-d" or a named component, never "we" in a way that implies a single organization. A few passages here read more like a product announcement than a community contribution (e.g., "We foresee this trend early on", "llm-d is not a reactive patch"). I'd suggest reframing these to let the technical content make the case, which it does well on its own.
Structure: The existing blogs tend to open with a tight 1-3 sentence problem statement and get to the feature quickly. Here, the first three paragraphs and the "Core Challenges" section cover overlapping ground before we reach time-slicing. Tightening the intro and cutting the repetition would help. Consider adding a "Key Takeaways" box at the top (like some of the other blog do).
Benchmark rigor: The other blogs are specific about their benchmark setups (GPU count, model, workload config, methodology caveats). The "Early Benchmarks" section here would benefit from those details, plus a limitations/caveats section — BLIS, heterogeneous inference, and networking blogs all include one.
Connection to llm-d architecture: the blog introduces several new components (Snapshot Agent, Accelerator Orchestrator, TimeSlice Client) but doesn't position them relative to the existing llm-d architecture (router/EPP, model servers, KV cache hierarchy). The other blogs are careful to show where new capabilities plug into the existing stack. Worth considering adding some wording making the connection.
Happy to discuss any of these in more detail. Specific suggestions in inline comments.
| --- | ||
|
|
||
| # RL Post-Training: Co-Operative Time-Slicing | ||
|
|
There was a problem hiding this comment.
This opening is strong on motivation but could be tightened. The existing blogs open with 1-3 sentences framing the problem, then move directly to the solution. Consider cutting this down to a single paragraph and moving straight to time-slicing by the second paragraph.
| # RL Post-Training: Co-Operative Time-Slicing | ||
|
|
||
| The math behind Reinforcement Learning (RL) post-training for Large Language Models is notoriously unforgiving. As frontier AI labs push the boundaries of reasoning and coding models using algorithms like GRPO, they routinely hit hard architectural and physical constraints. While much of the industry's focus remains on raw GPU count, the actual gatekeeper of experimental velocity is infrastructure efficiency. | ||
|
|
There was a problem hiding this comment.
This might read as a specific organization claiming foresight rather than a community project announcement. We are typically carefully and avoid this pattern (e.g. from other blogs "BLIS provides a faster inner loop") or the networking blog ("NIXL presents a single API"). Suggest a slight reframing.
| As established in large-scale systems research including ByteDance’s [HybridFlow](https://arxiv.org/abs/2409.19256) architecture (powering veRL) and Alibaba’s [RollMux](https://arxiv.org/abs/2512.11306), optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of RL Total Cost of Ownership (TCO). While these frameworks define the architecture, executing this at production scale is challenging. We foresee this trend early on and have been purposefully building llm-d to address efficiency bottlenecks in RL. llm-d is not a reactive patch, it is a mature, production tested engine that is being deployed across production RL workloads. | ||
|
|
||
| We have made targeted investments to expand llm-d into a comprehensive, composable RL infrastructure stack designed to meet researchers where they are i.e. whether they are using Slurm or Kubernetes based setups: | ||
|
|
There was a problem hiding this comment.
A couple of concerns with this section:
- Agent Sandbox (Sister Project) — we should clarify what is the relationship to llm-d? The other blogs don't reference related projects without clear context.
- Our Agent Sandbox provides... — "Our" is possessive in a way the other blogs avoid. Suggest: "The Agent Sandbox provides..."
- More broadly, this section tries to cover the entire RL stack in three bullet points before the blog has introduced its actual subject (time-slicing). Consider either expanding this into its own blog post or trimming it to a single sentence that links to a tracking issue.
On the RL scheduler specifically: The blog introduces the py-inference-scheduler as doing "intelligent routing and batching mechanics for the sampling step" — but this is also what the llm-d Router (EPP) does for inference workloads. The relationship between these two components isn't explained, and a reader familiar with llm-d's architecture would reasonably ask: why is there a second scheduler?
The relationship between py-scheduler and the EPP for RL workloads is still evolving and is active work within sig-rl. I think we have two options:
- Acknowledge it briefly.. e.g., "The RL scheduler complements the llm-d Router for RL-specific batch orchestration patterns (acquire/yield semantics, weight-version-aware dispatching) that are still being integrated into the core routing layer. This integration is an active area of work across sig-rl and sig-router."
- Omit the routing details entirely and keep this blog focused purely on time-slicing, which is its strongest content. The RL scheduler and weight propagation interface could be covered in their own blog posts when the EPP integration story is clearer.
I'd lean toward option 2, the time-slicing content is strong enough to stand on its own, and bundling three other half-introduced components dilutes the message.
|
|
||
| 1. **Throughput-Driven Inference**: The llm-d router maximises the RL rollout generation throughput, ensuring samplers can saturate the pipeline and prevent downstream optimization trainers from stalling. | ||
| 2. **Agent Sandbox (Sister Project)**: Fast tool use and isolated code execution are vital for generating reward signals. Our Agent Sandbox provides the secure, sub-second execution environments necessary to scale verifiable rewards and agentic rollouts without bottlenecking the GPUs. | ||
| 3. **Core Pipeline Primitives**: To combat generation stragglers, we shipped asynchronous batching and native inference schedulers, specifically designed to orchestrate complex, multi-turn RL workloads. |
There was a problem hiding this comment.
Suggest: "Today, llm-d introduces a new well-lit path for Co-operative Time-Slicing: the Snapshot Agent."
| 2. **Agent Sandbox (Sister Project)**: Fast tool use and isolated code execution are vital for generating reward signals. Our Agent Sandbox provides the secure, sub-second execution environments necessary to scale verifiable rewards and agentic rollouts without bottlenecking the GPUs. | ||
| 3. **Core Pipeline Primitives**: To combat generation stragglers, we shipped asynchronous batching and native inference schedulers, specifically designed to orchestrate complex, multi-turn RL workloads. | ||
|
|
||
| Today, we are introducing a new well-lit path in the **llm-d** project for Co-operative Time-Slicing : Snapshot Agent. Time-Slicing effort enables job interleaving into the Kubernetes platform layer to address the single greatest source of waste in modern RL: the Idle Accelerators. Snapshot agent is the first component that we are launching, other components will soon follow. |
There was a problem hiding this comment.
"The following two sections in the blog provide a high level overview..." — this kind of meta-narration ("in this blog we will...") is not used in the other posts. The reader can follow the structure from the headings. Suggest removing this paragraph.
|
|
||
| Today, we are introducing a new well-lit path in the **llm-d** project for Co-operative Time-Slicing : Snapshot Agent. Time-Slicing effort enables job interleaving into the Kubernetes platform layer to address the single greatest source of waste in modern RL: the Idle Accelerators. Snapshot agent is the first component that we are launching, other components will soon follow. | ||
|
|
||
| The following two sections in the blog provide a high level overview of problems in running RL workloads on scale followed by a brief overview of investments for solving those problems and then we dive deeper into the time-slicing for RL. |
There was a problem hiding this comment.
This section largely restates what was already covered in the opening paragraphs. The existing blogs state the problem once, then move to the solution. Suggest merging this with a trimmed intro so the problem is stated once and concisely.
| Furthermore, scaling distributed RL remains an engineering bottleneck. Teams face difficulties in configuring network primitives like NCCL and Intra-Cluster Interconnect (ICI), while simultaneously battling slow checkpointing operations and high-latency weight transfers across decoupled pools. | ||
|
|
||
| Finally, as organizations pivot toward Agentic RL use cases, they need isolation for untrusted code execution and tool calling during generation and reward evaluation steps during RL, all while struggling with reliability when running RL in distributed environments at scale. | ||
|
|
There was a problem hiding this comment.
Minor: space before the colon is non-standard. Also, "llm-d have been investing" should be "llm-d has been investing."
|
|
||
| To resolve these bottlenecks, llm-d have been investing in improving the ***efficiency, security, and observability*** for RL workloads. The solutions are composable meaning you can adopt one or more depending on your use case. | ||
|
|
||
| **Efficiency:** |
There was a problem hiding this comment.
The Weight Propagation Interface and RL scheduler get one-line descriptions with repo links but no explanation of how they integrate with the existing llm-d architecture. The other blogs are careful to show how new components plug into the stack (e.g., the networking blog shows exactly where NixlConnector sits in the vLLM → NIXL → network path).
|
|
||
| @orchestrator.on_accelerators(group_id="trainer-group") | ||
| def train_phase(model, trajectories): | ||
| return model.update(trajectories) |
There was a problem hiding this comment.
The other blogs specify exact benchmark configurations. Could you add:
- How many H100 GPUs?
- What model and size?
- What workload / batch configuration?
- How was the 41% baseline measured?
Also, "theoretical peak of ~95% under idealized phase alignments" could use a brief derivation or citation. Consider adding a limitations/caveats section — the heterogeneous inference blog has a "workload caveat" note, BLIS has a full "Limitations" section. E.g., what assumptions does the 95% number depend on? What workload characteristics would reduce the benefit?
| ### Help Shape the Future of SIG-RL | ||
|
|
||
| Building robust, highly optimized RL infrastructure requires tight collaboration with the engineers and researchers running these workloads at scale. | ||
|
|
There was a problem hiding this comment.
Broken link — missing closing parenthesis. Should be ...main).
| socials: | ||
| linkedin: abhisyant | ||
|
|
||
| bogdan: |
There was a problem hiding this comment.
bogdan and dolev are missing image_url, which is inconsistent with other author entries. Consider adding GitHub avatar URLs for consistency.
Auto-updated by llmd-site blog stamp on merge to main.
4 participants
What does this PR do?
New blog for time-slicing feature
Why is this change needed?
Time-slicing is launched, blog is needed to make folks aware of the change
How was this tested?
All of the below
npm test)npm run build:all)npm run check-links)npm run serve)Checklist
git commit -s) per DCOnpm test)npm run build:all)npm run check-links)Related Issues