Use vLLM's own server instead of TRL's

[qgallouedec]

trl vllm-serve ran a custom FastAPI app around LLM, with its own data-parallel fan-out and a worker extension for
weight sync. vLLM's server covers all of it, so the app is gone and the command is now a deprecated wrapper that
translates its flags into vllm serve and prints the equivalent command.

Changes

• Server: trl/scripts/vllm_serve.py goes from 1218 to ~130 lines! It adds the three settings TRL needs
  (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1), sets
  VLLM_SERVER_DEV_MODE=1, and forwards any other argument to vllm serve. fastapi, uvicorn and pydantic leave the vllm extra.
• Generation: /v1/completions for token IDs. No OpenAI-compatible endpoint takes token IDs and images, so
  multimodal prompts have their images processed on their own and the resulting features paired with the trainer's
  token IDs on /inference/v1/generate, which keeps multi-turn generation working with images.
• Weight sync: vLLM's NCCL weight-transfer engine. Tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request and one broadcast per tensor.
• Distillation: teacher logprobs come from prompt_logprobs.
• Docs: vllm serve is the documented way to start the server.

No change to training code: vllm_mode, vllm_server_base_url, vllm_group_port and the rest behave as before,
trl vllm-serve still accepts the same flags, and GRPO and RLOO are untouched.

Benchmark

GRPO in server mode on H100s, Qwen2.5-1.5B (text) and Qwen2.5-VL-3B (VLM), same seed and same data, server on its own GPUs and a two-process trainer.

Setup	main (two runs)	this PR (two runs)
Text, TP=1	1.08, 0.91 s/step	0.63, 0.62 s/step	1.59x
Text, TP=2	0.96, 0.96 s/step	0.62, 0.60 s/step	1.57x
VLM, TP=1	2.14, 2.13 s/step	1.48, 1.48 s/step	1.44x

Same behaviour: with a fixed generation seed the two implementations produce bit-identical completions (16/16 over 4 prompts x n=4, logprobs identical to 0.0 over 499 tokens). The training curves still do not lie on top of each
other, for reasons that have nothing to do with this PR.

Why the curves do not overlap exactly

The text curves diverge because each run talks to its own server process. Against the same server, two runs of
the same code are bit-identical, so the trainer itself is deterministic:

# repro_curve_divergence.py — needs two GPUs
#   VLLM_SERVER_DEV_MODE=1 CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-1.5B \
#       --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-logprobs -1 &
#   CUDA_VISIBLE_DEVICES=1 python repro_curve_divergence.py
from datasets import load_dataset

from trl import GRPOConfig, GRPOTrainer

STEPS = 10


def reward_len(completions, **kwargs):
    return [-abs(40 - len(completion)) for completion in completions]


def run(output_dir, group_port):
    config = GRPOConfig(
        output_dir=output_dir,
        per_device_train_batch_size=8,
        num_generations=8,
        max_completion_length=64,
        max_steps=STEPS,
        use_vllm=True,
        vllm_mode="server",
        vllm_server_base_url="http://127.0.0.1:8000",
        vllm_group_port=group_port,  # one port per run, the first is still bound when the second starts
        generation_kwargs={"seed": 42},  # same sampling seed in both runs
        seed=0,
        data_seed=0,
        temperature=1.0,
        bf16=True,
        logging_steps=1,
        report_to=[],
        save_strategy="no",
    )
    trainer = GRPOTrainer(
        model="Qwen/Qwen2.5-1.5B",
        reward_funcs=reward_len,
        args=config,
        train_dataset=load_dataset("trl-internal-testing/zen", "standard_prompt_only", split="train"),
    )
    trainer.train()
    return [entry["reward"] for entry in trainer.state.log_history if "reward" in entry]


first, second = run("/tmp/repro_run_1", 51216), run("/tmp/repro_run_2", 51217)
differing = [step for step, (a, b) in enumerate(zip(first, second, strict=True)) if abs(a - b) > 1e-9]
print(f"run 1: {[round(value, 2) for value in first]}")
print(f"run 2: {[round(value, 2) for value in second]}")
print(f"identical for the first {differing[0] if differing else STEPS} steps")
print(f"mean |difference|: {sum(abs(a - b) for a, b in zip(first, second, strict=True)) / STEPS:.2f}")

run 1: [-230.88, -224.12, -240.62, -259.38, -203.25, -280.38, -247.0, -232.5, -266.88, -246.25]
run 2: [-230.88, -224.12, -240.62, -259.38, -203.25, -280.38, -247.0, -232.5, -266.88, -246.25]
identical for the first 10 steps
mean |difference|: 0.00

Give each run its own server — which comparing two implementations necessarily does — and they drift apart, because a
fresh engine returns slightly different logits and one flipped token changes every later step. That drift is as large
between two runs of main as it is between main and this PR:

Comparison (text, TP=1, reward over 30 steps)	identical steps	mean abs difference
main vs main, rerun	0	5.99
this PR vs this PR, rerun	0	9.78
main vs this PR	5	6.05

The VLM curves sit apart for a real reason. The server returns the processed image already cast to the model
dtype, while handing the engine the image directly preprocesses in float32. The values are otherwise the same:

# repro_image_dtype.py — needs one GPU
#   VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct &
#   python repro_image_dtype.py
import base64
import io

import requests
import torch
from PIL import Image
from transformers import AutoProcessor
from vllm.entrypoints.scale_out.token_in_token_out.mm_serde import decode_mm_kwargs_item

MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"

image = Image.new("RGB", (64, 64), color=(30, 200, 30))
buffer = io.BytesIO()
image.save(buffer, format="PNG")
url = f"data:image/png;base64,{base64.b64encode(buffer.getvalue()).decode()}"

response = requests.post(
    "http://127.0.0.1:8000/v1/chat/completions/render",
    json={
        "model": MODEL,
        "max_tokens": 1,
        "messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": url}}]}],
    },
    timeout=300,
)
response.raise_for_status()
served = decode_mm_kwargs_item(response.json()["features"]["kwargs_data"]["image"][0])["pixel_values"].data

processor = AutoProcessor.from_pretrained(MODEL)
local = processor(images=[image], text=["<|vision_start|><|image_pad|><|vision_end|>"], return_tensors="pt")[
    "pixel_values"
]

print(f"server: {tuple(served.shape)} {served.dtype}")
print(f"local : {tuple(local.shape)} {local.dtype}")
print(f"max |difference|: {(served.float() - local.float()).abs().max().item():.3e}")
print(f"equal after casting the local one: {torch.equal(served, local.to(served.dtype))}")

server: (16, 1176) torch.bfloat16
local : (16, 1176) torch.float32
max |difference|: 2.747e-03
equal after casting the local one: True

So the image is the same image, rounded to the dtype the model runs in. That moves logprobs by ~0.04 and occasionally flips a greedy argmax: over three greedy generations the two implementations agreed on two and differed by one word on the third, both naming the colour correctly.

Validation

On H100s against vLLM 0.27.1:

• weight sync at TP=1, TP=2 and DP=2: a no-op sync leaves greedy output bit-identical, a perturbation propagates, restoring the weights restores the output
• GRPO, RLOO, VLM GRPO and server distillation trained end-to-end with the server on a separate GPU, plus the client-level weight sync checks (no-op, perturb, restore, single-tensor push, prefix cache)
• multimodal generation continued from token IDs across turns
• tests/test_vllm_client_server.py -m slow: 34 passed

Two fixes came out of this: vllm-project/vllm#52399, where
/inference/v1/generate drops completions when n > 1 (the client sets output_kind explicitly until it lands), and
the client here no longer builds its NCCL communicator on an unindexed device, which on main breaks a
single-process trainer with this nccl communicator is created to work on cuda, but the input tensor is on cuda:0.

---

Note

High Risk
Large change to server-mode vLLM integration (HTTP APIs, NCCL weight sync, multimodal routing); training configs stay the same but runtime depends on vLLM dev endpoints and correct server flags.

Overview
Replaces TRL's custom vLLM FastAPI server with vLLM's built-in vllm serve. trl vllm-serve is now a deprecated thin wrapper that maps flags to vllm serve, sets VLLM_SERVER_DEV_MODE=1, and forwards extra CLI args; the large in-process server implementation is removed. fastapi / uvicorn / pydantic drop out of the trl[vllm] extra.

VLLMClient no longer hits TRL-only routes (/generate/, /chat/, per-tensor /update_named_param/). It uses OpenAI-style /v1/completions and /v1/chat/completions, vLLM dev endpoints for NCCL weight transfer (/init_weight_transfer_engine, /start_weight_update, /update_weights, /finish_weight_update), and /inference/v1/generate for multimodal generation from trainer token IDs plus precomputed image features (image_features + parse_logprobs). Teacher sequence logprobs go through prompt_logprobs on /v1/completions instead of a custom binary API.

VLLMGeneration and Online DPO adopt the new client paths (batched packed weight sync, weight_update() context, chat for multimodal server prompts). Docs and examples now document vllm serve with the required TRL flags (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1).

^{Reviewed by Cursor Bugbot for commit ec6b582. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 659619393f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 92abffe. Configure here.}