[feat][Draft] Refactor slime to work with custom Megatron and vllm version #7
[feat][Draft] Refactor slime to work with custom Megatron and vllm version #7knlnguyen1802 wants to merge 13 commits intoSamitHuang:dev_vllmfrom
Conversation
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a backend separation plan to decouple the framework from a hard dependency on sglang, enabling support for vLLM. Key changes include renaming backend-agnostic arguments like --server-concurrency, introducing lazy imports for sglang-specific modules, and providing local fallbacks for weight-synchronization utilities to support environments without sglang. The PR also updates Megatron-Core parameter gathering to support strided layouts and includes a comprehensive RFC detailing the refactoring strategy. A critical issue was identified in the argument parsing logic where Hugging Face checkpoint validation was inadvertently disabled by hardcoding a skip flag to true.
knlnguyen1802
changed the title
slime/rollout/sglang_rollout.py
Outdated
|
|
||
| self.semaphore = asyncio.Semaphore( | ||
| args.sglang_server_concurrency * args.rollout_num_gpus // args.rollout_num_gpus_per_engine | ||
| args.server_concurrency * args.rollout_num_gpus // args.rollout_num_gpus_per_engine |
There was a problem hiding this comment.
it's more clear to name as rollout_server_concurrency ?
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
2 participants
This can run vllm 0.17.0 and Megatron-LM 0.16.1
How to run this
Base image for docker
nvcr.io/nvidia/pytorch:26.01-py3Download Megatron-LM 0.16.1
https://github.com/NVIDIA/Megatron-LM/releases/tag/core_v0.16.1Unzip and rename it to Megatron-LM folder and put under /root/
pip install vllm==0.17.0Clone the slime repo
For compatible