High-throughput LLM inference optimized for AMD Instinct MI300X.
- Batched inference with dynamic padding
- KV-cache for continuous batching
- ROCm 6.x native support
- Flash attention v2 for CDNA3
- Tensor parallelism (multi-GPU)
- OpenAI-compatible API server
| Model | MI300X | H100 | A100 |
|---|---|---|---|
| Llama-3-70B | 142 tok/s | 138 tok/s | 89 tok/s |
| Mixtral-8x22B | 98 tok/s | 91 tok/s | 62 tok/s |