Hardware: NVIDIA RTX 3090 · CUDA 13.0 · PyTorch 2.11.0
Each optimization is applied incrementally on top of the previous.
| Optimization | Tokens/s | TFLOPS/s | Peak TFLOPS/s | MFU (%) | Speedup |
|---|---|---|---|---|---|
| Baseline (fp32) | 10 455 | 9.54 | 35.58 | 26.8 | 1.0x |
| bf16 mixed precision | 24 007 | 21.90 | 142.6 | 15.4 | 2.30x |
| Flash Attention | 45 238 | 33.58 | 142.6 | 23.5 | 4.33x |
| torch.compile | 47 285 | 35.10 | 142.6 | 24.6 | 4.52x |
| Vocab padding | 48 178 | 35.77 | 142.6 | 25.1 | 4.61x |
| Liger CrossEntropy | 52 032 | 38.63 | 142.6 | 27.1 | 4.98x |
| Larger micro-batch | 53 945 | 40.05 | 142.6 | 28.1 | 5.16x |