BIGMAC

BigMac is an open-source toolkit for BigMac-style pipeline-parallel training of multimodal large language models. It keeps optimized LLM pipeline schedules intact, then nests encoder and generator work around them to reduce activation memory without reintroducing cross-module pipeline bubbles.

Schedule

Example BigMac schedule for 4 PP ranks, 2 VPP stages, 64 microbatches, and uneven multimodal inputs.

Try more settings and compare with other schedules in our playground: https://huggingface.co/spaces/Dots-Infra/bigmac-pp-simulator

Get Started

Install the root package in editable mode:

pip install -e .

Run the scheduler demo and check the generated schedule image:

python examples/scheduler/demo.py
ls outputs/scheduler_demo/bigmac_schedule.png

Components

This repository contains three main parts for building, running, and studying BigMac schedules:

• pp_scheduler builds BigMac-style global operator plans. Try the Get Started demo for an example schedule visualization.
• pp_executor executes operator plans produced by pp_scheduler. This repository provides a reference implementation for the open-source Megatron-Core backend, with Qwen tutorials showing how to integrate it into a training framework. Other backends should implement their own executor against the same schedule abstraction.
• pp_simulator helps iterate on pipeline strategies without launching full training jobs. It can visualize schedules and simulate per-operator duration imbalance from compute cost, input size, and uneven PP stage partitioning.

Tutorials

• Text-only Qwen3 training shows how to run a Megatron-Core Qwen3 smoke test with and without pp_executor.
• Multimodal Qwen3-VL training shows how to run the BigMac-style vision-language training path.
• PP profiler shows how to inspect and export per-operator executor traces for pipeline diagnosis.
• PP simulator provides an interactive playground for comparing BigMac schedules and visualizing pipeline bubbles without launching training.

Experimental Result

BigMac reports the optimal memory and compute efficiency across different MLLM training systems.

These results are reported in:

• Paper: BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
• arXiv: https://arxiv.org/abs/2605.25451

Citation

If you use this repository, please cite the BigMac paper:

@misc{zhang2026bigmac,
  title = {BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training},
  author = {Zili Zhang and Chengxu Yang and Shenglong Zhang and Chenyu Wang and Yufan Zhang and Tuo Dai and Zhouyang Li and Yuhong Ge and Chao Jin and Xin Jin and Yuliang Liu},
  year = {2026},
  eprint = {2605.25451},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  doi = {10.48550/arXiv.2605.25451}
}