TP roadmap

[youssef62]

Hey,

I saw TP is still marked 🚧 for vLLM/SGLang in the README while DP/EP already work. What's the current roadmap for TP support, and what are the main blockers there?

Me and @xzyaoi are working on cold start elimination and would be interested in contributing an implementation. We like Foundry's approach since it can be used with minimal privileges, as opposed to CRIU, which is nice for us on HPC infra.

Thanks!

[xenshinu]

Hi @youssef62, thanks for your interests in Foundry. We had some tests with foundry on TP internally, it worked well with torch symmetric memory as communication backend, while NCCL is kind of tricky as its initialization is indeterministic. Yet me and @libertyeagle currently don't have much time to polish the code, so there is no official implementation for TP right now.

That is to say, if you guys are interested to contribute, please feel free to submit PRs, and let me know if you encounter any issues.