异构集群（Heterogeneous Clusters）
1. HeteroG 参考文献： Yi X, Zhang S, Luo Z, et al. Optimizing Distributed Training Deployment in Heterogeneous GPU Clusters: Proceedings of the 16th Intern
3D并行（3D Parallelism）
参考文献： Narayanan D, Shoeybi M, Casper J, et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM: SC' 21, November 14-19
数据并行（Data Parallelism）
参考文献： Team D, Majumder R, President V, et al. DeepSpeed: Extreme-scale model training for everyone[J]. Microsoft, 2020. Rajbhandari S, Rasley J, Ruwas
最新文章
最新文章