异构集群（Heterogeneous Clusters）
1. HeteroG 参考文献： Yi X, Zhang S, Luo Z, et al. Optimizing Distributed Training Deployment in Heterogeneous GPU Clusters: Proceedings of the 16th Intern
3D并行（3D Parallelism）
参考文献： Narayanan D, Shoeybi M, Casper J, et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM: SC' 21, November 14-19
流水线并行（Pipeline Parallelism）
参考文献： Huang Y, Cheng Y, Chen D, et al. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism: 33rd Conference on Neural Inform
最新文章
最新文章