**3D Parallelism.** 3D parallelism is actually a combination of three commonly used parallel training techniques, namely data parallelism, pipeline parallelism [331, 332], and tensor parallelism [75]24. We next introduce the three parallel training techniques.