\(\bullet\)_Pipeline parallelism._ Pipeline parallelism aims to distribute the different layers of a LLM into multiple GPUs. Especially, in the case of a Transformer model, pipeline parallelism loads consecutive layers onto the same GPU, to reduce the cost of transmitting the computed hidden states or gradients between GPUs. However, a naive implementation of pipeline parallelism may result in a lower GPU utilization rate as each GPU has to wait for the previous one to complete the computation, leading to the unnecessary cost of _bubbles overhead_[331]. To reduce these bubbles in pipeline parallelism, GPipe [331] and PipeDream [332] propose the techniques of padding multiple batches of data and asynchronous gradient update to improve the pipeline efficiency.