Pipeline parallelism enables efficient training of Large Language Models (LLMs)  on large-scale distributed accelerator clusters. Yet, pipelinebubblesduring startup and tear-down reduce the utilization of accelerators.
Although efficient pipeline schemes with micro-batching and bidirectional pipelines have been proposed to maximize utilization, a significant number of bubbles cannot be filled using synchronous forward and backward passes.
To address this problem, we suggest thatextra workbe assigned to the bubbles to gainauxiliary benefitsin LLM training.
As an example in this direction, we proposePipeFisher, which assigns the work of K-FAC, a second-order optimization method based on the Fisher information matrix, to the bubbles toaccelerate convergence.
In Phase 1 pretraining of BERT-Base and -Large models, PipeFisher reduces the (simulated)  training time to 50-75% compared to training with a first-order optimizer by greatly improving the accelerator utilization and benefiting from the improved convergence by K-FAC.