\(\bullet\)_Tensor parallelism._ Tensor parallelism is also a commonly used technique that aims to decompose the LLM for multi-GPU loading. Unlike pipeline parallelism, tensor parallelism focuses on decomposing the tensors (the parameter matrices) of LLMs. For a matrix multiplication operation \(Y=XA\) in the LLM, the parameter matrix \(A\) can be split into two submatrices, \(A_{1}\) and \(A_{2}\), by column, which can be expressed as \(Y=[XA_{1},XA_{2}]\). By placing matrices \(A_{1}\) and \(A_{2}\) on different GPUs, the matrix multiplication operation would be invoked at two GPUs in parallel, and the final result can be obtained by combining the outputs from the two GPUs through across-GPU communication. Currently, tensor parallelism has been supported in several open-source libraries, _e.g._, Megatron-LM [75], and can be extended to higher-dimensional tensors. Also, Colossal-AI has implemented tensor parallelism for higher-dimensional tensors [335, 336, 337] and proposed sequence parallelism [338] especially for sequence data, which can further decompose the attention operation of the Transformer model.