In addition to the above library resources, existing deep learning frameworks (_e.g._, PyTorch [197], TensorFlow [198], MXNet [199], PaddlePaddle [200], MindSpore [136] and OneFlow [201]) have also provided the support for parallel algorithms, which are commonly used for training large-scale models.