With the development of machine learning, artificial neural networks have made great achievements in the development of recent years. Researchers have proposed many neural network structures for different scenarios, such as BP neural network[1]}, LeNet[2]}, LSTM[3]}, ResNet[4]}, DensNet[5]}, etc. Neural networks have been widely used in various fields and have extraordinary performance, such as medical treatment[6]}, transportation[7]}, computer vision[8]}, natural language translation[9]}, etc. As an important part of neural network, training algorithm has a significant influence on its performance. Poor training algorithms may lead to bad or even non-convergence of the network, failing to achieve the expected results.
