Back-propagation is the mainstay approach for DNN training.
Today's mainstream machine learning frameworks, such as PyTorch [1]} and TensorFlow [2]}, implement BP using dynamic/static computational graphs and significantly improve model deployment efficiency.
Compared to back-propagation AD, forward AD can reduce the stored intermediate variables and execute efficiently and straightforward in the specific numeric function \(f\) : \(\mathbb {R}^N \rightarrow \mathbb {R}^M,\ (N \le M)\) .
The activation functions are the typical element-wise functions with \(f\) : \(\mathbb {R}^1 \rightarrow \mathbb {R}^1\) , whose computation graphs can be optimized by FAD to reduce the DNN training's memory consumption.
