Computation efficiency 

Evidently, for cases \(f\)  : \(\mathbb {R}^N \rightarrow \mathbb {R}^M\) , \(N \le M\) , FAD is efficient for computing derivatives, and vise-versa for BP with \(N > M\)  [1]}.
According to our observation, manual differentiation is widely adopted by machine learning frameworks with tremendous optimization for high performance. TensorFlow and PyTorch utilize manual differentiation to compute the derivatives and fuse operations manually. For example, layer/batch normalization, convolution, and matrix multiplication are accelerated by the highly optimized library. That optimization exploits the Jacobian matrix sparsity due to their inner reduction (accumulation) operations and is incompatible with automatic differentiation, neither backward (BP) nor forward (FAD).
From the automatic differentiation perspective, the optimization is to minimize numbers of multiplication for each subregion. It is known as the Optimal Jacobian Accumulation (OJA) problem, which has been proved to be an NP-complete problem [2]}.
