We introduce forward mode automatic differentiation (Forward-AD, FAD) for gradient computation of activation functions to avoid the recomputation overhead and reduce the memory footprint.
Automatic differentiation [1]} (autodiff, AD) is a family of mathematics tools to automatically and accurately evaluate numeric function derivatives using computer programs.
There are two modes of autodiff: backward mode (i.e., back-propagation, BP) and forward mode (FAD).
