By injecting FAD into DNN frameworks, we can reduce the memory footprint and allow larger batches for the DNN training.
Therefore, we first adopt the metric of maximum supported batch size to represent the amount of memory footprint.
Fig. REF  depicts the maximum batch size results for the baseline and Forward-AD on TensorFlow and PyTorch.
TensorFlow adopts the static graph optimization for the self-attention mechanism [1]}, which is the main component of BERT model. As such, TensorFlow has less memory usage, and TensorFlow results are better than PyTorch ones with BERT. TensorFlow and PyTorch have similar ResNet implementation, except for the activation function. Thus, ResNet evaluation has close results for both frameworks.
