Many memory optimization approaches [1]}, [2]}, [3]}, [4]} have been proposed, but none of them is appropriate for activation functions without introducing extra computing overhead.
To reduce activation functions' footprint usage during model training, recomputation [4]}, [1]} reproduces the intermediate variables in the backward pass without saving them in memory.
However, the recomputation inserts new operators breaking the original computation graph and brings extra computing overhead, dropping the speed of training.
