Reversibility of neural networks relates to how well one can reverse the map from the activation of target neurons to the input neurons [1]}. Usually, neural networks are not reversible, due to a few reasons: non-reversible activation functions, for example ReLU [2]}, non-orthogonal weights, and lack of one-to-one relationships. One advantage of reversible neural networks, besides the reversing mapping, is its memory efficiency: while non-invertible networks need to store all the activations for the backward pass during training, reversible networks allows the storage of fewer activations to update the trainable parameters.
