The idea of reversible networks is first introduced by RevNets [1]}. Later on, [2]}, [3]}, [4]} invented different reversible architectures based on the ResNet [5]}. [6]} extended RevNets to the recurrent network, which is particularly memory-efficient. [7]}, [8]} conducted experiments with reversible Transformers by fixed point iteration. [9]} combined local sensitive hashing attention with reversible transformers to save memory in training with long sequences. An important application of reversible networks is the flow-based models [10]}, [11]}, [12]}. For sequence tasks, [13]} achieved success in non-autoregressive machine translation.
