Residual connections [1]} and layer normalization [2]} are crucial for convergence in training Transformers [3]}. To make our network fully reversible, these two tricks can not be used directly. Instead, we put the residual connections and layer normalization within each operation \(\tilde{o}(X) = \text{LayerNorm}(X + o(X))\) , except for Zero and Identity.
