For simple linear-based models, if \(\alpha \)  is initialized in zero, the input-output Jacobian will be initialized to 1, which makes the training stable. For more complex cases, experiments in [1]} also prove the effectiveness of ZeroRA. Since ZeroRA can replace the layer normalization in the transformer, it can also improve the expressive power of the model without degrading variances to early layers.
