[1]} is first proposed to train deep networks such as ResNet [2]}, which drops the entire block as a regularization method. Similarly, when training a vision transformer, stochastic depth randomly leaves some samples to be 0 after an attention or a FFN layer but before the residual connection. This step can be viewed as randomly replacing the networks by identity function for some samples.
