
We proposed a Layer Normalization method, which was used to train a BERT [1]} model with 1,000 layers, which to the best of our knowledge, is the deepest BERT model.

We proposed a Layer Normalization method, which was used to train a GPT [2]} model with 1,000 layers, which to the best of our knowledge, is the deepest GPT model.

