We train the baseline model on 2 trillion tokens in the left-to-right direction. Reverse training uses only half of these tokens (1 trillion), but trains in both the standard left-to-right direction, and in the right-to-left (reverse) direction with this same subset of the data. Hence it does model updates over 2 trillion tokens in total, i.e. 1 trillion tokens in each direction is passed through the model. We call this setup _compute-matched_ because both models processed the same amount of tokens in total and used the same compute resources. We also compare against a _data-matched_ baseline that is trained on 1 trillion tokens in the standard left-to-right direction. This model has been trained with half as many updates, but has seen the same data examples as the reverse trained model, but only in one direction. For reverse training, we employ entity-preserving reversal for 5% of the pre-train data, and the remainder uses word-reversal, mainly due to the extra computational cost of the entity reversal procedure, which we refer to as "entity"* in our results. We also test other reversal mechanisms in our ablation experiments.