We train a transformer model with \(dim=2048\), \(n\_layers=24\), and \(n\_heads=16\), resulting in 1.4B parameters. Training data and hyperparameter setup mostly repeats the one from Touvron et al. (2023b). To adapt for the relatively smaller model size, we increase the learning rate to \(4.0e-4\), and the global batch-size was capped at 2M due to the limited number of GPUs. During training, we observe a fixed gap in training perplexity between baseline models and reverse training (Figure 1). The loss of the baseline model is measured on data in the standard direction, while the reverse training loss covers data in both directions. We posit that the reverse training doesn't interfere with forward learning -- thus, the model's performance does not degrade on standard benchmarks in data-match conditions, and because we observe a match in the convergence rate of the reverse trained models with the baseline model when it's trained on about 50% of the data.