**100K length training.** We further increased the training sequence length to 100K from 32K and trained the models on Arxiv-math dataset. 100K training further decreased the perplexity score to **2.21** and **2.20** for _Linear_ and _Linear_\(+\)_Delta_ models.