 \hline \hline \end{tabular}
\end{table}
Table 4: Hyperparameters of pre-training of smaller models. Each line represents one model pre-trained completely from scratch with the certain number of tokens and its corresponding learning rate schedule.