Following Allen-Zhu & Li (2023b), we employ GPT2-small (12 layers, 12 heads, and 768 dimensions) for the bioS dataset and GPT2 with 12 layers, 20 heads, and 1280 dimensions for the bioR dataset. We also utilize the same AdamW optimizer with cosine learning rate decay (\(\beta_{1}=0.9,\beta_{2}=0.98,\epsilon=10^{-6}\)).