Previous studies have shown that distinct functionality exists in different layers of Transformer-based models when processing textual information Ethayarajh (2019); Van Aken et al. (2019). Consequently, we experiment with freezing specific layers of the model during continual pre-training. Specifically, we explore freezing the first ten layers and freezing the last ten layers, denoted as Freeze First 10 and Freeze Last 10, respectively.