While generic models Touvron et al. (2023); Taylor et al. (2022); Zhang et al. (2022); Gu and Dao (2023); Almazrouei et al. (2023); Zhang et al. (2024); Faysse et al. (2024) gain some exposure to legal data during their training, it typically only represents a minor fraction of the overall data. A straightforward method to enhance performance for legal tasks is to perform additional training focusing on legal data. This approach, particularly focused on decoder models, has been successfully used in various fields such as medicine Chen et al. (2023); Ji et al. (2023), translation Xu et al. (2023); Wu et al. (2024), and coding Roziere et al. (2023). The key advantage of this approach is its scalability and independence from the specific characteristics of the training data. Other research on domain adaptation has attempted to specialize language models via pretext tasks. However, these efforts often rely on smaller-scale approaches Niklaus and Giofre (2023), are computationally expensive Vu et al. (2020); Lu et al. (2023), or lack scalability Cheng et al. (2023); Cui et al. (2023); Nishida et al. (2019).