As AutoEncoder language models excel at Natural Language Understanding (NLU) tasks, many RALM architectures(Thulke et al., 2021) (Cheng et al., 2024) (Sachan et al., 2021)utilise them for specific tasks, such as judgement. One of the most commonly used models is BERT and its improved versions.Devlin et al. (2018) proposed the BERT model, which was inspired by closed tasks(Taylor, 1953). RoBERTa(Liu et al., 2019) is trained using dynamic masking, full sentences without NSP loss, large mini-batches, and a larger byte-level BPE to address the lack of training of Bert's model. According to Jiang et al. (2020), BERT heavily relies on global self-attention blocks, resulting in a large memory footprint and computational cost. Although all attention heads query the entire input sequence, some only need to learn local dependencies, leading to computational redundancy. To address this issue, They proposed a new span-based dynamic convolution to replace these self-attention heads and directly model local dependencies. The new convolutional head, along with other self-attentive heads, forms a hybrid attention block. Furthermore, Sanh et al. (2019) was able to decrease the size of the BERT model by 40% while maintaining 97% of its language comprehension abilities and achieving a 60% increase in speed by implementing knowledge distillation during the pre-training phase.