Language Models: We constructed two models, EduBERT and EduDistilBERT, which respectively refine <cite>BERT-base</cite> and DistilBERT , <cite>both of which</cite> were trained on general domain text from books and Wikipedia (<cite>Devlin et al., 2019</cite>) . Both models are initialized from their <cite>base model</cite> and finetuned on educational data, using the Transformers library .