From our findings, we believe that domain adaptation is important for the <cite>BERT</cite> architecture and we would like to investigate the effect of from scratch unsupervised pretraining on the supervised task as future work.