Literature shows substantial improvements are sometimes possible when pre-training on large generic corpora [1]}, [2]}. Given that pre-trained models are freely available, is it even necessary to pre-train on similar data as proposed above? We compare to publicly available ELECTRA models trained on 3.3 Billion tokens generic data. Note that the publicly available model we choose has the same model size. It is also pre-trained using the same hyper-parameters as ours, except that it is pre-trained longer than ours (1.45M vs.0.8M).
<TABLE>