Task-adaptive continual pre-training (TACP) refers to continual pre-training aiming to enhance performance on a targeted task. TACP has been studied in the context of smaller language models like BERT by pre-training the language model on labeled and unlabeled data from the task [8; 1; 6] showing improvements over the task. While task data is usually quite limited, TACP shows considerable effects on smaller language models like BERT. We benchmark TACP on our four financial evaluation tasks.