Our work leverages BERT [1]}, a transformer-based language model, pre-trained on general text from Wikipedia and BookCorpus.
We fine-tune BERT using its self-supervised objectives on unlabeled questions in SE formatted as (Title, Body).
Specifically, we concatenate the Title and Body of the questions of a domain to form documents for BERT.
We adopt the terminology of [2]} and call this
process domain-adaptive pre-training (DAPT) and the resulting
model BERT\(_{\scriptsize \textrm {DAPT}}\).
DAPT tailors the representation towards the particular vocabulary, syntax, and semantics of the problem space (DQD). It also adapts the representation to a specific domain (e.g. AskUbuntu).
Therefore, we expect BERT\(_{\scriptsize \textrm {DAPT}}\)  to produce richer initial representations
for our data.
