
Can a model fine-tuned with a subset of data outperform the model trained with general domain data?
The work of [1]} showed that performing fine-tuning on a subset of data (used to build the model) yields small improvements (and not statistically significant at level p=0.01). A limitation in their experiments is that, as BPE is not applied, the vocabulary of the adapted model remains the same as the general model. As in these experiments we are processing the data using BPE, the limitation of the vocabulary should disappear (as sub-words are considered rather than complete words). We are interested in exploring whether performing fine-tuning with a subset of the data (in which BPE was applied) can improve the base model.

Can a model fine-tuned with a subset of in-domain data outperform the model fine-tuned with the complete data set?
The general uses of fine-tuning [2]}, [3]} consist of using in-domain data set to adapt a model. However, we want to investigate whether applying data selection in smaller in-domain set can also lead to improvements.

Can a model fine-tuned with a dataset mixture of general-domain and in-domain data outperform the previous-mentioned models?
By considering both datasets (general and in-domain data), the number of candidate sentences is increased. This also poses a challenge to the transductive algorithm as most of the candidate sentences are not in-domain. We are interested in exploring whether these algorithms can successfully retrieve sentences that lead to improvements.

