RQ2: Does a combination of multiple synthetic datasets improve performance?We measure the impact of combining multiple synthetic styles with C4 for training. We consider two variants: combining in a 1:1 ratio meaning that there are two copies of C4 to match two synthetic styles (medium and QA), and 1:2 ratio which combines only one instance of the C4 dataset. For zero-shot QA tasks, our finding in Table 5-6 indicate lower performance than combining only QA and C4 data. Evaluations over the Pile are shown in Figure 4. We notice that both the 'Q/A' and 'Wikipedia' paraphrases help improve performance on certain domains. For example, 'Stackexchange', that has lots of question-answers benefits from the presence of synthetic data in Q/A style. Overall, we note that there is a small improvement on the average perplexity on the Pile by combining multiple styles.