Recently, the release ofInstructEvalhas provided valuable insights into the performance of large language models (LLMs)  that utilize encoder-decoder or decoder-only architecture. Interestingly, despite being introduced four years ago,T5-based LLMs, such asFlan-T5, continue to outperform the latest decoder-based LLMs, such asLLaMAandVicuna, on tasks that require general problem-solving skills. This performance discrepancy can be attributed to three key factors: (1)  Pre-training data, (2)  Backbone architecture, and (3)  Instruction dataset. In this technical report, our main focus is on investigating the impact of the third factor by leveragingVicuna, a large language model based onLLaMA, which has undergone fine-tuning on ChatGPT conversations. To achieve this objective, we fine-tunedVicunausing a customized instruction dataset collection calledFlan-mini.
This collection includes a subset of the
large-scale
instruction dataset known asFlan, as well as various code-related datasets and conversational datasets derived from ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand problem-solving skills. Our experimental findings strongly indicate that the enhanced problem-solving abilities of our model,Flacuna, are obtained through fine-tuningVicunaon theFlandataset, leading to significant improvements across numerous benchmark datasets inInstructEval.Flacunais publicly available athttps://huggingface.co/declare-lab/flacuna-13b-v1.0.