Large language models (LLMs)  with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones.
To this end, we carefully develop alargeset of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensurediversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions usinggpt-3.5-turbo.
Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both theencoder-decoderanddecoder-onlyfamilies, with varying sizes.
We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP)  benchmarks, as well as through human assessment. We also assess the model for hallucination and toxicity, and for the former, we introduce a new benchmark dataset for hallucination-inducing QA.
The results demonstrate that our proposed LaMini-LM models are comparable to strong baselines while being much smaller in size.111Our code, model checkpoints, and dataset are available athttps://github.com/mbzuai-nlp/LaMini-LM