\(\bullet\)_Instruction-tuned models mostly perform better than the base models._ Among all the compared open-source methods, the instruction-tuned models (_i.e.,_ LLaMA 2-Chat, Vicuna, Alpaca and ChatGLM) mostly perform better than non-instruction-tuned models (_i.e.,_ LLaMA 2, LLaMA, Pythia and Falcon). It indicates that instruction tuning is generally capable of improving the few-shot or zero-shot ability of LLMs in solving various tasks. However, after instruction tuning, Vicuna (7B) and Alpaca (7B) suffer from performance degradations on LAMBADA, a language modeling task. The reason may be that the instruction data mainly focuses on enabling LLMs to follow human instructions, which is not always useful for the general language generation task.