In particular, LoRA [145] has been widely applied to open-source LLMs (_e.g.,_ LLaMA and BLOOM) forparameter-efficient fine-tuning. Among these research attempts, LLaMA and its variants have gained much attention for parameter-efficient tuning. For example, Alpaca-LoRA [144] has been trained using LoRA as a lightweight tuned version of Alpaca [142] (a fine-tuned 7B LLaMA model with 52K human demonstrations of instruction following). There are extensive explorations of Alpaca-LoRA ranging in different languages or model sizes, which can be found in the collection page33. A recent study LLaMA-Adapter [409] inserts learnable prompt vectors into each Transformer layer, in which zero-initialized attention has been proposed to improve the training by mitigating the influence of under-fitted prompt vectors. They also extend this approach to a multi-modal setting, _e.g._, visual question answering.