AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models
Fine-tuning large-scale pre-trained language models to downstream tasks
require updating hundreds of millions of parameters. This not only increases
the serving cost to store a large copy of the model weights for every task, but
also exhibits instability during few-shot task adaptation. Parameter-efficient
techniques have been developed that tune small trainable components (e.g.,
adapters) injected in the large model while keeping most of the model weights
frozen. The prevalent mechanism to increase adapter capacity is to increase the
bottleneck dimension which increases the adapter parameters. In this work, we
introduce a new mechanism to improve adapter capacity without increasing
parameters or computational cost by two key techniques. (i) We introduce
multiple shared adapter components in each layer of the Transformer
architecture. We leverage sparse learning via random routing to update the
adapter parameters (encoder is kept frozen) resulting in the same amount of
computational cost (FLOPs) as that of training a single adapter. (ii) We
propose a simple merging mechanism to average the weights of multiple adapter
components to collapse to a single adapter in each Transformer layer, thereby,
keeping the overall parameters also the same but with significant performance
improvement. We demonstrate these techniques to work well across multiple task
settings including fully supervised and few-shot Natural Language Understanding
tasks. By only tuning 0.23% of a pre-trained language model's parameters, our
model outperforms the full model fine-tuning performance and several competing
methods.