@inproceedings{
le2024mixture,
title={Mixture of Experts Meets Prompt-Based Continual Learning},
author={Minh Le and An Nguyen The and Huy Nguyen and Thien Trang Nguyen Vu and Huyen Trang Pham and Linh Van Ngo and Nhat Ho},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=erwatqQ4p8}
}
Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms.
Mixture of Experts Meets Prompt-Based Continual Learning
Minh Le An Nguyen The Huy Nguyen Thien Trang Nguyen Vu Huyen Trang Pham Linh Van Ngo Nhat Ho