Adapter-based approaches [1]} introduce adapter modules for NLP task transfer, which achieve competitive results without full fune-tuning. We borrow this idea for layer adaptation by introducing an independent adapter module for each layer. Specifically, we adopt the recently proposed LoRA [2]} that shows promising results for task adaptation in GPT-3 [3]} as our layer adapter, as Figure REF (a) shows:
\(\centering W_{AdapterLA}^i=(W_{share} + W_{LoRA}^i)\) 
\(\centering W_{LoRA} = BA\) 
