As recent work [1]}, [2]}, [3]} uses prefix/prompt for task transfer, we investigate if such an approach can work for layer adaptation. As Figure REF (b) shows, we introduce \(L\)  tokens with learnable parameters as a specific prompt for each layer to adapt layers with tied weights. The prompts are only used for keys and values in attention modules, which will not introduce much inference overhead as long as \(L\)  is not set to be too large.
