Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors.
However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private.
We introduceproxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the result of directly tuning the model, but by accessingonly its predictionover the output vocabulary.
Our method instead tunes asmallerLM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the base model in the direction of tuning, while retaining the benefits of larger-scale pretraining.
In experiments, when we apply proxy-tuning toLlama2-70Busing proxies of only7Bsize, we can close 88% of the gap betweenLlama2-70Band its truly-tunedchatversion, when evaluated across knowledge, reasoning, and safety benchmarks.
Interestingly, when tested on TruthfulQA, proxy-tuned models are actually more truthful than directly tuned models, possibly because decoding-time guidance better retains the model’s factual knowledge.
We then demonstrate the generality of proxy-tuning by applying it for domain adaptation on code, and task-specific finetuning on question-answering and math problems.
Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.