LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
Fine-tuning pretrained language models (LMs) without making any architectural
changes has become a norm for learning various language downstream tasks.
However, for non-language downstream tasks, a common practice is to employ
task-specific designs for input, output layers, and loss functions. For
instance, it is possible to fine-tune an LM into an MNIST classifier by
replacing the word embedding layer with an image patch embedding layer, the
word token output layer with a 10-way output layer, and the word prediction
loss with a 10-way classification loss, respectively. A natural question
arises: can LM fine-tuning solve non-language downstream tasks without changing
the model architecture or loss function? To answer this, we propose
Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations
by conducting an extensive empirical study on a suite of non-language
classification and regression tasks. LIFT does not make any changes to the
model architecture or loss function, and it solely relies on the natural
language interface, enabling "no-code machine learning with LMs." We find that
LIFT performs relatively well across a wide range of low-dimensional
classification and regression tasks, matching the performances of the best
baselines in many cases, especially for the classification tasks. We report the
experimental results on the fundamental properties of LIFT, including its
inductive bias, sample efficiency, ability to extrapolate, robustness to
outliers and label noise, and generalization. We also analyze a few
properties/techniques specific to LIFT, e.g., context-aware learning via
appropriate prompting, quantification of predictive uncertainty, and two-stage
fine-tuning. Our code is available at
https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.