FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification
Modern deep learning systems are increasingly deployed in situations such as
personalization and federated learning where it is necessary to support i)
learning on small amounts of data, and ii) communication efficient distributed
training protocols. In this work we develop FiLM Transfer (FiT) which fulfills
these requirements in the image classification setting. FiT uses an
automatically configured Naive Bayes classifier on top of a fixed backbone that
has been pretrained on large image datasets. Parameter efficient FiLM layers
are used to modulate the backbone, shaping the representation for the
downstream task. The network is trained via an episodic fine-tuning protocol.
The approach is parameter efficient which is key for enabling few-shot
learning, inexpensive model updates for personalization, and communication
efficient federated learning. We experiment with FiT on a wide range of
downstream datasets and show that it achieves better classification accuracy
than the state-of-the-art Big Transfer (BiT) algorithm at low-shot and on the
challenging VTAB-1k benchmark, with fewer than 1% of the updateable parameters.
Finally, we demonstrate the parameter efficiency of FiT in distributed low-shot
applications including model personalization and federated learning where model
update size is an important performance metric.