@inproceedings{
konuk2024learning,
title={Learning from Offline Foundation Features with Tensor Augmentations},
author={Emir Konuk and Christos Matsoukas and Moein Sorkhei and Phitchapha Lertsiravarameth and Kevin Smith},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=VVd3iOKPMJ}
}
We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings  from a frozen foundation model, resulting in up to  $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute.    In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.
Learning from Offline Foundation Features with Tensor Augmentations
Emir Konuk Christos Matsoukas Moein Sorkhei Phitchapha Lertsiravarameth Kevin Smith