@inproceedings{
devvrit2024matformer,
title={MatFormer: Nested Transformer for Elastic Inference},
author={Fnu Devvrit and Sneha Kudugunta and Aditya Kusupati and Tim Dettmers and Kaifeng Chen and Inderjit S Dhillon and Yulia Tsvetkov and Hannaneh Hajishirzi and Sham M. Kakade and Ali Farhadi and Prateek Jain},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=fYa6ezMxD5}
}
Foundation models are applied in a broad spectrum of settings with different inference constraints, from massive multi-accelerator clusters to resource-constrained standalone mobile devices. However, the substantial costs associated with training these models often limit the number of unique model sizes that can be offered. Consequently, practitioners are compelled to select a model that may not be optimally aligned with their specific latency and cost requirements. We present MatFormer, a novel Transformer architecture designed to provide elastic inference across diverse deployment constraints. MatFormer achieves this by incorporating a nested Feed Forward Network (FFN) block structure within a standard Transformer model. During training, we  optimize the parameters of multiple nested FFN blocks with varying sizes, enabling the extraction of hundreds of accurate smaller models without incurring additional computational costs. We empirically validate the efficacy of MatFormer across different model classes (decoders and encoders) and modalities (language and vision), demonstrating its potential for real-world deployment. We show that a 850M decoder-only MatFormer language model (MatLM) allows us to extract multiple smaller models spanning from 582M to 850M parameters, each exhibiting better validation loss and one-shot downstream evaluations than independently trained counterparts. Furthermore, we observe that smaller encoders extracted from a universal MatFormer-based ViT (MatViT) encoder preserve the metric-space structure for adaptive large-scale retrieval. Finally, we showcase that speculative decoding with the accurate and consistent submodels extracted from  MatFormer can lead to significant reduction in inference latency.
MatFormer: Nested Transformer for Elastic Inference
Fnu Devvrit Sneha Kudugunta Aditya Kusupati Tim Dettmers Kaifeng Chen Inderjit S Dhillon Yulia Tsvetkov Hannaneh Hajishirzi Sham M. Kakade Ali Farhadi Prateek Jain