@inproceedings{
thomas2024retrieval,
title={Retrieval \& Fine-Tuning for In-Context Tabular Models},
author={Valentin Thomas and Junwei Ma and Rasa Hosseinzadeh and Keyvan Golestan and Guangwei Yu and Maksims Volkovs and Anthony L. Caterini},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=337dHOexCM}
}
Tabular data is a pervasive modality spanning a wide range of domains, and this inherent diversity poses a considerable challenge for deep learning. Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex tabular datasets, but have struggled to scale to larger and more complex ones. To address this limitation, we propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. Using TabPFN as the base model -- currently the best tabular in-context learner -- and applying our retrieval and fine-tuning scheme on top results in what we call a locally-calibrated PFN, or LoCalPFN. We conduct extensive evaluation on 95 datasets curated by TabZilla from OpenML, upon which we establish a new state-of-the-art with LoCalPFN -- even with respect to tuned tree-based models. Notably, we show a significant boost in performance compared to the base in-context model, demonstrating the efficacy of our approach and advancing the frontier of deep learning in tabular data.
Retrieval     Fine-Tuning for In-Context Tabular Models
Valentin Thomas Junwei Ma Rasa Hosseinzadeh Keyvan Golestan Guangwei Yu Maksims Volkovs Anthony L. Caterini