Meta-Learning a Real-Time Tabular AutoML Method For Small Data
We present TabPFN, an AutoML method that is competitive with the state of the
art on small tabular datasets while being over 1,000$\times$ faster. Our method
is very simple: it is fully entailed in the weights of a single neural network,
and a single forward pass directly yields predictions for a new dataset. Our
AutoML method is meta-learned using the Transformer-based Prior-Data Fitted
Network (PFN) architecture and approximates Bayesian inference with a prior
that is based on assumptions of simplicity and causal structures. The prior
contains a large space of structural causal models and Bayesian neural networks
with a bias for small architectures and thus low complexity. Furthermore, we
extend the PFN approach to differentiably calibrate the prior's hyperparameters
on real data. By doing so, we separate our abstract prior assumptions from
their heuristic calibration on real data. Afterwards, the calibrated
hyperparameters are fixed and TabPFN can be applied to any new tabular dataset
at the push of a button. Finally, on 30 datasets from the OpenML-CC18 suite we
show that our method outperforms boosted trees and performs on par with complex
state-of-the-art AutoML systems with predictions produced in less than a
second. We provide all our code and our final trained TabPFN in the
supplementary materials.