Transfer Learning with Deep Tabular Models
Recent work on deep learning for tabular data demonstrates the strong
performance of deep tabular models, often bridging the gap between gradient
boosted decision trees and neural networks. Accuracy aside, a major advantage
of neural models is that they learn reusable features and are easily fine-tuned
in new domains. This property is often exploited in computer vision and natural
language applications, where transfer learning is indispensable when
task-specific training data is scarce. In this work, we demonstrate that
upstream data gives tabular neural networks a decisive advantage over widely
used GBDT models. We propose a realistic medical diagnosis benchmark for
tabular transfer learning, and we present a how-to guide for using upstream
data to boost performance with a variety of tabular neural network
architectures. Finally, we propose a pseudo-feature method for cases where the
upstream and downstream feature sets differ, a tabular-specific problem
widespread in real-world applications. Our code is available at
https://github.com/LevinRoman/tabular-transfer-learning .