@inproceedings{
schambach2023scaling,
title={Scaling Experiments in Self-Supervised Cross-Table Representation Learning},
author={Maximilian Schambach and Dominique Paul and Johannes Otterbach},
booktitle={NeurIPS 2023 Second Table Representation Learning Workshop},
year={2023},
url={https://openreview.net/forum?id=IbiiNw4oRj}
}
To analyze the scaling potential of deep tabular representation learning models, we introduce a novel Transformer-based architecture specifically tailored to tabular data and cross-table representation learning by utilizing table-specific tokenizers and a shared Transformer backbone.
Our training approach encompasses both single-table and cross-table models, trained via missing value imputation through a self-supervised masked cell recovery objective.
To understand the scaling behavior of our method, we train models of varying sizes, ranging from approximately $10^4$ to $10^7$ parameters. 
These models are trained on a carefully curated pretraining dataset, consisting of 135 M training tokens sourced from 76 diverse datasets.
We assess the scaling of our architecture in both single-table and cross-table pretraining setups by evaluating the pretrained models using linear probing on a curated set of benchmark datasets and comparing the results with conventional baselines.
Scaling Experiments in Self-Supervised Cross-Table Representation Learning
Maximilian Schambach Dominique Paul Johannes Otterbach