@inproceedings{
cherepanova2023a,
title={A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning},
author={Valeriia Cherepanova and Roman Levin and Gowthami Somepalli and Jonas Geiping and C. Bruss and Andrew Wilson and Tom Goldstein and Micah Goldblum},
booktitle={NeurIPS 2023 Second Table Representation Learning Workshop},
year={2023},
url={https://openreview.net/forum?id=PxEY7pBb6F}
}
Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent over-fitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. We construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose an input-gradient-based analogue of LASSO for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.
A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning
Valeriia Cherepanova Roman Levin Gowthami Somepalli Jonas Geiping C. Bruss Andrew Wilson Tom Goldstein Micah Goldblum