TITLE: Retiring Adult: New Datasets for Fair Machine Learning 

ABSTRACT: Although the fairness community has recognized the importance of data,
researchers in the area primarily rely on UCI Adult when it comes to tabular
data. Derived from a 1994 US Census survey, this dataset has appeared in
hundreds of research papers where it served as the basis for the development
and comparison of many algorithmic fairness interventions. We reconstruct a
superset of the UCI Adult data from available US Census sources and reveal
idiosyncrasies of the UCI Adult dataset that limit its external validity. Our
primary contribution is a suite of new datasets derived from US Census surveys
that extend the existing data ecosystem for research on fair machine learning.
We create prediction tasks relating to income, employment, health,
transportation, and housing. The data span multiple years and all states of the
United States, allowing researchers to study temporal shift and geographic
variation. We highlight a broad initial sweep of new empirical insights
relating to trade-offs between fairness criteria, performance of algorithmic
interventions, and the role of distribution shift based on our new datasets.
Our findings inform ongoing debates, challenge some existing narratives, and
point to future research directions. Our datasets are available at
https://github.com/zykls/folktables.
Although the fairness community has recognized the importance of data,
researchers in the area primarily rely on UCI Adult when it comes to tabular
data. Derived from a 1994 US Census survey, this dataset has appeared in
hundreds of research papers where it served as the basis for the development
and comparison of many algorithmic fairness interventions. We reconstruct a
superset of the UCI Adult data from available US Census sources and reveal
idiosyncrasies of the UCI Adult dataset that limit its external validity. Our
primary contribution is a suite of new datasets derived from US Census surveys
that extend the existing data ecosystem for research on fair machine learning.
We create prediction tasks relating to income, employment, health,
transportation, and housing. The data span multiple years and all states of the
United States, allowing researchers to study temporal shift and geographic
variation. We highlight a broad initial sweep of new empirical insights
relating to trade-offs between fairness criteria, performance of algorithmic
interventions, and the role of distribution shift based on our new datasets.
Our findings inform ongoing debates, challenge some existing narratives, and
point to future research directions. Our datasets are available at
https://github.com/zykls/folktables.
Retiring Adult: New Datasets for Fair Machine Learning
http://arxiv.org/abs/2108.04884v3