TITLE: Algorithmic Fairness Datasets: the Story so Far 

ABSTRACT: Data-driven algorithms are studied in diverse domains to support critical
decisions, directly impacting people's well-being. As a result, a growing
community of researchers has been investigating the equity of existing
algorithms and proposing novel ones, advancing the understanding of risks and
opportunities of automated decision-making for historically disadvantaged
populations. Progress in fair Machine Learning hinges on data, which can be
appropriately used only if adequately documented. Unfortunately, the
algorithmic fairness community suffers from a collective data documentation
debt caused by a lack of information on specific resources (opacity) and
scatteredness of available information (sparsity). In this work, we target data
documentation debt by surveying over two hundred datasets employed in
algorithmic fairness research, and producing standardized and searchable
documentation for each of them. Moreover we rigorously identify the three most
popular fairness datasets, namely Adult, COMPAS and German Credit, for which we
compile in-depth documentation.
  This unifying documentation effort supports multiple contributions. Firstly,
we summarize the merits and limitations of Adult, COMPAS and German Credit,
adding to and unifying recent scholarship, calling into question their
suitability as general-purpose fairness benchmarks. Secondly, we document and
summarize hundreds of available alternatives, annotating their domain and
supported fairness tasks, along with additional properties of interest for
fairness researchers. Finally, we analyze these datasets from the perspective
of five important data curation topics: anonymization, consent, inclusivity,
sensitive attributes, and transparency. We discuss different approaches and
levels of attention to these topics, making them tangible, and distill them
into a set of best practices for the curation of novel resources.
Data-driven algorithms are studied in diverse domains to support critical
decisions, directly impacting people's well-being. As a result, a growing
community of researchers has been investigating the equity of existing
algorithms and proposing novel ones, advancing the understanding of risks and
opportunities of automated decision-making for historically disadvantaged
populations. Progress in fair Machine Learning hinges on data, which can be
appropriately used only if adequately documented. Unfortunately, the
algorithmic fairness community suffers from a collective data documentation
debt caused by a lack of information on specific resources (opacity) and
scatteredness of available information (sparsity). In this work, we target data
documentation debt by surveying over two hundred datasets employed in
algorithmic fairness research, and producing standardized and searchable
documentation for each of them. Moreover we rigorously identify the three most
popular fairness datasets, namely Adult, COMPAS and German Credit, for which we
compile in-depth documentation.
  This unifying documentation effort supports multiple contributions. Firstly,
we summarize the merits and limitations of Adult, COMPAS and German Credit,
adding to and unifying recent scholarship, calling into question their
suitability as general-purpose fairness benchmarks. Secondly, we document and
summarize hundreds of available alternatives, annotating their domain and
supported fairness tasks, along with additional properties of interest for
fairness researchers. Finally, we analyze these datasets from the perspective
of five important data curation topics: anonymization, consent, inclusivity,
sensitive attributes, and transparency. We discuss different approaches and
levels of attention to these topics, making them tangible, and distill them
into a set of best practices for the curation of novel resources.
Algorithmic Fairness Datasets: the Story so Far
http://arxiv.org/abs/2202.01711v4