TITLE: Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development 

ABSTRACT: Data is a crucial component of machine learning. The field is reliant on data
to train, validate, and test models. With increased technical capabilities,
machine learning research has boomed in both academic and industry settings,
and one major focus has been on computer vision. Computer vision is a popular
domain of machine learning increasingly pertinent to real-world applications,
from facial recognition in policing to object detection for autonomous
vehicles. Given computer vision's propensity to shape machine learning research
and impact human life, we seek to understand disciplinary practices around
dataset documentation - how data is collected, curated, annotated, and packaged
into datasets for computer vision researchers and practitioners to use for
model tuning and development. Specifically, we examine what dataset
documentation communicates about the underlying values of vision data and the
larger practices and goals of computer vision as a field. To conduct this
study, we collected a corpus of about 500 computer vision datasets, from which
we sampled 114 dataset publications across different vision tasks. Through both
a structured and thematic content analysis, we document a number of values
around accepted data practices, what makes desirable data, and the treatment of
humans in the dataset construction process. We discuss how computer vision
datasets authors value efficiency at the expense of care; universality at the
expense of contextuality; impartiality at the expense of positionality; and
model work at the expense of data work. Many of the silenced values we identify
sit in opposition with social computing practices. We conclude with suggestions
on how to better incorporate silenced values into the dataset creation and
curation process.
Data is a crucial component of machine learning. The field is reliant on data
to train, validate, and test models. With increased technical capabilities,
machine learning research has boomed in both academic and industry settings,
and one major focus has been on computer vision. Computer vision is a popular
domain of machine learning increasingly pertinent to real-world applications,
from facial recognition in policing to object detection for autonomous
vehicles. Given computer vision's propensity to shape machine learning research
and impact human life, we seek to understand disciplinary practices around
dataset documentation - how data is collected, curated, annotated, and packaged
into datasets for computer vision researchers and practitioners to use for
model tuning and development. Specifically, we examine what dataset
documentation communicates about the underlying values of vision data and the
larger practices and goals of computer vision as a field. To conduct this
study, we collected a corpus of about 500 computer vision datasets, from which
we sampled 114 dataset publications across different vision tasks. Through both
a structured and thematic content analysis, we document a number of values
around accepted data practices, what makes desirable data, and the treatment of
humans in the dataset construction process. We discuss how computer vision
datasets authors value efficiency at the expense of care; universality at the
expense of contextuality; impartiality at the expense of positionality; and
model work at the expense of data work. Many of the silenced values we identify
sit in opposition with social computing practices. We conclude with suggestions
on how to better incorporate silenced values into the dataset creation and
curation process.
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
http://arxiv.org/abs/2108.04308v2