TITLE: Right the docs: Characterising voice dataset documentation practices used in machine learning 

ABSTRACT: Voice-enabled technology is quickly becoming ubiquitous, and is constituted
from machine learning (ML)-enabled components such as speech recognition and
voice activity detection. However, these systems don't yet work well for
everyone. They exhibit bias - the systematic and unfair discrimination against
individuals or cohorts of individuals in favour of others (Friedman &
Nissembaum, 1996) - across axes such as age, gender and accent.
  ML is reliant on large datasets for training. Dataset documentation is
designed to give ML Practitioners (MLPs) a better understanding of a dataset's
characteristics. However, there is a lack of empirical research on voice
dataset documentation specifically. Additionally, while MLPs are frequent
participants in fairness research, little work focuses on those who work with
voice data. Our work makes an empirical contribution to this gap.
  Here, we combine two methods to form an exploratory study. First, we
undertake 13 semi-structured interviews, exploring multiple perspectives of
voice dataset documentation practice. Using open and axial coding methods, we
explore MLPs' practices through the lenses of roles and tradeoffs. Drawing from
this work, we then purposively sample voice dataset documents (VDDs) for 9
voice datasets. Our findings then triangulate these two methods, using the
lenses of MLP roles and trade-offs. We find that current VDD practices are
inchoate, inadequate and incommensurate. The characteristics of voice datasets
are codified in fragmented, disjoint ways that often do not meet the needs of
MLPs. Moreover, they cannot be readily compared, presenting a barrier to
practitioners' bias reduction efforts.
  We then discuss the implications of these findings for bias practices in
voice data and speech technologies. We conclude by setting out a program of
future work to address these findings -- that is, how we may "right the docs".
Voice-enabled technology is quickly becoming ubiquitous, and is constituted
from machine learning (ML)-enabled components such as speech recognition and
voice activity detection. However, these systems don't yet work well for
everyone. They exhibit bias - the systematic and unfair discrimination against
individuals or cohorts of individuals in favour of others (Friedman &
Nissembaum, 1996) - across axes such as age, gender and accent.
  ML is reliant on large datasets for training. Dataset documentation is
designed to give ML Practitioners (MLPs) a better understanding of a dataset's
characteristics. However, there is a lack of empirical research on voice
dataset documentation specifically. Additionally, while MLPs are frequent
participants in fairness research, little work focuses on those who work with
voice data. Our work makes an empirical contribution to this gap.
  Here, we combine two methods to form an exploratory study. First, we
undertake 13 semi-structured interviews, exploring multiple perspectives of
voice dataset documentation practice. Using open and axial coding methods, we
explore MLPs' practices through the lenses of roles and tradeoffs. Drawing from
this work, we then purposively sample voice dataset documents (VDDs) for 9
voice datasets. Our findings then triangulate these two methods, using the
lenses of MLP roles and trade-offs. We find that current VDD practices are
inchoate, inadequate and incommensurate. The characteristics of voice datasets
are codified in fragmented, disjoint ways that often do not meet the needs of
MLPs. Moreover, they cannot be readily compared, presenting a barrier to
practitioners' bias reduction efforts.
  We then discuss the implications of these findings for bias practices in
voice data and speech technologies. We conclude by setting out a program of
future work to address these findings -- that is, how we may "right the docs".
Right the docs: Characterising voice dataset documentation practices used in machine learning
http://arxiv.org/abs/2303.10721v1