In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development — through the adoption and widespread use of data statements, the field can begin to address critical scientific and ethical issues that result from the use of data from certain populations in the development of technology for other populations. We present a form that data statements can take and explore the implications of adopting them as part of regular practice. We argue that data statements will help alleviate issues related to exclusion and bias in language technology; lead to better precision in claims about how NLP research can generalize and thus better engineering results; protect companies from public embarrassment; and ultimately lead to language technology that meets its users in their own preferred linguistic style and furthermore does not mis- represent them to others.

 ** To appear in TACL ** As technology enters widespread societal use it is important that we, as technologists, think critically about how the design decisions we make and systems we build impact people -including not only users of the systems but also other people who will be affected by the systems without directly interacting with them. For this paper, we focus on natural language processing (NLP) technology. Potential adverse impacts include NLP systems that fail to work for specific subpopulations (e.g. children or speakers of language varieties which are not supported by training or test data) or systems that reify and reinforce biases present in training data (e.g. a resume-review system that ranks female candidates as less qualified for computer programming jobs because of biases present in training text). There are both scientific and ethical reasons to be concerned. Scientifically, there is the issue of generalizability of results; ethically, the potential for significant real-world harms. While there is increasing interest in ethics in NLP, 1 there remains the open and urgent question of how we integrate ethical considerations into the everyday practice of our field. This question has no simple answer, but rather will require a constellation of multi-faceted solutions.Toward that end, and drawing on value sensitive design BID22 , this paper contributes one new professional practicecalled data statements -which we argue will bring about improvements in engineering and scientific outcomes while also enabling more ethically responsive NLP technology. A data statement is a characterization of a dataset which provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software. In developing this practice, we draw on analogous practices from the fields of psychology and medicine that require some standardized information about the populations studied (e.g. APA 2009; BID41 BID26 BID40 . Though the construct of data statements applies more broadly, in this paper we focus specifically on data statements for NLP systems. Data statements should be included in most writing on NLP including: papers presenting new datasets, papers reporting experimental work with datasets, and documentation for NLP systems. Data statements should help us as a field engage with the ethical issues of exclusion, overgeneralization, and underexposure BID30 . Furthermore, as data statements bring our datasets and their represented populations into better focus, they should also help us as a field deal with scientific issues of generalizability and reproducibility. Adopting this practice will position us to better understand and describe our results and, ultimately, do better and more ethical science and engineering. 2 We begin by defining terms ( §2), discuss why NLP needs data statements ( §3) and relate our proposal to current practice ( §4). Next is the substance of our contribution: a detailed proposal for data statements for NLP ( §5), illustrated with two case studies ( §6). In §7 we discuss how data statements can mitigate bias and use the technique of 'value scenarios' to envision potential effects of their adoption. Finally, we relate data statements to similar emerging proposals ( §8), make recommendations for how to implement and promote the uptake of data statements ( §9), and lay out considerations for tech policy ( §10). As researchers and developers working on technology in widespread use, capable of impacting people beyond its direct users, we have an obligation to consider the ethical implications of our work. This will only happen reliably if we find ways to integrate such thought into our regular practice. In this paper, we have put forward one specific, concrete proposal which we believe will help with issues related to exclusion and bias in language technology: the practice of including 'data statements' in all publications and documentation for all NLP systems.We believe this practice will have beneficial effects immediately and into the future: In the short term, it will foreground how our data does and doesn't represent the world (and the people our systems will impact). In the long term, it should enable research that specifically addresses issues of bias and exclusion, promote the development of more representative datasets, and make it easier and more normative for researchers to take stakeholder values into consideration as they work. In foregrounding the information about the data we work with, we can work toward making sure that the systems we build work for diverse populations and also toward making sure we are not teaching computers about the world based on the world views of a limited subset of people.Granted, it will take time and experience to develop the skill of writing carefully crafted data statements. However, we see great potential benefits: For the scientific community, researchers will be better able to make precise claims about how results should generalize and perform more targeted experiments around reproducing results for datasets that differ in specific characteristics. For industry, we believe that incorporating data statements will encourage the kind of conscientious software development that protects companies' reputations (by avoiding public embarrassment) and makes them more competitive (by creating systems used more fluidly by more people). For the public at large, data statements are one piece of a larger collection of practices that will enable the development of NLP systems that equitably serves the interests of users and indirect stakeholders.
A practical proposal for more ethical and responsive NLP technology, operationalizing transparency of test and training data
{"src": "scitldr-AIC_train", "paper_id": "By4oPeX9f"}