Being surveyed is now a routine part of life for many people, especially students, who might fill out course evaluations, mental health checklists, and questionnaires about their substance use multiple times in a single year. Recall your thoughts the last time you were surveyed about something personal: Why are they asking me this? What are they going to do with this information? Why would I tell you? If these questions sound familiar, we wouldn’t be surprised if some of your responses were carefully crafted to be strategic rather than strictly truthful. In our current research, Melissa Wolf and I acknowledge how big a problem lying on school surveys can be (huge) and investigate why students do it. By better understanding the motivations of survey lying we hope to help researchers prevent and detect it. We argue that without a good idea about how much lying is going on, and what kind of lying, much school-based statistical research is vulnerable to distortion.
Finding the Liars
To study the phenomenon of survey lying (technically called “invalid response”) we used a procedure known as cognitive interviewing, a guided technique that helps people explain their mental processes in high detail. The stimuli we asked students to consider were based on a popular school mental health survey and a section of the California Healthy Kids Survey that included sensitive items pertaining to harassment, bullying, sexual orientation, and drug and alcohol use on campus. We started by asking students to identify items that were difficult for them to answer, and before long we were always talking about lying.
How Do You Lie?
Using this method, we found evidence of four basic types of invalid response behavior: mischievous responses, low-effort responses, reticent responses, and subversive responses. The first two types of responses were already well-known to researchers, while the last two were not previously identified. Mischievous response happens when people don’t take the research situation seriously, often reporting a highly unusual combination of demographics that they find amusing (e.g. they are blind, eight feet tall, orphaned, and in a gang) and then giving extreme responses to survey items (e.g. high rates of depression, high rates of drug use). Low-effort response happens when people are not invested in the survey and have little interest in giving thoughtful responses (e.g. choosing the “neutral” option on every question).
The two types of invalid response we introduce to the survey literature are reticent and subversive response. Reticent response is the withholding of truthful responses because of the personally sensitive nature of items. Examples of reticent responders may include suicidal teens who don’t want people to know that they are suicidal or students who don’t want to confess to being addicted to opiates. Subversive response happens when participants speculate about intended uses or interpretations of the survey and give misleading responses that they believe will bring about desirable results. For example, a student who wants more services for people at her school may untruthfully indicate that she needs these services on an anonymous survey. Individual students reported that they might use several of these response styles in a single survey.
Survey lying is common and there are a variety of motivations for doing it – so where do we go from here? Schools, we argue, should work towards earning the institutional trust of students and try to present tangible outputs from survey research. As one of our participants put it, “I would love to take [this survey] if it was like ‘OK, we’re going to look at these results and cater to the students… what do you guys need?’” For their part, academics should clearly explain to participants any beneficial outcomes of survey research and be clear about the way that confidential information will be handled. In our future work on survey lying we will continue to develop methods that researchers can use to identify “hot” items that attract a lot of lying and to adjust their statistic