Curbstoning and beyond: Confronting data fabrication in survey research <s> Analysis of duplicate data <s> 1. Introduction 2. Respondents' understanding of survey questions 3. The role of memory in survey responding 4. Answering questions about date and durations 5. Attitude questions 6. Factual judgments and numerical estimates 7. Attitude judgments and context effects 8. Mapping and formatting 9. Survey reporting of sensitive topics 10. Mode of data collection 11. Impact of the application of cognitive models to survey measurement. <s> BIB001 </s> Curbstoning and beyond: Confronting data fabrication in survey research <s> Analysis of duplicate data <s> Preface. Chapter 1. The Evolution of Survey Process Quality. 1.1 The Concept of a Survey. 1.2 Types of Surveys. 1.3 Brief History of Survey Methodology. 1.4 The Quality Revolution. 1.5 Definitions of Quality and Quality in Statistical Organizations. 1.6 Measuring Quality. 1.7 Improving Quality. 1.8 Quality in a Nutshell. Chapter 2. The Survey Process and Data Quality. 2.1 Overview of the Survey Process. 2.2 Data Quality and Total Survey Error. 2.3 Decomposing Nonsampling Error into Its Component Parts. 2.4 Gauging the Magnitude of Total Survey Error. 2.5 Mean Squared Error. 2.6 An Illustration of the Concepts. Chapter 3. Coverage and Nonresponse Error. 3.1 Coverage Error. 3.2 Measures of Coverage Bias. 3.3 Reducing Coverage Bias. 3.4 Unit Nonresponse Error. 3.5 Calculating Response Rates. 3.6 Reducing Nonresponse Bias. Chapter 4. The Measurement Process and Its Implications for Questionnaire Design. 4.1Components of Measurement Error. 4.2 Errors Arising from the Questionnaire Design. 4.3 Understanding the Response Process. Chapter 5. Errors Due to Interviewers and Interviewing. 5.1 Role of the Interviewer. 5.2 Interviewer Variability. 5.3 Design Factors that Influence Interviewer Effects. 5.4 Evaluation of Interviewer Performance. Chapter 6. Data Collection Modes and Associated Errors. 6.1 Modes of Data Collection. 6.2 Decision Regarding Mode. 6.3 Some Examples of Mode Effects. Chapter 7. Data Processing: Errors and Their Control. 7.1 Overview of Data Processing Steps. 7.2 Nature of Data Processing Error. 7.3 Data Capture Errors. 7.4 Post-Data Capture Editing. 7.5 Coding. 7.6 File Preparation. 7.7 Applications of Continuous Quality Improvement: The Case of Coding. 7.8 Integration Activities. Chapter 8. Overview of Survey Error Evaluation Methods. 8.1 Purposes of Survey Error Evaluation. 8.2 Evaluation Methods for Designing and Pretesting Surveys. 8.3 Methods for Monitoring and Controlling Data Quality. 8.4 Postsurvey Evaluations. 8.5 Summary of Evaluation Methods. Chapter 9. Sampling Error. 9.1 Brief History of Sampling. 9.2 Nonrandom Sampling Methods. 9.3 Simple Random Sampling. 9.4 Statistical Inference in the Presence of Nonsampling Errors. 9.5 Other Methods of Random Sampling. 9.6 Concluding Remarks. Chapter 10.1 Practical Survey Design for Minimizing Total Survey Error. 10.1 Balance Between Cost, Survey Error, and Other Quality Features. 10.2 Planning a Survey for Optimal Quality. 10.3 Documenting Survey Quality. 10.4 Organizational Issues Related to Survey Quality. References. Index. <s> BIB002
In remote areas, the task of fabrication prevention is more difficult and new methods are needed to improve and ensure data quality. Our first suggestion for improving the survey fabrication detection toolkit is a simple one: conduct an analysis of duplicate data. This refers to exploring the prevalence of repeated instances of identical data within two or more interviews (cases) in the same data file. Looking for duplicate data is not necessarily a new idea, but the practice does not appear to be in widespread use, based on the ease with which we discovered large amounts of duplicate data in publicly available datasets from reputable survey organizations. Analysis of duplicates bears similarities to recordlinkage methods, a field of inquiry with a long history which may hold promise for future improvements in fabrication detection. The duplicate data we discuss here take several forms. In some instances, there are lengthy "duplicate strings," which refers to consecutive variables within a case where response values are repeated in one or more other cases elsewhere in the data file. There are also "near-duplicate cases," where a very high percentage of a case's values are duplicated. In extreme cases, there are full duplicate cases, where the entire case appears more than once in the data file, with the exception perhaps of case IDs or other administrative data. And in the most absurd form of all, we see "duplicate blocks," where entire blocks of two or more consecutive cases are duplicated in the same data set. These forms overlap in some ways, which matter for their detection. For example, a full duplicate case could be seen as a duplicate string where the string length is equal to the entire case. A near duplicate case may be the result of a very long duplicate string, though it may not be if some of the variables were manually altered. Examining each form of duplication individually is useful because they may exist independently, and because the specific patterns in the data may suggest how the fabricated data was generated and why. For example, a duplicate block suggests a rectangular block of data was copied and pasted within a spreadsheet program. Near duplicates may indicate a copy and paste with some attempt to disguise the duplication. And duplicate strings pasted onto the end of what otherwise appear to be legitimate interviews may indicate interviewers taking shortcuts by fabricating only the end of an interview. Duplicate data represent a possible indicator of fabrication because one would expect to see some variation in how individuals respond to surveys. Based on the mechanics of answering questions, it would be quite unlikely for two individuals to give identical an-swers to a large number of questions. The mental processes respondents undertake when answering survey questions can be viewed as having multiple steps, each of which can contribute to variability in responses. Tourangeau et al. BIB001 present a useful, four-step model of cognition, retrieval, judgment, and response. Each step of the process contributes variation to survey response, meaning even two respondents who largely agree with one another would likely answer some questions differently. Misinterpretation of questions, imperfect recall, erroneous inference, and differing interpretations of scales offered for responses are among the many factors that would add even more variation to question responses. There will be still greater variation to the extent that people actually disagree with one another -as they often do. Further error, and hence variation, enters real data through such factors such as limited interviewer comprehension, poor transcription and mistakes in data entry BIB002 . All of this makes it virtually impossible that a large number of duplicate cases could result from legitimate interviews, especially in longer surveys and when the numbers of duplicates climb into the hundreds, as they do in instances presented in this paper. Robbins  argues there is no meaningful difference in the probability of a case where 99 percent of the data is duplicated vs a full duplicate, in a survey of at least moderate length. There are idiosyncrasies in every dataset which may lead to some pieces of duplicated data. Consecutive questions with little variance would be one example. In general, though, when lengthy duplicate strings or full duplicate cases appear in the data, especially clustered by interviewer, supervisor, or keypuncher, one may strongly suspect fabrication. To examine an instance where duplicate analysis can lead us to suspect fabrication, we consider several international surveys covering multiple countries. The sponsors for these surveys were a combination of major US universities and government agencies. They were fielded by large and reputable survey organizations and analyzed by experienced researchers. They did not fit the description offered by Lavrakas  , who writes "The most serious [of fabrication] cases seem to occur in small studies by researchers who do not have professional survey services at their disposal." Table 1 shows the prevalence of duplicate cases in a selection of international surveys. These surveys were chosen based on the ease of access to the raw data on the internet. This research did not include a systematic exploration of publicly available datasets, though this would be a task perhaps worthy of future research. These duplicates were found using the "identify duplicate cases" function in SPSS, with slight modifications. As is readily apparent, full duplicate cases are very prevalent in these datasets, representing hundreds of cases and between 28 percent and 66 percent of cases in each data file. It is strange that full duplicate cases are as easy to find as was the case in this analysis. Full duplicate cases are the simplest and easiest form of fabrication to detect. Some statistical packages even have menu commands to find duplicates. With more attention to the issue and public discussion, it may be that duplicates will become a far less frequent form of fabrication in the future, or at least one that is caught more frequently during data collection and processing. If so, it will eliminate one form of fabrication and one of the easiest to detect. Detection of duplicate data should only catch the least clever and most careless of fabricators. A clever fabricator may come up with more sophisticated methods that would evade this most basic of checks. This illustrates a basic and persistent challenge of fabrication detection. It is impossible to know for sure whether the fabrication that was discovered represents a full accounting or just the simplest and easiest to detect. Also present in these same datasets are "duplicate blocks". These are an even more exaggerated form of full duplicate cases, where identical sets of cases appear more than once, in the same order, in different parts of the data file. Instances of duplicate blocks are scattered throughout the data files outlined above. The most extreme case found in the course of this research showed a block of 13 consecutive cases that had been fully duplicated in another part of the data file. There is no conceivable way that duplication could occur on such a large scale with these patterns solely an artifact of individual interviewers fabricating their own interviews, without collaborating, and entirely via paper and pencil. Large duplicate blocks represent a high level of carelessness on the part of the apparent fabricator. We expect this will become rarer as the search for duplicate cases and blocks become common.