Sampling Methods for Web and E-mail Surveys <s> Bigger samples are not always better <s> OBJECTIVES ::: The Internet may provide a cost-effective means to collect outcomes data needed to improve the quality and efficiency of medical care. We explored the feasibility and methodology of a longitudinal outcomes study of Internet users who have ulcerative colitis (UC). ::: ::: ::: METHODS ::: We created an open-enrollment electronic survey of Internet users who have UC and recorded the number of respondents, their demographics, and their willingness to participate. ::: ::: ::: RESULTS ::: In a 2-month period, 582 users browsed the survey, 172 (30%) completed the questionnaire, and 162 (95%) reported willingness to enroll this study. Eighty-three percent were willing to release their medical records to verify their diagnosis. Most (> 70%) had the same E-mail address over 2 yr, suggesting that long-term follow-up could be performed electronically. In comparison with the male predominance of Internet users, respondents had gender distribution similar to that of patients who have UC. In comparison with the general population, respondents have higher education and higher household income. ::: ::: ::: CONCLUSIONS ::: The Internet community could serve as a resource for general population outcome studies. Selection bias due to limited availability and use of the networked computers may affect results. The Internet community, however, is expanding rapidly, so it should become more representative of the general population. <s> BIB001 </s> Sampling Methods for Web and E-mail Surveys <s> Bigger samples are not always better <s> Summary and Conclusions Use of the World Wide Web to conduct surveys provides enormous opportunities as well aschallenges. The cost advantages of collecting large amounts of data at very little cost means thatits use will escalate rapidly. At the same time, maintaining quality, whereby the results ofsurveys can be generalized to a large population is far from automatic.Respondent-friendly questionnaire design, found important to improving response to self-administered mail questionnaires, is also important for the development of web questionnaires.However, in a web context the term takes on a broader meaning. A respondent-friendly webquestionnaire is one that interfaces effectively with the wide variety of computers and browserspossessed by respondents. It also makes other aspects of the response task easy and interestingfor the respondent to complete. Respondent-friendly designed is aimed explicitly at reducingthree of the four types of error that typically prevent accurate surveys from being done, i.e.,nonresponse, measurement, and coverage of the survey population.Three important criteria for achieving respondent-friendly design have been stated. Theyinclude, first, compliancy with technology available to the respondent. If a browser cannotrecognize the programmed information being sent to it, the respondent cannot receive it.Questionnaires that use advanced programming that takes longer transmission times and requiresmore advanced browsers may result in unintended coverage and nonresponse error. A secondcriterion is the necessity of bridging between the logic by which respondents expect a question-naire to operate and the logic associated with operating a computer. It’s this connection thatprovides a questionnaire design challenge not previously faced by survey methodologists. Inaddition to these criteria, web designers must also consider the high likelihood that coverageproblems will require that most quality web surveys, for the foreseeable future, will require themixing of survey modes, whereby some respondents are surveyed by mail or telephone whileothers are surveyed via the web. <s> BIB002
With Internet-based surveys using a listbased sampling frame, rather than sending the survey out to a sample, researchers often simply send the survey out to the entire sampling frame. That is, researchers naively conducting (all electronic) Internet-based surveys -where the marginal costs for additional surveys can be virtually nil -often fail to recognize the 'trade-off between easy, low cost access to large numbers of patients [participants] and the representativeness in the population being studied' BIB001 . As we previously discussed, for both probability and non-probability-based samples, larger sample sizes do not necessarily mean the sample is more representative of any greater population: a sample can be biased whether it is large or small. One might argue that in these situations the researchers are attempting to conduct a census, but in practice they are forgoing a probability sample in favour of a convenience sample by allowing members of the sampling frame to opt into the survey. BIB002 summarized this practice as follows: '…the ease of collecting hundreds, thousands, or even tens of thousands of responses to web questionnaires at virtually no cost, except for constructing and posting, appears to be encouraging a singular emphasis on the reduction of sampling error'. By this Dillman et al. mean that researchers who focus only on reducing sampling error by trying to collect as large a sample as possible miss the point that it is equally important to reduce coverage, measurement, and nonresponse error in order to be able to accurately generalize from the sample data. A myopic focus on large sample sizesand the idea that large samples equate to sample representativeness which equates to generalizability -occurs with convenience sample-based web and e-mail surveys as well. 'Survey2000' is an excellent example of this type of focus. A large-scale, unrestricted, self-selected survey, conducted as a collaborative effort between the National Geographic Society (NGS) and some academic researchers, Survey2000 was fielded in 1998. The survey was posted on the National Geographic Society's website and participants were solicited both with a link on the NGS homepage and via advertisements in NGS periodicals, other magazines, and newspapers. Upon completion of the effort,  report that more than 80,000 surveys were initiated and slightly more than 50,000 were completed. While this is an impressively large number of survey completions, the unrestricted, self-selected sampling strategy clearly results in a convenience sample that is not generalizable to any larger population. Yet,  go to extraordinary lengths to rationalize that their results are somehow generalizable, while simultaneously demonstrating that the results of the survey generally do not correspond to known population quantities.