ers and survey researchers need to try out their questionnaires before using them on a full sample. During pretesting they try out different question wording and different question order with a few individuals to see whether they have trouble understanding the questions, whether the wording or order biases responses, or causes any other problems. If they have a particularly long or complex set of questions, they will pretest parts of the questionnaire and then conduct a pilot – ask a few individuals to answer all of the questions. A good pilot checks out all the phases of survey research – drawing a small sample, administering the questionnaire, and coding the responses for analysis. Doing so allows the lead researchers designing the survey project to make sure that any assistants who will be helping administer the questionnaire or code the data complete those steps as the lead researchers expect.
Reporting the Results
Polls are more likely to be reported in the media than surveys, though sometimes particular questions used in a survey are so interesting that journalists write about them and use the survey data in their stories.
Both polls and surveys ask a sample of people to estimate the attitudes in a larger population. Thus news stories seldom say that some percentage of those polled or surveyed gave some particular answer; they report that some percent of the relevant population (Americans, football fans, senior citizens, whatever) shares that opinion. Estimates of public opinion can be reported as point estimates – single numbers saying what percent of the population would answer the question a particular way (as in “39% of the voters in Ohio will vote yes on Question 1”) – or as an interval – a range of numbers giving the high and low estimate (as in “between 37 and 41% of the voters will vote yes on Question 1”). The point estimate treats the sample as if it exactly matches the population; the interval estimate acknowledges that even with a carefully-selected random sample that is not biased in any way it is still possible for the sample’s responses to be different than that of the whole population. Responsible pollsters and survey researchers never report a point estimate alone. Rather, they report xx things: the point estimate, a confidence interval around the point estimate that gives the highest and lowest value the actual population response could have, a confidence level stating how certain they are that the real population response is within the range of percentages covered by the confidence interval, and the size of (number of respondents in) the sample. 95% is the lowest confidence level commonly used; some researchers are more demanding and want to be 99% sure but this means having to widen the confidence interval. Sample size needs to be more than 100 for the confidence interval calculations to be correct; most pollsters prefer larger samples than that, particularly on national questions.
Calculating the confidence interval for 95% uses the formula that applies to proportions since polls are expressed in % of respondents answering a particular way. This formula is:
c.i. = Proportion of the sample + 1.96 multiplied by the square root of (.5 x .5) divided by the sample size. 1.96 is a constant that generated the interval matching a 95% confidence level and .5 is a constant used to make sure the interval is set at its maximum extent. (Those who want the explanation of why this works can consult Joseph F. Healey, Statistics for Social Research, 6th edition (2002), pp. 173-174 and background on pp. 165-168.)
Now you can see how an election can be “too close to call.” If pollsters survey their random samples of likely voters and 48% say they will vote for Arnold and 52% say they will vote for Condi but calculate that their sampling error is 3%, the pollsters have to acknowledge that Arnold could get anywhere from 45 to 51% of the vote and Condi anywhere from 49 to 55%. Condi looks like she has a lead, but with an overlap in confidence intervals, pollsters will report results but not predict a winner. They wi