The Science And Skill Of Public Polling
What to Know the Next Time Someone Tells you to Read a Poll that “Changes the Game”
In an age where everyone can serve as a political pundit on social media and companies consume data points by the millions, it is important to know how to sift through the clutter and identify a good public poll from a bad one. Most of the organizations that release the results of their polls are trustworthy sources. In fact, pollsters tend to follow similar rules when surveying an audience, regardless of the side of the aisle they sit on. It is the application of common practices that set pollsters apart and underscores the science, skill, and style behind the data.
As the 2018 elections approach, consumers of data will be presented with various polls from different sources that have varying results. The following are key reminders to help your team be effective consumers of public data:
Know How The Survey Was Conducted – Telephone vs. Online
There is a lot of contention in the survey world around whether telephone or online polls are better in the political world. Polls made available to the public will typically post how they were conducted at the bottom of the findings or in the description of the methodology. To be a good consumer of data, know how the poll was conducted and understand the benefits and detriments to each methodology.
Voter files, the lists that contain the contact information for most of the voters in the state (note: there is no nationwide voter file) are typically used for most political polls. For traditional pollsters and most campaigns, telephone polls continue to be the most effective way to guarantee you are speaking to the “right” person on the voter file, typically defined as someone with recent voting history, as there is little email information on existing voter files. Online polls are increasing their matching capabilities (confirming the person on the voter file is the same as the person on their list) with the voter file, however, the consistency is not as high as with telephone polls.
However, online surveys are an asset as Americans move further away from answering their telephones and want to spend less time taking surveys. Incidence rates, the percentage of people that answer the phone and complete a survey, continue to drop and pollsters are moving more towards gaining access to cellphone completes or going to mobile methods. Online surveys have better traction with younger populations, more mobile populations, and people of color – groups that continue to be underrepresented in many surveys. Pollsters often embrace “mixed-mode” surveys that capture responses from people via landlines, cellphones, and web-based applications to get an accurate representation of the population they are surveying.
Know What The Audience Is And How It Was Defined – Adults vs. Registered Voters vs. Likely Voters
The science behind polling is defining the audience you want to survey and ensuring the responses you receive are representative of the overall universe of the audience. For example, to capture the ballot for the Presidential race, a pollster is more likely to talk to people who consistently vote (likely voters) rather than a sample of American adults; as we know a large portion of them are either not registered to vote or have not cast a ballot in a recent election and thus less important when painting the picture of the electoral landscape. Polls with broader audiences, like American adults, tend to be younger and more diverse than likely voter samples.
During election season, it is important to find polls of likely voters rather than adults or even registered voters as they are proven to show up and vote.
Know The Size And The Demographic Breakdown Of The Audience – The Smaller The Audience, The Higher Probability For Error
The key rule to remember is that the larger the sample size, the lower the margin of error in the results. For example, the margin of error on a survey of 1,000 likely 2018 voters is +/-3%. This means each result could be within a 6-point swing; something that is often not mentioned when pushing out favorable poll results in close elections.
The skill of a pollster is seen in the way they anticipate the likely turnout and the demographic make-up of the population. A close election can be decided within a few percentage points and estimating millennial turnout at 15% when the actual turnout is 18% can throw off the results of an entire poll. The most effective way to determine the demographics of the turnout audience is to look at previous similar elections (for 2018, it is other midterm years like 2014 or 2010) and consider other outside factors – are there other competitive races that could drive turnout, what is the enthusiasm for one candidate over another, what role does the President play in the district. All of these factors are considered by a pollster when anticipating turnout. It is important for good consumers of public data to understand these dynamics, review the demographic make-up of the electorate, and then take account for the ballot. In some cases, more millennial representation in a poll could drive up support for one candidate over another; this is similar with anticipated turnout by party identification, race, and most other key demographic characteristics.
The biggest problem in polling is that consumers tend to believe it is a prediction on an election result. It is rather a snapshot in time based on what was happening at that moment. Elections can change on a dime in the world of 24/7 news.
The best thing to do is approach polls with the right questions to ensure you understand who the audience is, how the poll was conducted, and when the data was collected. View polls in this light and you will be able to tell someone if the survey truly “changed the game” or not.