vey Accuracy?
Imagine if your accuracy was instead +/-10%. Now the interval widens to 30% to 50%. Would you now feel as confident about your decisions based on the survey results? Obviously not.
Here’s another important reason to understand this. You’re likely trending the survey data over time. If next year, the “Percent Agree” was 43%, have the feelings of our target group really changed? I can guarantee you that most managers will conclude it has – especially if it means a bonus. However, with a +/-5% accuracy, the true trend might not be 40% to 43%, but instead it could have been 45% to 38% — if the two scores had actually fallen at the extremes of the 2 ranges!
We do have statistical procedures to assess whether the scores have really changed – a so called statistically significant difference – but we’ll keep the discussion simple here.
The point I want to stress here is that many readers of your survey statistics will think they have Truth – with a capital T. They don’t. They have an indication of the truth with some amount of sampling error. Decision makers shouldn’t go off half-cocked over small changes in survey scores without doing additional statistical analysis.
Where Does “Statistical Confidence” Enter the Picture?
I’ll close here adding one more piece to the puzzle that’s I’ve ignored to keep the discussion simpler. You may have noticed my dodgy statement earlier “we can be pretty certain that if we got data from our whole group…”
In reality, we have some level of confidence that the sample data provide a certain accuracy. So to be proper, the example I’ve been using is that we’re 95% confident of having an accuracy of +/-5%. Maybe you’ve seen a phrase like this: “95% +/-5%.” Now you know what that means.
Here’s the good part. By convention, we always use a 95% confidence level, which is why we seldom hear the confidence level presented. In some circumstances we might use a higher confidence level (say, 99%) or a lower one (say, 90%).
So, focus on the survey statistical accuracy.
To satisfy the statistician purest, here’s the correct interpretation of the example I’ve been using: If we conducted the identical survey using 20 different samples drawn from the same population, we’d expect the results from 19 of the 20 (95%) to fall within the accuracy interval (+/-5%).
So when we see the pollsters get an election prediction wrong, the reason could be that it was that one out of the 20.
Finally, I promised to define margin of error, sometimes abbreviated as MOE. It’s the same as statistical accuracy.
Survey Design and Data Analysis Workshops
For a more detailed discussion about this topic and all things related to a survey project…