Stats 101: Should You Believe All Medical Studies?
By alex hutchinson
Depending on who you listen to, eating red meat will either solve every health problem you have . . . or it will kill you. To make matters worse, both sides have the statistics to back up their points of view.
Confusing right? But sometimes those stats aren't all they're cracked up to be. A 2009 study in Archives of Internal Medicine found that red meat intake was associated with increased risk of cancer, heart disease, and death. The conclusions were "adjusted" to take into account possible confounding factors like study participants' age, education, and history of drinking alcohol. But if you've ever read Men's Health before, you know where we're going with this.
To figure out if it's possible to single out red meat as the culprit, it helps to understand how statistical adjustment works: When it's done correctly, the process can pick out the crucial variables from enormous piles of messy data to determine that it's red meat and not booze that's killing people.
More from MensHealth.com: Don't Fear Red Meat
But in practice, it's never quite that clear, says Kristin Sainani, Ph.D., a Stanford University professor who specializes in the use of statistics in clinical research. "There's always some residual error, and nutrition studies in particular are terrible for this."
We took a closer look to see how statistical adjustment really works—and help you decide if you should order the filet for dinner, or trade in steaks for salads.
Consider a recent study in Cancer Prevention Research that found that men who ate more than 2.5 eggs a week were 81 percent more likely to develop a lethal form of prostate cancer. Like the red meat study, its results were adjusted for a long list of other lifestyle traits like age and BMI.
More from MensHealth.com: Do Eggs Cause Prostate Cancer?
For simplicity, let's narrow the discussion to just one confounding factor: age. You could imagine that the older you get, the more eggs you eat and the more likely you are to get prostate cancer. One way to figure out the impact of eggs alone is to restrict your study to people who are all the same age. If everyone in the study is 50 years old and egg eaters still get more prostate cancer, then you can pin the blame on eggs.
Aha! But there's one big caveat—the study doesn't tell us anything about how eggs affect 20-year-olds or 70-year-olds. That's where regression analysis comes in: This statistical technique allows researchers to achieve the same effect, even with a study population that includes all ages.
So when researchers say their results are "controlled" or "adjusted" for age, it means that the risk they're quoting was calculated as if the age of the subjects was held constant.
The Problems
But—you guessed it—the method isn't perfect, says Sainani. For starters, study volunteers are notoriously bad at self-reporting, like remembering what and when they really ate. (Pop quiz: What did YOU have for breakfast last Tuesday?) These errors don't just skew the data—they can throw off the statistical adjustment, leading to erroneous conclusions about which risk factors are actually significant.
Even if the measurements are perfect, you have to consider problems like clustering: If the people who eat lots of eggs also eat lots of bacon, then statistical adjustment has a hard time distinguishing between the two. If it turns out that bacon is the real prostate-killer, the analysis would still identify eggs as a serious risk factor, in a form of statistical guilt-by-association.
More from MensHealth.com: The 125 Best Foods for Men
And in real life, unfortunately, lots of different risk factors tend to cluster together: For example, people who smoke and are overweight also tend to eat more fast food and be less proactive about health screening. That makes it very difficult to figure out exactly which risk factor is causing any given health problem.
The red-meat-will-kill-you study offers a classic example of the potential pitfalls of this type of analysis. A closer look at the statistics shows that eating more red meat also raises your risk of dying a sudden accidental death from factors like car crashes and guns.
"Unless red-meat eaters are swerving to avoid cows, that doesn't make any sense," Sainani says. Instead, the most likely explanation is that red-meat eaters take more risks in other areas of life. But the study didn't collect any data on the driving habits and gun collections of its volunteers, so the researchers were unable to adjust for these factors—and as a result, the conclusions were skewed.
More from MensHealth.com: The Other Reason Men Die in Car Crashes
So where does that leave you when you read "Red Meat Causes Cancer" or "Eggs Linked to Early Death"? Your solution isn't to ignore medical research entirely, but carry healthy skepticism when you read new studies. Don't make big changes in your diet and lifestyle based on one article you read, but do ask yourself these questions:
1. How reliable is the data? A questionnaire that asks people to recall their diet over the previous decade is far less useful than having study participants complete food diaries for several days at a time on multiple occasions.
2. Was the study a "fishing expedition"? Some studies aim to answer a specific question; others simply collect lots of data and then look for apparent patterns. If researchers measured 20 different risk factors, you can bet that one factor will appear significant just through random chance, even if there's no actual effect.
3. How big was the effect? In the end, this is your best defense. Bad data can easily generate an apparent risk increase of up to 60 percent, according to a research paper on statistical adjustment published in the journal PM&R.; Effects bigger than that are very difficult to explain without serious errors in the design of the study.
More from MensHealth.com: The Science Behind Your Diet
Four-Day Working Week Set to Be Trialed in the UK
A Doctor Explains Whether Jaw Trainers Work or Not
Popular Diabetes Drug Recalled Over Carcinogens
A Doctor Explains Kevin Durant's Knee Injury
What My Whoop 4.0 Taught Me About Weed and Sleep
10 Ways to Kick Back Without Drinking
3 Ways to Quit Nicotine, Caffeine, and Alcohol
9 Covid-19 Booster Side Effects to Expect
How a Doctor Wants You to Treat Covid at Home
Scientists Urge Spotify to Moderate Joe Rogan
The Stat Every Football Fan Should Know
We Believe in You
The Strangest Sex Studies in 2014
The Medications You Should Never Mix with Alcohol
Stay Strong Because I Believe in You
Stop Overeating, Stat!