How do we know that regression results are real and not the result of bias? Edit: this question was originally about political science alone but applies to any field that relies heavily on regressions.
I am an undergrad pursuing a double major in physics and political science. The combination gives me a really interesting view of how different fields do things. Physicists tend to view political scientists as &quot;not real scientists&quot;, so I was surprised to find that most of my entry-level political science classes go into the details of the scientific method (what makes a good theory, how to do research, etc.) quite a lot more than my physics classes.
But then I took a political science statistics class and now I'm filled with enormous doubt about the entire field. It seems like regressions are really easy to manipulate - even unconsciously - not just because of the never-ending problem of omitted variable bias, but also because of the huge number of different combinations of variables that can go into those regressions. You can easily get sixty different results using the same data.
So my question is, how do we know political scientists aren't just confirming their own biases? For example, this article on FiveThirtyEight shows how scientists in a bunch of studies looking at who got red cards in football games arrived at the same general conclusion despite having slightly different results. (They concluded that referees were more likely to give a red card to dark-skinned players than light-skinned ones.) I don't really doubt that result because I know about racism in general, and it makes sense that referees would be unintentionally racist with red cards. But how do we know that the scientists weren't just all finding a conclusion that lines up with what they thought the result should be? How do we know which studies were right in finding a relationship (vs. the ones that found no significant relationship)?
First, any statistical measure provides only evidence of the truth of something, not proof that it is true. Only a population study can provide proof and in many cases, especially with human studies, it is impossible to nail down a population (since the data changes over time as people grow, change, and die).
One can certainly design a &quot;study&quot; to come to a preconceived result. This is why experimental design is all important and must be disclosed in any research. Then, with all the details of what was done, a reader has the opportunity to replicate the study or to question it. But, assuming the &quot;researcher&quot; isn't just cheating, it is the research design (hypotheses and methodology) that one questions.
While you don't arrive at truth in statistics, you can approach it by replicating studies and also by researching the same hypotheses with different research designs. If they agree, then you have additional evidence.
In some studies, such as those involving racial bias, part of the design might need to be making the data blind in some way to the researchers.
Good papers provide complete design information including reasons for the decisions that were made about data (collection, analysis, possible alternative designs...).