<story><title>What&apos;s wrong with social science and how to fix it (2020)</title><url>https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/</url></story><parent_chain></parent_chain><comment><author>poszlem</author><text>We absolutely need to change the way we do social sciences. My favourite pet peeve in social sciences is idea laundering.&lt;p&gt;&amp;quot;It’s analogous to money laundering. 
Here’s how it works: First, various academics have strong moral impulses about something. For example, they perceive negative attitudes about obesity in society, and they want to stop people from making the obese feel bad about their condition. In other words, they convince themselves that the clinical concept of obesity (a medical term) is merely a story we tell ourselves about fat (a descriptive term); it’s not true or false—in this particular case, it’s a story that exists within a social power dynamic that unjustly ascribes authority to medical knowledge.&lt;p&gt;Second, academics who share these sentiments start a peer-reviewed periodical such as Fat Studies—an actual academic journal. They organize Fat Studies like every other academic journal, with a board of directors, a codified submission process, special editions with guest editors, a pool of credentialed “experts” to vet submissions, and so on. The journal’s founders, allies and collaborators then publish articles in Fat Studies and “grow” their journal. Soon, other academics with similar beliefs submit papers, which are accepted or rejected. Ideas and moral impulses go in, knowledge comes out. Voilà!&lt;p&gt;Eventually, after activist scholars petition university libraries to carry the journal, making it financially viable for a large publisher like Taylor &amp;amp; Francis, Fat Studies becomes established. Before long, there’s an extensive canon of academic work—ideas, prejudice, opinion and moral impulses—that has been laundered into “knowledge.” (source: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.wsj.com&amp;#x2F;articles&amp;#x2F;idea-laundering-in-academia-11574634492&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.wsj.com&amp;#x2F;articles&amp;#x2F;idea-laundering-in-academia-115...&lt;/a&gt;)&lt;p&gt;I was one of the extreme &amp;quot;trust the science&amp;quot; people, until I joined a startup that worked with academia. The amount of pettiness, vindictiveness, cutthroat power games I have seen  surpassed even the most hardcore of startups.</text></comment>
<story><title>What&apos;s wrong with social science and how to fix it (2020)</title><url>https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/</url></story><parent_chain></parent_chain><comment><author>matthewdgreen</author><text>Reading through this quickly I see several problems that make me wonder if the author has worked in academia.&lt;p&gt;1. &lt;i&gt;Publication venues aren’t created equal.&lt;/i&gt; People outside of academia don’t understand that anyone can and will start a journal&amp;#x2F;conference. If I want to launch the Proceedings of the of Confabulatory Results in Computer Science Conference, all I need is a cheap hosting account and I’m ready to go. In countries like the US this is explicitly protected speech, and the big for-profit sites run by Elsevier et al. will often go ahead and add my publications to their database as long as I make enough of an effort to make things look legitimate.&lt;p&gt;In any given field there are probably a handful of “top” venues that everyone in the field knows, and a vast long tail ranging from mid-tier to absolute fantasist for-profit crap. If you focus on the known conferences you might see bad results, but if you focus on the long tail you’re doing the equivalent of searching the 50th+ page of Google results for some term: that is, you’re deliberately selecting for SEO crap. And given the relatively high cost of peer-reviewing the good vs &lt;i&gt;not peer-reviewing&lt;/i&gt; the bad, unfiltered searches will always be dominated by crap (surely this is a named “law”?) Within TFA I cannot tell if the author is filtering properly for decent venues (as any self-respecting expert would do) or if they’re just complaining about spam. Some mention is made of a DARPA project, so I’m hopeful it’s not &lt;i&gt;too&lt;/i&gt; bad. However even small filtering errors will instantly make your results meaningless.&lt;p&gt;2. &lt;i&gt;Citations aren’t intended as a goddamn endorsement (or metric).&lt;/i&gt; In science we cite papers liberally. We do not do this because we want people to get a cookie. We don’t do it because we’re endorsing every result. We do it because we’ve learned (or been told) about a &lt;i&gt;possibly relevant&lt;/i&gt; result and we want the reader to know about it too. When it comes to citations, more is &lt;i&gt;usually&lt;/i&gt; better. Just as it is much better to let many innocent people go free rather than imprison one guilty one, it is vastly better to cite a hundred mediocre or incorrect papers than to miss one important citation. Readers should not see a citation and infer correctness or importance unless &lt;i&gt;the author specifically states this in the text&lt;/i&gt; at which point, sure, that’s an error. But most citation-counting metrics don’t actually read the citing papers, they just do text matching. Since most bulk citations are just reading lists, this also filters for citations that don’t mean much about the quality of a work.&lt;p&gt;The idea of using citation counts as a metric for research quality is a bad one that administrators (and the type of researchers who solve administrative problems) came up with. It is one of those “worst ideas except for any other” solutions: probably better than throwing darts at a board and certainly more scalable than reading every paper for quality. But the idea is artificial at best, and complaints like “why are people citing incorrect results” fundamentally ask citations to be something they&amp;#x27;re not.&lt;p&gt;Overall there are many legitimate complaints about academia and replicability to be had out there. But salting them with potential nonsense does nobody any favors, and just makes the process of fixing these issues much less likely to succeed.</text></comment>