How to Spot Dubious Claims in Scientific Papers
Synapse #8: Critical skills for navigating our landscape of unsubstantiated claims
This week’s issue of Synapse will be short as last week’s issue was quite long and dense. I’m particularly happy with the outcome of last week’s issue, so if you missed it, check it out here.
This week, I want to highlight the most interesting article I read: How to spot dubious claims in scientific papers by Gemma Conroy. It summarizes a few key points from the book Calling Bullshit: The Art of Scepticism in a Data-Driven World by Carl Bergstrom (whose is a fascinating follow on Twitter) and data scientist Jevin West. Here are three things to look out for when looking at scientific papers (especially those touted in the media):
1. Flawed Data
Often times science is talked about with language full of complicated jargon and math, but keep in mind that a lot of false claims can be spotted with simple common sense. For example, a 2016 preprint claimed that it had built a machine learning algorithm that could predict criminality in an individual based on a headshot with 90% accuracy. However, a cursory look at the images used in the paper showed that the criminals were mostly frowning in their headshot and the innocent slightly smiling. Thus, the algorithm really just learned to recognize smiles and did uncover any kind of causal relationship between facial features and the likelihood to be a criminal.
2. Data Censoring
A lot of times, important data can be left out of a study with dubious justification. I see this most often in clinical trials which may sometimes find ways to exclude unsuccessful patients for the sake of proving a drug worked. The example cited in the article was a 2016 study that claimed to have found that musicians of rap and hip hop disproportionally die at a young age compared to other genres like jazz and the blues. However, the study failed to account for musicians that are still alive. This is a problem because rap and hip hop are much younger genres, thus any musicians of these genres must have died young because rap and hip hop haven’t been around long enough for them to die old.
3. Spurious Correlations
Whenever a study crawls massive amounts of data in order to describe a relationship, it’s always a good idea to be skeptical. Given the number of confounding factors that occur, especially within larger and larger datasets, one can find a correlation between almost anything—including a correlation between the age of Miss America and the number of murders by steam, hot vapors, and hot objects—as Tyler Vigen points out in this book Spurious Correlations:
🤔 Ask Synapse
Like I said last week if you have a question about any previous issue of Synapse, just hit ‘reply’ to this email and I’ll be answering it in an upcoming article!
⚡️P.S. If you're new here and want to read more of the Synapse Newsletter each Sunday, subscribe below!⚡️
🐦 Follow me on Twitter