Part of the appeal of science is that it’s rigorous and methodical. We can assume that studies are generally right about what they claim. But, most studies focus on a narrow scope in order to control all input variables, and therefore the details matter a huge deal.
News, on the other hand, has to report an exaggerated version of reality because of its business model. Journalists are literally paid for capturing people’s attention, which is always easier with a bold claim. The bolder the claim, the more eyeballs the article will attract.
The problem with science is that it sounds wishy-washy. If you need to convince someone, “That thing MAY kill you” will beat “There’s evidence to support the opposite, but it can’t be ruled out that that thing kills you,” every time.
And then we have real life, which is complicated. It has variables that can never be controlled, and the topics that the public is interested in are way too broad for a single study. What’s left out from a study will be just as interesting as what it states.
Media actively exploits this phenomenon: advertisements and news outlets routinely cite scientists, studies and experiments to sound more credible. But if the decision-making part of our brain is switched off, who will fact check whether “4 out of 5 dentists” do in fact recommend a product?
To be a better reader of science journals, the solution is simple. Just keep your decision-making brain switched on.
Next time you read a study, keep asking questions such as:
- Does it apply? Cars get their safety ratings through crash tests — a relatively small set of controlled “accidents” that manufacturers can design cars around. How much does a crash test score tell us about a real-life crash? As you might have guessed, most models don’t do well in accidents they weren’t optimized for.
- How does it fit the bigger picture? We might be interested in the spread of a pandemic. This will encompass findings from all sorts of fields, from virology to network analysis to social sciences. A perfectly credible expert might tell us about whether a single virus can escape a face mask’s thin fabric without an issue — but we’d still need to know how many virus molecules are needed for an infection, or the size of the saliva droplets that virus molecules travel on.
- A mathematical model is just an opinion with a spreadsheet. Whenever life gets too complicated, we use models to estimate different outcomes. A city’s mayor might ask, “If we improve our roads and lower bus prices, how many people move into the suburbs?” — and the computer will give its best answer.
Prediction models are an amazing tool to discuss different options, but they’re more a visualization tool than a proven fact: they leave a lot to human judgment. Only believe the model as much as you believe the human presenting it.
Remember that experts can be wrong, science often changes its mind, and rational thinking is still the best we can all do. Being more critical about what you read will eventually help science journalism improve too.