A food causes cancer. A supplement adds years. A discovery changes everything. Most of these headlines begin with a real study. The distortion creeps in between the lab and the screen, and it follows patterns that are well documented. These six checks catch most of it.
1. Find the study behind the story
Exaggeration often starts before a journalist touches the story. Researchers in the BMJ examined 462 health press releases issued by 20 leading UK universities in 2011. They found that 40 percent contained exaggerated advice, 33 percent exaggerated causal claims, and 36 percent exaggerated the step from animal research to humans.
The news followed the press releases. When a release exaggerated causal claims, 81 percent of the related news stories did too, against 18 percent when the release did not. For animal to human claims the figures were 86 percent against 10 percent. And there was "little evidence that exaggeration in press releases increased the uptake of news." The hype did not even clearly buy more coverage.
Exaggeration in news is strongly associated with exaggeration in press releases.
2. Ask whether it has been peer reviewed
Many headlines now come from preprints, papers posted publicly before review. The preprint servers say so themselves. medRxiv, the server for health research, states: "Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information." Peer review is not a guarantee, but a preprint is openly a first draft.
3. Ask whether it was in people
A result in cells or mice is a lead, not a conclusion. The National Center for Advancing Translational Sciences at NIH estimates that going from a drug target to an approved drug takes about 14 years on average, and "the failure rate during this process is more than 95 percent." That covers the whole pipeline, not only the jump from animals to humans, but the lesson is the same: early promise rarely survives intact.
4. Correlation or cause?
The National Library of Medicine puts it simply: "Often in the health sciences, finding a correlation between two variables is not enough, as correlation does not necessarily imply causality." Observational studies watch what people already do. They cannot fully separate a habit from everything that travels with it.
The National Cancer Institute offers a cautionary example. Observational studies of antioxidant supplements gave mixed results. Then nine randomized controlled trials found no evidence that the supplements help prevent a first cancer, and some found harm: in two large trials, beta carotene was linked to more lung cancer. In the institute's words, "Randomized trials are considered to provide the strongest and most reliable evidence of the benefit and/or harm of a health-related intervention."
5. Relative or absolute risk?
This is the most common trick in health headlines, usually by accident. NIH gives a worked example. If a drug lowers the risk of a disease from 2 in 100 to 1 in 100, the absolute risk falls by 1 percentage point. One hundred people would need the drug to prevent one case. The same result can be reported as a 50 percent reduction, "since 1 is half of 2."
Many times, the relative risk sounds much greater than the absolute risk, which can be confusing.
When a headline gives only a percentage change, look for the starting risk. A doubling of a 1 in a million risk is still 2 in a million.
6. Significant is not the same as important
"Statistically significant" sounds like "big" or "proven." It means neither. The American Statistical Association's 2016 statement on p values is explicit: "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result." It adds that "scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." Ask how large the effect was, in real units.
One study is a starting point
In 2015, the Open Science Collaboration repeated 100 studies from three psychology journals. Of the original studies, 97 percent had reported statistically significant results. Of the repeats, 36 percent did, and the average effect was about half as large. By other measures the picture was less stark: 47 percent of original effects fell within the repeat's confidence interval, and 68 percent stayed significant when the original and repeat data were combined.
That does not mean the originals were false. The National Academies of Sciences, Engineering, and Medicine note that results fail to replicate "for a number of reasons that do not necessarily reflect that something is wrong." It does mean that a single new finding deserves interest, not certainty.
