Ed Yong on Science, Replication, and Journalism
What this covers
Ed Yong, a science writer, discusses with economist Russ Roberts the structural failures that allow false findings to survive in published literature and spread into public understanding. The conversation centers on three interacting problems: researcher degrees of freedom—the analytical choices scientists make during data collection that can manufacture statistical significance from noise—publication bias toward positive results, and the underuse of direct replication to catch errors. Yong illustrates these mechanisms with concrete examples: Joe Simmons showing that arbitrary analytical choices could "prove" that hearing the Beatles made people physically younger, Theodore Sterling's finding that 97% of psychology journal studies report significant results, and the failure to replicate John Bargh's influential priming study about age-related word cues. The discussion moves from the science itself to its gatekeeping and communication—how conceptual replication, which treats many weak studies as validation of a concept, can amplify rather than correct bias, and how the replication crisis touches not just psychology but high-stakes fields like cancer research.
The conversation then turns to science journalism's role in laundering uncertainty into false certainty. Yong argues that journalists routinely treat individual studies as isolated points instead of placing them within strength in numbers)—the accumulated body of evidence that epidemiology actually requires. He contrasts this with how the alcohol-cancer link should be reported: not on the basis of any single study but on decades of observational study data plus biological plausibility. Roberts and Yong also explore the deeper incentive structures: how scientists, journalists, and readers all claim to seek truth yet operate under pressures—publication, tenure, audience attention—that often point elsewhere. The exchange touches on the distinction between limits of science and limits of scientists, whether psychology's openness to self-criticism sets a model for other fields, and whether the internet accelerates both the spread of false claims and the ability to scrutinize them—though with no guarantee that corrections ever catch up to errors.
Yong argues that replication is the under-appreciated, under-rewarded backbone of science, and that a combination of conceptual replication, researcher degrees of freedom, and publication bias allows false findings to persist—failures that science journalists too often launder into public 'truth' instead of scrutinizing.
- Direct replication is rare and disincentivized while 'conceptual replication' lets weak findings prop each other up
- Researcher degrees of freedom plus publication bias can manufacture statistically significant false positives
- Journalists treat each new study as an isolated, authoritative point rather than placing it in the accumulated body of evidence
Subconscious bias, not fraud, is science's core problem—researchers unconsciously reinterpret results until randomness confirms their hypothesis.
- Outright fraud is not the real problem in science because fraud is detectable once found; the deeper problem is subconscious bias, where a researcher who believes a hypothesis is true reinterprets a null result as a flawed sample or wrong specification and re-runs the study with altered terms until randomness produces a significant result, which then invalidates the classical tests of significance.
“fraud is not the real problem. Fraud, you can find fraud and you can actually see fraud when you do find it. The real problem is subconscious fraud.”
- Most academics, scientists, and journalists genuinely see themselves as searchers for truth, but this is a form of self-deception, because they are simultaneously affected by powerful incentives—to be published, get tenure, be famous, respected—that often do not push them toward truth; the healthier stance is to acknowledge one's biases openly rather than maintaining a false veneer of objectivity, since admitted bias is more powerful than striving for an unachievable neutrality.
“most academics, most medical researchers, most physicists see themselves as searchers for truth. But they don't act that way... we of course are affected by hundreds of other things--our incentives to be published, to get tenure, to be famous, to be lauded, to be respected.”
- Scientists are often defensive about publicly criticizing scientific results because they fear it opens the door to religion and superstition, but this is misguided: there is a crucial difference between the limits of science (a great achievement of the human mind) and the limits of scientists (humans subject to incentives and biases), and being honest about the latter is far better than deceiving the public about reliability.
“they are afraid that if we find that science isn't as scientific as we thought it was, maybe that opens the door toward religion and superstition... Science is one of the great achievements of the human mind; but what it consists of that we can act on and rely on and use is much more complicated.”
Replication failures across biomedical research and methodologically suspect studies demonstrate how bias corrupts even high-stakes science.
- A large UK study (over 100,000 women) claiming even one drink a week raises cancer risk was methodologically suspect because it discarded non-drinkers from the sample—justified by the 'sick quitter' concern that some abstainers are former heavy drinkers—yet the same contamination logic applies to heavy drinkers (who may be recent starters), and the discarded non-drinkers actually had higher cancer rates than modest drinkers, making the exclusion a sign the analysis was junk.
“they said: We threw out the people who didn't drink... So they didn't drink the last month or the last year, but maybe they became teetotalers... So you can't partially justify. And of course, strangely enough, the people who didn't drink had different cancer rates. They were higher than the modest drinkers!”
- A study by Begley identified 53 landmark cancer research studies and found that 47 of them could not be replicated, and in one case the original author admitted he had run the experiment six times, got the result once, and published that single instance because it made the best story—demonstrating that the replication problem is not confined to soft sciences but extends to high-stakes biomedical research.
“this was a study that identified 53 landmark studies in cancer research and allegedly 47 of these 53--47!--could not be replicated.”
Most people care more about convenient facts as weapons than truth, so corrections lag behind errors due to human nature.
- The deeper reason corrections lag behind errors is not the internet but human nature: most people don't really care about the truth and instead use convenient facts as weapons to grind an axe or feel good, so an inconvenient statistic is simply replaced by another lying around—though Yong holds that professional journalists, unlike the average person, have no excuse because caring about truth is their job.
“Most of us don't really care about the truth; which is human. We are interested in grinding an axe, feeling good about ourselves, feeling angry about something... it was laying there; I picked it up, that statistic. And if told me it wasn't true, I'd feel bad about it; I'd find a different weapon, which was lying around elsewhere.”
The Beatles study was a great musical demonstration.
- The Beatles study was a great musical demonstration.
“It's a miracle. I always said the Beatles were a great musical group. But it turns out they are even better than that.”