What this covers
Ian Ayres and Russ Roberts debate the reliability of statistical analysis versus human judgment across domains from medicine to criminal justice. Ayres, drawing on his book Super Crunchers, argues that randomized experiments and data-driven prediction consistently outperform expert intuition, especially when many causal variables are at play. Roberts, the host, grants that randomized trials have merit but argues that most statistical work on contentious social questions—guns, abortion, the death penalty—amounts to faith-based reasoning where researchers massage specifications until results confirm their priors. The exchange ranges from concrete cases (wine quality prediction, Capital One's A/B testing, Supreme Court voting patterns) to methodological foundations: whether regression-based causal inference can ever truly isolate causation or merely correlate, and what distinguishes honest statistical work from confirmation bias dressed in mathematics.
Ayres grounds his case in measurement and scale. He shows how crude algorithms beat legal experts at predicting Supreme Court decisions, how evidence-graded medicine lets statistically untrained physicians rely on systematic research instead of intuition, and how randomized trials at companies like Capital One produce transparent, repeatable knowledge. Roberts and Ayres both invoke Ed Leamer's critique of econometric practice—the danger of trying many specifications and reporting only those that survive—but they part on remedy. For Roberts, non-experimental regression on complex social data is inherently vulnerable to researcher bias and unmeasured confounders; Ayres counters that the solution is more and better experiments, and that even crude statistics beat human judgment when prediction space grows large. The discussion touches measurement traps too: inflation's true rate, living standards trends, and hospital rules (like the four-hour pneumonia antibiotic mandate) that produce unintended costs even when the underlying finding is sound.
Ayres argues that statistical analysis—especially randomized experiments and 'super crunching' of large datasets—routinely outperforms expert human intuition, while Roberts counters that for complex social questions, regression-based causal inference is often faith-based pseudoscience vulnerable to researcher bias and reverse causation.
- Randomized experiments rely on the law of large numbers to produce transparent, trust-free causal results
- Crude statistical algorithms beat human experts precisely when there are many causal variables
- Non-experimental regression work on contentious issues (guns, abortion, Walmart) tends to confirm researcher priors and cannot reliably isolate causation
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Capital One combines historical data mining (correlating customer attributes to predict products before an operator even answers, and computing counter-offers for rate negotiations) with massive randomized experiments on mailings to identify which promotions cause more new accounts.
“before you even an operator picks up your call before the operators eyes the screen flashes the predicted products that and services that you're likely to be most interested in”
John Lott helped change the norms of data sharing in economics, contributing to a shift in which leading journals now require empirical articles to post their data publicly or explain why not—building on Ed Leamer's earlier call to 'take the con out of econometrics.'
“he's played an important role in changing the norms of data sharing... now leading journals are starting to require empirical articles to post their data publicly or give a good explanation why they are not”
Before roughly 1990 physicians rarely looked up patient-specific information because it was too time-intensive and they couldn't read the statistics; the internet now gives every physician a massive library, and graded treatments let even statistically unskilled doctors calibrate how much discretion to exercise.
“in the old days like 1990 very few physicians would ever go out and look up information for specific patients”
In a 2002 study, a crude statistical algorithm using only structural factors (was the government a party, region of origin, whether the lower court was liberal/conservative) predicted Supreme Court justices' affirm/reverse votes better than 83 legal experts, because the experts—though many had clerked on the Court—couldn't bring themselves to weight known patterns heavily enough (e.g., that the Court tends to reverse the Ninth Circuit).
“in one corner they had 83 legal experts who were trying to predict for each of the nine justices whether it would be a vote to affirm or reverse and in the other corner they had a incredibly crude statistical algorithm”
Sophisticated statistical techniques like two-stage least squares (instrumental variables) are designed to correct for reverse causation and confounding, but they are often applied to data that is simply not good enough for the task, so we can never control for all factors and may falsely attribute correlations to an unobserved variable.
“the techniques for solving these problems as glamorous and elegant as these techniques are they're often being used with data that are simply not up to the task”
Researchers' practice of trying many specifications and discarding those that fail, then reporting only the surviving result, invalidates the standard reliability measures (confidence intervals, 95% significance), because what appears in the paper is the last of hundreds of attempts rather than a first effort—a point Roberts draws from Ed Leamer.
“because researchers try so many different variables and so many different specifications that fail and they just throw those out... doing that gives you a very misleading measure of statistical reliability”
The central claim is relative, not absolute: statistical prediction is not invariably accurate or precise, but in case after case it does better than human prediction—and counterintuitively, the more subtle and complex the prediction (more than ten causal variables), the worse humans do relative to even crude statistics, because humans cannot bring themselves to put the right weights on the big variables.
“when you have more than ten underlying causal variables that's actually when the humans do relatively poorly compared to statistical prediction”
With only non-experimental data and temporal orderings, causal conclusions rest on attaching compelling narratives to correlations—'correlation is in the data but causation is in the mind of the observer'—making such inference literature and wisdom (faith-based decision-making swayed by advocates' rhetoric) rather than science (quoting Leamer).
“correlation is in the data but causation is in the mind of the observer”
Heckman's work with Donohue showed the 1964 Civil Rights Act had a dramatic and independent statistical impact on hiring of African Americans in southern textile industries, beyond pre-existing trends of progress—an example of careful microeconomics establishing a causal effect.
“the 64 Civil Rights Act has said it had a dramatic statistical impact on the amount of hiring in the in southern textile industries”
Measurement difficulties limit where statistical analysis can work: when you can't measure the outcome you care about (standard of living, happiness), lack a large enough comparable historical dataset, or can't run a randomized experiment, statistical prediction cannot answer the question.
“you can't measure the things you care about you don't have adequate sized data set on historical comparable examples you can't run a randomized experiment on whether you're going to do a moonshot or not”
The law of large numbers ensures that randomly assigned groups are statistically identical not just on average but in their full distributions, so any difference in outcomes can be attributed to the one variable being manipulated—making randomized results transparent and not requiring trust in the statistician.
“something called the law of large numbers is going to pretty much assure that the two groups of applicants who are prospects are identical on every dimension except for the fact that the one dimension that that Kaplan controls”
Regression output can report not just a point prediction but its precision (e.g., a 95% confidence range), so when data quality is low or the outcome is inherently random, the statistics themselves disclose that the prediction is imprecise.
“the regression output not only makes a prediction for you but it will simultaneously tell you the precision of that prediction”
Semmelweis's discovery that doctors moving from the morgue to childbirth without washing hands killed thousands of women was resisted partly because of his arrogance and poor marketing of the findings, illustrating that statistics presented poorly fail to persuade even when correct.
“one of the tragedies of Semmelweis is he was very arrogant and very confident and did a very quick statistical analysis that confirmed his hypothesis and he didn't spend a lot of time marketing the findings to the doctors”
Physicians still fail to wash hands enough between patients despite scrubbing before surgery, and routinized clean-hands projects that force handwashing at fixed checkpoints have been shown to save lives.
“routinized clean hand projects they could force physicians to wash their hand every time they go by a certain spot has been shown to to save life”
Evidence-based medicine has produced a systematic grading of the quality of evidence behind treatments, which creates competition among researchers to fill gaps and lets physicians who lack statistical training still rely on evidence grades rather than pure intuition.
“there is now a systematic grading of the quality of evidence behind just about every treatment or potential treatment that is out there”
Randomized trials are far less vulnerable to author bias than regressions because it is easier to 'cook the books' on regressions; the Moving to Opportunity experiment exemplifies this—its funders expected vouchers letting poor families move to middle-class neighborhoods to dramatically improve life outcomes, but preliminary results across ~40 life-chance measures over years show little improvement.
“it's easier to cook the books on regressions than it is on randomised trials”
Statistical analysis of growing conditions (rainfall, temperature) via multivariate regression predicts Bordeaux wine quality more accurately than traditional expert tasters, because the wine is unpalatable and imprecise to judge in its immature early months.
“Ashton Tucker over the last decade has been doing a better job and even Parker while he's been dismissive of Ashton halter has more and more been incorporating the weather conditions into his own predictions of wine quality”
A hospital mandate requiring pneumonia patients to get antibiotics within four hours leads to over-testing—every patient who might have pneumonia, including those with mere colds, gets an expensive chest x-ray—illustrating that data-derived rules can have hidden costs even when the underlying finding is sound.
“because of that mandate every patient who comes in who might have pneumonia gets a chest x-ray to make sure they don't miss anybody”
Even simple economic facts are elusive: claims that US living standards have stagnated since the late 1970s rely on real average hourly earnings, but inflation measurement excludes fringe benefits and demographic composition effects, so even this 'simple' data is highly controversial.
“the measurement of inflation is problematic it doesn't include fringe benefits it doesn't include population demographic changes that would through composition effects alter that”
LoJack (a hidden radio chip enabling police to track stolen cars) reduces car theft rates across an area, not just for equipped vehicles, and—unlike concealed handguns—there is no plausible theoretical mechanism by which LoJack could increase crime, since it is never used as an offensive weapon.
“there is one big theoretical distinction between LoJack and concealed weapons and that is that LoJack is never used as an offensive weapon”
Friedman's 'A Monetary History of the United States' changed the economics profession's view of inflation—shifting it from multiple competing explanations toward the consensus that changes in the money supply cause inflation—through laboriously meticulous (though statistically unsophisticated) analysis.
“if you ask me for the gold standard and empirical work in economics I would say the monetary history of the United States by Milton Friedman”
Multiple alternative approaches increasingly show that women who would have preferred to abort but did not take less good care of those children later, visible not only in the children's criminality but in a variety of other factors—supporting the Donohue-Levitt abortion thesis though it remains disputed.
“there's now tons of alternative approaches that are tending to show that women that would have preferred to have aborted... don't take as good care of the kids later on”
There is an 'iron law of resistance' whereby traditional experts (wine tasters, baseball scouts, physicians) resist yielding their status to statistical number-crunchers, not purely on the merits but due to ego and dislike of outsiders telling them what to do.
“there's almost an iron law of resistance that the traditional experts don't like yielding their status their to this new breed of number cruncher”
In contentious empirical debates (death penalty, guns, abortion), findings tend to confirm the researcher's prior bias because the vast range of possible regressions and techniques lets researchers keep crunching until they get a result that confirms their hypothesis, discarding the many that don't.
“most of the findings confirm the bias of the researcher and that is because of the incredible range of regressions you can run... they keep crunching until they do”
Randomized testing of store layouts (planograms) shows that placing higher-markup toothbrushes at chest-high prime viewing level increases purchases, demonstrating that controlled experiments reveal causal merchandising effects.
“if you put those more at chest high level in the prime viewing area of customers you it's it sparks more purchases”
The clash of competing studies, especially in contentious areas, drives progress toward eventual consensus by drawing attention and accumulating studies; emerging consensus suggests concealed-weapons laws and the death penalty (vs. life imprisonment) have little impact either way, while the immigration-wages question remains unresolved.
“the consensus is starting to develop that no concealed weapons laws don't have much of an impact one way or another and indeed that the death penalty relative to life imprisonment has not had much of an impact”
Ayres predicts the next EBM revolution will strip physicians of their 'front-end' discretion over diagnosis and test ordering, as digital medical records let data determine which classes of patients should receive which tests based on when medical benefit exceeds cost.
“the next EBM revolution that we're just in the midst of of beginning is physicians I predict are going to lose their front-end discretion”
In any institution with many frontline employees, roughly half will be below average, so designing the institution around the top 10% is irrational; routinizing decisions and removing some discretion is therefore likely to improve outcomes.
“if you're running any institution with a thousand frontline employees... about half of them are going to be below average... and for you to build your institution for those that are the top 10% is crazy”
Whether concealed weapons deter or increase crime is fundamentally an empirical question that theory cannot resolve, because theory is ambiguous about the net effect and cannot tell you the magnitude—so conflicting empirical studies (Ayres vs. Lott) are the only path to an answer.
“theory just is ambiguous as to whether unobserved precautions with regard concealed weapons will be it's an empirical question”
A good test for intellectual honesty is whether you can name a statistical study you believe but don't like; if you only believe results you like, you're a biased consumer of evidence.
“what is the statistical study that you believe but don't like and if you only believe the ones that you like you know your something is you've got to be a biased consumer”
Statistical analysis should be made contestable—Ayres argues businesses adopting number-crunching should hire 'statistical auditors' and allow people to re-analyze the same questions with alternative assumptions, mirroring the academic clash-of-studies model.
“It'll be a good sign when businesses start hiring statistical auditors or start allowing people to come to the same questions with alternative assumptions”
Roberts proposes that to address bias, empirical work might be restricted to actual experiments, to cases where statistical techniques give less scope for researcher creativity, or conducted as 'empirical tournaments' where both ideological sides analyze data simultaneously and in the open.
“maybe we should have empirical tournaments where all the data analysis would be conducted simultaneously by people on both sides of the ideological fence out in the open”
Walmart shifts pop-tart inventory upward ahead of forecasted hurricanes because data mining of past consumer responses revealed that people demand the non-refrigerated, gooey comfort food after hurricanes.
“after a hurricane people tend to like pop-tarts and that you don't need to refrigerate them... so Walmart has taken to when a hurricane is predicted to come in they will start pumping up the Pop Tart inventory”
Welcome and introduction to EconTalk and the guest Ian Ayres.
“my guest today is Ian Ayres the William K Townsend professor at Yale Law School”