Paul Pfleiderer on the Misuse of Economic Models
What this covers
Paul Pfleiderer, a finance professor at Stanford, sits down with Russ Roberts to dissect what he calls Chameleon Models—economic and financial models that are logically sound in isolation but treat their assumptions as validated descriptions of reality without scrutiny. The conversation centers on a core tension: models are presented as delivering real-world policy guidance, yet when their assumptions are questioned, they retreat to the shelter of being "just logical exercises." Pfleiderer argues this illegitimate double-life lets flawed reasoning masquerade as science, and he traces the problem not to individual error but to the profession's collective unwillingness to filter conclusions through the assumptions that generate them.
The episode ranges across several weaknesses in how economics deploys models. Pfleiderer shows how theoretical cherry-picking works—with enough degrees of freedom in calibration, one can reverse-engineer assumptions to produce almost any desired result, so matching observed data proves nothing about whether those assumptions are correct. He challenges Milton Friedman's famous defense that assumptions need not be realistic (As-If Reasoning) by distinguishing contexts: the argument holds for repeated, fast-feedback activities like baseball, but fails for rare, complex decisions like capital structure where feedback is slow and ambiguous. The conversation also examines why falsification rarely occurs in economics—the field is non-stationary, natural experiments are scarce, and practitioners add new assumptions to explain away contradictions rather than revise their priors. Roberts and Pfleiderer discuss George Akerlof's Market for Lemons as an example of a Bookshelf Model that usefully illuminates forces like asymmetric information without claiming literal realism. The remedy Pfleiderer proposes is neither to abandon modeling nor to accept results uncritically, but to cultivate institutional humility and insist that PhD programs and professional incentives reward scrutiny of assumptions as rigorously as the construction of elegant models.
Pfleiderer argues that many economic and finance models are 'chameleons'—logical exercises built on cherry-picked, untested assumptions that are then illegitimately treated as descriptions of the real world, and that the profession lacks the humility to filter models through their assumptions before drawing policy conclusions.
- With enough degrees of freedom in assumptions and calibration, almost any desired result can be produced, so matching reality does not validate the assumptions.
- The Friedman 'as if' defense only works where agents get repeated, fast feedback (pool, baseball), not in rare, complex decisions like corporate capital structure.
- Economics is non-stationary and natural experiments are scarce, so theories are rarely disconfirmed and priors are rarely revised.
Economic models cannot observe all real-world variables, so they systematically mis-capture what drives decisions.
- Because we cannot observe the full set of variables driving real decisions (a Hayekian point), models don't merely fail to capture what's going on—they mis-capture it, which becomes especially damaging when those omitted underlying variables change in systematic ways; the lesson is humility.
“because we don't have all the variables and the information--this is a very Hayekian point, obviously--then it's not just that, well, we don't really capture what's going on. We mis-capture it.”
- Bookshelf models like Akerlof's lemons model are valuable not as literal descriptions but as guides to where to look in the real world—they reveal the underlying forces (e.g. asymmetric information) and thereby explain why markets like used cars actually survive through mechanisms such as warranties and third-party information.
“the virtue of bookshelf models--I'll be critical of them later--is that it tells you where to look to understand why in this particular case this market does work.”
The discipline resists disconfirmation by explaining away failed predictions rather than revising core assumptions.
- After WWII government spending fell roughly 60% yet no major depression occurred, contradicting Samuelson's 1943 prediction; tellingly, Keynesians did not revise their underlying assumptions but instead explained the failure away (e.g. via pent-up demand), illustrating how economics resists disconfirmation.
“Government spending fell, I think by about 60%. And there was no recession. Certainly not the worst depression--Samuelson had said it would be the worst depression of American history.”
- Competing explanations for the weak post-2008 labor market—uncertainty, high marginal taxes distorting hiring, insufficient aggregate demand, and technology—can each marshal confirming evidence, leaving little scientific basis to choose among them beyond ideological predisposition.
“each side is totally capable of providing evidence, which seems to confirm the underlying assumptions of the model... Where is the science in any of that other than cherry picking both assumptions and empirical evidence?”
Economics inflates its authority by presenting contingent work as scientific truth to policymakers and the public.
- The core problem is not that decision-makers must act on limited evidence, but the profession's sociological tendency to endow its work with more scientific merit than it deserves, giving conclusions a grandeur and authority that misleads lay audiences in policy debates.
“The sociological tendency of our profession to endow what we do with more scientific merit than I think it deserves.”
Treating each episode as unique prevents generalization and makes economists indistinguishable from historians.
- If every economic episode is treated as unique (e.g. 2008 had shadow banking, the Depression did not), then one cannot generalize and is effectively a historian rather than an economist; the hope that 30 more recessions would yield enough data to predict is unrealistic.
“If every case is unique, you are just an historian. You can't be an economist.”
- If there were private money or a constant-monetary-growth rule, a central banker like Yellen would not have to make discretionary decisions or her decision-making power would be much narrower.
“if we had private money, for example, she wouldn't have to make those decisions. Or if we had a monetary rule of a constant monetary growth her decision-making power would be much narrower.”