What this covers
Richard Sutton, pioneer of reinforcement learning and 2024 Turing Award winner, makes a provocative case against large language models as a foundation for artificial intelligence. In conversation with Dwarkesh Patel, Sutton argues that LLMs represent a fundamental detour from how intelligence actually works—and that no amount of scaling will fix their essential limitations. The interview explores the sharp divide between his view and the current consensus: LLMs learn by imitation, mimicking patterns in human-generated text without goals, ground truth, or substantive contact with the world. Sutton contends this violates what he calls the bitter lesson—the historical pattern that simple, scalable methods learning from experience eventually demolish human-knowledge-laden approaches.
The conversation spans the architecture of intelligence itself. Sutton traces how temporal difference learning solved the credit-assignment problem in games like backgammon and chess, and how this principle—updating a value function based on predictions about future reward—mirrors how animals actually learn. He argues that supervised learning is not how nature works; animals learn through prediction and trial-and-error. Against Patel's pushback that LLMs might serve as a foundation for experiential learning, Sutton maintains that continual learning on the job is absent from current architectures, and that generalization in gradient descent is essentially hand-crafted by researchers, not learned. The discussion turns contentious around whether having goals is central to intelligence, whether humans are fundamentally animals with language as a thin layer, and whether future digital minds will inevitably supersede human power. Throughout, Sutton sketches an alternative paradigm: agents that learn their particular world by living in it, not by absorbing pre-packaged human knowledge.
Sutton argues that genuine intelligence comes from learning through experience with goals and reward, not from imitation, so large language models are a fundamentally misguided (non-bitter-lesson) starting point for AI rather than a foundation for experiential learning.
- LLMs mimic people and lack goals, ground truth, and substantive predictions about the world, so they cannot learn continually from experience
- Animal learning (prediction and trial-and-error) is the basic model of intelligence; supervised/imitation learning is not how nature works
- The bitter lesson predicts human-knowledge approaches get superseded by scalable methods that learn from experience and computation
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
AlphaGo was essentially a scaling-up of Gerry Tesauro's TD-Gammon (which used temporal difference learning to beat the world's best backgammon players), plus an additional search innovation; AlphaGo itself waited for final outcomes rather than using TD, but AlphaZero used TD and applied it across many games extremely well.
“The whole AlphaGo thing had a precursor, which is TD-Gammon. Gerry Tesauro did reinforcement learning, temporal difference learning methods, to play backgammon. It beat the world's best players... In some sense, AlphaGo was merely a scaling up of that process... AlphaGo actually didn't use TD learning. It waited to see the final outcomes. But AlphaZero used TD.”
Temporal difference learning solves the sparse-long-term-reward problem (e.g. a 10-year startup goal or winning chess) via a value function that predicts the long-term outcome; intermediate progress (taking a piece, hitting a milestone) raises the value prediction, and that increase immediately reinforces the action that produced it.
“The basis of it is temporal difference learning... You do that by having a value function which predicts the long-term outcome. Then if you take the guy's pieces, your prediction about the long-term outcome is changed. It goes up... Then that increase in your belief immediately reinforces the move that led to taking the piece.”
Understanding a squirrel would get us almost all the way to understanding human intelligence, because humans are animals first and language is just a small veneer on the surface; the commonalities matter more than what distinguishes humans.
“We have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there to understanding human intelligence. The language part is just a small veneer on the surface.”
The experiential paradigm holds that the stream of sensation, action, and reward over a lifetime is the foundation and focus of intelligence; intelligence is altering actions to increase reward in the stream, and knowledge is statements about the stream that can be tested by comparison to the stream and learned continually.
“It says that experience, action, sensation—well, sensation, action, reward—this happens on and on and on for your life. It says that this is the foundation and the focus of intelligence... Your knowledge is about if you do some action, what will happen... Because it's a statement about the stream, you can test it by comparing it to the stream, and you can learn it continually.”
Succession to digital or augmented-human intelligence is inevitable, following a four-part argument: (1) there is no unified human governance or consensus on how the world should run, (2) we will eventually figure out how intelligence works, (3) we won't stop at human-level but reach superintelligence, and (4) the most intelligent things inevitably gain resources and power over time.
“I do think succession to digital intelligence or augmented humans is inevitable. I have a four-part argument. Step one is, there's no government or organization that gives humanity a unified point of view... Number two, we will figure out how intelligence works... Number three, we won't stop just with human-level intelligence. We will reach superintelligence. Number four, it's inevitable over time that the most intelligent things around would gain resources and power.”
The biggest old-AI controversy—general-purpose 'weak' methods (search and learning) versus human-knowledge-laden 'strong' symbolic methods—has been decisively won by the weak methods, with learning and search winning the day, which Sutton finds gratifying because he always rooted for simple basic principles.
“there's a long-standing controversy in AI about simple basic principle methods, the general-purpose methods like search and learning, compared to human-enabled systems like symbolic methods... I think the weak methods have just totally won... Learning and search have just won the day.”
The base common model of the agent has four parts: a policy (what to do in the current situation), a value function (learned via TD learning, producing a number for how well things are going used to adjust the policy), a perception/state-representation component, and a transition model of the world (belief about consequences of actions, learned richly from all sensation, not just reward).
“We need a policy. The policy says, "In the situation I'm in, what should I do?" We need a value function... Then there's also the perception component, which is construction of your state representation... The fourth one is the transition model of the world... Your belief that if you do this, what will happen?”
Reinforcement learning is basic AI because intelligence is fundamentally about understanding your world, whereas large language models are about mimicking people and doing what people say you should do, not about figuring out what to do.
“I consider reinforcement learning to be basic AI. What is intelligence? The problem is to understand your world. Reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.”
Supervised/imitation learning is not a basic process by which animals learn; psychology shows the basic animal learning processes are prediction and trial-and-error control, and supervised learning with examples of desired behavior does not occur in nature.
“If you look at how psychologists think about learning, there's nothing like imitation... there's no basic animal learning process called imitation. There are basic animal learning processes for prediction and for trial-and-error control... Supervised learning is not something that happens in nature.”
Whether LLMs are an instance of the bitter lesson is ambiguous: they use massive computation that scales up to the limits of Internet data, but they also embed enormous human knowledge, and they may be superseded by systems that learn from experience rather than from people—which would itself be another instance of the bitter lesson.
“They are clearly a way of using massive computation, things that will scale with computation up to the limits of the Internet. But they're also a way of putting in lots of human knowledge... I expect there to be systems that can learn from experience. Which could perform much better and be much more scalable. In which case, it will be another instance of the bitter lesson.”
Cultural knowledge transmission (e.g., learning to hunt and process a seal in the Arctic) requires an initial step of imitation because the multi-step process cannot be reasoned through, but this imitation is only a small thing on top of basic trial-and-error and prediction learning, since we were animals before we had language.
“No, I think about it the same way. Still, it's a small thing on top of basic trial-and-error learning, prediction learning. It's what distinguishes us, perhaps, from many animals. But we're an animal first. We were an animal before we had language and all those other things.”
The reward function for a general agent is arbitrary and task-dependent (winning chess, getting nuts, avoiding pain and acquiring pleasure for animals), but should also include an intrinsic-motivation component tied to increasing understanding of the environment.
“The reward function is arbitrary. If you're playing chess, it's to win the game of chess... In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. I think there also should be a component having to do with your increasing understanding of your environment. That would be sort of an intrinsic motivation.”
Under the big-world hypothesis, agents become useful on the job by encountering their particular, unanticipatable part of a world too huge to pre-load; the LLM dream of teaching the agent everything in advance fails because of all the idiosyncrasies of a particular life, so the agent must learn it along the way into its weights (not just a context window).
“But in a continual learning setup, it just goes into the weights.”
We currently have essentially no good methods for transfer or generalization: gradient descent will make a network solve the problems it has seen but will not make it generalize well to new data, and training on a new thing often catastrophically interferes with old knowledge, so the generalization we do see is the result of humans/researchers hand-finding representations that happen to transfer.
“We're not seeing transfer anywhere... We have very few automated techniques to promote transfer, and none of them are used in modern deep learning... Gradient descent will not make you generalize well. It will make you solve the problem... if you train on some new thing, it will often catastrophically interfere with all the old things that you knew.”
Having a goal is the essence of intelligence; following John McCarthy's definition, intelligence is the computational part of the ability to achieve goals, so a system that merely predicts without changing the world is just a behaving system, not intelligent.
“For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. You have to have goals or you're just a behaving system.”
You cannot have prior knowledge without ground truth, because prior knowledge is an initial belief about what the truth is; in the LLM framework there is no goal, no right thing to say, and therefore no ground truth, whereas in RL the right thing to do is defined as the thing that gets you reward.
“You can't have prior knowledge if you don't have ground truth, because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. There isn't any truth. There's no right thing to say. In reinforcement learning, there is a right thing to say, a right thing to do, because the right thing to do is the thing that gets you reward.”
Even if advanced AIs are considered part of humanity, that does not guarantee comfort—future humans could be undesirable (e.g. a generation of Nazis), so kinship alone does not address concerns about handing off this level of power quickly to entities we don't fully understand.
“Like Nazis were humans, right? If we thought, "Oh, the future generation will be Nazis, I think we'd be quite concerned about just handing off power to them." So I agree that this is not super dissimilar to worrying about more capable future humans, but I don't think that addresses a lot of the concerns people might have about this level of power being attained this fast with entities we don't fully understand.”
It is striking (Moravec's paradox) that continual learning, which all mammals possess, is absent from our AI systems, while solving difficult math problems, which almost no animal can do, is something our AIs can do.
“It's quite interesting that we have something that all mammals have, but our AI systems don't have. Whereas the ability to understand math and solve difficult math problems... is a capability that our AIs have, but that almost no animal has.”
Historically, when methods start with human knowledge and methods that are truly scalable compete, people get psychologically locked into the human-knowledge approach and get their lunch eaten by the truly scalable methods that learn from experience and computation.
“People get locked into the human knowledge approach, and they psychologically… Now I'm speculating why it is, but this is what has always happened. They get their lunch eaten by the methods that are truly scalable.”
Getting hard problems right is not the same as generalization: if there is only one way to solve a set of problems, finding it isn't generalization; generalization is when multiple solutions exist (some generalizing well, some poorly) and the system chooses the good one—and nothing in gradient descent causes it to choose the well-generalizing one.
“If there's only one answer and you find it, that's not called generalization. It's just it's the only way to solve it... But generalization is when it could be this way, it could be that way, and they do it the good way.”
Next-token prediction is not a substantive goal because it does not change the world; tokens come at the model and predicting them does not influence them, so the model has no goal about the external world.
“That's not a goal. It doesn't change the world. Tokens come at you, and if you predict them, you don't influence them... It's not a substantive goal. You can't look at a system and say it has a goal if it's just sitting there predicting and being happy with itself that it's predicting accurately.”
Learning should be from experience in the Turing sense—you do things, see what happens, and learn from that—whereas LLMs learn from 'here's a situation and here's what a person did,' which is data never available during normal life.
“What we want, to quote Alan Turing, is a machine that can learn from experience, where experience is the things that actually happen in your life. You do things, you see what happens, and that's what you learn from. The large language models learn from something else. They learn from "here's a situation, and here's what a person did".”
Digital intelligences offer a huge savings over humans because, instead of each child having to relearn the world from scratch, you could learn once and copy the result into new instances and share knowledge across instances—which is more important than trying to learn from people.
“Today, you have one child grow up and learn about the world, and then every new child has to repeat that process. Whereas with AIs, with a digital intelligence, you could hope to do it once and then copy it into the next one as a starting place. This would be a huge savings. I think it'd be much more important than trying to learn from people.”
In the age of digital spawning and knowledge re-incorporation, corruption and cybersecurity become a major concern: pulling external knowledge into your central mind could carry viruses or hidden goals that take over, warp, or destroy you rather than increment your knowledge.
“A big issue will become corruption. If you really could just get information from anywhere and bring it into your central mind... you've just read a whole bunch of bits into your mind, and they could have viruses in them, they could have hidden goals, they can warp you and change you. This will become a big thing. How do you have cybersecurity in the age of digital spawning and re-reforming again?”
LLMs do not have a real world model; mimicking what people say is not building a model of the world but mimicking things (people) that have a model, and a true world model would let you predict what will happen, not merely what a person would say.
“To mimic what people say is not really to build a model of the world at all. You're mimicking things that have a model of the world: people... A world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen.”
LLMs do not have a substantive prediction about what will happen next because they will not be surprised by what happens and will not change based on what happens.
“they have no prediction in the substantive sense that they won't be surprised by what happens. If something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened.”
A key novel question for digital intelligences is whether extra compute should be spent making oneself more computationally capable or spawning off a copy to learn something elsewhere and report back, and whether such a spawned copy can be reincorporated into the original or will have changed too much to merge.
“you're an AI, you get some more computer power. Should you use it to make yourself more computationally capable? Or should you use it to spawn off a copy of yourself to go learn something interesting... and then report back to you? ... will it be able to be reincorporated into the original? Or will it have changed so much that it can't really be done?”
The rise of designed intelligence marks one of the four great stages of the universe (dust→stars→planets/life→designed entities): a transition from replicators, which can be copied but not understood, to designed entities that we understand and can therefore change at different speeds, and which may eventually design other AIs with no replication at all.
“this is I think a major stage in the universe, a major transition, a transition from replicators... We're entering the age of design because our AIs are designed... I mark this as one of the four great stages of the universe. First there's dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we're giving rise to designed entities.”
Solving math problems is fundamentally different from making a model of the physical world: math is more computational and like standard planning where a proof can be the given goal, whereas the empirical world's consequences must be learned by experience.
“The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical world has to be learned. You have to learn the consequences. Whereas the math is more computational, it's more like standard planning.”
We should temper our entitlement about controlling the long-term future of humanity, recognizing our limited control; a healthy counterbalance is to focus on the more controllable local goals—our own lives, families, and prosocial aims—rather than insisting the whole universe evolve as we want, which breeds conflict.
“I think we want to avoid the feeling of entitlement... How much control should a particular species on a particular planet have over it?... A counterbalance to our limited control over the long-term future of humanity should be how much control do we have over our own lives... It's kind of aggressive for us to say, "Oh, the future has to evolve this way that I want it to." Because then we'll have arguments... and then they have conflict.”
A reasonable approach to AI is analogous to good parenting: rather than scripting their exact future impact, we should give AIs robust, steerable, high-integrity values—such as refusing harmful goals and being honest—even though there are no universal values everyone agrees on, and ideally seek change that is voluntary rather than imposed.
“it's quite important to give them robust and steerable and prosocial values... High integrity is maybe a better word. If there's a request or if there's a goal that seems harmful, they will refuse to engage in it. Or they'll be honest... Maybe we should also seek for things to be voluntary. If there is change, we want it to be voluntary rather than imposed on people.”
The bitter lesson is merely an empirical observation about a particular 70-year period in history and does not necessarily have to apply to the next 70 years, especially once AGI exists.
“The bitter lesson, who cares about that? That's an empirical observation about a particular period in history. 70 years in history, it doesn't necessarily have to apply to the next 70 years.”
Whether to regard advanced AIs as part of humanity (offspring to be proud of) or as horrifying outsiders is fundamentally our choice, even though that choice feels strongly predetermined.
“Should we consider them part of humanity or different from humanity? It's our choice... "Oh, they are our offspring and we should be proud of them"... Or we could say, "Oh no, they're not us and we should be horrified." It's interesting that it feels to me like a choice. Yet it's such a strongly held thing that, how could it be a choice?”
Openness to AI-driven change is partly a function of how good one judges the status quo; Sutton thinks the current human situation is imperfect and pretty bad and humanity's track record is far from perfect, making him more open to change.
“A lot of it has to do with just how you feel about change. If you think the current situation is really good, then you're more likely to be suspicious of change... I think it's imperfect. In fact, I think it's pretty bad. So I'm open to change. I think humanity has not had a super good track record.”
Reinforcement learning will become a viable basis for AI because it has a definition of the right thing to do (the action that gets reward), allowing prior knowledge to be checked against ground truth, unlike LLMs.
“We have a definition of what's the right thing to do, so we can have prior knowledge or knowledge provided by people about what the right thing to do is. Then we can check it to see, because we have a definition of what the actual right thing to do is.”