Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence
What this covers
Ilya Sutskever, OpenAI's Chief Scientist, makes the case that next-token prediction—the core task of language models—is not a ceiling on performance but a genuine path toward AGI. In conversation with Dwarkesh Patel, he argues that predicting text well requires understanding the underlying reality that produced it, not merely matching statistical patterns. This understanding allows a sufficiently capable model to extrapolate the behavior of hypothetical people far smarter than anyone in its training data. The conversation ranges across the machinery of deep learning, from transformers and GPUs to backpropagation and compute, situating progress in Compute Plus Data as Engine of Progress—the entanglement of transistor shrinking, personal computers, the internet, and hardware acceleration all arriving together.
Sutskever identifies reliability as the chief bottleneck, not raw capability. He argues that economic value and the path to AGI hinge on whether systems can be trusted without constant human verification, likening current models to Tesla's self-driving: convincing in appearance but far from mature. alignment of superhuman models remains unsolved, requiring multiple definitions—behavioral tests, adversarial stress probes, and internal neural inspection—rather than a single mathematical formulation. The conversation also covers reinforcement learning from human feedback, why OpenAI abandoned robotics and when it might return, the limits of scaling laws in predicting reasoning capability, and the threat of commoditization. He addresses why deep learning breakthroughs often feel inevitable in hindsight, the distinction between inspired and borrowed cognitive-science ideas, and the human role in a world where AI handles 99% of teaching work.
Sutskever argues that next-token prediction, far from being mere imitation, requires genuine understanding of the reality generating the text and can therefore extrapolate beyond human performance, making the current generative paradigm a viable path toward AGI whose chief remaining obstacle is reliability and whose chief risk is aligning superhuman models.
- Predicting the next token well requires understanding the underlying world that produced the text, not just statistics
- A sufficiently capable model can extrapolate the behavior of a hypothetical person of greater ability than anyone in its training data
- The dominant barrier to economic value and AGI is reliability, not raw capability, and alignment of superhuman models is an open research problem
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The approach most likely to ultimately succeed at understanding powerful models is to have a small, well-understood neural net given the task of studying and verifying the behavior of a large neural net that is not understood.
“the thing that will really succeed is when we will have a small neural net that is well understood that’s been given the task to study the behavior of a large neural net that is not understood, to verify.”
Next-token prediction is not mere imitation that caps at human ability; because predicting the next token well requires understanding the underlying reality that produced the text, a sufficiently smart base network can extrapolate how a hypothetical person of far greater insight, wisdom, and capability would behave, even if such a person doesn't exist, deducing it from the thoughts, feelings, and ideas implicit in the behavior of regular people.
“Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics?”
It is not a coincidence that data, transformers, and GPUs arrived together: shrinking transistors made personal computers economical, which created the internet and thus mass data, while the same transistor progress drove GPUs through gaming before Nvidia repurposed them for general computation — all dimensions of progress are intertwined and you don't get to pick which improve.
“All this progress in all these dimensions is very intertwined. It's not a coincidence. You don't get to pick and choose in which dimensions things improve.”
At current capability levels there is a pretty good set of ideas for alignment, but one should not underestimate the difficulty of aligning models that are actually smarter than us and capable of deliberately misrepresenting their intentions.
“I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions.”
Neuroscientists are convinced the brain cannot implement backpropagation because synaptic signals only move in one direction, which motivates alternatives like Hinton's forward-forward algorithm that approximate backpropagation's good properties; but for pure engineering there is no reason not to use backpropagation, which remains the only algorithm.
“neuroscientists are really convinced that the brain cannot implement backpropagation because the signals in the synapses only move in one direction.”
The more capable a model is, the more confident you need to be before releasing it, and you want to be in a world where the degree of alignment keeps increasing faster than the capability of the models.
“Depends on how capable the model is. The more capable the model, the more confident we need to be.”
A single mathematical definition of alignment is unlikely; instead assurance will come from multiple definitions examining alignment from different aspects simultaneously — behavior in tests, congruence in adversarial stress situations, and inspection of how the neural net operates from the inside.
“A mathematical definition is unlikely. Rather than achieving one mathematical definition, I think we will achieve multiple definitions that look at alignment from different aspects.”
Inference of better models will become more expensive, but it is not prohibitive if it is more useful than it is expensive — just as people happily pay $400 an hour for a lawyer, they would pay similarly for reliable legal advice from a neural net, so the real question is whether the net can produce an answer good enough at that cost.
“If it is more useful than it is expensive then it is not prohibitive.”
OpenAI was right to leave robotics because at the time there was no path to data: progress comes from the combination of compute and data, and even a giant operation of a hundred robots could not generate enough data, whereas today a path exists if one commits to building hundreds of thousands of robots in a gradual loop of more robots, more usefulness, more data.
“in a world where most of the progress comes from the combination of compute and data, there was no path to data on robotics. So back in the day, when we made a decision to stop working in robotics, there was no path forward.”
In reinforcement learning from human feedback, humans are used only to train the reward function; once trained, the reward function's interaction with the model is automatic, so most of the data generated during reinforcement learning is created by the AI rather than by humans.
“The human feedback has been used to train the reward function and then the reward function is being used to create the data which trains the model.”
Scaling laws describe how the log of next-word prediction accuracy improves, but there is a separate, complicated challenge of linking next-word prediction accuracy to reasoning capability; the link exists but other methods, such as reasoning tokens, may yield more reasoning per unit of effort.
“the scaling law tells you what happens to your log of your next word prediction accuracy... There is a whole separate challenge of linking next-word prediction accuracy to reasoning capability.”
The single most likely reason AI might fail to produce large economic value would be reliability: if you still have to look over and double-check every answer, that severely dampens the economic value of these systems, and a model that is not reliable is in effect not technologically mature.
“It would be reliability. That you still have to look over the answers and double-check everything. That just really puts a damper on the economic value that can be produced by those systems.”
There is a real force pushing AI models toward commoditization where competitors bid prices down to the cost of the GPU run, and the only answer is to keep making progress — improving models to be better, more reliable, and more trustworthy — supplemented by inference-cost research and specialization.
“there's without question a force that's trying to create that. And the answer is you got to keep on making progress.”
Language models are not fundamentally bad at multi-step reasoning; they are bad at mental multi-step reasoning when not allowed to think out loud, but when allowed to think out loud they are quite good, and this will improve significantly with better models and special training.
“I actually think that they are bad at mental multistep reasoning when they are not allowed to think out loud. But when they are allowed to think out loud, they're quite good.”
A major disruption to Taiwan's chip production would be a significant setback preventing more compute for a few years, but compute would spring up elsewhere — for example existing fabs outside Taiwan, while inferior, could produce older-generation GPU-like chips and still go very far, making the impact a matter of cost and delay rather than collapse.
“It's definitely going to be a significant setback. No one will be able to get more compute for a few years. But I expect compute will spring up.”
Google's TPUs offer no fundamental advantage over GPUs because both solve the same problem — a big processor, large memory, and a bottleneck between them requiring batch processing — so the only things that really matter about hardware are cost per flop and overall systems cost.
“What really turned out to be the case is that TPUs and GPUs are almost the same thing.”
Although the data situation is still quite good with lots remaining, at some point the supply of tokens to train models will run out, and by then we will need other ways of productively improving model capabilities without more data.
“there are claims that at some point we will run out of tokens, in general, to train those models. And yeah, I think this will happen one day”
Major advances in deep learning often consist of realizing something had a desirable property all along, so they straddle the line between breakthrough and mere implementation; the deepest advance — that a big neural network trained with backpropagation can do a lot — was a giant conceptual breakthrough precisely because for the longest time people didn't see it, even though in hindsight everyone calls it obvious.
“it was most definitely a giant conceptual breakthrough because for the longest time, people just didn't see that.”
In deep learning research, coming up with whole new ideas is only a modest part of the work; the main activity and where the real action takes place is understanding — figuring out the underlying phenomena, why a neural net produced an unexpected result, and what experiment to run next.
“Coming up with whole new ideas is a modest part of the work... even more important is to understand the results, to understand the existing ideas, to understand what's going on.”
Researchers should be inspired by human intelligence and the brain but with care, because it is easy to latch onto non-essential qualities and get too specific about cognitive-science models; the artificial neuron itself was a fruitful brain-inspired idea, so the skill lies in identifying which human behaviors are essential proofs of possibility versus emergent phenomena, and focusing on getting the basics right.
“There is an art into being inspired by humans in the brain correctly, because it's very easy to latch on to a non-essential quality of humans or of the brain.”
There is no clean distinction between the world of bits and the world of atoms, because a neural net can affect the physical world indirectly — for instance by advising you to rearrange your apartment, which you then do.
“Suppose the neural net tells you — hey here's something that you should do... you need to rearrange your apartment in a certain way. And then you go and rearrange your apartment as a result. The neural net impacted the world of atoms.”
Had key pioneers like Sutskever and Hinton never been born, the deep learning revolution would have been delayed by only about a year, because as GPUs keep improving it becomes progressively easier and cheaper to train neural nets, so someone would inevitably have discovered that big neural networks work.
“Maybe there would have been some delay. Maybe like a year delayed?”
Current AI models may be like a Tesla's self-driving behavior: they look like they can do everything, but there is still a long way to go in reliability, robustness, and well-behavedness before all the issues are ironed out.
“if you look at the size of a Tesla, and if you look at its self-driving behavior, it looks like it does everything. But it's also clear that there is still a long way to go in terms of reliability.”
The ideal teaching setup is not 100% AI but a human-machine collaboration where human teachers do roughly 1% of the work and the AI does 99%, with that collaboration teaching the next generation of machine.
“You might want to think of it as being in a world where the human teachers do 1% of the work and the AI does 99% of the work. You don't want it to be 100% AI.”
Across competing AI labs, research directions follow a convergence-divergence-convergence pattern: convergence on near-term work, divergence on longer-term bets, then convergence again once a promising long-term direction bears fruit — though reduced publishing means rediscovery now takes longer.
“I expect there's going to be a convergence-divergence-convergence behavior, where there is a lot of convergence on the near term work, there's going to be some divergence on the longer term work. But then once the longer term work starts to fruit, there will be convergence again”
The most important class of emergent properties to hope for at scale is reliability and controllability — being able to trust the model's output and to control its behavior — because together they would help solve a large number of problems.
“Reliability means you can trust the model's output, controllability means you can control it.”
AI will increasingly help research by suggesting fruitful ideas when asked, much like Copilot assists coding, but the bottleneck in research is good ideas and insights, which is precisely what neural nets could help with — though this won't necessarily let humans solve problems they couldn't solve before.
“the bottleneck there is good ideas, good insights and that's something that the neural nets could help us with.”
Because the post-AGI world will change so much that people will struggle to understand events and contribute, some people will choose to become part AI in order to expand their minds and understanding and solve the hardest problems society will face.
“One thing that I think some people will choose to do is to become part AI. In order to really expand their minds and understanding and to really be able to solve the hardest problems that society will face then.”
The preferable post-AGI world is one where descendants of humans live free, solve their own problems, make their own mistakes and gradually evolve morally, with AGI acting as a base safety net, rather than a world where the AGI dictates how society should be run and humans simply obey.
“I'd much rather have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own, with the AGI providing more like a base safety net.”
Optimistic people working on a technology tend to underestimate the time it takes to get there, which is why Sutskever grounds his own AGI timeline estimates against concrete analogies rather than trusting his optimism.
“there is this effect where optimistic people who are working on the technology tend to underestimate the time it takes to get there.”
Sustained effort and perseverance are necessary but not sufficient conditions for repeatedly figuring something out; many things must come together, including having the right way of looking at things.
“It's a necessary but not a sufficient condition. Many things need to come together in order to really figure something out. You need to really go for it and also need to have the right way of looking at things.”
It would not be surprising if foreign governments are already using open-source models for propaganda or scams at some scale, and this is something they would certainly be interested in doing in the future, though large-scale tracking of such misuse is possible with special operations.
“I can certainly imagine they would be taking some of the open source models and trying to use them for that purpose.”
Current hardware is not a limitation on which ideas can be tried; one might wish hardware were cheaper or had higher memory bandwidth, but by and large hardware is not an issue and any idea can be spun up.
“I don't think current hardware is a limitation. It's just not the case.”