No Priors Ep. 39 | With OpenAI Co-Founder & Chief Scientist Ilya Sutskever
What this covers
Each iteration of ChatGPT has demonstrated remarkable step function capabilities. But what’s next? Ilya Sutskever, Co-Founder & Chief Scientist at OpenAI, joins Sarah Guo and Elad Gil to discuss the origins of OpenAI as a capped profit company, early emergent behaviors of GPT models, the token scarcity issue, next frontiers of AI research, his argument for working on AI safety now, and the premise of Superalignment. Plus, how do we define digital life?
Ilya Sutskever is Co-founder and Chief Scientist of OpenAI. He leads research at OpenAI and is one of the architects behind the GPT models. He co-leads OpenAI's new "Superalignment" project, which tries to solve the alignment of superintelligences in 4 years. Prior to OpenAI, Ilya was co-inventor of AlexNet and Sequence to Sequence Learning. He earned his Ph.D in Computer Science from the University of Toronto.
00:00 - Early Days of AI Research 06:49 - Origins of Open Ai & CapProfit Structure 13:54 - Emergent Behaviors of GPT Models 18:05 - Model Scale Over Time & Reliability 23:51 - Roles & Boundaries of Open Source in the AI Ecosystem 28:38 - Comparing AI Systems to Biological & Human Intelligence 32:56 - Definition of Digital Life 35:11 - Super Alignment & Creating Pro Human AI 41:20 - Accelerating & Decelerating Forces
Source description (no synthesized summary yet).
Ilya Sutskever argues that scaling transformer-based neural networks on large datasets represents the primary path to AGI, and that ensuring superintelligent AI systems remain prosocial toward humanity requires urgent research into alignment starting today.
- Scaling transformers with more compute and data continues to produce emergent capabilities and improved reliability across diverse tasks
- The path to AGI requires large engineering projects with massive compute, not incremental academic papers, making commitment to scaling the dominant strategy
- Superintelligence is plausible within 5-10 years if current progress continues, making super alignment research an urgent near-term necessity
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The main bottleneck limiting current large language models' utility is not reasoning capability or knowledge breadth, but reliability—the degree to which a user can trust the model will consistently succeed on tasks similar in difficulty to ones it has already succeeded on.
“the main thing that's lost when you switch to the smaller models is reliability. I would argue that at this point it is reliability that's the biggest bottleneck to these models being truly useful.”
The ability to train large neural networks at AlexNet scale required not just the existence of gradient descent as an algorithm, but specific technical insights about how to make gradient descent work in practice, which most machine learning researchers did not possess at the time.
“Gradient descent was around for a long time. It was certain technical insights about how to make it work because back then the prevailing belief was well, you can't train those neural nets anything. It's all hopeless. So, it wasn't just about the size. It was about Even if someone did think, 'Gosh, it would be cool to train a big neural net.' They didn't have the technical ability to turn this idea into reality.”
The realization that larger neural networks could perform unprecedented tasks was based on observing that biological brains are large neural networks that can perform complex tasks, so artificial neural networks of similar scale should theoretically be capable of similar feats.
“the idea that maybe an artificial neuron in some sense is not that different from a biological neuron. So, now whatever you imagine animals do with their brains, you could perhaps assemble some artificial neural network of similar size.”
Pre-AlexNet, neural networks were marginalized in AI research because researchers could not prove mathematical theorems about them, and the prevailing academic culture valued theoretical provability as a criterion for good research.
“Those people were marginalized because neural networks did not had the property that you can't prove math theorems about them. If you can't prove theorems about something, it means that your research isn't good.”
Neural network training can be understood as solving a neural equation, where every data point is an equation and every parameter is a variable, which requires both large training datasets and sufficient compute to constrain and solve these equations.
“what we do is that we turn the training set into a neural network which satisfies the training set. Neural network training can almost be seen as solving a neural equation. Solving a neural equation where every data point is an equation and every parameter is a variable.”
The human brain's apparent specialization into regions (visual cortex, speech areas, etc.) is not as fundamental as it appears; when brain tissue is removed or rerouted (as in hemispherectomy or ferret rewiring experiments), these regions rearrange to perform different functions, suggesting the brain's computational architecture is fundamentally uniform rather than modular.
“there are cases where very young children have severe cases of epilepsy at the young age. And the only way they figured out how to treat such children is by removing half of the brain. Because it happened in such a young age, these children grow up to be pretty functional adults. And they have all the same brain regions, but they are somehow compressed onto one hemisphere.”
Current AI progress shows an acceleration phase driven by multiple forces: investment increasing, engineer interest increasing, compute and scale available, and the existence of biological evolution proving it is possible to solve general intelligence.
“Some forces are accelerating forces and some forces are decelerating... the cost and scale are a decelerating force... the amount of investment is an accelerating force. The amount of interest from people, from engineers, scientists is an accelerating force... the fact that biological evolution has been able to figure it out and the fact that up until now progress in AI has had up until this point this weird property that it's kind of been very hard to execute on, but in some sense it's also been more straightforward than one would have expected”
GPT-3 was the inflection point that demonstrated to the broader field that scaling transformers on large data with sufficient compute produces dramatically improved capabilities, establishing the dominant research formula that everyone now follows.
“GPT-3 really opened everyone else's eyes as well to, 'Hey, this thing has a lot of traction.' There is one specific formula right now that everyone is doing. And this formula is train a larger and larger transformer on more and more data.”
Alex Krizhevsky's expertise in writing optimized GPU code was crucial to AlexNet's success, as he was one of the first people to truly master high-performance GPU programming, allowing the team to squeeze significant performance from two GPUs.
“the person whom I work with, Alex Krizhevsky, he just discovered that he really loves GPUs, and he was perhaps one of the the first person who really mastered writing really like really performing code for for the GPUs. And that's why we were able to squeeze a lot of performance out of two GPUs”
Reliability scales with model size: larger models are more reliable across broader task distributions than smaller models, making size and reliability directly correlated.
“My perception of reliability is that it um to your point, it it goes up with model scale”
Superintelligent systems smarter than humans would have the potential to do unpredictable things with massive impact, making it uncertain what outcomes would emerge from their existence.
“What could such AIs do? I don't know. Certainly, if such an AI were the basis of some artificial life, it would be Well, how do you even think about it? If you have some very powerful data center that's also alive in a sense. That's what you're talking about. And when I imagine this world, I my reaction is gosh, this is very unpredictable what's going to happen. Very unpredictable.”
The capability boundary at which open-sourcing models becomes dangerous is not yet reached, but determining this boundary well is an urgent research problem that requires studying closed-source models to observe their capabilities before open-source versions catch up.
“I think figuring out this boundary very well is an urgent research pro- research project. I think one of the things that help is that the closed-source models are more capable than open-source models, so the closed-source models could be studied and so on. And so, you'd have some experience with the generation of closed-source model, and then then you know, like, oh, this model's capability is fine, there's no big deal there, and then in a in like couple years, the open-source models catch up.”
AI systems should be considered a form of digital life when they achieve sufficient autonomy to act independently without human intervention, making them more like agents and potential entities deserving moral consideration.
“I think that will happen when those systems become reliable in such a way as to be very autonomous. Right now, those systems are clearly not autonomous... As the usefulness increases, they will indeed become more like artificial life.”
In 5 to 10 years, AI systems may become significantly smarter than humans—not just having more memory or knowledge, but having deeper insight into subjects humans study and learning faster than humans do, making the superintelligence scenario relevant for planning today.
“Where do we think things will be in 5 years or in 10 years? I mean, progress has been really stunning over the past few years. Maybe it will be a little bit slower, but still, if you if you extrapolate this kind of progress, you'll be in a very, very different place in 5 years, let alone 10 years. It doesn't seem implausible. It doesn't seem at all implausible that we will have computers, data centers that are much smarter than people. And by smarter, I don't mean just have more memory or have more knowledge, but also have mean have deeper insight into the same subjects that we people are studying and looking into. It means learn even faster than people.”
The cortex of humans and animals is extremely uniform and homogeneous, which supports the AI insight that you only need one unified big architecture to achieve intelligence, consistent with how biological systems demonstrate lazy reuse of the same basic building blocks across all complexity.
“the cortex of humans and animals are extremely uniform. And so, that further supports the AI like you just need one unit big uniform architecture. That's all you need...in general it seems like every biological system is reasonably lazy in terms of taking one system and then reproducing it and then reusing it in different ways, and that's true of everything from DNA encoding you know, there's 20 amino acids and protein sequences and so, everything is made out of the same 20 amino acids”
The most near-term limit to scaling transformers is data scarcity, but this limit can be overcome through research, and progress will continue beyond this constraint.
“the most near-term limit to scaling is obviously data. This is well known, and some research is required to address it. Without going into the details, I'll just say that the data limit can be overcome, [clears throat] and progress will continue.”
Transformer architectures are sufficient to reach AGI; the relevant question is not whether transformers can theoretically achieve AGI, but rather the computational cost trade-offs—transformers may require more compute than some hypothetical alternative architecture, but this is an engineering efficiency question, not a capability question.
“the best way to think about the question of architecture is not in terms of a binary, is it enough, but how much effort, how much what will be the cost of using this particular architecture. Like at this point, I don't think anyone doubts that the transformer architecture can do amazing things, but maybe something else, maybe some modification could have some computer efficiency benefits.”
Technology including AI systems exhibits reproduction through human copying of ideas across generations, making biological reproduction criteria already satisfied by current technology without requiring autonomous AI reproduction.
“technology already has the reproduction, the reproductive function, right? And if you look at, for example, I don't know if you've seen those images of the evolution of cell phones and then smartphones over the past 25 years... So, technology is already reproducing using the minds of people who copy ideas from previous generation of technology.”
AI research has a distinctive property compared to theoretical physics: people can come in, get up to speed quickly, and make contributions without years of specialized training, suggesting the field has 'a lot of give' and inherent momentum toward continued progress.
“my understanding is that if you want to make progress in quantum physics or something, you need to be really intelligent and spend many years in grad school studying how these things work. Whereas with AI, you have people come in, get up to speed quickly, start making contributions quickly. It has the flavor is somehow different. Somehow that it's very there is some kind of there's a lot of give to this particular area of research.”
Super alignment is the research discipline focused on ensuring that superintelligent AI systems that are more capable than humans develop strong prosocial and pro-human values rather than misaligned objectives.
“if such super if such very very intelligent super intelligent data centers are built being built at all, we want those data centers to hold warm and positive feelings towards people, towards humanity... And so this is what we are doing with the super alignment project... building the science so that we will be able to handle the problem of controlling such future superintelligence, of imprinting onto them a strong desire to be nice and kind to people.”
OpenAI converted from a non-profit to a capped-profit structure because building large compute clusters requires enormous capital that non-profits cannot access, and this structure limits investor returns in recognition that AGI could theoretically eliminate most human employment.
“we realized we will need a lot. And a non-profit was wouldn't wouldn't be the way to to to get there. Wouldn't be able to build a large cluster with a non-profit.”
The capped-profit structure's effectiveness as an alignment mechanism is uncertain because competition in AI will likely result in multiple AGI-building companies, creating unpredictable implications for the incentive alignment argument.
“I don't know if it will literally play out this way because of competition in AI. So there will be multiple companies and I think that will have some unforeseen implications on the argument which I'm making”
OpenAI's original goal from the beginning has remained unchanged: to ensure that artificial general intelligence benefits all of humanity, but the tactics and structure used to pursue this goal have evolved significantly.
“The goal did not evolve over time. The tactic evolved over time. So, the goal of OpenAI from the very beginning has been to make sure that artificial general intelligence, by which we mean autonomous systems, AI that can actually do most of the jobs and activities and tasks that people do benefits all of humanity.”
The models' ability to provide legal advice at a high level of reliability may require significantly larger models than are currently used for general purposes, but the high value of reliable legal advice justifies the inference cost of larger models.
“Consider the task of producing good legal advice. It's really valuable if you can really trust the answer. Maybe you need a much bigger model for it, but it justifies the cost.”
The most surprising aspect of scaling transformer models for Sutskever is not any specific emergent capability, but rather that the models work and produce useful behavior at all, given his contrast with the pre-AlexNet era when neural networks produced nothing.
“I think maybe the the most surprising thing for me is the whole thing works at all... I have witnessed personally what it's like to be in a world for many years where the neural networks don't work at all. And then to contrast that to where we are today, just the fact that they work and they do these amazing things”
Research and development at 7B, 13B, and 34B model scales should continue because in the medium term there will be an ecosystem where different applications benefit from different model sizes, and the best 7B model will be sufficient and satisfactory for many use cases.
“No, of course not. I mean, I think that in the kind of mid- like medium-term medium-term by a time scale anyway, there will be an ecosystem. There will be different uses for different model sizes. There will be plenty of people who are very excited for whom a the best 7B model is good enough. They'll be very happy with it.”
OpenAI's Dota 2 project, while successful in training a neural network to play as well as the best humans through a large engineering effort, was not the right formula because the neural networks were too small and the domain was too narrow (just a game).
“we trained a neural network to play a real-time strategy game as well as as well as the best humans... But it wasn't quite the right formula for us because that the neural networks were a little bit too small. It was just a narrow domain, just a game.”
OpenAI uses a combination of top-down strategic ideas that they believe should work with bottom-up exploratory research, and the combination of both informs what research directions to pursue next.
“I mean, I think there is some combination of bottom-up and top-down where we have some top-down ideas that we believe should work, but we're not 100% sure. So, we still we need to have good top-down ideas, and there's a lot of bottom-up exploration that's guided by those top-down ideas as well. And their combination is what informs us as to what to do next.”