Rich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025
What this covers
As AI has become a huge industry, to a large extent it has lost its way. What is needed to get us back on track to true intelligence? We need agents that learn continually. We need world models and planning. We need knowledge that is high-level and learnable. We need to meta-learn how to generalize. The Oak architecture is one answer to all these needs. In overall outline it is a model-based RL architecture with three special features: 1) all of its components learn continually, 2) each learned weight has a dedicated step-size parameter that is meta-learned using online cross-validation, and 3) abstractions in state and time are continually created in a five-step progression: Feature Construction, posing a SubTask based on the feature, learning an Option to solve the subtask, learning a Model of the option, and Planning using the option's model (the FC-STOMP progression). The Oak architecture is rather meaty; in this talk we give an outline and point to the many works, prior and co-temporaneous, that are contributing to its overall vision of how superintelligence can arise from an agent's experience.
Source description (no synthesized summary yet).
The OAK (Options and Knowledge) architecture enables domain-general artificial superintelligence by learning from runtime experience through hierarchical abstraction discovery, where agents autonomously create subproblems based on state features, solve them as options, and use the resulting models for planning—avoiding the brittleness of design-time knowledge and scaling with computational resources rather than human expertise.
- Runtime learning beats design-time learning because the world is vastly larger than any agent and requires continuous adaptation to encountered environments
- Agents must autonomously generate subproblems from discovered state features to achieve open-ended abstraction without domain-specific engineering
- The OAK architecture unifies options, general value functions, and learned models into a cycle that produces increasingly sophisticated abstractions limited only by computation
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Planning is required because the world is big, and agents must use planning to prepare for different possible goals and values that may become relevant in the future, even when the underlying world dynamics don't change
“planning is required because the world is big”
The path to strong AI runs through reinforcement learning rather than through non-experiential approaches like large language models, and the fundamental bottleneck is inadequate learning algorithms rather than insufficient computational power or data
“good for this group that the path to full AI strong AI runs through reinforcement learning and not I think through things like uh to non-experential things like large language models. The biggest bottleneck is strangely is we have inadequate learning algorithms.”
Domain-general AI design should not depend on or reflect the specific world it will encounter; instead, the agent's architecture should be domain-independent and learn all world-specific details at runtime through experience
“Should the agents design reflect the world in which it's expected to be used? Good thing about this question is both answers are wrong. I mean both answers are right...But if you want a good design, so I'm gonna say no. My quest, my quest is that the design should not depend on the world at all. Okay. Uh it should be domain general.”
Metalearning is the ability to learn how to learn—to try one learning approach, try another, notice which works better, and then adopt the superior approach—and this critically requires runtime learning rather than design-time learning
“if we if we one last thing about runtime learning, it does enable metalarning. Metal metal learning is where you like try learning one way and then you try learning another way and you notice that oh this way works better in the future I will do this. If you were if you were doing everything in one shot you couldn't do that. This idea of becoming better at learning requires uh one time you're doing learning another time you're trying a different way of learning and you pick the better one. So metalarning really requires this to be done at the at the runtime.”
The reward hypothesis—that all goals and purposes can be understood as maximizing the expected value of cumulative scalar reward—is sufficient for generality and does not require modifications like multiple objectives, constraints, or risk sensitivity to achieve general intelligence
“the reward hypothesis is this that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal called reward.”
David Marr's theory of computation defines three levels of analysis (computational, algorithmic, implementation), and OAK should be understood as a computational theory of intelligence at Marr's highest level
“if you know about cognitive uh David Maher, then we would say that oak is a computational theory of intelligence.”
The crucial missing capability in today's AI systems is the ability to plan with learned models, and OAK provides a vision for integrating planning with learned models across multiple levels of abstraction
“It's a vision for how to do planning with a learned model which is a key missing ability for today's AIS.”
An option is a pair consisting of a policy (way of behaving) and a termination function (decision to stop behaving), and the agent learns what happens when it follows each option to form a high-level transition model that enables planning with larger temporal jumps
“an option is a pair. Well, actually, you may think it's a triple. I sort of have dropped the initiation set in in in my for many last couple decades. Um, so for me, it's a pair of a policy, a way of behaving, and a way of deciding to stop behaving”
Planning with option models follows the same value iteration principle as standard planning, but substitutes options for primitive actions and option models (transitions and cumulative rewards under the option) for single-step models, maintaining the same mathematical structure
“the interesting thing about planning with option models is that it's the same... Although life is lived one step at a time, we have to plan it at a higher level... we have to change the actions into options and we still talk about the reward for following that option and the probability of each next state under the option... value duration is almost unchanged. We just change the actions into options”
The world is vastly larger and more complex than any agent—containing billions of other agents and infinite intricate details—making it impossible for an agent to hold a complete world state in its mind or know all abstractions needed for any given environment
“The big world perspective or the big world hypothesis...The idea is simply that the world is bigger, more complex than the agent. Uh and it's much bigger really. It's bigger than this. It's it's big, you know, it's really big. And it's got to be much much bigger than the agent because the world contains, you know, billions of other agents and um and all of course all the atoms and all the intricacies of uh the objects spread around.”
Animals naturally create their own subproblems through play and exploration—infants exploring objects, young orangutans learning to swing, orcas figuring out how to manipulate novel objects—and this autonomous subproblem creation is the mechanism underlying open-ended cognitive development
“here's an orang ba a young orangutang um playing swinging. And so I What is he doing? Like he's not getting food. He's just interested in what it feels like when he swings...he got it once and now he's trying to get it again and understand how to control it.”
Anything that can be learned can also be planned—general value functions used to learn models and values can equally be used to plan, because planning just applies value iteration to a different set of states, meaning there is mathematical unity across learning and planning
“we could use just standard off-the-shelf algorithms standard offtheshelf usually off policy algorithms for learning general value functions like GTD and emphatic TDD and retrace ABQ uh these are prediction learning methods for generic uh GVFs and so we can use that to learn the main problem how to get reward we can learn that for learning diagrams for the sub problems. We can find the transition models of the options with these methods and the planning can also be done with standard algorithms applicable to all GDFs. And this enables us to say that anything that can be learned can also be planned.”
The path to full artificial general intelligence runs through reinforcement learning and experiential learning rather than through non-experiential approaches like large language models
“the path to full AI strong AI runs through reinforcement learning and not I think through things like uh to non-experential things like large language models”
Generating new state features is an old, unsolved problem dating to Minsky and Selfridge's work on representation learning and the 'new terms problem' in the 1960s, and gradient descent alone has not solved it despite backpropagation in 1986
“the other problematic uh or the other step that I wanted to talk about where we having to do with um creating generating new state features. This is this is also a super old problems going back to the 1960s like Minsky and Selfridge would talk about this. They would talk about representation learning. They would talk about the new terms problem”
Runtime learning always wins over design-time learning because: (1) design time can't cover every case while runtime customizes to the actual encountered world, (2) runtime learning scales with available compute while design time scales with human expertise, and (3) historically, scaling with compute wins in the long run
“Runtime learning I think always wins over design time because the the world is much bigger than the agent, the big world perspective. Um design time can't cover every case. Runtime learning can customize to the part of the world actually encountered. Runtime learning scales with available compute whereas design time learning or anything done at design time scales with the available human expertise at design time... historically scaling with compute wins in the long run”
Backpropagation (1986) was supposed to solve representation learning through gradient descent, but it does not actually solve the feature generation problem, suggesting that methods based on generate-and-test (generating features and testing their utility) may be more promising
“Backrop back in '86 was supposed to solve this. We were supposed to you know learning representations by gradient descent. Um but it really just doesn't. And um I think we're we we accept we recognize that unless we are still in love and think that gradient descent is enough for everything. Um most of the other methods other than gradient descent are based on generate and test ideas where you like generate a bunch of features and then you test them to see if they're useful”
Simple reward—a single scalar—is sufficient to specify goals and does not need to be supplemented with constraints, multiple objectives, or risk sensitivity to achieve generality or intelligence.
“I want things to be simple. You know, that's like a really high high uh uh deserata desire. I want things to be simple and I might even simplify them a little bit too far in order to uh to uh be clear and uh so anyway, I want things to be simple. Do we need to do we need all these things to get generality? That's the real question.”
The subproblem is to maximize the expected sum of rewards received while following an option plus a reward-weighted contribution of the feature value at termination, minus any penalty from bad terminal states, creating a trade-off between feature attainment and reward preservation.
“An option is a pair. It's a policy pi termination function gamma that maximizes the value of the i feature at termination while respecting the rewards and values... in each state you're trying to maximize... you want lots of rewards. You want to be the feature to be true but you know it's got to be traded off the rewards. And you also care about the state that you're in at the time of termination. You don't want to like find a really good way to to uh I don't know get some coffee but has the consequence that you have to break your leg.”
Current deep learning methods do not support reliable continual learning—despite decades of work on catastrophic forgetting and recent work on catastrophic loss of plasticity—and this is a major limitation of applying runtime learning to complex real-world problems
“However, today's deep learning methods, runtime deep learning methods, continual learning, they don't work very well. Okay, this is this is a a a big bitter thing for me.”
Perception can operate without reference to human labels or external world knowledge by being understood as conceptual frameworks formed to solve sub-problems that are the basis of perception
“What is the purpose of perception? We're answering the question of how we perception can operate without reference to a a human label or to an external world. Perception um can be concepts that have been formed to solve problems that are the basis of uh subpros”
Problems and solutions interact in a cycle: problems are proposed, solved, which creates new features that become the basis for new problems, generating new features in an endless cycle, illustrated by play behavior in young animals where curiosity about novel sensations leads to learning new control capabilities
“we have problems and solutions and these interact with each other. We propose a problem to work on. We work on it. We solve it. As as a part of solving it we will make new features and those features will then be the basis for new sub problems and and and then the sub problems will have to be solved new features and so on in an endless cycle”
The reward hypothesis states that all goals and purposes can be understood as maximization of the expected value of the cumulative sum of received scalar signals called reward, and this is not a quick choice but has been thoughtfully considered and validated by work such as 'Settling the Reward Hypothesis'
“the reward hypothesis is this that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal called reward”
Animals learn skills for specific environments through runtime experience during their lifetimes, not through pre-built domain knowledge, as evidenced by the diversity of animal behaviors across different ecological niches.
“you go to you go to work, you're you're you're AI agent that's supposed to be play a productive role in society and it goes to work and it meets its co-workers. It has to remember the name of the guy it's working with. You know, that was not in the domain knowledge. It has to remember uh the work that they've done on that project... everything you do in your life uh is is is un could not have been foreseen”
Generating new state features from existing features is marked red because there is no specific solution proposal yet, though this is a long-standing challenge in AI going back to Minsky and Selfridge's work on the representation problem and the credit assignment problem
“The second one uh is red because we don't really have a solution. We have lots of ideas but we don't have a specific proposal”
AI is a grand intellectual quest comparable to the origin of life on Earth—understanding how intelligence works, building intelligent systems, and making ourselves more powerful represents a fundamental milestone in human civilization
“AI is a grand quest. Uh we're trying to understand how people work. We're trying to make people. We're trying to make ourselves powerful. Um and this is a profound intellectual milestone. It's going to change everything.”
Large language models perform all important learning at design time and do nothing at runtime when deployed, whereas the OAK approach emphasizes doing all important things at runtime online on the job.
“a large language model everything is done as design time and when it goes out to be used in the world it doesn't do anything. My emphasis is going to be the other way around. We want I want to do all the important things at runtime online on the job.”
In the big world context, while world dynamics themselves may not change, observed values necessarily change because the agent's wants change—for instance, it's always true you can walk to the restroom, but not always true that you want to; planning prepares the agent for these future value changes
“the world changes or appears to change... most of the world's dynamics or in many cases the world's dynamics including the reward parts don't really change but the values nevertheless change. Like it's always true that I can walk over there and find the restroom but it's not always true that I want to go to the restroom... to prepare for these later wants um these these different values you plan”
A reward-respecting sub-problem of feature attainment takes a feature index i, a kappa parameter (intensity of wanting that feature), and creates a problem to drive the world to a state where the feature is high without losing too much main reward, formalized as a policy and termination function that maximize cumulative reward plus the feature value at termination
“a reward respecting subpros of feature attainment... you have a feature I a feature index I and you have kappa which is how intensely you want that feature... the sub problem is to drive the world to a state where the feature is high without losing too much reward because you will lose some reward if you're not doing what you normally do because what you normally do is uh maximize reward”
The most important capabilities of a super intelligence grown from experience at runtime are: (1) learning, (2) modeling sub-problems, (3) discovering options from sub-problems, (4) learning models from options, and (5) discovering state features—forming a virtuous open-ended cycle.
“his most important capabilities and does it act learn plan model learning sub problems the options with um all the rest and the discovery of of the state features and thereby of the problems, options and models leading into this virtuous open-ended cycle of discovery”
The three main design goals for intelligent agents should be: domain generality (nothing specific to any particular world), experientiality (mind grows from runtime experience not from special training phases), and open-endedness (ability to form any concepts needed to deal with whatever world it encounters, limited only by computational resources)
“It should be domain general. It should be experential. That is the mind should grow from runtime experience not from a special training phase. And third it should be open-ended in its sophistication in its abstractions. So that um it's a it can it can form any any concepts in its mind that are needed to deal with whatever world it's connected to limited only by its computational resources”
The purpose of play is to discover and work on sub-problems that structure cognition, allowing agents to practice problem formulation and solution before facing high-stakes real-world challenges.
“What is the purpose of play to find these these uh subpros which structure our our cognitive um our cognition?”
Perception should be understood as grounded in problem-solving experience rather than in human labels or direct access to the external world; perceptual concepts are features discovered while solving sub-problems.
“What is the purpose of perception? ... Perception um can be concepts that have been formed to solve problems that are the basis of uh subpros... perception can operate without reference to a a human label or to an external world”
The big world perspective states that the world is much larger and more complex than the agent—containing billions of other agents and countless atomic details—making it impossible for the agent to hold complete world knowledge, thereby necessitating approximate value functions, policies, and transition models
“the world is bigger, more complex than the agent. Uh and it's much bigger really. It's bigger than this. It's it's big, you know, it's really big. And it's got to be much much bigger than the agent because the world contains, you know, billions of other agents and um and all of course all the atoms and all the intricacies of uh the objects spread around”
The key open questions about sub-problems are: (1) what should the sub-problems be?, (2) where do they come from?, (3) how can the agent generate its own sub-problems?, and (4) how do sub-problems help on the main problem?—OAK proposes answers to all these questions
“The key open questions about subpros, which are what should the subpros be? Where do they come from? How can the agent or can the agent generate its own subpros and how do the sub problems help on the main problem?... the contribution of oak is to an is to propose answers to all these questions”
The OAK architecture requires solving reliable continual learning for deep neural networks, which currently fails due to catastrophic forgetting and loss of plasticity, with some solution approaches including continual backprop and meta-learning of new features
“the oak architecture requires reliable continual learning, continual deep learning... Do we have reliable continual learning? Well, we do have reliable continual learning for the linear case, for the tabular case, but for the nonlinear case, for the deep learning case, we can't have reliable we don't yet”
The biggest bottleneck in achieving AI is inadequate learning algorithms—existing deep learning algorithms are very crude and need to be substantially better, contrary to the assumption that deep learning is the solved piece of the puzzle
“The biggest bottleneck is strangely is we have inadequate learning algorithms. You may think we have our deep learning and we that's the one thing we know but I think it's not like that at all. I think it's it's more like we they are very our algorithms are very crude”
In the OAK architecture, the agent learns a high-level transition model of the world by understanding what happens when following different options, enabling planning with larger temporal jumps and carving the world at its natural joints
“the agent is meant to learn a high level transition model of the world that enables planning with larger jumps and hopefully carves the world at its joints”
The OAK architecture is a computational theory of intelligence in the sense of David Marr—providing a mechanistic account of how minds compute understanding and behavior.
“if you know about cognitive uh David Maher, then we would say that oak is a computational theory of intelligence.”
If a deep learning method could perform everything current methods do but also learn continually, it would be transformative and would likely take over most of what people do with deep learning, and such a breakthrough seems within reach
“If we had a deep learning method that was can do everything we're doing now but can also learn continually that would just um be a really big thing and I think there's no reason why it couldn't happen. Um and so I think it will”
Most methods for generating features beyond gradient descent are based on generate-and-test approaches, and continual backdrop (which generates candidate features and tests them by utility) could be a key part of solving representation learning
“most of the other methods other than gradient descent are based on generate and test ideas where you like generate a bunch of features and then you test them to see if they're useful and so you could generate them randomly and you could test by their utility and u continual backdrop is an instance of that.”
State features should be discovered through a feedback cycle where later processing steps (option solving, model learning, and planning) provide feedback about which features proved useful, informing the perception process to generate more useful features over time
“all of these these three uh pillars the the later three all use features to to do their job right...when you uh learn the models of the options you're going to look at the state...say oh that feature I found useful that other feature was useless to me um and so and and then when you use the models you will find some models are useful And that will sort of trickle back to evaluate the choice of the options. And that will also trickle back to evaluate the choice of the feature attainment problems.”
The perfect AI design would be conceptually simple—fitting in about five pages of pseudocode—containing only essential domain-independent elements yet capable of generating arbitrary open-ended abstractions limited only by computational resources
“I think of a design for an AI, the perfect design. It would not be a huge thing. It would not be like an encyclopedia or a library worth of knowledge. It would be like uh well, when I make for me actually it almost fits on a slide. You write in the pseudo code, you know, maybe it's three slides. Okay. I thought I think something of that that order. five pages for your uh description of all the essential elements that are domain independent and and and and yet are capable of arbitrary um open-ended abstractions. Okay.”
The perception process in OAK is responsible for constructing interesting state features from raw sensory input and action history, serving as the foundation for all subsequent learning.
“the perception process is going to is responsible for constructing interesting state features. Um the play process or the problem posing process problem posing and solving. That's where you do the sort of core reinforcement learning things”
Reinforcement learning and the OAK architecture offer the first plausible mechanistic answers to fundamental cognitive science questions: how high-level knowledge is learned from low-level experience, where concepts come from, how reasoning works, the purpose of play, and how perception can operate without external labels
“I think arguably reinforce Enforcement learning and oak offer the first plausible mechanistic answer to several important questions. How can high level knowledge be learned from low-level experience? Where do concepts come from? How do we reason? What is reason? Perhaps reason is just planning in this way. What is the purpose of play to find these these uh subpros which structure our our cognitive um our cognition? And what is the purpose of perception? We're answering the question of how we perception can operate without reference to a a human label or to an external world.”
The OAK architecture contains eight main steps executed in parallel at runtime: learning reward policy and value, generating new state features, creating subproblems from features, solving subproblems to create options, learning transition models for options, ordering features by utility, maintaining metadata and pruning, and planning with the learned models
“These these what eight steps done in parallel at runtime. Okay, it's a lot of steps.”
A truly domain-independent deep learning method that can do everything current deep learning does but also learn continually would be transformative and could take over everything people have done with deep learning
“If we had a deep learning method that was can do everything we're doing now but can also learn continually that would just um be a really big thing and I think there's no reason why it couldn't happen. Um and so I think it will”
Leslie Kearling pioneered work on continual backdrop in her 1993 PhD thesis, and this historical algorithm may be key to solving modern representation learning and feature discovery problems
“Leslie Keelbing did her uh did some of this work in her PhD thesis in 1993.”
An option in reinforcement learning is a pair consisting of a policy (a way of behaving) and a termination function (a way of deciding to stop behaving), with the initiation set dropped from the classical definition
“an option is a pair. Well, actually, you may think it's a triple. I sort of have dropped the initiation set in in in my for many last couple decades. Um, so for me, it's a pair of a policy, a way of behaving, and a way of deciding to stop behaving”
Value iteration updates the value of a state by maximizing over possible actions, taking the immediate reward, discounting, and taking the expected value of next state values weighted by the probability of reaching each next state
“value iteration... says I'm trying to improve the values of some states. I look at the possible actions and I'm going to maximize over them. I'm going to look at the immediate reward and I'm going to discount and then take the probability or the really the expected value of of the value of the next state... the probability of each next state. You wait by that you take the value of the next state”
Value iteration under function approximation requires replacing exact state values V(s) with approximate values V(s,w) parameterized by weight vector w, and replacing exact models with approximate models R-hat and P-hat, introducing computational complexity that simplifications would require discussing.
“we have to have function approximation. So I don't know I don't think it's that helpful to go through these equations but yeah the value V of S will become the approximate value of a state given a parameter vector W and also your model of the world will become R hat and P hat they also become parametric and then after you've done that um things are much the same there is more complications which I will skip having to do with what's computationally expensive”
Model-free reinforcement learning where agents construct policy and value functions should be extended by adding models of the world and planning with those models, which would potentially yield better results than model-free approaches alone
“obvious starting place is model free reinforcement learning basic reinforcement learning where the agent constructs a policy and value function at runtime... Uh but still better uh would be to make a model of the world and use that model to plan with potentially better”
Approximate value functions and approximate models are necessary in OAK because exact state values cannot be computed for infinite state spaces
“I'm sure you are concerned because I've been talking about via s which is the value of an individual state and uh we can't do that of course we have to have function approximation. So I don't know I don't think it's that helpful to go through these equations but yeah the value V of S will become the approximate value of a state given a parameter vector W”
AI development is a profound intellectual milestone comparable to the origin of life on Earth, representing humanity's attempt to understand how people work and make ourselves more powerful
“this is a profound intellectual milestone. It's going to change everything... as an intellectual milestone I think it'll be comparable to the origin of life on the earth”
The OAK architecture has spent 50 years in development, with Barto having spent his career designing and seeking better reinforcement learning algorithms for learning from experience, following the Alberta Plan for AI research developed by Barto, Mike, and Patrick.
“myself uh I've tried to think deeply about this intelligence for half a century. Every day I'm sort of in the trenches designing algorithms, trying to design algorithms, seeking better algorithms for reinforcement learning... And I follow this Alberta plan for AI research which you may know about. Mike and Patrick and I did it a couple years ago.”
The speaker has been working on designing reinforcement learning algorithms for half a century, doing this work 'every day...in the trenches', and this deep historical engagement informs his perspective on what matters for AI
“myself uh I've tried to think deeply about this intelligence for half a century. Every day I'm sort of in the trenches designing algorithms, trying to design algorithms, seeking better algorithms for reinforcement learning, for learning from experience.”
The speaker has followed the Alberta Plan for AI research co-developed with Mike and Patrick a couple years before this talk
“I follow this Alberta plan for AI research which you may know about. Mike and Patrick and I did it a couple years ago.”
Creating sub-problems for highly-ranked features is conceptually understood (blue checkmark) but waiting to be practically implemented once the problem of continual deep reinforcement learning with meta-learning is solved
“I think that is is is almost done. If it was done, there would be a green check mark, but it's blue. And blue means it would be done if we could do this continual deep reinforcement deep learning thing with metalarning”
The IDBD (Incremental Delta-Bar-Delta) algorithm, which is 'really old,' will likely be a key part of solving the continual learning problem for deep learning
“I also think that something like uh my algorithm called IDBID or IDBD and it's really old will be a key part of that uh solving this problem”
This talk is new, prepared the night before, and represents the first public presentation of OAK to an audience, making it unpolished and potentially repetitive
“this talk like hey um I was up till like the like four o'clock last night making making these slides. Um this is this is um this is sort of a new talk for me. This is the first time you're the you're the first guys hearing it.”
This talk is new and newly prepared, not previously delivered, reflecting ongoing changes to Barto's thinking developed throughout the week of the conference after talking to many people.
“this is um this is sort of a new talk for me. This is the first time you're the you're the first guys hearing it... I was up till like the like four o'clock last night making making these slides... this talk is new and it's not quite polished”