What does the next training paradigm look like?

A couple years after DeepMind released AlphaZero, researchers trained EfficientZero to be very data-efficient, where if given two hours to play an unseen Atari game, the model would beat a novice human, but this doesn't necessarily mean the model was more sample-efficient than humans—it depends on how you measure, because EfficientZero plays dozens of simulated games in its head for each real-world game step.

“Let's go back into history a bit. A couple years after DeepMind released AlphaZero, a group of researchers trained a model called EfficientZero, and the whole point of this model is to be very efficient with data. So if this model and a human both got two hours to play against a simulator of an Atari game that they hadn't seen before, this model would actually probably beat the novice human. Does this mean that the model was more sample-efficient than the humans? Well, that was the goal of the training, but it depends on how you measure sample efficiency. Because for each step in the real game, EfficientZero is playing dozens of simulated games in its head.”

Architectural innovations like sparse attention and KV cache compaction could provide intermediate representations to solve the memory-scaling problem, and a new architectural innovation is proposed almost every week in recent research.

“I talked before about how we already have many different working ideas for this kind of thing, from sparse attention to KV cache compaction. And every week, somebody releases a new paper suggesting some kind of other architectural optimization.”

Even if AIs could develop expert-level knowledge (like Henry Ford or Albert Einstein) from extended in-context experience, this knowledge would be ephemeral and wasted if it could not be distilled back into the weights, where it could persist across sessions.

“And even if, after enough in-context experience, the AIs could become like Henry Ford or Albert Einstein, all that would be ephemeral and wasted if you couldn't get those learnings back into the weights.”

Another more speculative idea for addressing sample efficiency is 'dreaming'—where AIs build a good simulation of reality to rehearse new skills, try alternative strategies, and reinforce what works, allowing AIs to experience orders of magnitude more simulated samples in the same wall-clock time.

“But there's also another much more speculative idea. Let's call it dreaming. If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what actually works, then AIs could experience orders of magnitude more simulated samples in the same wall-clock time.”

Just as pretraining created a base intelligence smart enough to become a competent agent with enough RLVR on top, RLVR has created an agent competent enough to actually be broadly deployed in the world, and from this broad deployment the agent can learn on the job once the training recipe for continual learning arrives.

“Just as pretraining created a base intelligence that was smart enough to become a competent agent with enough RLVR on top, so RLVR has created an agent that is competent enough to actually be broadly deployed in the world, and from this broad deployment to learn on the job once the training recipe for continual learning actually arrives.”

For many real-world skills like building a business from scratch, winning court cases, having a profitable day trading, or helping a candidate win an election, the RL rollout requires interacting with the actual real world and cannot be recreated within a datacenter, and the outer-loop verification may take months or even years of real-world actions.

“But for so many other different kinds of skills that we need AIs to have, we simply can't do this. How do we train an AI to get really good at building a business from scratch? How about winning court cases, or having a profitable day of trading in the markets, or helping a candidate win an election? The rollout here requires interacting with the real world, and you can't recreate it from just within a datacenter. The outer-loop verification here may take months or even years of real-world actions to elicit”

The most valuable bits of information that a model could learn from are revealed only during deployment, including what organizations are actually using the model for, what kinds of mistakes the model makes in the real world, and domain-specific tacit knowledge; yet deployed AIs are not able to use this information to improve themselves.

“And it's even worse than it sounds, because it is only in deployment that the most valuable bits of information which your model could learn from are actually revealed. What's actually happening in the organizations where I'm being used? What are they using me for? And what kinds of mistakes do I tend to make in the real world?”

The question of whether RLVR can generalize so strongly that spending trillions on RL environments could produce fully human-like general intelligence is an empirical question; Dario's quote that model performance degrades when serving at longer context lengths than trained suggests short-horizon RL training may not generalize to long-horizon performance.

“Now, whether RLVR can generalize this well is an empirical question. If the labs went from spending billions of dollars on RL environments to a trillion dollars, would you get the kind of thing that is a fully human-like general intelligence within the context window? Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not infinitely strong. When he was explaining why model performance tends to degrade at long context, he said: "There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations." Now, maybe I'm reading too much into this, but it seems like he's saying that short-horizon RL training doesn't necessarily generalize to long-horizon RL performance.”

OPSD has two advantages over RLVR: (1) it doesn't require outer-loop verifiable rewards, just a model that can learn in-context; (2) it provides denser supervision than naive RL because you can train on per-token probability discrepancies rather than projecting a single reward through the entire trajectory.

“This is better than RLVR for two reasons. One, OPSD doesn't require us to have some outer-loop verifiable reward. We just need a model that can learn the right things within the context window. And as long as we have that, we can train the base model to match our veteran teacher model, which has built up all this experience during the session. And two, OPSD provides a much denser supervision signal than naive RL. Instead of projecting a single reward through the whole trajectory, you can train on the per-token probability discrepancy between the teacher and student.”

Coding can be trained efficiently because you can define a deterministic container with a software repo and missing feature, and run a thousand parallel agents on identical copies from the same starting point, but this doesn't work for computer use because you can't have thousands of bots hitting Amazon's checkout flow without being detected and shut down.

“If you're trying to make a model better at coding, you can define some container that has a software repo with some missing feature that you have tasked the AIs with creating. And then you have a thousand parallel agents go at the problem, each of which has an identical copy of the container. But this doesn't work with computer use, at least not trivially. You can't just have a thousand agents go try the same checkout flow on Amazon to get better at using websites, because Andy Jassy will find your bots and shut your ass down.”

Unless a domain can produce a very replayable training target, models will struggle to make much progress, and the reason for this is that models are incredibly sample-inefficient during training.

“unless you can build a very replayable training target for a domain, the models will struggle to make much progress. And the reason this is true, of course, is that the models are incredibly sample-inefficient during training.”

Continual learning requires going back to update the weights; AIs cannot simply keep building up a larger and larger KV cache as they learn from more users, because that is not scalable and is not how humans learn—humans have no clean separation between parameters and activations, and the brain doesn't expand as we learn.

“But this kind of continual learning requires going back to the weights. AIs can't just keep building up a bigger and bigger KV cache as they learn from more and more users. That's just not scalable, and that's also not how humans do it. There's no clean separation in our brain between parameters and activations, and it's not like some part of your skull keeps expanding as you learn more things throughout your lifetime.”

RL training does not suffer from the SFT failure mode of encoding irrelevant data because RL concentrates updates only on what is relevant to getting the outcome right; RL updates are incredibly sparse, which is important for continual learning because you don't want to overwrite everything else the model knows.

“RL training doesn't suffer from this failure mode. RL is great at concentrating the update to only what is relevant to getting the outcome right. That's why the updates from RL are incredibly sparse. And this is a very important property for continual learning, because as you're learning on the job, you don't want to overwrite and forget all the other things that the base model knows.”

All major AI labs are making a research bet that training AIs on millions of verifiable tasks across thousands of diverse RL environments will produce AGI, because such training creates general problem-solving agents capable of making progress on open-ended tasks for weeks despite errors, ambiguity, and mistakes.

“So here's a big research bet that all the labs are making. They think that if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we will have basically built AGI, because this kind of training will have created a kind of problem-solving agent: the kind of thing that can make progress on open-ended tasks for weeks on end in the face of errors and mistakes and ambiguity.”

The current approach is analogous to having a genius graduate student who has never been allowed to take a real internship, given only classroom case studies in the form of RL training on environments, even though the student could be deployed in the economy and privy to so much domain- and organization-specific tacit knowledge.

“We've got some genius grad student who's never been allowed to take a real internship, and we keep giving it more and more classroom case studies in the form of RL training on environments. It's so bizarre that we have AIs that are broadly deployed through the economy already, and are participating in so many different kinds of tasks, and are privy to so much domain- and organization-specific tacit knowledge, and they're not able to make use of it.”

When humans learn, there is compression and this aids generalization and grokking; some autistic savants have high-fidelity recall of information (like models do in context), and such volume cripples their ability to understand abstractions and metaphors.

“When we learn stuff, there's clearly some kind of compression, and this aids our generalization and grokking. There are, in fact, some humans who have this autistic-savant-type ability to recall random tables of numbers or nonsense syllables years later — basically the kind of fidelity of information that models have in context. And such sheer volume cripples these humans' ability to understand abstractions and metaphors.”

Every time a user interacts with an AI, the AI will be smarter not only because it's learning from that user's previous sessions, but also because it's learning from all other interactions across the entire user base—this is very scary, exciting, and different from how AI currently improves.

“Every time that you interact with an AI, it'll be smarter, not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world. And that's very scary and exciting and different from the way that AI improves right now.”

Continual learning—where model weights are updated based on what the model learns from deployment—may not be necessary if in-context learning can get so good across longer and longer time horizons that you don't need to distill deployment learnings back into weights.

“Similarly, people would say, look, continual learning — this capability I keep harping about, where the model's weights get updated based on what it's learning from deployment — may simply not be necessary. Because if in-context learning gets so good across longer and longer time horizons, then you don't need to distill everything the model is learning on the job back into the weights.”

Approximately 30 to 50 percent of a lab's compute currently goes to inference, and this inference compute is not playing any productive role in helping improve the model, which represents a huge waste.

“Around 30 to 50 percent of a lab's compute goes to inference, and that compute is currently not playing any productive role in helping improve the model. This seems like a huge waste.”

For example, the Cursor Tab model can do online learning by predicting the same objective (which edits the user accepted) across over 400 million requests a day, but this works only because everyone is learning the same thing.

“For example, the Cursor Tab model online-learns by predicting the same exact objective for over 400 million requests a day. The objective here is which edits actually got accepted by the user.”

After a week of work at a deployment, the AI receives a thumbs up or thumbs down (work review), and the base model distills everything learned during the session via OPSD, dreaming, or combinations thereof, allowing the AI to get better at domains adjacent to what it was trained for, starting a cycle of expanding capabilities.

“Now let's say at this point, the effective context lengths have expanded such that AIs can jam and co-work with you for a full week of wall-clock time. At the end of a week, you give it a thumbs up or a thumbs down, you give it a work review. And if you give it a thumbs up, the base model distills everything that the AI learned during the session, and it may use OPSD, it may use dreaming, it may use some other technique that we aren't even aware of, or it'll use a combination of all of the above. And AI can get better at domains that are adjacent to what it was explicitly trained for beforehand with RLVR.”

Computer use has made much slower progress than other verifiable domains like coding and math, despite being clearly verifiable (e.g., 'did the package get delivered?', 'is the venue booked?', 'have taxes been submitted?'), which is unusual and reveals something important about AI progress constraints.

“Why has progress on computer use been so much slower than other domains? Computer use is so clearly verifiable. You could ask a question like: did the desired Etsy item I ordered get delivered? Is the venue for an event I'm trying to organize booked? Have my taxes been submitted? So isn't it weird that computer use has been making so much slower progress than coding and math and these other verifiable domains?”

OPSD is one approach to solving the sample-efficiency problem by squeezing all signal from scarce real-world experience into a tiny, well-targeted weight update.

“OPSD is one way to attack the sample-efficiency problem. You take this scarce real-world experience, and you squeeze all the signal into a tiny, well-targeted update.”

Once AIs become good enough at coding to build high-fidelity clones of Slack, Gmail, and other applications, progress on computer use will accelerate, and this process of building clones also serves as a great RL objective for coding itself, killing two birds with one stone.

“You can solve this by making clones of Slack and Gmail and all the other common applications and websites. But at least currently, this is a very labor-intensive and unscalable way to build environments. Of course, once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure computer use will make quicker progress than it is right now. And you're also killing two birds with one stone with this kind of procedure, because getting AIs to rebuild whole applications from scratch is also a great RL objective for coding.”