What this covers
Talk given by Yann LeCun at ETH Zürich during "Frontiers of Embodied AI".
Source description (no synthesized summary yet).
LeCun argues that achieving human-like intelligence requires world models trained via joint embedding predictive architectures (JEPA) using energy-based learning, not generative models or large language models, because the former enables efficient adaptation to new tasks through abstract representation and planning.
- Generative models fail at video prediction because they cannot represent infinite plausible futures; joint embeddings solve this by predicting in abstract representation space rather than pixel space
- Human intelligence is characterized by rapid adaptation to novel tasks with few samples, not accumulation of knowledge or skills, which requires grounded learning through interaction with the world
- Energy-based models with inference-time optimization (planning) are computationally more powerful than forward propagation for reasoning and control of complex systems
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Finding an abstract representation of input to enable prediction is natural because trying to explain all details at lower levels of description (quantum field theory, particle physics) is impractical; science progresses by inventing abstractions at multiple levels (quantum → atoms → molecules → cells → organisms → societies)
“In principle, I could explain or simulate everything that takes place in this room at the moment at the level of quantum field theory or particle physics, right? Could simulate the trajectory of every particle in this room. But of course that's completely impractical. And then you know what we do in science is that we invent abstractions to allow us to make predictions and those abstractions ignore a lot of details about the state the underlying state of the system.”
Intelligence requires grounding—a connection to the physical or sensory world—and is not achieved through language models alone, which operate without embodied experience
“we really have to go beyond this if you believe that a intelligence requires grounding. Of course, some philosophers, and certainly some language people, don't believe that's necessary, but I think it is.”
The Moravec paradox demonstrates that tasks simple for humans (like driving a car in a few hours or a 10-year-old doing domestic tasks) are extremely difficult for machines, while tasks difficult for humans (chess, symbolic computation) are easy for computers
“the Moravec paradox that you know, things that are simple are difficult for computers and things that are complicated for humans turn out to not be that difficult for computers. Uh like playing chess, computing integrals symbolically, and you know, solving equations, proving math theorems, etc.”
Machine learning has a fundamental gap compared to human and animal learning: people and animals can learn new tasks extremely quickly with very few trials and samples, while current AI techniques struggle with this rapid adaptation
“if when we compare the learning abilities of uh machines with humans and animals, uh clearly there is a big gap. Um you know, people and animals can learn new tasks uh extremely quickly and with very few trials, very few samples.”
Hierarchical planning is essential to intelligent systems and humans do it naturally—decomposing high-level goals into subgoals, gradually moving down to elementary actions—but this is an unsolved problem in AI and an open research question for roboticists and agentic AI researchers
“Ultimately, what you want though is something that can do hierarchical planning. All of us do hierarchical planning. Animals do hierarchical planning.”
World models are needed for controlling complex systems where dynamics cannot be reduced to equations, such as humanoid robots interacting with the real world, complex industrial systems, or turbojets, where you must learn an energy model of the system and environment interaction
“you want world models to control complex systems where you cannot model the dynamics of the system by writing a bunch of equations. Okay, if you have a humanoid robot or a you know, any kind of robot, you can just write down the dynamical equations and then simulate the dynamics of the robot and you can get your humanoid robot to do somersaults and and kung fu and whatever, right? That's simple. Uh as soon as the robot starts to interact with the real world, that's a lot more complicated.”
Seymour Papert, in a late-1970s debate between Jean Piaget and Noam Chomsky on language acquisition, argued for the learning possibility of neural networks despite having written a 1969 book critiquing the limitations of perceptrons, showing a reversal in his thinking about the value of neural network research.
“He had a debate with uh Noam Chomsky uh in France in the late 1970s, where um you know, they were debating whether language was innate or learned...there was transcriptions of that debate with, you know, people participating in it, and one of them was Jean uh was uh um uh was uh a a guy who had worked with Jean Piaget, who was a professor at MIT, and was talking about the perceptron, uh kind of saying, you know, there's those simple machine learning models that are you know, capable of uh learning surprisingly complex uh tasks...that may be kind of evidence for for fact that learning is possible, contrary to what Chomsky was saying. Um This guy was Seymour Papert. He was professor at MIT, and 10 years before that, he had written a book that basically killed the entire field of neural net, including the this basically pointing out the limitation of the perceptron.”
AGI (Artificial General Intelligence) is a meaningless concept because human intelligence is specialized; the real question is not 'can you know how to do everything?' but rather 'can you learn quickly how to do a wide spectrum of things?'
“the notion of AGI is complete nonsense. Uh human intelligence is specialized. The characterization of human intelligence is that it's very quickly adaptive and we can learn new task.”
Video is more redundant than text, which is actually a feature rather than a bug for self-supervised learning because redundancy in data is necessary for learning; without redundancy you cannot learn anything
“of course you're going to say, well, video is much more redundant than than text. But in fact, that's a feature not a bug. If you want to train a system, particularly using self-supervised learning, you need redundancy in the in the data. If you don't have redundancy, you can't learn anything. So, redundancy is a good thing.”
Energy-based models provide a unified framework for explaining self-supervised learning and all learning in general; they are more general than probabilistic models, which are a special case with particular energy forms
“the way to properly explain how you how you know, how you can train self-supervised learning systems and and every learning system really is a framework I call energy-based models that I've been advocating for 20 years or so.”
A fundamental challenge with information maximization methods is that proper definitions of information content require knowing the distribution of vectors, which we do not have; empirical measures are all upper bounds, not lower bounds, so methods use good upper bounds and cross their fingers
“we don't actually have objective measures of information content because all the proper definitions are based on knowing the distribution of the the vectors or whatever that you want to measure the information content of. And we don't know the distribution.”
An important property of intelligent systems is the mode of inference: forward propagation through fixed layers (as in LLMs) is less powerful than inference through energy minimization (searching for outputs compatible with observations)
“an important property of an intelligent system is the mode of inference. Does it compute its output by propagating through a fixed number of layers of some neural net? Or consider the alternative. The alternative is computing the output of a system by searching for an output that is most compatible, if you want, with the input.”
If you want to make real progress in AI for the real world (physical AI), do not work on LLMs or generative models; this view makes LeCun unpopular in Silicon Valley
“if you're interested in making real progress in AI, in sort of grounded, you know, AI for the real world, if you want, physical AI, don't work on LLMs. Don't work on generative models, either. So, as you can probably guess, this does not make me very popular in Silicon Valley.”
Reinforcement learning should be minimized in use (though not completely abandoned) because it is inefficient in sample efficiency; RL should be used only when desperate and no other approach works, and is most effective after learning good representations through observation and world model training.
“I've been saying for ever to abandon reinforcement learning. I don't really mean abandon. I mean, minimize its use because it's so so horribly inefficient in terms of sample efficiency... RL is like what you do when you're desperate and there is nothing else you can do... most of the learning, you know, in by observation... you can use RL on top of it because you already have the good representations.”
Training world models using generative approaches (predicting pixel-level details) fails because it is impossible to predict all plausible futures in video; there are infinite possible continuations and the system learns to predict the average, resulting in blurry outputs
“you simply cannot predict everything that takes place in a video. There's an infinite number of plausible things. In text, it's easy because there is only a finite number of words, and so you can get the system to produce a probability distribution over all possible words or tokens in your dictionary. But, you can't do this with video, right?”
Current AI techniques do not handle the real world well because it is characterized by continuous, high-dimensional, noisy data, whereas language is comparatively simple and easier for AI to process
“they don't really handle the real world. They don't handle continuous, high-dimensional, noisy data. Language is easy by comparison.”
Joint-embedding systems can collapse by ignoring inputs and producing constant representations; preventing collapse is the central problem in self-supervised learning for these systems and requires either contrastive methods (pushing up energy of non-data points) or regularized methods (minimizing volume of low-energy regions)
“a system like this can can collapse, and how can it collapse? It can essentially completely ignore the inputs, produce constant representations and other prediction problem is trivial.”
Distillation-based methods in JEPA use two encoders with identical architecture but different update rates: the left encoder gets gradients and updates normally, while the right encoder uses exponential moving average of the left's weights, stabilizing training
“you don't propagate gradient through the encoder on the right. Okay, those are two encoders with identical architectures and they kind of share the weights but the funny thing is that the encoder on the right use a exponential moving average over time of the weights of the encoder on the left. The encoder on the left gets gradient and gets updated all the time. The encoder on the right gets updated slower essentially and share the weights.”
The best way to explain how a baby's view of the world changes when moving its head is that the world is three-dimensional; this explains parallax and leads babies to derive 3D understanding
“One thing a baby can learn really quickly is that the world is three-dimensional. Why? Because the fact that an object has a distance every every uh point in the world has a distance from us is the best way to explain how our view of the world changes when we move our head.”
A 10-year-old child can accomplish tasks that domestic robots cannot (with minimal training and zero-shot capability) because children possess grounded intelligence developed through embodied interaction with the physical world.
“how is it that a 10-year-old can uh basically do what you would like a domestic robot to do and do most of those tasks without actually being trained to do them the first time you ask them, they can do it. They may not want to do it, but they can.”
Video generation systems (which can produce cute-looking videos) work by predicting in representation space then passing predictions through a decoder, not by predicting all plausible futures; the system only needs to produce one coherent video, not represent the distribution of all plausible videos—this is a much simpler problem than world modeling.
“you're going to tell me, 'Oh, but we can train system to produce cute videos, right? Video generation.' Yes, but this prediction usually is done in representation space, not in pixel space. It's only a second stage that actually turns the predictions into high-resolution uh high-frame-rate uh uh videos. And the system only needs to produce one cute-looking video. It doesn't need to actually represent all plausible videos.”
LLM-style auto-regressive token prediction is not a good model of reasoning; humans reason internally and not in token space/language, and tricking LLMs into reasoning by generating more tokens does not reflect how reasoning actually works
“That's auto-regressive prediction, and every token involves the computation of, you know, a fixed amount of computation running through a fixed number of layers of some neural net. Uh, this is not a good model. It's not a good model of reasoning. The way you coerce an LLM to do reasoning is that you trick it into generating more tokens. But, that's not the way we reason. We reason internally, we don't reason in token space, in language even.”
Model predictive control (MPC)—planning by recursively applying a world model to predict multiple steps ahead and optimizing action sequences to minimize energy—is a classical optimal control technique from the 1960s that provides the theoretical foundation for planning with world models.
“if you have a world model, certainly a lot of robotics system optimal control people in the room, you can apply this world model multiple time steps and, you know, each action an action sequence can be decomposed into a sequence. The guardrails can be applied to all the the steps in the sequence. Okay, that's the way you would use a world model. And the way you plan by optimization there is akin to model predictive control, MPC, very classical stuff in optimal control going back to the 1960s.”
Academics should not work on LLMs because there is nothing new to contribute; LLM development is dominated by industry with massive resources, making academic research in this area unproductive
“if you work on LLM, of course. But you should not work on LLM. Uh at least if you're in academia, you should absolutely not work in LLM. There is nothing you can bring to the table.”
A world model-based system can be made intrinsically safe by incorporating guardrail objectives that must be optimized alongside task objectives for every output, unlike LLMs which can only be fine-tuned and are susceptible to jailbreaking
“a system constructed this way can be made intrinsically safe because it has to obey and, you know, optimize the guardrail objective with every output it produces. This is not the case for an LLM. An LLM, the only way an LLM can be made, uh, you know, safe or non-toxic or whatever you want to call it, is by fine-tuning it. And there is always a way to break the conditioning, if you want, to jailbreak the the system. Here, you can't jailbreak a system like this.”
The violation of expectation method used by psychologists to measure whether babies have learned world concepts (e.g., showing a 6-month-old a floating car produces no surprise, but showing a 10-month-old the same scene produces shock) can be adapted to test whether machine learning systems have acquired common sense understanding.
“a 6-month-old will barely pay attention, hasn't learned about gravity yet. Uh a 10-month-old will go very surprised, like the little girl. Uh and that that's actually how psychologists measure uh whether a baby has learned a particular concept about the world, uh which is a the violation of expectation. Um we actually we can actually use those techniques to test whether machine learning system has have acquired some, you know, some notion of common sense.”
Generative models (autoencoders) trained on images do not produce good representations for downstream tasks; joint-embedding architectures are superior and all the best self-supervised image and video representation systems use joint-embedding, not reconstruction
“So, the values is when you train systems of this type to learn representations of images, you don't get good representations. If you use the representation of images obtained this way, you feed it to a downstream uh task that you train supervise, okay, you train a head supervise. The results you get are not great. To get good results, you have to use joint-embedding architectures. All the best systems that use self-supervised learning to train an image or video representation systems system, all use joint-embedding. None of them uses reconstruction.”
V-JEPA (video JEPA) trained on masked video masking predicts representation of full video from partially masked video and learns common sense about physical plausibility: prediction error spikes when impossible events occur (e.g., ball disappearing, car not falling)
“so one thing we can do V-Jeppa because we train it to predict what's going to happen next in the video. We can try to predict her to do that. We can measure its internal prediction error. We can show it a video and monitor the internal prediction error at every every time step. And the cool thing is that if you show it a video where something impossible occurs, something unphysical the prediction error shoots to the roof.”
Video generation systems are not world models; they are fundamentally different from world models that are designed for planning and control, and conflating them confuses the research agenda.
“so, a lot of people are working on video generation and they call this world models. They They're not world models. They're video generation systems. Uh so, one one big message from my talk is that if if you want to use world models, do not work on video generation. This is a different problem, okay? If you want to produce cute videos, work on video generation. But if you want to like control robots or industrial processes or understand the world, do not work on generation.”
Joint Embedding Predictive Architecture (JEPA) encodes both input X and future observation Y before prediction, allowing the system to eliminate unpredictable information and produce more abstract, accurate representations compared to generative models that reconstruct raw pixels
“with uh JEPA, you observe X and Y and A, but you encode both X and Y, and the prediction takes place in that representation space. Okay? Major difference. What the system can do is essentially eliminate from the input uh, by constructing a representation of Y, it can eliminate all the information about about Y that is simply not predictable. Right? And that makes the prediction more abstract, with fewer details, but more accurate in a way.”
Information can be maximized by ensuring all rows of a representation matrix are different (making each sample distinct) or all columns are different (making each variable independent), with the latter (dimension contrastive methods) being preferable because independent variables are more informative
“There's two ways to make that matrix informative. One way is to make sure all the rows are different. And other way is to make sure all the columns are different. You want to make sure the columns are different because if all the columns are the same, that means every variable in the representation carries the same information.”
Sigreg (Sketch Isotropic Gaussian Regularization) makes representation distributions isotropic Gaussian by projecting points along many directions and ensuring marginal distributions match a Gaussian cumulative distribution, which guarantees the joint distribution becomes isotropic Gaussian
“you run a batch of samples through your encoders, and what you get is a bunch of points in the vector space of dimension whatever the dimension of your representation space is. We're going to try to make the distribution of those points as a tropic Gaussian with the same uh variance in all dimensions. Why? Because an isotropic Gaussian is a distribution where all the variables are independent.”
A 4-year-old has been exposed to approximately 16 hours of awake time per day, with about 30 minutes of that being YouTube-like video content, receiving data through 2 million optic nerve fibers at about 1 byte per second each, totaling approximately 10^14 bytes of sensory data—the same volume as current LLMs trained on 400,000 years of text
“a 4-year-old has seen during his or her life. That's about 16 hours of wake time. And uh which by the way the small amount of video is about 30 minutes of YouTube uploads. And uh we have 2 million optic nerve fibers carrying about 1 byte per second each. So, the data volume uh that for your world I see through vision and probably through touch as well is about 10 to the 14 bytes.”
Techniques like MNCR, NCR squared, WMSE, Sigreg, VICReg, and Barlow Twins all implement information maximization for joint-embedding systems, with JEPA gaining popularity (1,700 papers mention it on Google Scholar)
“there's about 1,700 papers that mention joint embedding predictive architecture spelled out uh on Google Scholar.”
Guardrails for world model systems require learning small projection heads that map from representation space to constraint/task objectives; these are trained with very small sample sizes since they are minimal neural network layers.
“No, you would have to learn uh a very small head on top of your representation that maps uh that to your you know, the constraint that you're interested in. Uh so, that part has to be trained, but you can train it with a very small number of samples because it's a tiny basically it's just a projection.”
Planning with world models trained on DINO encodings can optimize complex simulated environments with non-trivial dynamics, achieving goals in less than 25 steps, demonstrating practical effectiveness of the world model + planning approach
“you have a initial state here of a kind of simulated environment that has pretty complex dynamics. And you have goals at the top and at the bottom what you see is the sequence of actions of a planner that uses this trained world model to get the world to a configuration as close as possible to the original one in less less than 25 steps.”
Jean Piaget's definition of intelligence—'Intelligence is not what you know, it's what you do when you don't know'—is actually apocryphal and was never directly stated by Piaget, but other psychologists distilled his thinking into this formulation
“Jean Piaget says, "Intelligence is not what you know, it's what you do when you don't know." In fact, he never actually said this. This is apocryphal. Uh but there are other psychologists who basically kind of distilled his thinking into this sentence, which he never said.”
Babies learn that objects have permanent existence (object permanence) and stability/rigidity concepts very quickly, while learning intuitive physics concepts like gravity takes 9 months of development
“basic concepts like object permanence is this is learned really quickly. Uh notions of stability, rigidity, and things like that. But then, uh what we would consider intuitive physics, uh things like uh inertia, gravity, that actually takes 9 months for uh human infants.”
Distillation methods came from ideas at Google DeepMind for stabilizing variance in reinforcement learning (using exponential moving average of weights), which were realized to apply to self-supervised learning from images; BYOL (Bootstrap Your Own Latent) pioneered this approach.
“This is derived from some intuitive ideas some people at Google DeepMind who are using techniques like this to stabilize the variance in reinforcement learning and they realized you could apply this to self-supervised learning from images. They call this BYOL bootstrap your own latent”
LeCun left Meta at the end of the previous year and formed a new company called Ami Labs focused on AI for the real world (physical AI)—robotics, control of industrial processes, and high-dimensional continuous noisy problems where LLMs are helpless
“I left Meta, as many of you probably know, at uh the end of last year and formed a new company called Ami Labs. And the purpose of uh Ami Labs is sort of AI for the real world, like, you know, physical AI. Uh robotics is a use case, but it's not just that. It's control of uh industrial processes. Like, anything that is high-dimensional, continuous, and noisy, for which LLMs are completely helpless.”
Technical working papers on world models and energy-based learning by LeCun (written with collaborators, published 2022) are available via QR codes shown in the presentation.
“I've I've sort of uh settled on this kind of uh I idea or architecture uh about 5 years ago. I wrote a long paper about it uh that I put online in 2022 uh with some general architecture, etc. If you want to take pictures, here are QR codes, you can get to it.”