What this covers
The Next AI Revolution: Yann LeCun’s Vision Beyond LLMs
At the AI Action Summit in Paris, Yann LeCun underscored a fundamental shift in artificial intelligence—one that moves beyond the brute-force approach of large language models (LLMs). Instead of systems that merely predict the next token, the future of AI hinges on *world models*—structured, adaptive representations that can infer, reason, and plan.
This vision holds immense potential for fields like healthcare and biology, where complexity defies exhaustive computation.
🔹 1. Prioritize Key Insights Over Exhaustive Generation In medicine, capturing every molecular interaction is infeasible. The focus should be on critical variables—key biomarkers, for instance—that shape disease progression and treatment response. This is where JEPA (Joint Embedding Predictive Architecture) thrives: predicting essential relationships rather than generating redundant details.
🔹 2. Replace Probability Overload with Efficient Scoring Traditional AI wastes resources computing endless probabilities. Instead, energy-based models assess how likely or “normal” a given state is. In healthcare, this translates to identifying anomalous symptoms or lab results instantly—without brute-force calculations.
🔹 3. Move Beyond Contrastive Learning to More Direct Approaches Contrastive learning hinges on distinguishing “positive vs. negative” examples, often requiring vast datasets. Simpler, more direct methods can recognize meaningful patterns with less data—an advantage in medical research, where data is often scarce.
🔹 4. Shift from Trial-and-Error to Model-Driven Discovery Blindly testing drugs or protein interactions is slow and costly. AI-driven world models can predict biological behavior, allowing experiments to focus only on deviations from expected outcomes. This approach accelerates breakthroughs while reducing inefficiencies.
🔹 5. LLMs Alone Won’t Achieve Human-Level Intelligence While LLMs excel at summarizing and automating documentation, true comprehension—such as understanding disease mechanisms—requires AI that grasps causality, not just linguistic patterns. The next frontier isn’t about scaling transformers but building models that think.
LeCun’s vision challenges conventional AI wisdom: instead of merely making models bigger, make them *smarter*. In healthcare and beyond, this shift could redefine how we diagnose, predict, and treat complex conditions. The future of AI isn’t just about processing data—it’s about understanding the world.
If you would like to support the channel, please join the membership: https://www.youtube.com/c/AIPursuit/join
Subscribe to the channel: https://www.youtube.com/c/AIPursuit?sub_confirmation=1
The video is reposted for educational purposes and encourages involvement in the field of AI research. Source: AI Action Summit 2025
Source description (no synthesized summary yet).
LeCun argues that human-level AI requires energy-based world models trained on sensory data to learn causal understanding of physics and planning, not large language models that merely predict tokens, and that this approach must be developed through open-source collaborative platforms.
- Current LLMs lack causal models of the world and fail at tasks animals solve trivially (domestic robotics, physical reasoning), revealing a fundamental architectural mismatch with human intelligence
- Energy-based joint embedding predictive architectures (JEPAs) can learn abstract representations of world dynamics through self-supervised learning on video, enabling planning without explicit reward signals
- Open-source distributed platforms are necessary because no single entity can build foundation models that span all languages, cultures, and values, and secret research causes technological decline
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Hierarchical planning allows AI systems to decompose complex goals into multiple levels of abstraction: high-level planning works with macro actions and abstract representations over longer timescales (e.g., 'go to the airport'), while lower levels plan detailed actions over shorter timescales (e.g., 'walk to the elevator, push button, descend')—humans do hierarchical planning constantly, but learning machines have almost no capability to do this; nearly all robots use hand-crafted representations at each hierarchy level rather than learned abstractions.
“if I want if I'm sitting at my office at NYU and I decide to go to Paris um I can decompose that task into two sub tasks go to the airport and catch a plane okay now I have a sub goal going to the airport um I'm in New York city so going to the airport consist in going down on the street and haing a taxi how do I go down in the street well I need to uh get to the elevator push the button go down go out the building how do I go to the elevator well I need to stand up for my chair pick up my bag open the door walk to the elevator avoid all the obstacles”
Current LLMs operate by autoregressive token prediction where a causal transformer repeatedly takes a window of previous tokens and predicts the next token, and this creates a fundamental problem: each token prediction has some probability of being wrong, and if one is, there is no way to fix it afterwards, leading to exponential divergence and hallucination issues.
“they basically um produce one token after the other autor regressively right so you have a sequence of tokens which are subo units but it doesn't matter what they are a sequence of symbols and then you have a predictor that is repeated over the sequence that Bic basically take a window of previous tokens and predict the next token”
Classical inference in statistics, structure prediction, and classical AI uses a different paradigm than LLMs: these domains use an energy function that measures the degree of incompatibility between observations and proposed outputs, and inference consists of finding the output that minimizes this energy/incompatibility—this is more akin to human System 2 thinking (deliberate reasoning about actions before taking them) than LLM System 1 (automatic pattern matching without reasoning).
“the way inference works is is not that way in uh In classical AI in statistics uh in structure prediction a lot of different domains the way it works is that you have a function that measures the degree of compatibility or incompatibility between your observation and a proposed output and then the inference process consist in finding the value of an output that minimizes this incompatibility measure”
Autoregressive token prediction in LLMs produces exponential divergence errors because each generated token carries independent error probability, and if any token falls outside the set of reasonable answers, there is no way to recover afterward, leading to hallucinations
“the way you train the uh those systems uh then you can use it to generate text by just Auto aggressively producing a token shifting it into the input and then producing the second token shifting that in ETC that's Auto prediction Not A NEW Concept at all obviously um and there's an issue with this which is that um the U the that process is basically Divergent every time you produce a token there is some chance that the token is not within the set of reasonable answers and take you outside a set of reasonable answers and if it does that there is no way to fix it afterwards”
Infants and young humans accumulate substantial background knowledge about the world in the first few months of life through mostly observation, including notions of object permanence, solidity, rigidity, and natural object categories before they even understand language, and they develop understanding of intuitive physics like gravity and inertia around nine months of age—this development is driven by observations and interactions, not language.
“infants accumulate a huge amount of background knowledge about the world in the first few months of life um Notions like object permanence um solidity rigidity natural categories of objects before children understand language they do understand the difference between the table and the chair um that kind of develops naturally and they understand intuitive physics notion like gravity inertia and things of that type around the age of nine months”
A particularly interesting finding is that systems trained on masked video frame prediction show anomaly detection capability: when the system slides a prediction window over a video, it detects frames where anomalous events occur (objects disappearing, changing shape spontaneously) through sharp spikes in prediction error, indicating the system has learned intuitive physics and common sense despite the simplicity of the training objective.
“if you show that system um videos where something really strange happens that system actually is capable of telling you my prediction error is going through the roof there is something strange going on in that window so you you take a you take a video and you take the 16 video Frame Window you slide it over the video and you measure the prediction error of the system and if something really strange happen like an object spontaneously disappears or change shape um the prediction error shoots up so what that tells you is that that system despite its Simplicity has learned some level of Common Sense he can tell you if something really strange in the world is happening”
A typical modern LLM is trained on approximately 30 trillion tokens (3×10^13), which equals roughly 10^14 bytes of data—equivalent to all publicly available text on the internet—and would take a human almost half a million years to read through, yet a four-year-old child awake for 16,000 hours has received approximately the same data volume (10^14 bytes) through visual perception via 2 million optical nerve fibers at roughly 1 byte per second, and thus humans achieve far greater intelligence from equivalent data volume because they use multimodal sensory input rather than text alone.
“A typical llm nowadays is trained on on the order of 30 trillion tokens three 10 to the 13 uh tokens that's two to the 13 words roughly each token is about three bytes um so the data volume is roughly 10 to the 14 bytes uh it would take any of us uh almost half a million years to read through all that material it's basically all the publicly available text on the internet”
Training a system to predict what will happen next in a video is a natural approach for learning world understanding because if a system can predict future video frames it presumably has learned the underlying structure of the world; however, this approach for video fails where it works for text: with text there are a finite number of possible next tokens so prediction can succeed, but with video there are infinite possible pixel configurations making exact frame prediction intractable.
“very natural idea for for getting systems to understand how the world works is um using the same process that we used to um to to train system for natural language and apply this to let's say video okay if a system is capable of predicting what's going to happen in a video you show it A short segment of video and you ask it to predict what's going to happen next presumably it would have understood the underlying structure of the world um and so training it to make that prediction might actually cause the system to understand the annoing structure of the world it works for text because predicting words is relatively simple why is predicting words simple because words um there's only a finite number of possible words”
Current AI systems can solve abstract intellectual tasks like chess, math problems, and theorem proving because those are pattern-matching problems that can be regurgitated from training data, but when you change the problem statement slightly, systems still produce the same answer because they have no real mental model of the underlying structure or physics involved.
“when you formulate a problem you pose a problem to an llm and if the problem is kind of a standard puzzle the answer will be regurgitated in just a few seconds if you change the statement of the problem a little bit the system will still produce the same answer that it had before because it has no real mental model what goes on um in the in the puzzle”
Current LLMs cannot distinguish between their own knowledge and hallucinations because they lack mental models of the domain; when a problem statement is altered, they produce the same answer despite the change, indicating they are regurgitating memorized patterns rather than reasoning about the problem structure.
“if the problem is kind of a standard puzzle the answer will be regurgitated in just a few seconds if you change the statement of the problem a little bit the system will still produce the same answer that it had before because it has no real mental model what goes on um in the in the puzzle”
A world model is a system that, given a current estimate of the world state (in abstract representation) and an imagined sequence of actions, predicts the resulting world state that will occur after those actions—this allows planning by running the world model multiple steps forward with different action sequences to find those that lead to desired outcomes.
“World model is given given a current estimate of the state of the world which is in an abstract representation space and given an action sequence that you imagine taking uh your world model predicts the the resulting state of the world that will um occur after you take that sequence of actions okay that's what a world model is”
DINO (a distillation-based method developed by Fair Paris researchers led by Mathilde Caron) and similar approaches learn generic visual features that transfer well to downstream tasks without needing negative samples, and these methods have been extended to video (DINO v2, v3) to learn temporal representations from video segments.
“there's a particular way of implementing this AIO distillation called IA there's another one called called Dino or Dino uh which I I skipped a little bit um and um so Dino um is V2 people are working on on V3 this is a method produced by some some of my colleagues at at Fair Paris um team led by Max Maximo cab um and then a slight different version um called IA V JEA by also Fair people in in Montreal and Paris mostly so no need for negative samples there and those those kind of those systems learn generic features that you can then learn for any Downstream task and the features are really good”
Infants accumulate vast background knowledge about the physical world in the first few months of life through observation—developing concepts like object permanence, solidity, rigidity, natural object categories, and intuitive physics (gravity, inertia) around nine months—and the amount of interaction required for this learning is astonishingly small
“infants accumulate a huge amount of background knowledge about the world in the first few months of life um Notions like object permanence um solidity rigidity natural categories of objects before children understand language they do understand the difference between the table and the chair um that kind of develops naturally and they understand intuitive physics notion like gravity inertia and things of that type around the age of nine months”
Current LLMs operate through autoregressive token prediction—producing one token at a time by predicting the next token given a fixed window of previous tokens—which creates exponential divergence because each token prediction has some probability of error, and once an incorrect token is generated, there is no way to correct it, leading to hallucinations.
“the way you train the uh those systems uh then you can use it to generate text by just Auto aggressively producing a token shifting it into the input and then producing the second token shifting that in ETC that's Auto prediction Not A NEW Concept at all obviously um and there's an issue with this which is that um the U the that process is basically Divergent every time you produce a token there is some chance that the token is not within the set of reasonable answers and take you outside a set of reasonable answers and if it does that there is no way to fix it afterwards um and if you assume there is some probability for that... then you get exponential Divergence uh which is why you know we have with those models hallucination issues”
We need human-level AI not just as an interesting scientific question but as a product need for ubiquitous smart devices like smart glasses that will mediate all our interactions with the digital world and require AI assistants with human-like intelligence because that is what humans are familiar with interacting with and understand intuitively.
“we do need human level AI um and it's not just because it's an interesting scientific question it's also sort of a product need um we are going to be uh wearing smart devices like smart glasses and things of that type in the future and in in those smart U devices we'll be able to um access AI assistants that will be with us at all times”
Current machine learning techniques are fundamentally inadequate compared to human and animal learning because they lack the right characteristics: animals and humans have background knowledge allowing them to learn new tasks extremely quickly, understand how the world works, reason and plan based on common sense, and are driven by objectives—capabilities that current AI systems do not possess.
“the problem is machine learning sucks compared to what we observe in humans and animals uh we don't really have the techniques that would um allow us to build machines that have the the same type of uh learning abilities and Common Sense and understanding of the physical world”
Training JEPA architectures requires designing cost functions that produce low energy on training data but high energy outside the training manifold; there are only two classes of methods for this: (1) contrastive methods push down energy on real data and push up energy on synthetic negative samples, but they don't scale well to high dimensions and require many negative samples; (2) regularized methods use regularizers to minimize the volume of low-energy space, which scales better.
“we need this to be low on the training data but we need also needed to be large outside the training set okay so this is you know this kind of energy function here that has kind of uh Contours of equal equal energy we need to make sure the energy is high outside of the manifold of data and I only know two classes of methods for this one set of method is called contrastive it consists in um having uh data points which are those those blue dark blue dots pushing the down the energy of those and then generating you know those flashing green dots and then pushing the energy up the problem with this type of method Contra method is that they don't scale very well in high dimension if you have too many dimensions in your space of Y you're going to need to push up in lots of different places”
A major danger is that geopolitical rivalry will incentivize governments to make the release of open-source models illegal under the impression that countries stay ahead by keeping science secret; this would be a huge mistake because research in secret always falls behind—the rest of the world advances and overtakes you, which is currently happening as open-source models are slowly but surely overtaking proprietary models.
“the danger I see um in Europe and in other places is that geopolitical rivalry will U entice governments to basically make the release of Open Source model illegal because there are under the impression that a country will stay ahead if he keeps uh its science secret that's that would be a huge mistake when you do research in secret you fall behind that's inevitable what will happen is that the rest of the world we go up and and will overtake you that's currently what's what's happening the open source models are overtaking uh slowly but surely uh proprietary models”
There is danger that geopolitical rivalry will incentivize governments to ban open-source AI model releases under the mistaken belief that keeping science secret confers competitive advantage, but research in secret causes countries to fall behind as the rest of the world overtakes them—open-source models are already overtaking proprietary ones
“the danger I see um in Europe and in other places is that geopolitical rivalry will U entice governments to basically make the release of Open Source model illegal because there are under the impression that a country will stay ahead if he keeps uh its science secret that's that would be a huge mistake when you do research in secret you fall behind that's inevitable what will happen is that the rest of the world we go up and and will overtake you that's currently what's what's happening the open source models are overtaking uh slowly but surely uh proprietary models”
Current machine learning systems are fundamentally deficient compared to animal and human learning because they lack techniques that enable machines to build background knowledge, common sense understanding of the physical world, and the capacity to reason and plan—capabilities that allow humans and animals to learn new tasks extremely quickly and understand how the world works
“machine learning sucks compared to what we observe in humans and animals uh we don't really have the techniques that would um allow us to build machines that have the the same type of uh learning abilities and Common Sense and understanding of the physical world”
Current machine learning systems are fundamentally inadequate compared to human and animal learning because they lack the background knowledge, common sense, and causal understanding of the physical world that allows humans and animals to learn new tasks extremely quickly and reason about the world.
“the problem is machine learning sucks compared to what we observe in humans and animals uh we don't really have the techniques that would um allow us to build machines that have the the same type of uh learning abilities and Common Sense and understanding of the physical world”
Humans and animals learn through the development of mental models of the world—abstract causal models that allow them to understand object permanence, solidity, intuitive physics (gravity, inertia), and natural categories before or independently of language learning, and these models are formed through observation and minimal interaction.
“how do um humans infants learn how the world works and you know infants accumulate a huge amount of background knowledge about the world in the first few months of life um Notions like object permanence um solidity rigidity natural categories of objects before children understand language they do understand the difference between the table and the chair um that kind of develops naturally and they understand intuitive physics notion like gravity inertia and things of that type around the age of nine months”
A world model is a learned predictor that takes a current estimate of the world state (in abstract representation) and an imagined sequence of actions and predicts the resulting state of the world that will occur after those actions are taken, enabling planning without explicit reward signals.
“World model is given given a current estimate of the state of the world which is in an abstract representation space and given an action sequence that you imagine taking uh your world model predicts the the resulting state of the world that will um occur after you take that sequence of actions”
Model predictive control—using an internal model to optimize actions at runtime by simulating forward and finding action sequences that achieve goals—is a classical technique from optimal control (dating to the 1960s) that is being rediscovered in modern AI as 'test-time computation', despite being a well-established approach.
“um you know test time computation and blah blah blah as if it was something new this is completely classical in optimal control this is called Model preductive control it's been around with us for about the same time that I've been around all right um the first paper is on you know planning using using models of this type using optimization are from the early 60s um the the ones that actually learned the model are more recent they're more from the 70s from France actually um it's called edcom”
We observe Moravec's Paradox: tasks that humans find simple and take for granted, like domestic robots clearing a dinner table or understanding cat-level intelligence, are extraordinarily difficult for current AI, while tasks humans find cognitively demanding, like proving theorems or passing the bar exam, are relatively easy for LLMs.
“we keep bumping into this Paradox called Mor Paradox right things that we take for granted um because humans and animals can do it we think it's not complicated it's actually very complicated and the stuff that we think is uniquely human like manipulating and generating language playing chess playing go playing poker producing poetry and this kind of stuff turn that to be easy relatively”
Reference to 'test time computation' and related ideas as recent innovations is misleading: this is classical optimal control under the name Model Predictive Control (MPC), which has existed since the early 1960s; variants that learned the dynamics model emerged in the 1970s from France in work like ECCOM, and the concepts are well-established in optimal control literature.
“you know people are excited about um um you know test time computation and blah blah blah as if it was something new this is completely classical in optimal control this is called Model preductive control it's been around with us for about the same time that I've been around all right um the first paper is on you know planning using using models of this type using optimization are from the early 60s um the the ones that actually learned the model are more recent they're more from the 70s from France actually um it's called edcom um some people in optimal control might know about this um but you know it's very simple concept this works amazingly well”
Moravec's Paradox reveals that tasks humans find effortless—like domestic robots clearing a dinner table, self-driving vehicles operating at human level, or cats understanding physics and opening doors—are actually computationally complex, while tasks we consider uniquely human like proving theorems, playing chess, or generating poetry are relatively easy for machines
“so that tells you we're missing something really big um yet we have systems that can pass the bar exam do math problems prove theorems but no domestic robots so we keep bumping into this Paradox called Mor Paradox right things that we take for granted um because humans and animals can do it we think it's not complicated it's actually very complicated and the stuff that we think is uniquely human like manipulating and generating language playing chess playing go playing poker producing poetry and this kind of stuff turn that to be easy relatively”
Current LLM inference is computationally wasteful because they apply a fixed number of transformer layers regardless of question complexity: answering 'does 2+2=4?' and 'does P=NP?' consume identical computation, whereas classical inference (statistics, classical AI, structure prediction) uses energy-based optimization that allocates computation proportionally to problem difficulty
“right now the way an llm uh performs inference is by running through a fixed number of layers of anet a transformer then producing a token injecting that token on the input and then running through a fixed number of layers again and the problem with this is that if you ask a simple question or complex question and you ask the system to answer by yes or no like does 2 and two equal four yes or no or does p equal NP yes or no it's going to spend the exact same amount of computation to answer those two questions”
LeCun's core recommendations for AI research: (1) Abandon generative models in favor of JEPA, (2) Skip probabilistic models in favor of energy-based models, (3) Replace contrastive methods with regularization methods, (4) Abandon reinforcement learning except as a last resort when models or cost functions are inaccurate
“I'm having a number of uh recommendations abandon generative models the most popular method today that everybody is working on startop working on this you work on jads those are not generative models they predict in representation space probably seek models because it's intractable use energy based models uh M have had like a 20 year contentious discussion about this um abandon contractive methods in favor of those regularized methods abandon reinforcement learning but that I've been saying for a long time we know it's inefficient”
We will never achieve human-level intelligence by training exclusively on text because the information integration rate favors multimodal perception; text-only approaches are fundamentally constrained regardless of scale
“we're never going to get to human level intelligence by just turning on text it's not just not happening despite what you know some people who are have a vested interest in this happening are telling us we're going to reach you know PhD level intelligence by next year it's just not happening”
Predicting words works well for LLM training because there are a finite number of possible words/tokens, allowing probability distributions over the entire vocabulary; predicting video frames cannot use this approach because the space is too high-dimensional and intractable.
“training it to make that prediction might actually cause the system to understand the annoing structure of the world it works for text because predicting words is relatively simple why is predicting words simple because words um there's only a finite number of possible words certainly a finite number of possible tokens and so we can't predict exactly which word will follow another word or what what word is missing in the text but we can produce a probability distribution or score for every possible word in the dictionary we cannot do this for images for video frames we do not have good ways of representing distributions of our video frames um every attempt to do this uh basically bumps into mathematical intract abilities”
Information content regularization methods prevent encoder collapse by maintaining information in the encoder outputs: one approach is to compute the covariance matrix of representation vectors across a batch and try to make it equal to the identity matrix, which forces each variable to be individually informative (high variance) and decorrelated with others.
“an indirect way of doing this is maintain the information content coming out of the encoder Okay so so you're going to have a training objective function which is a negative information content if you want because we minimize in machine learning we don't maximize uh one way to do this is to basically take the um vectors representation vectors that come out of the encoder over a batch of samples um and make sure they contain information how you can you do this you can take that Matrix of representation vectors and compute the product of that matrix by its transposed you get aarian Matrix and you try to make that coari Matrix equal to Identity”
We need human-level AI not just as an interesting scientific question but as a product necessity for ubiquitous smart assistants (smart glasses, wearables) that must mediate all digital interactions and work for populations unfamiliar with technology, requiring the same level of intelligence humans expect when interacting with other humans
“we do need human level AI um and it's not just because it's an interesting scientific question it's also sort of a product need um we are going to be uh wearing smart devices like smart glasses and things of that type in the future and in in those smart U devices we'll be able to um access AI assistants that will be with us at all times”
Distillation-based methods for representation learning use two encoders where one is an exponential moving average of the other, forcing the slower encoder to change weights more gradually; this prevents encoder collapse without explicit contrastive samples, and while there is some theoretical work on why this works, the mechanism is somewhat mysterious and LeCun is personally uncomfortable with it, though it empirically works well when implemented carefully.
“those are based on distillation so again you have two encoders it's still a joint Ting productive architecture you have two encoders they kind of share the same weights but not really so the encoder on the right uh gets a version of the weights of the enod on the left that are obtained through a um exponential moving average okay a moving average so basically you force the encoder on the right to uh change its weights more slowly than the one on the left and for some reason that prevents collapse there's some theoretical work on this um in fact uh this one that jum just finished writing um but it's a little bit mysterious why this works”
Distillation-based methods for training joint embeddings (like BYOL and Dino) use two encoders where one is updated via exponential moving average of the other's weights, which prevents collapse and works well empirically, though the theoretical explanation for why this works is not fully understood.
“there is another set of method that to some extent has been slightly more successful over the last couple years and those are based on distillation so again you have two encoders it's still a joint Ting productive architecture you have two encoders they kind of share the same weights but not really so the encoder on the right uh gets a version of the weights of the enod on the left that are obtained through a um exponential moving average okay a moving average so basically you force the encoder on the right to uh change its weights more slowly than the one on the left and for some reason that prevents collapse there's some theoretical work on this um in fact uh this one that jum just finished writing um but it's a little bit mysterious why this works and frankly I'm a little uncomfortable with this method”
Dino (self-supervised vision model using knowledge distillation) and similar vision models trained on image masking learn generic visual features that can be used without retraining for any downstream task (action classification in videos), achieving strong results without task-specific optimization.
“so Dino um is V2 people are working on on V3 this is a method produced by some some of my colleagues at at Fair Paris um team led by Max Maximo cab um and then a slight different version um called IA V JEA by also Fair people in in Montreal and Paris mostly so no need for negative samples there and those those kind of those systems learn generic features that you can then learn for any Downstream task and the features are really good um so this works really well”
The reason large language models fail at tasks requiring causal understanding (like solving puzzles) is that they have no real mental model—they simply regurgitate training data when given standard problem formulations and produce identical answers when problems are reformulated, despite the logical reformulation.
“if you formulate a problem you pose a problem to an llm and if the problem is kind of a standard puzzle the answer will be regurgitated in just a few seconds if you change the statement of the problem a little bit the system will still produce the same answer that it had before because it has no real mental model what goes on um in the in the puzzle”
Energy-based models capture the dependency between observations (X) and outputs (Y) through an energy function that takes low values when X and Y are compatible and higher values when they are incompatible, requiring inference to find a Y that minimizes this energy rather than computing Y directly from X
“energy based models which I'm not going to have time to get into too much detail but basically you capture the dependency between variables let's say observations X and uh outputs uh y through an energy function that takes low value where when X and Y are compatible and then larger values when X and why are not compatible you don't want to just compute y from X as we just saw you just want an energy function that measures the degree of incompatibility and then you know given an X find a y that has low energy for that X”
LeCun's practical recommendations for AI research include: abandon generative models, work on JEPAs and representation-space prediction instead, abandon probabilistic models (they are intractable for high-dimensional outputs), use energy-based models, abandon contrastive methods in favor of regularized methods, abandon reinforcement learning as a primary approach (use it only as a last resort when model or cost functions are inaccurate).
“so I'm having a number of uh recommendations abandon generative models the most popular method today that everybody is working on startop working on this you work on jads those are not generative models they predict in representation space probably seek models because it's intractable use energy based models uh M have had like a 20 year contentious discussion about this um abandon contractive methods in favor of those regularized methods abandon reinforcement learning but that I've been saying for a long time we know it's inefficient um you have to use reinforcement learning really as a last result when your model is inaccurate or or your cost function is inaccurate”
If you are an academic interested in human-level AI, do not work on LLMs because you will be in direct competition with hundreds of researchers backed by tens of thousands of GPUs from well-capitalized companies—there is nothing an academic can contribute to that effort
“just don't work on llm there's no point I mean in fact if you are in Academia don't work on LM because you're in competition with like hundreds of people with tens of thousands of gpus like there's nothing you can bring to the table do something else”
Contrastive learning methods do not scale well in high-dimensional output spaces because they require many negative samples to push energy up across the space, whereas regularization methods (using information content constraints) work better by using a regularizer that prevents low-energy regions from expanding
“the problem with this type of method Contra method is that they don't scale very well in high dimension if you have too many dimensions in your space of Y you're going to need to push up in lots of different places and um it it doesn't work so well you need a lot of contrastive samples for this to work”
VICReg (Variance-Invariance-Covariance Regularization) uses a loss function that forces individual encoder output variables to maintain non-zero variance, decorrelates variables with each other via covariance regularization, and makes the resulting covariance matrix approximate the identity matrix.
“you have a matrix coming out of your encoder it's got a number of samples um and each Vector is a separate variable what we're going to try to do is going to try to make each variable individually uh informative so we're going to try to prevent the the variance of the variable from going to to zero force it to be one for example and then we're going to decorrelate the variables with each other and that means Computing The coverance Matrix of this Matrix is transpose multiply by itself and then try to make the resulting coar Matrix as close to the identity uh Matrix as possible”
Information content estimation by computing the covariance matrix of encoder outputs and enforcing it to be identity (by forcing individual variable variance to 1 and decorrelating variables) is an upper bound on actual information content under strong assumptions about variable dependencies.
“the bad news with this which is that this basically approximates the information content by making very strong assumptions about the the nature of the dependencies between the variables and in fact it's an upper bound on information content and we're pushing it up crossing our fingers that the actual information contain which is below is going to follow okay so it's slightly uh uh irregular uh theoretically but but it works”
Energy-based models represent dependencies between variables (observations and outputs) through a function that takes low values when variables are compatible and high values when they are incompatible, avoiding the need to model full probability distributions.
“the appropriate um sort of semi theoretical framework to um explain this is energy based models which I'm not going to have time to get into too much detail but basically you capture the dependency between variables let's say observations X and uh outputs uh y through an energy function that takes low value where when X and Y are compatible and then larger values when X and why are not compatible you don't want to just compute y from X as we just saw you just want an energy function that measures the degree of incompatibility”
Hierarchical planning decomposes complex tasks into multiple levels of abstraction: high-level abstract actions (e.g., 'go to Paris') decompose into sub-goals (e.g., 'go to airport', 'catch plane'), which further decompose into lower-level actions (e.g., 'stand up', 'open door'), eventually reaching direct actions requiring no planning (e.g., muscle movements).
“if I want if I'm sitting at my office at NYU and I decide to go to Paris um I can decompose that task into two sub tasks go to the airport and catch a plane okay now I have a sub goal going to the airport um I'm in New York city so going to the airport consist in going down on the street and haing a taxi how do I go down in the street well I need to uh get to the elevator push the button go down go out the building how do I go to the elevator well I need to stand up for my chair pick up my bag open the door walk to the elevator avoid all the obstacles and then at some point I get to a level where I don't need to plan I can just take the actions”
Model Predictive Control (MPC) is a classical control method from ~1960s that plans through optimization using forward models; more recent versions (1970s, from France, called EDCOM) learn the models, making test-time computation via optimization well-established and not new despite current enthusiasm about 'test-time compute'
“people are excited about um um you know test time computation and blah blah blah as if it was something new this is completely classical in optimal control this is called Model preductive control it's been around with us for about the same time that I've been around all right um the first paper is on you know planning using using models of this type using optimization are from the early 60s um the the ones that actually learned the model are more recent they're more from the 70s from France actually um it's called edcom”
Joint-Embedding Predictive Architecture (JEPA) solves the video prediction problem by training encoders on both observations (X) and future observations (Y) to predict in abstract representation space rather than pixel space, enabling the system to learn representations that eliminate unpredictable details and simplify the prediction problem
“a new architecture I call John tedding predictive architecture or jepa and that's because generative architecture simply do not work for producing videos so here's what the JEA is the idea is that you run both the observation and the output which is the next observation into an encoder so that the prediction does not consist in predicting pixels but basically predicting an abstract representations of what goes on in the video”
Video prediction systems trained to minimize prediction error can learn implicit common sense about physics by detecting high prediction error anomalies—when something physically impossible happens (objects spontaneously disappearing or changing shape), the prediction error spikes, indicating the system has learned some causal understanding of the world.
“this is a paper that we just submitted um if you show that system um videos where something really strange happens that system actually is capable of telling you my prediction error is going through the roof there is something strange going on in that window so you you take a you take a video and you take the 16 video Frame Window you slide it over the video and you measure the prediction error of the system and if something really strange happen like an object spontaneously disappears or change shape um the prediction error shoots up”
Hierarchical planning—decomposing high-level goals into increasingly concrete sub-goals across multiple levels of abstraction (e.g., 'go to Paris' → 'go to airport' → 'get taxi' → 'get to street' → 'get to elevator' → 'stand up') with abstract world models at higher levels and detailed models at lower levels—is essential for human-level AI but is almost entirely unsolved in machine learning; current robots use hand-crafted representations at each hierarchical level.
“if I want if I'm sitting at my office at NYU and I decide to go to Paris um I can decompose that task into two sub tasks go to the airport and catch a plane okay now I have a sub goal going to the airport um I'm in New York city so going to the airport consist in going down on the street and haing a taxi how do I go down in the street well I need to uh get to the elevator push the button go down go out the building how do I go to the elevator well I need to stand up for my chair pick up my bag open the door walk to the elevator avoid all the obstacles and then at some point I get to a level where I don't need to plan I can just take the actions um but we do those type of this type of hierarchical planning absolutely all the time and I tell you we have no idea how to do this with learning machines almost every robot does hierarchical planning but the the representations at every level of the hierarchy are hand handcrafted”
Geopolitical rivalry and the assumption that keeping AI research secret maintains technological leadership is fundamentally mistaken; countries that conduct research in secret fall behind because the rest of the world advances openly, a pattern currently visible with open-source models overtaking proprietary models.
“the danger I see um in Europe and in other places is that geopolitical rivalry will U entice governments to basically make the release of Open Source model illegal because there are under the impression that a country will stay ahead if he keeps uh its science secret that's that would be a huge mistake when you do research in secret you fall behind that's inevitable what will happen is that the rest of the world we go up and and will overtake you that's currently what's what's happening the open source models are overtaking uh slowly but surely uh proprietary models”
Recommendation 4: Abandon reinforcement learning except as a last resort when models or cost functions are inaccurate; RL is inefficient and should not be the primary learning mechanism.
“abandon reinforcement learning but that I've been saying for a long time we know it's inefficient um you have to use reinforcement learning really as a last result when your model is inaccurate or or your cost function is inaccurate um but if you are interested in human level AI just don't work on llm”
Open-source AI foundation models are necessary because universal virtual assistants mediating all digital interactions cannot be allowed to come from only a handful of companies from the US west coast or China; such platforms must be shared, multilingual, culturally diverse, and developed collaboratively or in a distributed fashion.
“in the future we'll have Universal virtual assistants they'll be with us at all times they will mediate all our interaction with the digital world we cannot afford to have those systems come from a handful of companies from the west coast of the US or China uh which means the platforms on top of which we build those systems need to be open source and widely available they are expensive to train but once you have a foundation model fun tuning it for a particular application is relatively cheap and a lot of people afford to do this so the platforms need to be shared they need to speak all the the world languages understand all the world's cultures all the value systems all the centers of Interest no single entity in the world can train a foundational model of this type”
Joint embedding architectures trained through image corruption/transformation have existed since the 1990s through siamese networks, with more recent variations, and adding predictors to this framework is more recent work enabling better learned representations.
“and the way um we we started testing them several years ago um maybe five six years ago was um to train them to learn representations of images so you take one image you corrupt it or transform it in some ways and you run the original image and the corrupted version in identical encoders and you train a predictor to predict the representation of the original image from the corrupted one once you're done training the system you remove the predictor and you use a representation at the output of the encoder as input to a simple um like a linear classifier or something of that type that you train supervised uh so as to verify that the representations that are learned are good and this idea is very old it goes back to the 198 90s and things like uh we used to call SES networks um and some more recent work on on those joint embedding architectures”
Dogs and cats naturally perform hierarchical planning and possess causal world models, understanding how to open doors and taps through observation, demonstrating that hierarchical planning and world modeling are not unique to humans but are foundational animal cognition.
“but we're missing something really big because uh you know never mind trying to reproduce human intelligence we can even reproduce cat intelligence or rat intelligence let alone dog intelligence they can do amazing feits they understand the physical world um um you know any house cat can plan very highly complex um actions um and they have causal models of of the world some of them know how to open doors and and Taps and things of that type”
Joint Embedding Predictive Architecture (JEPA) solves video prediction by running both observation X and future observation Y through encoders (which may be identical) and predicting the representation of Y from representation of X in abstract space, rather than predicting pixels directly, causing the encoder to learn representations that eliminate unpredictable details
“so the solution to that problem is an AR a new architecture I call John tedding predictive architecture or jepa and that's because generative architecture simply do not work for producing videos you may have seen video generation systems that produce pretty amazing stuff there's a lot of hacks that go be Beyond them uh behind them and they don't really understand physics um they don't need to they just need to to predict pretty pictures they don't need to actually have kind of accurate model of the world okay”
Joint-Embedding Predictive Architecture (JEPA) solves the indeterminacy problem of video prediction by encoding both current and future observations into abstract representation spaces (using potentially identical encoders) and predicting abstract representations rather than pixels, which eliminates unpredictable details and simplifies the prediction problem.
“so the solution to that problem is an AR a new architecture I call John tedding predictive architecture or jepa and that's because generative architecture simply do not work for producing videos”
A typical modern LLM is trained on approximately 30 trillion tokens (3×10^13), representing roughly 10^14 bytes of data—equivalent to almost half a million years of reading. By contrast, a four-year-old child has been awake for only 16,000 hours and experiences approximately 10^14 bytes of visual data through ~2 million optical nerve fibers each carrying ~1 byte per second, demonstrating that humans achieve similar information volumes through multimodal perception while LLMs rely exclusively on text
“A typical llm nowadays is trained on on the order of 30 trillion tokens three 10 to the 13 uh tokens that's two to the 13 words roughly each token is about three bytes um so the data volume is roughly 10 to the 14 bytes uh it would take any of us uh almost half a million years to read through all that material”
Guardrail objectives are explicitly implemented task constraints that the system must satisfy during inference optimization for safety purposes, and cannot be jailbroken through prompts because they are hardwired into the inference procedure rather than learned through fine-tuning like LLM constraints
“we also can set some uh guardrail objectives think of them as constraints that need to be satisfied for the system to behave in a safe manner right so those guardes will be explicitly implemented and the way the system proceeds is by optimization it's looking for an action sequence that minimizes the task objective and the uh guard rail objectives at runtime”
Predicting video frames through generative models (pixel-level prediction) fails because: (1) there are multiple valid future frames and probabilistic distributions over video frames are mathematically intractable, (2) generative video systems use hacks beyond basic architecture and don't truly understand physics, (3) they only need to predict plausible pictures, not build accurate world models
“Training it to make that prediction might actually cause the system to understand the annoing structure of the world it works for text because predicting words is relatively simple why is predicting words simple because words um there's only a finite number of possible words certainly a finite number of possible tokens and so we can't predict exactly which word will follow another word or what what word is missing in the text but we can produce a probability distribution or score for every possible word in the dictionary we cannot do this for images for video frames we do not have good ways of representing distributions of our video frames um every attempt to do this uh basically bumps into mathematical intract abilities”
Video prediction using generative models fails to produce accurate physics understanding because video generation systems can produce visually plausible but physically incorrect frames through hacks, without requiring genuine causal models of the world.
“generative architecture simply do not work for producing videos you may have seen video generation systems that produce pretty amazing stuff there's a lot of hacks that go be Beyond them uh behind them and they don't really understand physics um they don't need to they just need to to predict pretty pictures they don't need to actually have kind of accurate model of the world”
Current AI systems need to be controllable and safe by design (through hardwired guardrail constraints), not by fine-tuning after training, because systems using design-level safety guarantees cannot be jailbroken by adversarial prompts since the guardrails are intrinsic to the optimization problem, not learned behaviors.
“we also can set some uh guardrail objectives think of them as constraints that need to be satisfied for the system to behave in a safe manner right so those guardes will be explicitly implemented and the way the system proceeds is by optimization it's looking for an action sequence that minimizes the task objective and the uh guard rail objectives at runtime okay we're not talking about learning here we're just talking about inference um and that will guarantee the safety of the system because uh the guard rails guarantee safety and there is no way you can Jailbreak that system by giving it a prompt that will you know have it ES Escape its guardwire objectives”
Recommendation 1: Abandon generative models (the most popular method today) and work on JEPA variants instead, which predict in representation space rather than generating pixels.
“I'm having a number of uh recommendations abandon generative models the most popular method today that everybody is working on startop working on this you work on jads those are not generative models they predict in representation space”
A typical modern LLM is trained on approximately 30 trillion tokens (about 10^14 bytes), which would take a human nearly half a million years to read, yet a 4-year-old child has received approximately 10^14 bytes of visual data through the optical nerve (2 million fibers at 1 byte per second over 16,000 waking hours), meaning we cannot reach human-level intelligence by scaling text-only systems.
“A typical llm nowadays is trained on on the order of 30 trillion tokens three 10 to the 13 uh tokens that's two to the 13 words roughly each token is about three bytes um so the data volume is roughly 10 to the 14 bytes uh it would take any of us uh almost half a million years to read through all that material... a four-year-old has been awake a total of 16,000 hours which by the way is only 30 minutes of YouTube uploads um we have 2 million optical nerve fibers Each of which carries about 1 B per second maybe a bit less but it doesn't matter so the data volume is about 10 to the 14 in four years”
The correct inference model for AI systems should not be autoregressive token generation but rather optimization over an energy function that measures compatibility between observations and proposed outputs, allowing systems to spend more computation on harder problems (system two thinking) rather than spending identical computation on simple yes-or-no questions and complex questions.
“the way um a lot of inference in statistics for example that's going to make Mike happy actually um the way inference works is is not that way in uh In classical AI in statistics uh in structure prediction a lot of different domains the way it works is that you have a function that measures the degree of compatibility or incompatibility between your observation and a proposed output and then the inference process consist in finding the value of an output that minimizes this incompatibility measure okay let's call it an energy function”
Training joint embedding architectures requires ensuring that the energy function (divergence measure between predicted and target representations) is low on the training data manifold but high outside it, which can be achieved through two classes of methods: contrastive methods (explicitly pushing negative samples up) and regularized methods (constraining the manifold volume using regularizers like correlation or covariance penalties).
“we need to train those systems and we need to figure out how to train those jepa architectures and tells that to not be completely trivial because you need to train the the cost function in this JEA architecture that measures the the Divergence between the representation of Y and the predicted representation of Y we need this to be low on the training data but we need also needed to be large outside the training set okay so this is you know this kind of energy function here that has kind of uh Contours of equal equal energy we need to make sure the energy is high outside of the manifold of data and I only know two classes of methods for this one set of method is called contrastive it consists in um having uh data points which are those those blue dark blue dots pushing the down the energy of those and then generating you know those flashing green dots and then pushing the energy up”
A world model is a learned function that takes a current estimate of world state (in abstract representation space) and an imagined action sequence, then predicts the resulting state after those actions are taken, enabling mental simulation for planning
“what is a world model World model is given given a current estimate of the state of the world which is in an abstract representation space and given an action sequence that you imagine taking uh your world model predicts the the resulting state of the world that will um occur after you take that sequence of actions okay that's what a world model is if I tell you imagine a cube floating in the air in front of you okay now rotate this Cube by 90 degrees around a vertical axis um what does it look like”
Foundation models must speak all world languages and understand all world cultures, value systems, and centers of interest; no single entity globally can build such models, requiring collaborative or distributed development across countries and institutions
“they need to speak all the the world languages understand all the world's cultures all the value systems all the centers of Interest no single entity in the world can train a foundational model of this type this probably will have to be done in a collaborative fashion or distributed fashion again some work for Applied mathematicians who are interested in distributed algorithms for large scale optimization”
Major open problems requiring research include: training world models with large-scale data, improving planning algorithms (which are inefficient), learning cost modules/objectives because most cannot be hand-designed (important for applied math and optimization), hierarchical planning (completely unsolved), planning under uncertainty, and exploration strategies.
“there's a number of problems to solve U training those things with you know large scale data blah blah blah planning algorithms are kind of inefficient we have to come up with better methods so if you are like into optimization applied math it's great um J with latent variables planning under uncertainty hierarchical planning which is completely unsolved um learning cost module because probably most of them you can't build by hand you need to learn them and then there is issues exploration Etc”
Foundation models must understand all world languages and cultures, all value systems, and all centers of interest, and no single entity can train such a model alone, requiring collaborative or distributed training efforts
“they need to speak all the the world languages understand all the world's cultures all the value systems all the centers of Interest no single entity in the world can train a foundational model of this type this probably will have to be done in a collaborative fashion or distributed fashion”
DINO (self-distillation with no labels) and its variants learn generic image features that transfer well to downstream tasks by using the distillation approach without negative samples, enabling efficient self-supervised learning
“there's a particular way of implementing this AIO distillation called IA there's another one called called Dino or Dino uh which I I skipped a little bit um and um so Dino um is V2 people are working on on V3 this is a method produced by some some of my colleagues at at Fair Paris um team led by Max Maximo cab um and then a slight different version um called IA V JEA by also Fair people in in Montreal and Paris mostly so no need for negative samples there”
Unsolved problems requiring research effort: (1) Training large-scale JEPA systems, (2) Developing efficient planning algorithms beyond current methods, (3) Hierarchical planning and representation learning at multiple abstraction levels, (4) Learning cost functions/objectives (most cannot be handcrafted), (5) Exploration strategies
“there's a number of problems to solve U training those things with you know large scale data blah blah blah planning algorithms are kind of inefficient we have to come up with better methods so if you are like into optimization applied math it's great um J with latent variables planning under uncertainty hierarchical planning which is completely unsolved um learning cost module because probably most of them you can't build by hand you need to learn them and then there is issues exploration Etc”
Once foundation models are trained, fine-tuning them for particular applications is relatively cheap and affordable by many entities, so shared open-source platforms enable democratic participation in AI development and deployment
“they are expensive to train but once you have a foundation model fun tuning it for a particular application is relatively cheap and a lot of people afford to do this so the platforms need to be shared”
The proposed AMI architecture integrates perception (observation encoding), memory (past state estimates), world models (action-conditioned state prediction), objectives (task cost functions), guardrail constraints (safety specifications), and optimization-based inference to ensure controllability and safety by design.
“feed okay hang on make a more radical Choice here okay this doesn't look like it was a good idea oh I know what I [19:39] that's nice okay I think we're going to have human level intelligence before we have audio video that actually works okay um so so if we have this world model which is able to predict the result of a sequence of actions um we can feed it to an objective which is a task objective that measure to what extent the predicted final State U satisfies a goal that we set for ourselves it's just a cost function um and we also can set some uh guardrail objectives think of them as constraints that need to be satisfied for the system to behave in a safe manner right so those guardes will be explicitly implemented”
Smart glasses and wearable AI assistants will eventually have displays and will allow users to interact through voice, electromyography (muscle signals), and other interfaces, requiring human-level intelligence because users are most familiar with human intelligence and systems need to be easy for non-technical populations.
“we are going to be uh wearing smart devices like smart glasses and things of that type in the future and in in those smart U devices we'll be able to um access AI assistants that will be with us at all times and we'll be interacting with them either through voice or through U uh electron um electrogram CMG um the glasses will eventually have displays although currently they don't and um and we need those system to have human level intelligence because that's what we're the most familiar um interacting with we're familiar with interacting with other humans uh we are familiar with the level of intelligence that we expect in a in a human and uh it would be more you know easier to interact with systems that have kind of similar forms of intelligence um so you know those ubiquitous assistants um are going to mediate all of our interactions with the digital world”
Researchers interested in human-level AI should not work on large language models, particularly in academia, because they are in competition with well-resourced industry labs with tens of thousands of GPUs, and individual researchers have nothing of value to contribute to that specific area.
“if you are interested in human level AI just don't work on llm there's no point I mean in fact if you are in Academia don't work on LM because you're in competition with like hundreds of people with tens of thousands of gpus like there's nothing you can bring to the table do something else”
Once a foundation model is trained (expensive), fine-tuning it for particular applications is relatively cheap and accessible to many people, making shared foundation models more democratizing than training separate models from scratch.
“they are expensive to train but once you have a foundation model fun tuning it for a particular application is relatively cheap and a lot of people afford to do this so the platforms need to be shared”
Training data requirements and computational complexity create major open problems in machine learning: training JEPAs at large scale, designing more efficient planning algorithms, learning cost functions rather than hand-crafting them, and exploration strategies.
“there's a number of problems to solve U training those things with you know large scale data blah blah blah planning algorithms are kind of inefficient we have to come up with better methods so if you are like into optimization applied math it's great um J with latent variables planning under uncertainty hierarchical planning which is completely unsolved um learning cost module because probably most of them you can't build by hand you need to learn them and then there is issues exploration Etc”
Experiments demonstrate that DINO world models can plan object manipulation by optimizing action sequences: pushing T-shaped objects into target positions and manipulating scattered blue chips, showing learned models capture interaction dynamics without explicit physical simulation
“let me play that video again okay so at the top you see what actually happens in the real world when you take a sequence of actions that is planned and what you see at the bottom is the internal mental prediction of what the system of the sequence of actions the system was planning”
Energy-based inference aligns with psychological System 2 thinking—deliberate, controlled reasoning where one thinks through actions before taking them—whereas LLMs correspond to System 1—fast, automatic, subconscious processing.
“this type of inference would be more akin to what psychologists call system two in uh sort of human U mind if you want system two is when you think about what action or sequence of actions you're going to take before you you you take them you think about something before doing it and the system one is when you can do the thing without thinking about it you know it becomes sort of subconscious so llms are system one what I'm proposing is system two”
Distillation-based methods prevent encoder collapse by using two encoders where one receives exponential moving average (EMA) updates from the other, forcing slower weight changes that empirically prevent collapse, though the mechanism remains somewhat mysterious and requires empirical care to implement
“there is another set of method that to some extent has been slightly more successful over the last couple years and those are based on distillation so again you have two encoders it's still a joint Ting productive architecture you have two encoders they kind of share the same weights but not really so the encoder on the right uh gets a version of the weights of the enod on the left that are obtained through a um exponential moving average okay a moving average so basically you force the encoder on the right to uh change its weights more slowly than the one on the left and for some reason that prevents collapse”
Joint embedding architectures with predictors are a recent development; this general approach goes back to the 1990s as 'siamese networks' and earlier work on joint embedding approaches, but adding the predictor (to prevent collapse without contrastive samples) is more recent and represents an evolution that enables better scaling to high dimensions.
“and this idea is very old it goes back to the 198 90s and things like uh we used to call SES networks um and some more recent work on on those joint embedding architectures and then adding the predictor is more is more recent um so s clear which is from from Google is a contrastive method derived from s Nets”
LeCun published 'A Path Towards Autonomous Machine Intelligence' approximately 3 years before the GPT craze, and the paper (now titled 'Advanced Machine Intelligence') argues where AI research should focus; the emergence of GPT models has not changed these research directions because the fundamental limitations of LLMs were already understood
“so with all those Reflections about 3 years ago I wrote a long paper where I kind of explained sort of where where I think AI research should be focusing on so this so before the whole GP CH GPT craze um I haven't changed my mind about this CH GPT hasn't Chang anything we wereing Els before that so we knew what was coming anyway um this is the paper um a path towards autonomous machine intelligence”
A robot trained to push a T-shaped object to target positions using world model planning learns Dynamics (how objects interact) purely from state-action-next-state observations and can accomplish multi-step pushing tasks with learned trajectory planning, demonstrating that the approach works for contact-rich manipulation.
“okay so at the top you see what actually happens in the real world when you take a sequence of actions that is planned and what you see at the bottom is the internal mental prediction of what the system of the sequence of actions the system was planning and this is run to a decoder that produces a pictorial representation of the internal state but that is trained separately there's no image generation um let me skip to the more interesting one so here is one where you have an initial state which is a bunch of Blue Chips randomly thrown on the floor and the target state is at the top and what you see here are the actions that are resulted from planning and the robot like accomplishing those actions the Dynamics of this environment is actually fairly complicated because those blue Chiefs kind of interact with each other and and everything um the system has just learned this through you know observing a bunch of uh uh State action next state”
The lossless encoder-based training approach (where one image is corrupted/transformed and both original and corrupted versions are encoded to identical encoders, then a predictor learns to predict the original from corrupted) dates to the 1990s with siamese networks and has been rediscovered and enhanced with predictor modules.
“this idea is very old it goes back to the 198 90s and things like uh we used to call SES networks um and some more recent work on on those joint embedding architectures and then adding the predictor is more is more recent um so s clear which is from from Google is a contrastive method”
Causal architectures in autoregressive generation prevent cheating (using future tokens to predict past ones) by restricting the transformer to only look at previous tokens in the sequence.
“you put the sequence at the at the input and...the way you train the system to just reproduce its input on its output and because it has a causal structure um it cannot cheat and use a particular input to predict itself it has to only look at the symbols that are to the left of it that's called causal architecture”
Classical AI was primarily about reasoning and search, and most computational problems can be reduced to optimization or search problems.
“classical AI is all about reasoning and uh search and therefore optimization pretty much any computational problem can be reduce an optimization problem essentially or search problem”
Practitioners in deep learning often build systems that work without fully understanding why they work—this is characteristic of good engineering—but there is theoretical discomfort with such disconnect between practice and theory, illustrating a fundamental gap in the field.
“there's some theoretical work on this um in fact uh this one that jum just finished writing um but it's a little bit mysterious why this works and frankly I'm a little uncomfortable with this method but we have to um accept the fact that actually works um if you if you're careful um you know real Engineers buildings without necessarily knowing why they work that's good engineers and then the usual joke in France that everybody here should should learn is that students that come out of e poly technique when they build something it doesn't work but they can tell you why”
Alternative regularized methods include MCR-squared (Maximal Coding Rate Reduction) by Yuanzhi Ma and colleagues, and MMCR (Modular MCR) from NYU neuroscience collaborators, representing a broader family of scalable regularization approaches.
“there been kind of similar methods proposed by uh yima and his team called MCR squar and then another method by uh some colleagues from NYU called mmcr from neuroscience”
VICReg uses variance-invariance-covariance regularization with specific loss functions for the covariance matrix; MCR-squared (by Yima and team) and MMC-Red (by NYU colleagues) represent similar methods in this class of regularization approaches.
“we have um a method of this type called viag that means variance in variance Co variance regularization and it's got particular loss functions for this ciance Matrix um there been kind of similar methods proposed by uh yima and his team called MCR squar and then another method by uh some colleagues from NYU called mmcr from neuroscience”
SimCLR is a contrastive method derived from SimNet networks; distillation-based methods like BYOL, DINO, and IJEPA have been more successful in recent years than contrastive approaches.
“s clear which is from from Google is a contrastive method derived from s Nets um but again the dimension is is restricted so the regularized method uh worked...there is another set of method that to some extent has been slightly more successful over the last couple years and those are based on distillation”
Meta prefers the term 'Advanced Machine Intelligence' (AMI) over 'Artificial General Intelligence' (AGI) because human intelligence is actually quite specialized, making 'AGI' a misnomer; AMI is pronounced like the French word for 'friend'.
“we call this Advanced machine intelligence at meta we don't like the term AGI artificial general intelligence the reason being that that human intelligence is actually quite specialized and so calling it AGI is kind of a misnomer um so we call this Ami we actually pronounce it Ami which means friend in French”
LeCun wrote a detailed paper 3 years ago (before the GPT era) titled 'A Path Towards Autonomous Machine Intelligence' outlining where AI research should focus, and he has not changed his mind despite GPT's emergence because the fundamental ideas were understood before the GPT hype and the hype has not altered his technical priorities.
“so with all those Reflections about 3 years ago I wrote a long paper where I kind of explained sort of where where I think AI research should be focusing on so this so before the whole GP CH GPT craze um I haven't changed my mind about this CH GPT hasn't Chang anything we wereing Els before that so we knew what was coming anyway um this is the paper um a path towards autonomous machine intelligence that we now call Advanced machine intelligence because autonomous just scares people um and it's on open review it's not on archive and there's various versions of this talk that I've I've given various ways”
Current AI terminology uses 'Artificial General Intelligence' (AGI) misleadingly because human intelligence is actually quite specialized rather than general; Meta uses the term 'Advanced Machine Intelligence' (AMI, pronounced as 'ami' meaning friend in French) as a more accurate descriptor of the goal
“so we call this Advanced machine intelligence at meta we don't like the term AGI artificial general intelligence the reason being that that human intelligence is actually quite specialized and so calling it AGI is kind of a misnomer um so we call this Ami we actually pronounce it Ami which means friend in French”
Meta's terminology is 'Advanced Machine Intelligence' (AMI, pronounced 'ami' meaning 'friend' in French) rather than 'Artificial General Intelligence' (AGI) because human intelligence is highly specialized, making AGI a misnomer
“so we call this Advanced machine intelligence at meta we don't like the term AGI artificial general intelligence the reason being that that human intelligence is actually quite specialized and so calling it AGI is kind of a misnomer um so we call this Ami we actually pronounce it Ami which means friend in French”
LeCun wrote 'A Path Towards Autonomous Machine Intelligence' (now called Advanced Machine Intelligence) three years prior to the GPT revolution, and the paper has not changed his thinking because the broad research direction was already clear before GPT became mainstream.
“with all those Reflections about 3 years ago I wrote a long paper where I kind of explained sort of where where I think AI research should be focusing on so this so before the whole GP CH GPT craze um I haven't changed my mind about this CH GPT hasn't Chang anything we wereing Els before that so we knew what was coming anyway”
LeCun expects to see more regularized self-supervised learning methods in the future, as they work well and avoid the scalability issues of contrastive methods.
“I really like those methods and I I think and they work really well I expect to see more of them in the future”