What this covers
Yann LeCun, Executive Chairman, Advanced Machine Intelligence, New York University.
Source description (no synthesized summary yet).
LeCun argues that the next revolution in AI requires abandoning generative models and large language models in favor of joint embedding predictive architectures (JEPA) that learn abstract world models to enable planning and reasoning about physical environments.
- LLMs are fundamentally limited to discrete symbol sequences and fail on high-dimensional continuous data like video, images, and physical sensory inputs
- World models must predict in abstract representation space rather than directly reconstructing sensory details, enabling efficient planning through model predictive control
- Self-supervised learning with information-maximization methods prevents collapse and trains systems to extract only predictable information, matching how humans and animals learn from observation
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
A 4-year-old child is trained on approximately 16,000 waking hours of sensory data (vision, touch, hearing), which through the optic nerve (2 million fibers, each carrying ~1 byte/second) results in about 10^14 bytes of raw data—the same volume as the largest LLMs trained on all publicly available text on the internet.
“a human child, a 4-year-old, is trained from sensory data, you know, vision, touch, audition, uh in 16,000 wake hours uh during the first 4 years... the small amount of of of video that corresponds to about 30 30 minutes of YouTube uploads. Um this child gets a huge amount of information through the visual cortex, through uh the optic nerve. Optic nerve has about 2 million optical nerve fibers, each carrying about 1 byte per second. And so, do the arithmetics and the volume is about 10 to the 14 bytes. So, a 4-year-old has seen the same amount of data of raw data as uh you know, the biggest LLM trained on the entire corpus of publicly available text on the internet.”
The Moravec paradox, articulated by roboticist Moravec in 1988, describes the counter-intuitive observation that complex intellectual tasks like proving theorems and playing chess are easy for computers, while simple sensorimotor tasks like manipulating objects remain fundamentally difficult despite decades of robotics research.
“So we keep bumping into this paradox which um actually is very well known in in AI called the Moravec paradox. Moravec is uh is a roboticist in 1988. He said he says that you know, kind of paradoxical that a lot of you know, complex intellectual tasks like you know, proving theorems, uh, you know, playing chess and things like that. Seem easy for computers. But then the things that we take for granted like, you know, just manipulating an object, uh, we're still completely unable to reproduce this with uh, with computers. With robots.”
Babies and animals learn primarily through observation and self-supervised learning, acquiring basic concepts like object permanence at 9 months, intuitive physics (gravity, inertia) by age 10 months, and the difference between animate and inanimate objects without explicit instruction.
“babies learn basic concepts about the world, things like the difference between animate and inanimate objects, um, the fact that the table and the chair are different even though, you know, babies don't have language, they they learn those those categories spontaneously. Um, concepts, very simple concepts like uh, object permanence, the fact that when an object is hidden behind another one, it still exists. Um, and then around the age of 9 months, uh, uh, infants can uh, learn basic notions of intuitive physics like gravity, inertia, and things of that type.”
Hierarchical planning in world models is necessary because humans cannot plan complex actions like traveling from New York to Paris at the level of millisecond-by-millisecond muscle control—instead, they operate at multiple levels of abstraction, from high-level goals (catch a plane) to progressively lower-level sub-goals (reach the street, hail a taxi).
“There is no way I can plan my entire trip from New York to to Paris in terms of millisecond by millisecond muscle control. It's too complicated first of all. I also don't have the information. I don't have position of every obstacle on my way, right?”
Generative models that attempt to predict every pixel or detail of future video or high-dimensional continuous data produce blurry, averaged predictions and fail because most details in sensory data are fundamentally unpredictable.
“If you mask a piece of the video and you ask a system to predict the missing piece of the video, it doesn't really work. It's uh at least with a generative model, because most of the details in a video are completely unpredictable... you train the system to make one prediction and and it can't really figure out among all the possible futures which one will actually occur, so it predicts the average.”
Effective altruism (EA) in AI safety is a 'cult' that originated in Silicon Valley promoting the view that potentially dangerous technologies should be stopped, and practitioners like Dario Amodei use safety claims as a marketing strategy to suppress open-source AI competition.
“it's basically a cult that started in Silicon Valley. Uh it's called effective altruism of people who are claiming to to kind of try to make long-term prediction about the future of humanity and and basically they focus on the idea that if a technology is potentially dangerous, um you know, or powerful, necessarily dangerous, then we should stop working on it. I mean, it's just completely insane. And and a lot of people who are working on AI today come from that community. People like Dario Amodei come from that community... Dario Amodei makes claims, you know, very recently in the last two weeks that, you know, Claude is so powerful that we don't want to put it in the hands of everyone because, you know, it's too dangerous potentially, you know, it can create cyber attacks. It turns out after investigation, it's complete BS. It's just completely false.”
Open-source AI is essential for sovereign AI development, particularly for regions like Tunisia, North Africa, Europe, and parts of Asia that would otherwise be dependent on closed-source proprietary systems controlled by US companies.
“There's another point which I think is super important for a lot of regions in the world including you know, Tunisia, North Africa more more generally, but Europe as well and and very you know, a lot of places in Asia. Which is that we need open source AI. The only way to have any level of sovereignty in AI is is is to have access to open source, you know, powerful open source, AI systems. Uh and so, you know, if you are a proprietary AI company like like Anthropic or OpenAI, uh you want to stop that. Um you have a commercial interest in stopping it.”
Current AI systems are fundamentally limited compared to animals and humans because they cannot perform basic real-world tasks like clearing a dinner table, cleaning a house, or learning to drive in a few hours despite being trained on millions of hours of human driving data.
“AI basically can deal with data at the moment that is represented as a sequence of discrete symbols. At least the type of AI we're talking about, generative AI. But it really doesn't work at all when the data is high-dimensional, continuous, and potentially noisy.”
There are two fundamentally different modes of reasoning: System 1 (immediate action prediction based on current state, like driving while experienced) and System 2 (planning using world models to anticipate consequences and find action sequences, requiring full cognitive resources).
“Um so, the first type of um AI system or even uh uh human behavior is what psychologists call system one. Okay, given the current idea of the state of the world, can I predict an action that I should take uh right now, okay? In the context of uh reinforcement learning, that's called a policy, right? Given the state, what action should I take? Um but uh humans don't do this uh uh exclusively. That's what psychology calls system one. That means if you are accomplishing a task you are very very familiar with, you don't need to think about it. You can do it, you know, subconsciously. When you drive and you're an experienced driver, you don't need to think about it. You can talk to someone at the same time and things like that.”
Joint embedding architectures that do not attempt to reconstruct perform much better than reconstruction-based methods (autoencoders, VAEs, masked autoencoders, diffusion models) for learning image representations that transfer well to downstream tasks.
“And what we figured out is that um the the architectures that perform the best in this context are architectures that are joint embedding that do not attempt to reconstruct. So, people have tried uh uh reconstruction-based uh methods like this like autoencoders, variational autoencoders, masked autoencoders, which is a special form of denoising autoencoder, uh and and diffusion models, the representations you get out of those things are not very good. The the ones that are really good are the ones that are used joint embedding and do not attempt to reconstruct.”
Model Predictive Control and world models are not new concepts but rather established techniques from the control theory community dating back to the 1970s and earlier, so the innovation is applying learned models to replace hand-written differential equations.
“this idea of world model is very old. It goes back to the late 50s, early 60s and the the whole idea of model predictive control also goes back a long time time 1970s. Uh this is not a new concept. This is not invented by me or by Jürgen Schmidhuber or by the machine learning community. It really was invented by the optimal control community... it's about as old as I am is actually.”
Reinforcement learning should be minimized or abandoned in favor of world models because it is extremely sample-inefficient and impractical for achieving human or animal-level intelligence.
“I've also been vocal about saying for the last 12 years abandon reinforcement learning or at least minimize its use because it's so inefficient in terms of samples. And so if you're interested in really pushing AI towards human or animal level, you should really not work on LLMs. You should you know, kind of try you know, try to work on world model.”
Abstract representations are fundamental not just for building intelligent AI systems but for all of science: in physics, chemistry, and biology, scientists always identify abstractions (particles, atoms, molecules, organisms) that allow tractable modeling and prediction without simulating underlying details like quantum mechanics.
“This notion that we need to find abstract representations of the world is very natural, not just you know to build intelligent systems, but but also in science when we when we model the world when we you know as applied mathematicians, of course you're all familiar with the concept of you know writing down a model of reality, right? Let's say you want to simulate the the flow of air a fluid around an airplane or or something like that, right? You you model the state of the of the air in in the little cube by velocity, pressure, density, viscosity perhaps, you know, temperature. And then you solve the Navier-Stokes equations, right? The the the partial differential equations. This is already an abstraction.”
AI development should not be feared due to inherent dangers from AI itself, but society should exercise normal caution as with any powerful technology, regulating for safety and beneficial impact without being captured by asymmetric risk narratives.
“We should be careful about AI uh because it's you know, powerful technology with a big impact. Uh we have to make sure the impact is uh beneficial to to all of us. Uh but like society is used to doing this. I mean, we do this with every new technology that comes out. We we have uh you know, particularly if it's consumer-facing, we have regulation to make sure it's safe... but we should not be afraid of it for the reasons that a lot of people are claiming we should be afraid of it. So, uh the idea somehow that AI is qualitatively different from other technology and and is intrinsically dangerous at the research level, that's just completely false.”
Anthropic and other proprietary AI companies are engaging in a strategy to slam the door behind them: they lobby governments to regulate AI research while claiming their own systems are safe, and they oppose open-source AI development, all to protect their commercial advantage based on a superiority complex.
“But their strategy is basically to slam the door behind them. They think they're ahead of everyone. They have a huge superiority complex. And basically, they're they're they're telling governments regulate everybody else. Don't regulate us because we're smart. We know what we're doing. But like the plebe, right? The unwashed masses you know, don't know what to do with AI. So, you should not make it available to them. I mean, I think it's incredibly you know, that is hypocritical. You know, like you know, I think I think it's insane.”
LLMs fail on basic physical reasoning (e.g., whether to walk to a car wash 100m away) and exhibit no real understanding of physical intuition, instead performing retrieval-based responses fine-tuned on human feedback.
“There is a a story that has circulated recently uh, on on social networks where where someone said, um, Okay, I need to wash my car. Um, I need to get my car washed. And the car wash place is 100 100 m from my house. Should I walk? And most most LLMs, you know, ChatGPT, the latest one, and Claude, you know, all of those, uh, Meta AI, they say, No, you should I mean, yes, you should walk because it's only 100 m away. Like not realizing you're not going to have your car with you, okay?”
Large language models appear intelligent in only narrow domains (mathematics, coding, law) where reasoning is substrate-based (symbol manipulation itself aids thought) and where answers can be self-corrected (code can be tested, proofs verified).
“There is only two or three domains where LLMs exhibit sort of baffling intelligence if you want. And those are uh uh computer code generation uh mathematics and perhaps law... the reasoning can be uh basically supported by the language itself... the reason why they seem to be intelligent by answering all the questions we ask them is a bit of an illusion. Uh, they basically do retrieval. Okay. So, those systems are not particularly intelligent, but they are extremely good at accumulating an enormous amount of knowledge and then regurgitating that knowledge at the right time and maybe adapting it in in minor ways.”
LeCun left Meta at end of 2023 and founded AMI Labs (Advanced Machine Intelligence) to develop world models because most of Silicon Valley remains focused on LLMs while he believes world models represent the next revolution in AI.
“I left uh Meta uh in early January and and um I started a new company called Advanced Machine Intelligence or AMI Labs... a lot of what we're doing at AMI Labs... since most of Silicon Valley is really focused on LLM, I decided to leave Meta at the end of last year and create a company called Emina Labs Advanced Machine Intelligence that really is focusing on developing this this thing.”
Chomsky's theory of universal grammar and hardwired language modules has been disproven by the success of data-driven neural networks like LLMs, which can achieve good language understanding without explicit linguistic structures.
“if LLMs have proved something, it's that you can do extremely good language understanding and manipulation without appealing to any concept of traditional linguistics or any concept that, you know, Chomsky has argued for for decades. Uh so, that battle is over and the the Chomskians have lost.”
Predictive coding—the theory that the brain predicts sensory input and adjusts predictions based on prediction errors—is a plausible computational model for how biological systems learn, and self-supervised learning in AI mirrors this fundamental learning mechanism.
“there's a a theory called predictive coding where where, you know, the brain predicts something and then confronts the prediction with reality and sort of adjusts itself to make a better prediction. That's a very abstract concept, but the question is how you reduce this to practice.”
Double descent is an empirical phenomenon where test error first increases then decreases as model size increases beyond the interpolation threshold, contradicting classical statistical intuition that overfitting increases with model capacity.
“every textbook in statistics tell you, if you try to fit a function to a bunch of points, you don't want too many parameters because your system is going to overfit... that argument turns out to be completely false empirically initially, okay? And now we need to understand this theoretically. So, neural nets are way over parameterized... the number of parameters in neural nets is enormous compared to the amount of training samples. Yet, those systems generalize pretty well... So, neural nets are way over parameterized... the phenomenon, an empirical phenomenon called double descent, where uh the the error on the separate test set of a model is going to go down as you increase the number of training samples... then... as you increase the power of the model... it's going to increase, okay? And this is where overfitting occurs. But then, as you increase the the power of the model, the number of parameters... it actually decreases again.”
JEPA (joint embedding predictive architecture) is an architecture that predicts abstract representations of future states rather than reconstructing raw sensory details, eliminating the need to predict unpredictable information by focusing only on predictable, relevant information.
“What I'm proposing, the Jepa architecture, basically says, you simply cannot predict all the details of Y. So, let's learn an abstract representation of Y and make predictions in that abstract representation space. And that abstract representation space will eliminate all the details about the input video that I simply cannot predict. That is simply unpredictable.”
LLMs perform autoregressive token prediction by generating probability distributions over a finite token dictionary and sampling from them sequentially, which is fundamentally different from the optimization-based inference required for planning with world models.
“So you have a sequence of discrete tokens, and you train a system to predict the next token in a sequence from enormous amounts of data. And then you can use that system to do auto to do auto regressive prediction, right? You You get That's That's the way that LLM works. And And also VLA, which are form forms of LLMs that are used to produce actions for agentic systems. So, you take a sequence of actions, you run it through your LLM, your predictor, you run through a fixed number of layers of a neural net transformer, and then you predict a distribution over all possible tokens in your dictionary. You only have a finite number, maybe 100,000 or so. So you you you produce a score for each of them. And that gives you a distribution, and then you sample from that distribution, and you get a discrete predictive token, XT + 1.”
Claims by some AI company leaders (e.g., Dario Amodei) that current LLMs like Claude pose severe cybersecurity risks have been debunked by the computer security community—the risks are not new and were deliberately manufactured through careful prompt engineering to attack known vulnerabilities.
“Uh Dario Amodei makes claims, you know, very recently in the last two weeks that, you know, Claude is so powerful that we don't want to put it in the hands of everyone because, you know, it's too dangerous potentially, you know, it can create cyber attacks. It turns out after investigation, it's complete BS. It's just completely false. I mean, there's nothing new to the cyber you know, security capabilities of Claude they couldn't do with existing tools. It's just that, you know, they carefully it's a strategy it's a marketing strategy basically. They carefully kind of propped the system into attacking you know, known software in ways that they know was vulnerable. And and, you know, it was completely debunked by the by the the computer security community basically.”
V-JEPA can acquire a level of physical common sense by learning to predict at the representation level, as demonstrated by the system being 'surprised' (high prediction error) when shown impossible physical scenarios like objects floating or changing shape.
“we tried to uh figure out to what extent the system has learned what is possible and what is not possible in the real world. So, you show it a video where something impossible occurs like, you know, a ball is thrown in the air and follows the usual trajectory, but then at some point it disappears. Or it turns into a cube. Or, you know, something else that's impossible occurs. And inside internally to the system, you can measure the prediction error. When such a impossible event occurs, the prediction error shoots through the roof. And so, the system basically knows, you know, is surprised by things that are impossible. And it's the first time that we observe this kind of phenomenon in a in a in a system that has spontaneously through just observation being trained to predict at the representation level basically has acquired some level of physical common sense.”
Quantum computing is unlikely to have significant impact on AI within the next two decades; the only medium-term application is simulation of quantum systems, and scaling, error correction, and efficiency remain highly uncertain.
“I'm extremely skeptical about quantum computing. I think the only application of quantum computing in the medium term, I don't even talk about short term, medium term, is the simulation of quantum systems... I think it's essentially useless for anything else. There is no useful algorithm... other than factorization... The technology is finished finished complicated. It's not clear it's going to scale up. It's not clear that error correction is going to work... in terms of an industry, I'm very skeptical and I certainly do not believe there's going to be any significant impact of quantum computing on AI within the next, let's say, two decades.”
DINO (Self-Supervised Vision Transformers for Image Representation) is the state-of-the-art method for extracting generic features from images without labels, now at version DINO V3, and represents the best system globally for image representation learning.
“Another uh such distillation method is called DINO. Uh this is uh a work by uh our colleagues at former colleagues at Meta in Paris. Uh uh the latest one being DINO V3 and it's basically the the best uh way of uh extracting generic features out of images that you can you can use as as image representation for any task uh you want. This is, you know, the best system in the world for for this and it's, you know, open source. Um extremely popular.”
A world model is an AI system that predicts the consequences of actions before they are executed, enabling an AI system to accomplish new tasks without being explicitly trained on them by planning sequences of actions.
“world models might enable AI systems to solve new problems without being trained explicitly to solve those problems. Uh this is really kind of the essence of intelligence in in animals and humans, which is that when we face a new situation we can apprehend it and figure out a way to accomplish a task uh without, you know, being trained uh to do that.”
Sigreg (Sketch anisotropic Gaussian regularization) prevents JEPA collapse by maximizing information content in encoder outputs through a regularization method based on projecting distributions along multiple axes and ensuring marginal distributions match isotropic Gaussians.
“the technique we arrived at very recently is uh is this technique that consists in uh given a bunch of data points that come out of your encoder... if you were able to make that distribution to train the encoder so that the distribution of those points over a batch of samples were a isotropic uh uh multivariate Gaussian, then all the variables would be independent, right? Of each other... the system will not be collapsed to zero because the the variance has to be one... So, a joint Gaussian is actually the maximally the maximum entropy distribution for a given uh variance.”
Intelligence is the ability to acquire new skills extremely quickly with zero-shot or minimal training (few-shot), not a collection of specialized skills, which explains why benchmarking AI systems on static test sets misses the essence of intelligence.
“intelligence is not a collection of skills. It's an ability to acquire new skills extremely quickly. Either zero-shot with no training you face a new situation you know what to do or with minimal training like like driving okay in only a few hours so this ability to acquire new skills is really what defines intelligence or to solve new problems”
Energy-based models offer a better paradigm than probabilistic models for capturing dependencies between variables in high dimensions because representing exact joint distributions is intractable.
“How do you capture this dependency between X and Y? So, of course you could say, Well, I'm going to learn a a joint distribution between X and Y. But that may actually be extremely difficult, particularly in high dimension. Estimating distribution in high dimension is basically impossible and intractable. Uh you have to represent it by an energy function... the normalization constant is intractable... so, it's basically intractable. Okay, so how about something that's simpler? And that's energy-based model, which consists in can I learn a contrast function that will give me which is scalar scalar outputs, right?... if the pair of X and Y is in the region of high data density, it gives me a low value, let's say zero. And if I move away from the manifold of data... the value of this uh scalar function increases... it's a much better way, actually, a much simpler way, also, of capturing dependencies than probabilistic modeling.”
V-JEPA implicitly learns 3D geometric structure (depth) without being explicitly trained on 3D annotations, as evidenced by strong depth estimation performance when a simple supervised head is added to the learned representation.
“the VideoPoet 2.1 system, if you train a very simple head on top of the representation to predict the depth of every pixel in an image in a single image, it does a pretty good job at it. Um and it's it's kind of interesting because basically this system has learned to represent the 3D world without being told anything about the fact that the world is three-dimensional. It's just being trained on video. But it's learned probably that the best way to explain how our view of the world changes as we move the camera is to basically give a depth to every point in the image.”
LeCun's 2023 claim that ChatGPT cannot understand basic physics (objects moving with tables) was later 'debunked' when ChatGPT was fine-tuned on this specific question after his podcast appeared, demonstrating fine-tuning on individual examples rather than learning understanding.
“I was made fun of uh in the past year or two because on a interview with uh Lex Fridman, uh you know, a pretty popular podcast, I said, Look, you know, LLMs don't have any physical in If you put a if you put a an object on the table and you push the table, you know the object will move with the table, right? That's kind of completely obvious physical intuition. But if you have if you ask, you know, ChatGPT or one of its successor, uh it's not going to be able to, you know, understand the physical reality behind this and make the right prediction. And of course, 6 months later, like so if you ask a question to ChatGPT at the time, it would not answer correctly. But if you ask the next version of ChatGPT, it did answer correctly. And the reason is it's not that it, you know, understood physical reality any better. It's that it was explicitly fine-tuned to answer that particular question because as soon as as, you know, the podcast was published, hundreds of people actually asked the question to ChatGPT to verify the answer was incorrect. Um and so that became part of the fine-tuning set now and and, you know, OpenAI integrated that that question in the fine-tuning set. And of course, you know, any particular question that you ask any AI system, you can fine-tune it to answer it correctly. But it's retrieval.”
Geometric deep learning (graph neural networks, manifold learning) should be applied to JEPA: training JEPA architectures on graph-structured inputs has not been done yet but would be very interesting and promising to explore.
“Okay, so let me ask let me let me answer first the question about manifold learning. Okay, so the the whole idea that I presented of information maximization, uh you know, to train a GeP to prevent the collapse using this uh isotropic Gaussian criterion, that basically is manifold learning. I mean, you're trying to train a system to learn the low-dimensional structure uh of the the data in the ambient input space, right? And map it to essentially a Gaussian distribution. Uh a flat manifold, right? With the uh Gaussian distribution. Um so that's a form of uh manifold learning. Now, the connection with graph neural nets, I don't think anyone has done this yet, like train a Jepa whose input is, you know, a graph, essentially. Uh I think that would be super interesting to experiment with.”
The hope for AMI Labs is that hierarchical JEPA-based architectures will enable training of universal causal models of any complex phenomenon, powering intelligent systems of the future across diverse domains.
“And basically, you you solve all the problems in the world world that uh AI currently uh really is not so good for. Uh and uh the hope is that, you know, maybe hierarchical Japa hierarchical planning will allow us to to train sort of universal causal models of any complex phenomenon and be the business for, you know, any intelligent uh system uh in the future.”
Complex systems like turbojet engines, chemical plants, oil refineries, and patients with chronic conditions are too complex to model with reductionist equations but can be modeled phenomenologically using trained world models with abstract representations.
“a turbojet engine typically has something like 1,000 sensors, right?... you'd like is being able to extract an abstract representation of the the state of the of the engine... at this abstract level predict what the next state of the engine will be as a consequence of the previous state and the action you imagine taking. If you have such a model then you can plan a sequence of actions or interventions, the controls, that will optimize metrics like fuel consumption, efficiency, longevity... This is really classical optimal control but but with a model that is trained from data as opposed to written in the reductionist approach... You can apply this of course to more complex systems like say a chemical plant or oil refinery... Or something like a patient which is an extremely complex system.”
Humans are not particularly good at causal inference—we regularly attribute hidden causes to unexplained phenomena (attributing thunder to Zeus), suggesting that if humans were truly good at causal reasoning, religion would not exist.
“And so, you know, I'm going to say something slightly offensive, but if humans were so good at causal inference, uh religion would not exist.”
Mathematics and theoretical computer science need more deep engagement with machine learning because the mathematical problems in deep learning are very interesting but complicated, and they have historically been neglected by mathematicians in favor of simpler frameworks (kernel methods, SVMs, Bayesian inference).
“Uh and I've always tried to bring in interest from mathematicians to machine learning and deep learning in particular, because the mathematical problems there are are really in my view are very interesting, but also very complicated. And traditionally, uh people interested in machine learning who have, you know, a bit of a interest and background in mathematics have focused in the past on simpler uh machine learning uh paradigms, things like kernel methods, support vector machines, Bayesian inference and things like that, where the theory is easier to write. Neural nets are messy, and so theorists have reluctantly approached deep learning because it's very it's very complicated, right? The functions we optimize are non-convex.”
Training JEPA architectures faces the 'collapse' problem: if you simply minimize prediction error in representation space, the system can cheat by predicting constant representations, making the prediction trivial and useless.
“when you train a system like this and you just minimize the prediction error in representation space, it can cheat. It can say, well, I'm just going to predict a constant representation. And then my prediction error is going to be zero all the time. Okay? Make ignore the input and then make make SX and SY just constant. Uh and and now your prediction problem is trivial. That's called a collapse.”
The I-JEPA method prevents collapse through momentum-based weight updates—maintaining a second encoder whose weights are an exponential moving average of the primary encoder's weights, which unexpectedly prevents representation collapse, though the theoretical explanation remains incomplete.
“the way we compute the weights of the the encoder on the right is that we take a sequence of previous weight vectors of the encoder on the left during training. We compute an exponential moving average, basically a running average of the past weights of that encoder, and that's what we set the weights of this encoder to. And that somehow prevents collapse. Why? We don't really know.”
Joint embedding predictive architecture has become very popular: searching 'joint embedding predictive architecture' on Google Scholar between quotes yields approximately 1,300 papers, indicating a large and growing research community interested in this approach.
“This whole idea of JAPA actually has become quite popular over the last you know four or five years since I I proposed it. If you search joint embedding predictive architecture on Google Scholar between double quotes, you get about 1300 papers. So there's a whole community really interested in this which is growing.”