What Is Yann LeCun Cooking? JEPA Explained Simply
What this covers
Warp is the agentic development environment born out of the terminal. Download Warp for free today at → https://go.warp.dev/bycloudythoa
For the longest time, Yann LeCun has been pioneering this idea called JEPA. With its rapid advancements as of recent, it has become a spotlight in the research field, especially for world modeling. So in today's video, I'll be covering the main idea of JEPA, how it works, and what makes it promising.
my latest project: Intuitive AI Academy We just wrote a new piece on RL & RLHF! https://intuitiveai.academy/ limited time code "EARLY" for 40% off yearly plan
My Newsletter https://mail.bycloud.ai/
My Patreon https://www.patreon.com/c/bycloud
Sauces [Original JEPA paper] https://openreview.net/pdf?id=BZ5a1r-kVsf [V-JEPA] https://arxiv.org/abs/2506.09985 [I-JEPA] https://arxiv.org/abs/2301.08243 [EMA for Self-Supervised ViT] https://arxiv.org/abs/2104.14294 [Infomax] https://pubmed.ncbi.nlm.nih.gov/7584893/ [SimCLR] https://arxiv.org/abs/2002.05709 [Barlow Twins] https://arxiv.org/abs/2103.03230 [VICReg] https://arxiv.org/abs/2105.04906 [LeJEPA] https://arxiv.org/abs/2511.08544 [EchoJEPA] https://arxiv.org/abs/2602.02603 [DINO v2] https://arxiv.org/abs/2304.07193
Try out my new fav place to learn how to code https://scrimba.com/?via=bycloudAI
This video is supported by the kind Patrons & YouTube Members: 🙏Spam Maj, Alex, Chris LeDoux, DX Research Group, Poof N' Inu, Deagan, Robert Zawiasa, Ryszard Warzocha, Tobe2d, Louis Muk, Akkusativ, Kevin Tai, Mark Buckler, NO U, Tony Jimenez, Ângelo Fonseca, jiye, Anushka, Asad Dhamani, Binnie Yiu, Calvin Yan, Clayton Ford, Diego Silva, Etrotta, Gonzalo Fidalgo, Handenon, Hector, Jake Disco very, Michael Brenner, Nilly K, OlegWock, Daddy Wen, Shuhong Chen, Sid_Cipher, Stefan Lorenz, Sup, tantan assawade, Thipok Tham, Thomas Di Martino, Thomas Lin, Richárd Nagyfi, Paperboy, mika, Leo, Berhane-Meskel, Kadhai Pesalam, mayssam, Bill Mangrum, nyaa, Toru Mon, Lame Plane, Matej Macak, Len Mo, saylikhapekar, ZyanSheep, THEVIERAOS, Ricardo Raphael Corona-Moreno, C
[Discord] https://discord.gg/NhJZGtH [Twitter] https://twitter.com/bycloudai [Patreon] https://www.patreon.com/bycloud [Business Inquiries] bycloud@smoothmedia.co [Profile & Banner Art] https://twitter.com/pygm7 [Video Editor] @aduckchicken2 [Ko-fi] https://ko-fi.com/bycloudai
Manim Animations created with Manimate https://www.manimate.ai/
Source description (no synthesized summary yet).
JEPA (Joint Embedding Predictive Architecture) represents a fundamentally different approach to self-supervised learning that predicts high-dimensional latent representations rather than pixels or tokens, making it particularly suited for vision and medical imaging but not for language, where autoregressive token prediction already solves the core problem efficiently.
- JEPA works by predicting abstract representations in latent space rather than reconstructing pixels, which removes unpredictable sensory noise from the learning objective
- Language lacks the low-level noise problem JEPA solves because text is already a compressed symbolic representation, making autoregressive training superior for LLMs
- JEPA is especially promising for medical imaging where it can focus on clinically meaningful anatomical structures rather than wasting capacity on noise artifacts
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
When views are compressed into latent space, the target embedding naturally removes all noise and contains only compressed abstraction and semantics, allowing the predictor to model only stable structure between views without noise interference.
“But when you compress these views into a latent space, the target embedding will naturally remove all the noises and only contain this compressed abstraction and semantics. So, when the predictor minimizes distance to the target embedding, it is encouraged to model only the stable structure between views, and the noise will not interfere, which is the culprit that makes the model predictions harder than it should.”
JEPA performs world simulation directly in latent space where action sequences and state transitions are represented as embeddings, saving compute compared to generating full video frames by operating in representation space rather than pixel space.
“So, with Jepa, you do not need to materialize anything into pixel space to simulate the world. So, you're basically saving the compute for generating the pixels. So, instead of generating the full video frames to predict the future, Jepa performs the simulation directly in latent space, where action sequences and state transitions are represented as embeddings. And because the computations happen in representation space rather than pixel space, this process is significantly more efficient while still capturing the underlying dynamics of the environment.”
For complex JEPA applications, the latent space must be structured to support planning, with the predictor learning smooth transitions between states through conditional actions, so changes in latent state make sense and predicted trajectories don't drift or accumulate errors.
“On top of that, for more complicated Jepa applications, the latent space must be structured in a way that supports planning. The predictor needs to learn smooth transitions between states through conditional actions, so the changes in the latent state would make sense. So, if the geometry of the embedding space is poorly structured, then the predicted trajectories would drift, accumulate errors, or stop corresponding to real-world outcomes.”
Representation collapse is a failure mode where both the context and target encoders in JEPA can learn to output the same constant embedding for all inputs, with the predictor simply outputting a constant embedding that always matches, resulting in extremely low training loss but learning absolutely nothing about the world.
“Since Jepa does not reconstruct pixels or have any inherent noise, nothing stops the encoder from outputting the same vector for every input, especially when they are two encoders used for Jepa, the context and the target encoder. So, the model could discover a very easy shortcut. Instead of learning meaningful representations of the scene, both encoders could simply output the same constant embedding for everything. A cat, a car, or a building would all produce the exact same embedding. And now, the predictor's job becomes very easy. Because no matter what the context is, it would just output the same constant embedding, which will always match the target embedding. So, the training loss becomes extremely small, but the model has learned absolutely nothing about the world, and everything looks identical in the latent space. This failure mode is called representation collapse”
LeJepa can train JEPA-style models without relying on EMA while still avoiding collapse and learning strong representations, achieving competitive or better performance compared to VICReg, Barlow Twins, and SimCLR on standard vision benchmarks, with accuracy levels similar to state-of-the-art DINO on ImageNet.
“In their experiments, LeJepa was able to train Jepa-style models without relying on EMA, while still avoiding collapse and learning strong representations. Despite using a simpler objective, it achieves competitive or better performance compared to earlier self-supervised methods, VICReg, Barlow Twins, and SimCLR on standard vision benchmarks, even achieving accuracy levels similar to Dino on ImageNet, which is state-of-the-art”
A view in JEPA is a transformation, masking, or partial observation of an input that preserves the underlying semantic state while hiding or removing some information, such as different image crops, masked versions, camera angles, or video frames of the same scene.
“A view is a transformation, masking, or partial observation of an input that preserves the underlying semantic state while hiding or removing some information. So, let's say there's a cat sitting on a couch. You could observe an image, the left half of the image, the right half of the image, a zoomed-in crop, a masked version of the image, a different camera angle of the same scene, a video, a different frame from the same video.”
The idea of a cat sitting on a couch can be captured in a high-dimensional latent space with all kinds of sentences or images that show this idea pointing to that spot in the latent space, demonstrating that JEPA's representation captures semantic invariants across diverse inputs.
“So, the idea of a cat sitting on a couch can be captured in a high-dimensional latent space with all kinds of sentences or images that show this idea will point to that spot in the latent space”
The first practical solution researchers used to prevent representation collapse was updating the target encoder with exponential moving average (EMA), where instead of both encoders learning freely, the target encoder is updated very slowly as a delayed version of the context encoder.
“The first practical solution researchers used was updating the target encoder with something called EMA, short for exponential moving average. So, instead of letting both encoders learn freely and can instantly copy each other's behavior, the target encoder is updated very slowly. If the context encoder suddenly changes its representation, the target encoder does not immediately follow. It only moves gradually over time, acting like a delayed version of the context encoder. You can think of it like chasing a slowly moving target.”
EMA is heuristic-based because stability depends on EMA schedules, weight sharing, and training dynamics, which can be fragile and require manual tuning, motivating research into other methods for preventing representation collapse.
“It is also heuristic-based, because stability depends on EMA schedules, weight sharing, and training dynamics, which can be fragile and require manual tuning. This is why later research still explored other methods for preventing representation collapse.”
At inference, the target encoder is typically not needed; instead, the trained context encoder serves as a feature extractor that can be used for classification, retrieval, similarity search, or downstream supervised fine-tuning in a representation extraction application.
“Then at inference, the target encoder is typically not needed. What is then generally used is the trained context encoder, which now serves as a feature extractor or latent state estimator. And there are multiple ways to apply Jepa. One is representation extraction, where you feed an input like an image, video clip, etc. into the context encoder and obtain an embedding. And that embedding can be used for classification, retrieval, similarity search, or even downstream supervised fine-tuning.”
Early JEPA experiments like I-JEPA (image generation), V-JEPA (video generation), and DINO (self-supervised vision) all utilized exponential moving average, though EMA is ultimately a training trick rather than a principled objective since there is no loss function for EMA to minimize.
“In the early experiments of Jepa, like I-Jepa, where Jepa is applied to image generations, and V-Jepa, where Jepa is applied to video generation, or even Dino, which is a self-supervised vision method for learning image representations without labels, they are all utilizing EMA. However, EMA is ultimately a training trick, rather than a principled objective, as there does not exist a loss function for EMA, which prevents it from being minimized directly.”
InfoMax is a major research direction for preventing representation collapse that tries to ensure the representation contains information uniquely about the input, rather than collapsing into something trivial.
“With one major direction known as the InfoMax approach, which tries to ensure the representation itself contains information uniquely about the input. The idea of InfoMax first appeared in 1995 from Bell and Sejnowski, and the main idea is simple. A good representation should retain as much information about the input as possible.”
A typical JEPA learning setup contains three components: a context encoder that takes visible content and produces an embedding, a target encoder that takes masked or future content and produces a target embedding, and a predictor that takes the context embedding and tries to produce the target embedding.
“A typical Jepa learning setup contains three components: a context encoder that takes the current context view like visible image blocks or text and produces an embedding, a target encoder that takes the target view like masked blocks or a future segment and produces the target embedding. This embedding represents what actually happens next or what is missing, but it is still within the same representation space. And lastly, a predictor component that bridges the two state, which is a representation of the current situation and a representation of the future or missing part. What this component does is that it takes the context embedding and tries to produce the target embedding”
Direct pixel or token reconstruction objectives contain full entropy from noise sources like text variations without right answers and lighting changes in images, forcing models to accommodate unpredictable details even though they are fundamentally irrelevant to meaning.
“As the problem of directly working with pixels or tokens is that the objective is full of entropy. For example, text variations that does not have a right answer while all meaning roughly the same thing, and lighting changes of background within an image that doesn't even matter, but the reconstruction error will always be full of noise. This results in the actual model having the need to accommodate for everything, even for things that are fundamentally unpredictable.”
The problem JEPA is designed to solve (removing unpredictable sensory noise from prediction) does not exist as strongly in text, and autoregressive training works perfectly already with high signal feedback, making JEPA probably not a better design for predicting text.
“So, when an LLM predicts the next token, it is already operating at a fairly high semantic level. In other words, the problem Jepa is designed to solve, which is removing unpredictable sensory noise from prediction, does not exist as strongly in text. And of course, autoregressive training works perfectly already. It's an objective that can provide high signal feedback to the model. And Jepa is probably not going to be a better design to predict text”
The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.
“But, the downside of this approach is that it relies heavily on negative samples. So, to properly separate representations, the model needs a large batch of other images to push away from. This means you often need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.”
Techniques like Barlow Twins and VICReg implement dimension contrastive methods by measuring correlation between embedding dimensions and penalizing the model if multiple dimensions start carrying the same signal, avoiding the need for large batches of negative samples.
“So, techniques like Barlow Twins and VICReg implement this by measuring the correlation between embedding dimensions and penalizing the model if multiple dimensions start carrying the same signal. And this was already a big step forward, because the model no longer needed large batches of negative samples to achieve the contrastive method.”
Contrastive methods like SimCLR (published 2020) train models by creating two augmented views of the same image and training the model to recognize they come from the same sample, pulling their embeddings closer together while pushing embeddings of other images farther apart.
“The first is simple contrastive methods where the model is encouraged to make embeddings of different samples distinct from one another. Methods like SimCLR, published in 2020, long before Jepa, work this way by creating two augmented views of the same image and training the model to recognize that they come from the same underlying sample. The model basically learns to pull the embeddings of those two views closer together, while pushing embeddings of other images further apart.”
A prominent direction of JEPA research is using it for medical imaging, with EchoJepa applying the JEPA framework to echocardiography videos (ultrasound videos of the heart).
“A really prominent direction of Jepa I've been seeing though is using it for medical imaging. There is this research called EchoJepa, which applies the Jepa framework to echocardiography videos, which are basically ultrasound videos of the heart.”
LeJepa constrains the overall geometry of the embedding space to follow an isotropic Gaussian distribution, meaning representation space spreads information evenly across dimensions so no direction collapses or dominates.
“Instead of simply decorrelating dimensions, LeJepa takes a more direct approach, where it constrains the overall geometry of the embedding space itself. It specifically encourages the embeddings to follow an isotropic Gaussian distribution, meaning the representation space spreads information evenly across dimensions, so no direction collapses or dominates.”
When Yann LeCun said LLMs are doomed, he was taken out of context; LLMs are only meant to work with languages because language is a symbolic system with much less noise, and if you want to truly scale beyond language (especially for natural or real-life data), his point is valid.
“So, when Yann LeCun said LLM is doomed, I low-key took it out of context. LLM is only meant to work with languages due to how it is a symbolic system with a lot less noise. So, if you want to truly scale beyond language, especially for research on natural or real-life data, he he he's got a point.”
Despite the progress of dimension contrastive methods, they still relied on multiple loss terms and carefully balanced hyperparameters to keep representation stable, motivating Yann LeCun's latest bet called LeJepa released in November 2025.
“But, these methods still relied on multiple loss terms and carefully balanced hyperparameters to keep the representation stable. And this is the latest bet that Yann LeCun is making called LeJepa comes in. Released in November 2025”
JEPA (Joint Embedding Predictive Architecture) is one of the most convoluted research topics that is difficult to follow because it involves predicting high-dimensional representations in a learned latent space rather than tokens or noisy pixels.
“I think it is one of the most convoluted research topics that's just impossible to follow. Like for LLMs, we predict tokens. For image generation, we predict a less noisy image. But for Jepa, we are literally predicting a high-dimensional representation in a learned latent space”