The first practical solution researchers used to prevent representation collapse was updating the target encoder with exponential moving average (EMA), where instead of both encoders learning freely, the target encoder is updated very slowly as a delayed version of the context encoder.

JEPA (Joint Embedding Predictive Architecture) is one of the most convoluted research topics that is difficult to follow because it involves predicting high-dimensional representations in a learned latent space rather than tokens or noisy pixels.

Direct pixel or token reconstruction objectives contain full entropy from noise sources like text variations without right answers and lighting changes in images, forcing models to accommodate unpredictable details even though they are fundamentally irrelevant to meaning.

Techniques like Barlow Twins and VICReg implement dimension contrastive methods by measuring correlation between embedding dimensions and penalizing the model if multiple dimensions start carrying the same signal, avoiding the need for large batches of negative samples.

When Yann LeCun said LLMs are doomed, he was taken out of context; LLMs are only meant to work with languages because language is a symbolic system with much less noise, and if you want to truly scale beyond language (especially for natural or real-life data), his point is valid.

The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.

EMA is heuristic-based because stability depends on EMA schedules, weight sharing, and training dynamics, which can be fragile and require manual tuning, motivating research into other methods for preventing representation collapse.

A view in JEPA is a transformation, masking, or partial observation of an input that preserves the underlying semantic state while hiding or removing some information, such as different image crops, masked versions, camera angles, or video frames of the same scene.

When views are compressed into latent space, the target embedding naturally removes all noise and contains only compressed abstraction and semantics, allowing the predictor to model only stable structure between views without noise interference.

Despite the progress of dimension contrastive methods, they still relied on multiple loss terms and carefully balanced hyperparameters to keep representation stable, motivating Yann LeCun's latest bet called LeJepa released in November 2025.

The problem JEPA is designed to solve (removing unpredictable sensory noise from prediction) does not exist as strongly in text, and autoregressive training works perfectly already with high signal feedback, making JEPA probably not a better design for predicting text.

At inference, the target encoder is typically not needed; instead, the trained context encoder serves as a feature extractor that can be used for classification, retrieval, similarity search, or downstream supervised fine-tuning in a representation extraction application.

JEPA performs world simulation directly in latent space where action sequences and state transitions are represented as embeddings, saving compute compared to generating full video frames by operating in representation space rather than pixel space.

For complex JEPA applications, the latent space must be structured to support planning, with the predictor learning smooth transitions between states through conditional actions, so changes in latent state make sense and predicted trajectories don't drift or accumulate errors.

Representation collapse is a failure mode where both the context and target encoders in JEPA can learn to output the same constant embedding for all inputs, with the predictor simply outputting a constant embedding that always matches, resulting in extremely low training loss but learning absolutely nothing about the world.

LeJepa constrains the overall geometry of the embedding space to follow an isotropic Gaussian distribution, meaning representation space spreads information evenly across dimensions so no direction collapses or dominates.

Early JEPA experiments like I-JEPA (image generation), V-JEPA (video generation), and DINO (self-supervised vision) all utilized exponential moving average, though EMA is ultimately a training trick rather than a principled objective since there is no loss function for EMA to minimize.

InfoMax is a major research direction for preventing representation collapse that tries to ensure the representation contains information uniquely about the input, rather than collapsing into something trivial.

Contrastive methods like SimCLR (published 2020) train models by creating two augmented views of the same image and training the model to recognize they come from the same sample, pulling their embeddings closer together while pushing embeddings of other images farther apart.

A prominent direction of JEPA research is using it for medical imaging, with EchoJepa applying the JEPA framework to echocardiography videos (ultrasound videos of the heart).