Claims

JEPA (joint embedding predictive architecture) is an architecture that predicts abstract representations of future states rather than reconstructing raw sensory details, eliminating the need to predict unpredictable information by focusing only on predictable, relevant information.

Generative models that attempt to predict every pixel or detail of future video or high-dimensional continuous data produce blurry, averaged predictions and fail because most details in sensory data are fundamentally unpredictable.

Training JEPA architectures faces the 'collapse' problem: if you simply minimize prediction error in representation space, the system can cheat by predicting constant representations, making the prediction trivial and useless.

Sigreg (Sketch anisotropic Gaussian regularization) prevents JEPA collapse by maximizing information content in encoder outputs through a regularization method based on projecting distributions along multiple axes and ensuring marginal distributions match isotropic Gaussians.

DINO (Self-Supervised Vision Transformers for Image Representation) is the state-of-the-art method for extracting generic features from images without labels, now at version DINO V3, and represents the best system globally for image representation learning.

Reinforcement learning should be minimized or abandoned in favor of world models because it is extremely sample-inefficient and impractical for achieving human or animal-level intelligence.

Energy-based models offer a better paradigm than probabilistic models for capturing dependencies between variables in high dimensions because representing exact joint distributions is intractable.

The credit assignment problem is the fundamental challenge that computational systems must solve: determining which parameters to adjust and by how much to achieve a desired output

A neuron's predicted activity is given by the weighted sum of activities of upstream neurons, determined by synaptic weights which are typically followed by a nonlinear activation function (like sigmoid or ReLU), though the speaker simplifies this for pedagogical clarity

If predictive coding networks were allowed to freely adjust every parameter (both neural activities and weights), they would naturally settle to a zero energy state which would be trivial and not perform any meaningful computation, so in practical implementations and likely in the brain, certain neurons are clamped to specific values

Repeating the iterative relaxation process across diverse examples gradually refines the network's internal model of the world, developing compressed representations of data

In real models with nonlinear activation functions, the update rules for opposing synapses are not mathematically identical, but research suggests perfect symmetry may not be essential; approximate symmetry emerging from independent learning is sufficient for effective network function.

Backpropagation with gradient descent is the workhorse algorithm that powers virtually the entire field of machine learning today

Predictive coding's local autonomy makes the algorithm extremely parallelizable and in certain settings more efficient than back propagation, and theoretical considerations suggest that resulting updates may lead to better solutions than back propagation.

The downside of contrastive methods is they rely heavily on negative samples, requiring a large batch of other images to push away from, meaning you need very large batch sizes or memory banks, which makes training computationally expensive and harder to scale.

LeJepa can train JEPA-style models without relying on EMA while still avoiding collapse and learning strong representations, achieving competitive or better performance compared to VICReg, Barlow Twins, and SimCLR on standard vision benchmarks, with accuracy levels similar to state-of-the-art DINO on ImageNet.

The idea of a cat sitting on a couch can be captured in a high-dimensional latent space with all kinds of sentences or images that show this idea pointing to that spot in the latent space, demonstrating that JEPA's representation captures semantic invariants across diverse inputs.

The manifold hypothesis states that all natural data lies on a low-dimensional manifold within the high-dimensional space where it is encoded; this is accurate for any perception problem including MNIST digits, human faces, voice sounds, and natural language.

Within a manifold, it is always possible to interpolate between two inputs by morphing one into another via a continuous path along which all points fall on the manifold; the ability to interpolate between samples is the key to understanding generalization in deep learning because it allows making sense of points never seen by relating them to nearby points on the manifold.

The power to generalize in deep learning is much more a consequence of the natural structure of your data than a property of your model; you will only be able to generalize if your data follows the manifold hypothesis, and the more informative and less noisy your data, the better you can generalize.

A listener named Shawn solved the Jane Street puzzle by decomposing it into two parts: first identifying which layers pair together by looking for negative diagonal patterns in weight matrices, then ordering the pairs by sorting blocks by residual contribution size, combined with local swaps for refinement.

Training world models using generative approaches (predicting pixel-level details) fails because it is impossible to predict all plausible futures in video; there are infinite possible continuations and the system learns to predict the average, resulting in blurry outputs

Video is more redundant than text, which is actually a feature rather than a bug for self-supervised learning because redundancy in data is necessary for learning; without redundancy you cannot learn anything

Joint Embedding Predictive Architecture (JEPA) encodes both input X and future observation Y before prediction, allowing the system to eliminate unpredictable information and produce more abstract, accurate representations compared to generative models that reconstruct raw pixels

Generative models (autoencoders) trained on images do not produce good representations for downstream tasks; joint-embedding architectures are superior and all the best self-supervised image and video representation systems use joint-embedding, not reconstruction

Joint-embedding systems can collapse by ignoring inputs and producing constant representations; preventing collapse is the central problem in self-supervised learning for these systems and requires either contrastive methods (pushing up energy of non-data points) or regularized methods (minimizing volume of low-energy regions)

Techniques like MNCR, NCR squared, WMSE, Sigreg, VICReg, and Barlow Twins all implement information maximization for joint-embedding systems, with JEPA gaining popularity (1,700 papers mention it on Google Scholar)

A fundamental challenge with information maximization methods is that proper definitions of information content require knowing the distribution of vectors, which we do not have; empirical measures are all upper bounds, not lower bounds, so methods use good upper bounds and cross their fingers

Information can be maximized by ensuring all rows of a representation matrix are different (making each sample distinct) or all columns are different (making each variable independent), with the latter (dimension contrastive methods) being preferable because independent variables are more informative

Sigreg (Sketch Isotropic Gaussian Regularization) makes representation distributions isotropic Gaussian by projecting points along many directions and ensuring marginal distributions match a Gaussian cumulative distribution, which guarantees the joint distribution becomes isotropic Gaussian

Distillation-based methods in JEPA use two encoders with identical architecture but different update rates: the left encoder gets gradients and updates normally, while the right encoder uses exponential moving average of the left's weights, stabilizing training

V-JEPA (video JEPA) trained on masked video masking predicts representation of full video from partially masked video and learns common sense about physical plausibility: prediction error spikes when impossible events occur (e.g., ball disappearing, car not falling)

Video generation systems (which can produce cute-looking videos) work by predicting in representation space then passing predictions through a decoder, not by predicting all plausible futures; the system only needs to produce one coherent video, not represent the distribution of all plausible videos—this is a much simpler problem than world modeling.

Distillation methods came from ideas at Google DeepMind for stabilizing variance in reinforcement learning (using exponential moving average of weights), which were realized to apply to self-supervised learning from images; BYOL (Bootstrap Your Own Latent) pioneered this approach.

Energy-based models provide a unified framework for explaining self-supervised learning and all learning in general; they are more general than probabilistic models, which are a special case with particular energy forms

Large language models will likely get much better when they're multimodal—trained on images as well as words. GPT-4 was trained with images, and it's possible Google is doing the same. When multimodal, these models could learn much more than humans.

Distillation is much more effective when the teacher provides probabilities for all output classes rather than just the correct answer. When a teacher gives probabilities for 1024 output categories (which sum to 1), training a student to match those probabilities provides far more information per training example than just telling the student the right class label (which only provides 10 bits of information).

The relative probabilities of wrong answers often contain more information than the correct answer. For example, a car classification system saying a BMW has 0.8 probability, Audi 0.1, garbage truck 1-in-a-million, and carrot 1-in-a-billion reveals classification structure: the ratio between garbage truck and carrot tells you that vehicles get much higher probabilities than vegetables.

Digital computers have a vastly better way to share knowledge than biological systems: weight/gradient sharing. Multiple copies of the same neural network run on different computers, each examines different data to compute desired weight changes, then all computers agree to average those weight changes so everyone learns from everyone else's data—effectively pooling knowledge at scale.

For decades many researchers assumed neural networks would get stuck in local minima during training. People never actually checked if this was true—they just assumed it. In fact, it's not true, and even if it were, good local minima would make the assumption irrelevant. This shows the importance of empirical testing over pure theorizing.

Activity perturbation can learn MNIST-level problems reasonably fast, and can learn CIFAR-10 if you're very determined, but imagenet with millions of images is just too slow to scale activity perturbation learning to large networks.

Hinton spent years trying to develop biologically-plausible learning algorithms as good as backpropagation and finally concluded there probably aren't any—backpropagation is actually better, which was a major relief because it resolved a long research puzzle.

Backpropagation requires knowing the properties of the forward pass through layers. If you don't know the hardware properties, you need a different learning algorithm. Activity perturbation (perturbing neuron inputs and measuring the effect) works better than weight perturbation because there are fewer neurons than weights, so it has much less variance.

One way to scale learning beyond single global objectives is to have gazillions of local objective functions where each small neural unit secretes its own objective function that could be spatially local. You're never trying to learn lots of parameters all at once from one objective function.

You can generate as much synthetic data as you want from games because the system can play against itself, and games also have very clear metrics—win conditions and score maximization—which are useful from an AI perspective for optimization.

Current machine learning systems are fundamentally inadequate compared to human and animal learning because they lack the background knowledge, common sense, and causal understanding of the physical world that allows humans and animals to learn new tasks extremely quickly and reason about the world.

Current LLMs operate through autoregressive token prediction—producing one token at a time by predicting the next token given a fixed window of previous tokens—which creates exponential divergence because each token prediction has some probability of error, and once an incorrect token is generated, there is no way to correct it, leading to hallucinations.

A world model is a learned predictor that takes a current estimate of the world state (in abstract representation) and an imagined sequence of actions and predicts the resulting state of the world that will occur after those actions are taken, enabling planning without explicit reward signals.

Video prediction using generative models fails to produce accurate physics understanding because video generation systems can produce visually plausible but physically incorrect frames through hacks, without requiring genuine causal models of the world.

Distillation-based methods for training joint embeddings (like BYOL and Dino) use two encoders where one is updated via exponential moving average of the other's weights, which prevents collapse and works well empirically, though the theoretical explanation for why this works is not fully understood.