Distillation-based methods in JEPA use two encoders with identical architecture but different update rates: the left encoder gets gradients and updates normally, while the right encoder uses exponential moving average of the left's weights, stabilizing training

factualpending

Speaker

Yann LeCun

Evidence Quote

the encoder on the right use a exponential moving average over time of the weights of the encoder on the left

Source

Yann LeCun: World Models: Enabling the next AI revolutionComputer Vision and Geometry Group, ETH Zurich
Created: 8/12/2026, 5:59:01 PM

My Notes

Loading notes...