Distillation-based methods for training joint embeddings (like BYOL and Dino) use two encoders where one is updated via exponential moving average of the other's weights, which prevents collapse and works well empirically, though the theoretical explanation for why this works is not fully understood.
factualpending
Speaker
Yann LeCunEvidence Quote
“there is another set of method that to some extent has been slightly more successful over the last couple years and those are based on distillation so again you have two encoders it's still a joint Ting productive architecture you have two encoders they kind of share the same weights but not really so the encoder on the right uh gets a version of the weights of the enod on the left that are obtained through a um exponential moving average okay a moving average so basically you force the encoder on the right to uh change its weights more slowly than the one on the left and for some reason that prevents collapse there's some theoretical work on this um in fact uh this one that jum just finished writing um but it's a little bit mysterious why this works and frankly I'm a little uncomfortable with this method”
Created: 8/11/2026, 7:11:43 AM
My Notes
Loading notes...