The first practical solution researchers used to prevent representation collapse was updating the target encoder with exponential moving average (EMA), where instead of both encoders learning freely, the target encoder is updated very slowly as a delayed version of the context encoder.
factualpending
Evidence Quote
“The first practical solution researchers used was updating the target encoder with something called EMA, short for exponential moving average. So, instead of letting both encoders learn freely and can instantly copy each other's behavior, the target encoder is updated very slowly.”
Created: 8/12/2026, 6:16:15 PM
My Notes
Loading notes...