Distillation methods came from ideas at Google DeepMind for stabilizing variance in reinforcement learning (using exponential moving average of weights), which were realized to apply to self-supervised learning from images; BYOL (Bootstrap Your Own Latent) pioneered this approach.