Distillation is much more effective when the teacher provides probabilities for all output classes rather than just the correct answer. When a teacher gives probabilities for 1024 output categories (which sum to 1), training a student to match those probabilities provides far more information per training example than just telling the student the right class label (which only provides 10 bits of information).