Superposition explains polysemantic neurons

Superposition is the hypothesis that models represent more concepts than they have neurons/dimensions by packing many almost-orthogonal features into the space — explaining polysemanticity (neurons responding to many unrelated things) — and gradient descent implicitly searches over sparse models and folds them into dense computation that GPUs run efficiently.