A key limitation of the matchbox-and-beads learning system was that every possible situation or board state required a separate box that a human would have to select; to truly mimic a brain, machines needed their own ability to recognize patterns on their own, called abstraction.

In Transformer networks generating music, each colored line is a different attention head and the weight of the line is the amount of attention it gives to each location; each attention head looks for different kinds of patterns in the music, and the more attention heads a network has, the more powerful it becomes.

The hard part was realizing that training neural networks to predict the next token is a worthwhile goal at all; this realization greatly influenced subsequent thinking and approaches in the field.