The attention mechanism in theory has an infinite window to reference from given enough compute resources, allowing it to use the entire context of a sequence while generating text, unlike RNNs which cannot access words generated earlier.
factualpending
Speaker
Unidentified Speaker — Illustrated Guide to Transformers Neural Network: A step by… [4Bdc55j80l8]Evidence Quote
“the attention mechanism in theory and given enough compute resources have an infinite window to reference from therefore being capable of using the entire context of the story”
Created: 8/13/2026, 9:51:15 AM
My Notes
Loading notes...