The attention scores are scaled down by dividing them by the square root of the dimension of the queries and keys in order to allow for more stable gradients, as multiplying values can have exploding effects.
causalpending
Speaker
Unidentified Speaker — Illustrated Guide to Transformers Neural Network: A step by… [4Bdc55j80l8]Evidence Quote
“the scores get scaled down by getting divided by the square root of the dimension of the queries and the keys this is to allow for more stable gradients as multiplying values can have exploding effects”
Created: 8/13/2026, 9:51:15 AM
My Notes
Loading notes...