Stacking encoder layers allows each layer to learn different attention representations, potentially boosting the predictive power of the transformer network.

In the decoder, attention scores for a word like 'am' have values for itself and all other words before it, but zero attention scores for future words like 'fine', which tells the model to put no focus on those words.

Transformers must add positional encoding information to input embeddings because unlike recurrent neural networks, transformers have no recurrence and therefore need explicit position information to understand word order.

In theory, each head in multi-headed attention learns something different, therefore giving the encoder model more representation power through multiple perspectives on the same input.

Transformers are breaking multiple NLP records and pushing the state-of-the-art in applications like machine translation, conversational chat bots, and search engines.

Transformers leverage the power of the attention mechanism to make better predictions compared to recurrent neural networks which suffer from short-term memory, particularly for encoding or generating longer sequences.

The encoder layer's job is to map all input sequences into an abstract continuous representation that holds the learned information for that entire sequence, containing two sub-modules: multi-headed attention followed by a fully connected network.

To achieve self-attention, the input is fed into three distinct fully connected layers to create query, key, and value vectors, which follow the same concept as a retrieval system where queries are mapped against keys to find relevant matches.

RNNs suffer from short-term memory because they have a shorter window to reference from compared to GRUs and LSTMs, which have greater capacity to achieve longer-term memory with a longer reference window.

The attention mechanism in theory has an infinite window to reference from given enough compute resources, allowing it to use the entire context of a sequence while generating text, unlike RNNs which cannot access words generated earlier.

The 'Attention is All You Need' paper introduced a novel attention-based encoder-decoder neural network architecture called Transformers that demonstrated superior performance compared to previous models.

The encoder in a transformer maps an input sequence into an abstract continuous representation that holds all the learned information of that input, while the decoder takes this representation and step-by-step generates a single output while also being fed previous outputs.

Word embedding layers function as lookup tables that map each word to a vector with continuous values representing that word, since neural networks learn through numbers and cannot directly process words.

Positional encoding uses sine and cosine functions where for every odd time step a cosine vector is created and for every even time step a sine vector is created, then these vectors are added to their corresponding embedding vectors to give the network position information.

Sine and cosine functions were chosen for positional encoding because they have linear properties that allow the model to easily learn to attend to positional information.

Encoder layers contain residual connections around each of the two sub-modules (multi-headed attention and fully connected network) followed by layer normalization.

Self-attention allows a model to associate each individual word in the input to other words in the input, enabling the model to learn relationships like associating the word 'you' with 'how' and 'are' in a question pattern.

In self-attention, queries and keys undergo a dot product matrix multiplication to produce a score matrix that determines how much focus should be placed on each word relative to other words, with higher scores indicating more focus.

The attention scores are scaled down by dividing them by the square root of the dimension of the queries and keys in order to allow for more stable gradients, as multiplying values can have exploding effects.

The softmax function is applied to scaled attention scores to produce attention weights with probability values between 0 and 1, where higher scores are heightened and lower scores are depressed, allowing the model to be more confident about which words to attend to.