Attention Is All You Need: The Transformer Architecture Explained
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$
The Transformer architecture is structured around two key blocks:
1. **Scaled Dot-Product Attention**: Projects input embeddings into Query ($Q$), Key ($K$), and Value ($V$) matrices. The inner product between queries and keys measures semantic relevance, scaled by $\sqrt{d_k}$ to prevent gradient vanishing in large dimensions.
2. **Multi-Head Attention**: Runs the attention function $h$ times in parallel with distinct learned parameter matrices, enabling the model to jointly attend to information from different representation subspaces at different positions.
3. **Positional Encoding**: Injects order awareness using sinusoidal functions of varying frequencies, allowing the network to distinguish word position without recurrent state passing.
Frequently Asked Questions
Why did the Transformer replace RNNs like LSTM?
RNNs must process tokens one by one in sequence ($O(N)$ sequential operations), which prevents parallel GPU training. The Transformer processes all tokens simultaneously in parallel matrix multiplications.
What is the role of scaling by $\sqrt{d_k}$ in self-attention?
For large projection dimensions $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions with extremely small gradients. Dividing by $\sqrt{d_k}$ stabilizes gradient variance.