Attention Is All You Need: The Transformer Architecture Explained
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$
The Transformer architecture is structured around two key blocks:
1. **Scaled Dot-Product Attention**: Projects input embeddings into Query ($Q$), Key ($K$), and Value ($V$) matrices. The inner product between queries and keys measures semantic relevance, scaled by $\sqrt{d_k}$ to prevent gradient vanishing in large dimensions.
2. **Multi-Head Attention**: Runs the attention function $h$ times in parallel with distinct learned parameter matrices, enabling the model to jointly attend to information from different representation subspaces at different positions.
3. **Positional Encoding**: Injects order awareness using sinusoidal functions of varying frequencies, allowing the network to distinguish word position without recurrent state passing.