Mixture of Experts (MoE) Architectures
Sparsity at scale: Fine-grained routing, load balancing loss, expert parallelism, and memory management.
Pillar Architectural Overview
### Sparse Computation and Compute Efficiency
Mixture of Experts (MoE) decouples total parameter capacity from compute cost per token. By routing each token through only a subset of specialized feed-forward networks (FFNs), models achieve the knowledge capacity of 500B+ parameter models at the inference cost of a 40B dense model.
Key architectural innovations:
1. **Fine-Grained Experts**: Dividing feed-forward capacity into dozens of smaller experts with shared base layers.
2. **Auxiliary-Loss-Free Routing**: Preventing expert collapse using dynamic bias adjustments rather than punitive auxiliary loss terms.
3. **EP Communication Overlap**: Hiding All-to-All network communication latency behind GPU kernel execution.