High-Resolution Image Synthesis with Latent Diffusion Models
Authors: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
$L_{\text{LDM}} = \mathbb{E}_{\mathcal{E}(x), \epsilon \sim \mathcal{N}(0,1), t} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y)) \right\|^2 \right]$
The architecture decouples the generative modeling task into two distinct components:
1. **Perceptual Compression (Autoencoder $\mathcal{E}$ and $\mathcal{D}$)**: A pretrained autoencoder compresses images from pixel space into a lower-dimensional latent representation ($z = \mathcal{E}(x)$), stripping imperceptible high-frequency noise.
2. **Latent Diffusion & Cross-Attention Conditioning**: A U-Net denoiser operates exclusively within this compressed latent space, guided by text embeddings via cross-attention layers ($\tau_\theta(y)$).