Why Generation Happens in a Smaller Space

A one-second clip at 512×512 with 24 frames contains roughly 18.9 million pixel values per color channel, and diffusion must run its denoiser hundreds of times over all of them. A variational autoencoder, or VAE, solves this by learning a compressed latent representation: the encoder maps the pixel video into a much smaller tensor, and the decoder reconstructs pixels from it. Generation then happens entirely in that latent space, and only the final result is decoded back to pixels. The compression is lossy but preserves the structure that matters — layout, motion, appearance — while discarding high-frequency detail the denoiser would otherwise waste compute on.