Spatial self-attention operates within one frame. The frame's latent is divided into patches, and each patch produces a query, a key, and a value. Every patch compares its query against all other patches' keys, and the resulting weights decide how much information it pulls from each value. This is how a region that only sees a patch of fur can still represent a whole animal: it gathers evidence from the patches around it. Without this exchange, each patch would be denoised on its own local evidence, and the frame would look like unrelated tiles rather than one scene.
How Text-to-Video AI Works Under the Hood