How Conditioning Enters the Generator | Visual AI Learning Guide

So how does that sequence actually steer the video? It enters through cross-attention. Picture the spatial grid of the video as a set of queries, and the prompt tokens as keys and values.

0:00 / 0:00

Auto-play

The conditioning embeddings reach the generator through cross-attention. In cross-attention, the spatial positions of the video latent act as queries, while the prompt token embeddings act as keys and values. Each position computes how strongly it matches each token and pulls in a weighted mixture. That is why different regions of the frame can attend to different words: the region containing the boat queries "boat" most strongly, while the region containing rain queries "rainy". The prompt is therefore a control signal applied at every generation step, not a key that retrieves a stored video.

References

[1]Attention Is All You Need (Vaswani et al., 2017)arxiv.org
[2]High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., 2022)arxiv.org