Temporal attention runs along the time axis. For a given spatial location, the patch at that location in frame one, frame two, frame three, and onward are treated as a sequence, and each one attends to the others. A patch representing an eye in frame one can therefore pull in the appearance of that same eye in later frames, so the representation stays anchored to one identity instead of being re-decided from scratch each frame. This is what keeps a face recognizable as it turns and a shirt the same color as the person walks.
How Text-to-Video AI Works Under the Hood