Spatial and temporal attention can be arranged in two ways. Factorized attention runs them as separate passes: first each frame attends over its own patches, then each location attends over frames. Joint spatio-temporal attention instead lets every position attend to every other position across both space and time in a single operation. Joint attention can capture relationships that factorized attention misses, but its cost grows with the square of the total number of positions — frames multiplied by patches — so doubling the frame count roughly quadruples the work. Factorized attention keeps the two passes linear in their own dimension, which is why many video models use it.
How Text-to-Video AI Works Under the Hood