A temporal model needs to combine two kinds of evidence. The first is appearance: what the scene looks like in each frame, which is exactly what the image-side machinery from earlier chapters already extracts. The second is motion: how the scene is changing, which optical flow supplies.
The standard conceptual arrangement is two parallel pathways. One pathway processes a small stack of raw frames and learns what objects and scenes are present. The other processes a stack of flow fields and learns what movements are happening. Each pathway produces its own summary of the clip, and the two summaries are merged into a single decision.
The reason for keeping them separate is that they fail in different situations. Appearance is reliable when the camera is still and the object is clear, but it struggles when the same object appears in many poses. Motion is reliable when the movement is distinctive, but it struggles when the camera itself is moving, because camera motion contaminates the flow. A model that sees both can lean on whichever is more trustworthy for the clip in front of it.