Action recognition takes a sequence of frames as input and produces one label for the entire clip. It is classification with a temporal input, not a new kind of output.
Why averaging the frames loses the action
Sitting down and standing up contain the same person and the same chair, so their frames overlap heavily. Averaging collapses the sequence into one static-looking summary in which the direction of the change is gone. The model then has no basis for choosing between the two labels.
A clip label answers what happened, not who or where. Localizing the action in space and time is a separate capability.