The action categories that benefited the most by plugging in gsf to the backbone cnn are reported in Fig. REF . The Y-axis labels show the true labels (predicted by gsf) and the labels predicted by tsn, separated by “/". From the plot, it can be seen that the baseline tsn confuses actions that are of similar nature but following a different (reversed) order, such as unfolding something and folding something, moving something and something closer to each other and moving something and something away from each other, etc.With gsf modules in the backbone cnn, the model is capable of encoding the temporal ordering of frames to correctly recognize the action. This can further be seen from the visualization of the saliency tubes [1]} for the samples from the most improved classes as shown in Fig. REF . From the figure, one can see that gsf predicts the action category by focusing on the relevant frames where the corresponding action is taking place as opposed to the baseline tsn. In Figs. REF  and REF , even though the baseline model is focusing on the relevant spatio-temporal features, the lack of temporal reasoning renders the model to make wrong predictions.
<FIGURE><FIGURE>