1. A method for determining motion directions for video dataset object images, the method comprising: estimating directions of motion, through an optical flow process executing on a processor, for each of a plurality of object images that are extracted from a source training video dataset input, for each of a plurality of different camera scene viewpoints, wherein the object images are collected from each of the different camera scene viewpoints; representing each space-time point in the estimated optical flow directions of motion of the objects appearing for each respective camera viewpoint by a four-dimensional vector, the vector comprising a location of the each space-time point in an image plane, a magnitude and a direction of its optical flow; and categorizing the plurality of object images into a plurality of optical flow map clusters as a function of similarities of their determined motion directions by: discarding the space-time points that have an optical flow magnitude that is above or below certain respective fixed thresholds as noise; after the discarding the noise points, randomly sub-sampling and clustering a remainder of the space-time points into the optical flow map clusters by clustering that automatically selects a scale of analysis and a total number of the clusters; and representing different values of the directions of motion of the objects appearing in the scene viewpoint of each optical flow map cluster by a dominant direction of motion of the points within each optical flow map cluster and by a location of the cluster in the image plane.