1. A computer-implemented method for classifying a moving object in a field of view, comprising: using a processor to perform the steps of: receiving a sequence of scene imagery representing said field of view from at least one image capturing device; identifying said moving object in said sequence of scene imagery; classifying said moving object in accordance with one or more extracted spatio-temporal features of said moving object, said classifying including comparing the moving object's spatio-temporal features to the spatio-temporal features of one or more learned objects; and generating an alert in response to a newly detected moving object by comparing the extracted spatio-temporal features to one or more learned events, the learned events being stored as histograms of motion-based spatio-temporal features for every pixel in an image of the learned event, the histograms representing the probability distribution of spatio-temporal features due to normal activity in the field of view.