1. A computer implemented method for deriving, from video data, an association between at least two entities in motion, comprising: extracting the entities from the video data; tracking trajectories of the entities based on the video data to form two or more tracklets; deriving one or more associations between the entities by: detecting an event based on at least one spatio-temporal motion correlation between the entities; calculating a similarity measure of the closeness of the tracklets; identifying entity behaviors comprising at least one of spatial actions and behavioral action; performing pattern analysis to group the tracklets and sites; and merging an event ontology with hierarchical weighted graph matching to reduce candidate space wherein the candidate space comprises all entities to be tracked.