1. A computer-implemented method for tracking foreground objects depicted in a sequence of video frames captured by a video camera, the method comprising: identifying a first foreground patch in a current video frame of the sequence of video frames; determining a motion flow field for a first foreground patch relative to at least a previous video frame of the sequence of video frames; upon determining the motion flow field for the first foreground patch corresponds to a motion history of a first foreground object tracked in the sequence of video frames, associating the first foreground patch with the first foreground object; and upon determining the motion flow field for the first foreground patch corresponds to a motion history of the first foreground object and a motion history of least a second foreground object tracked in the sequence of video frames, computing a covariance matrix for the foreground patch in the current video frame and each of the first and second foreground objects as depicted in the previous video frame.