1. A computer-implemented method for tracking a detected foreground patch that depicts a foreground object in a sequence of video frames captured by a video camera, the method comprising: receiving a video frame in the sequence of video frame; detecting a foreground patch that depicts a foreground object in the video frame to produce a detected first foreground patch; receiving a motion flow field for the video frame that includes an angle and magnitude value for the first detected foreground patch in the video frame relative to the first detected foreground patch in a previous video frame; categorizing the first detected foreground patch as being associated with an existing tracked foreground patch or as a discovered foreground patch based on a combination of the motion flow field and features of the first detected foreground patch; and updating tracking data that is used to track the foreground object in the sequence of video frames.