1. A computer-implemented method for detecting motion in a sequence of video frames, the method comprising: identifying residual pixels from the sequence of video frames, to provide estimated indication of a moving object or a static three-dimensional (3-D) structure; sorting the identified residual pixels to isolate pixels that correspond to the moving object comprising: applying at least two geometric constraints to the identified residual pixels, wherein the at least two geometric constraints represent at least a variation in relative depth of the residual pixels; detecting a disparity of the residual pixels to the applied at least two geometric constraints; and based on the detected disparity, filtering the residual pixels to remove pixels that do not correspond to the moving object; and tracking the moving object in the video frames comprising: representing the detected disparity in a probabilistic likelihood model; accumulating the probabilistic likelihood model within a number of frames during the filtering; and based on the accumulated probabilistic likelihood model, extracting an optimal path of the moving object.