1. A system for estimating ego-motion from video scenes, the system comprising: one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of: for consecutive frames in a video captured by a moving camera, determining a first ego-translation estimate between the consecutive frames from a first local minimum (h 0 ,v 0 ); determining a second ego-translation estimate from a second local minimum (h′ 0 ,v′ 0 ); if the second ego-translation estimate does not equal the first ego-translation estimate, determining an optimal solution by minimizing a cost function until the first ego-translation estimate and the second ego-translation estimate are a matching estimate; estimating ego-motion of the camera using the optimal solution; and detecting independent moving objects in the video after compensating for ego-motion of the camera.