1. A method for solving for surface normals of objects in a video stream, comprising: sampling the video stream to generate a set of keyframes; selecting a plurality of image points in the set of keyframes; generating corresponding 3D feature points to the plurality of image points in a scene imaged by the set of keyframes using the plurality of image points in the set of keyframes; generating hypothesis surface normals corresponding to each of the 3D feature points; warping image pixel patches corresponding to 3D feature points in a first keyframe to a viewpoint of a second keyframe with a warping matrix computed from each of the hypothesis surface normals; scoring warping errors resulting from each hypothesis surface normal between the first and second keyframes; and discarding hypothesis surface normals that produce scores for warping errors between the first and the second keyframes above a predefined level.