1. A method for solving for surface normals of objects in a video stream, comprising: sampling the video stream to generate a set of keyframes; selecting corresponding image points in the set of keyframes; generating 3D feature points in a scene imaged by the set of keyframes using the corresponding image points in two or more keyframes; generating hypothesis surface normals for the 3D feature points; warping image pixel patches corresponding to 3D feature points in a first keyframe to the viewpoint of a second keyframe with a warping matrix computed from each of the hypothesis surface normals; scoring warping errors resulting from each hypothesis surface normal between the first and second keyframes; and discarding hypothesis surface normals that produce high warping errors between the first and second keyframes.