1. A method for reconstructing scene features of an object in three-dimensional (3D) space, the method comprising: acquiring, using a camera, a first camera frame at a first camera position; extracting image features from the first camera frame; initializing a first set of 3D points from the extracted image features; calculating a position and variance for each 3D point in the first set of 3D points; acquiring, using the camera, a second camera frame at a second camera position; predicting a second set of 3D points having corresponding 3D positions and variances by converting the corresponding 3D positions and variances of the 3D points to the second camera position; projecting the predicted 3D positions and variances to an image plane of the second camera position to obtain two-dimensional (2D) predictions of the image features; measuring an innovation of the 2D predictions; and updating estimates of 3D points based on the measured innovation to reconstruct the scene features of the object in the 3D space.