1. A method comprising: receiving a set of three dimensional (3D) points estimated from a plurality of images; determining a contribution of each of the 3D points in the set to constraining each of a plurality of camera parameters; selecting a subset of the 3D points that, as a group, provide at least a desired amount of constraint on each of the plurality of camera parameters, the subset excluding one or more of the 3D points in the set; determining that each of the plurality of camera parameters are sufficiently constrained according to a threshold and that not enough of the 3D points have been selected to perform a non-linear optimization; updating the threshold to repeat said selecting the subset of the 3D points to sufficiently constrain each of the plurality of camera parameters to the updated threshold; and performing the non-linear optimization over the selected subset of the 3D points to recover the plurality of camera parameters and the 3D points.