1. A method for aligning a target image to a reference image in the presence of lens distortion, the method comprising: receiving the target image and the reference image, each of the target image and the reference image being captured by a lens having lens distortion parameters; detecting a first plurality of visual features appearing in the target image at first image feature coordinates in a two-dimensional image space, and a corresponding second plurality of visual features appearing in the reference image at second image feature coordinates in the two-dimensional image space; transforming, based on the lens distortion parameters, the first and second image feature coordinates from the two-dimensional image space to a three-dimensional spherical space to generate respective first spherical feature coordinates and second spherical feature coordinates; applying, by a processor, a rotation to the target image in the three-dimensional spherical space to generate a rotated target image, the applying of the rotation comprising (i) aligning at least a subset of the first spherical feature coordinates to a corresponding subset of the second spherical feature coordinates, and (ii) determining the subset of the first spherical feature coordinates and the subset of the second spherical feature coordinates as background features in the target image and the reference image, respectively; and inverse transforming, based on the lens distortion parameters, the rotated target image to the two-dimensional image space; wherein the determining of the subset of the first spherical feature coordinates and the subset of the second spherical feature coordinates as background features comprises: determining feature tracks representing a distance along a spherical arc and a direction from each of the first spherical feature coordinates to a corresponding one of the second spherical feature coordinates; clustering the feature tracks by length to determine a plurality of feature track clusters; and determining from the plurality of feature track clusters, a background cluster of feature tracks corresponding to the background features.