1. An imaging system comprising: an image capture device comprising a lens and an image sensor, the lens configured to direct light from an environment surrounding the image capture device to the image sensor, the image sensor configured to: sequentially capture a first plurality of image segments of an image based on the light from the environment, the image representing a field of view (FOV) of the image capture device, the FOV comprising a portion of the environment and including a plurality of sparse points, and sequentially capture a second plurality of image segments, the second plurality of image segments captured after the first plurality of image segments and forming at least another portion of the image; non-transitory data storage configured to sequentially receive the first and second plurality of image segments from the image sensor and store instructions for estimating at least one of a position and orientation of the image capture device within the environment; and at least one hardware processor operably coupled to the non-transitory data storage and configured by the instructions to: identify a first group of sparse points based in part on a corresponding subset of the first plurality of image segments, the first group of sparse points identified as the first plurality of image segments are received at the non-transitory data storage, determine at least one of the position and orientation of the imaging device within the environment based on the first group of sparse points, identify a second group of sparse points based in part on a corresponding subset of the second plurality of image segments, the second group of sparse points identified as the second plurality of image segments are received at the non-transitory data storage, and update the at least one of the position and orientation of the imaging device within the environment based on the first and second group of sparse points.