1. A vision-aided inertial navigation system (VINS) comprising: an image source to produce image data at a first set of time instances along a trajectory within a three-dimensional (3D) environment, wherein the image data captures feature observations within the 3D environment at each of the first time instances; an inertial measurement unit (IMU) to produce IMU data for the VINS along the trajectory at a second set of time instances that is misaligned in time with the first set of time instances, wherein the IMU data indicates a motion of the VINS along the trajectory; and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates for poses of the IMU at each of the first set of time instances and poses of the image source at each of the second set of time instances along the trajectory, wherein the estimator computes each of the poses for the image source as an interpolation from a subset of the poses for the IMU along the trajectory.