1. A method comprising: receiving a first set of image frames that are captured by a first set of camera modules at a particular time, wherein (1) the first set of camera modules includes a first camera module and a second camera module that have an overlapping field of view, (2) the first set of image frames includes a first image captured by the first camera module and a second image captured by the second camera module, and (3) the first image includes a first sub-image that overlaps with a second sub-image of the second image on the overlapping field of view; interpolating a first virtual camera between the first camera module and the second camera module; determining a first set of disparity maps between the first set of camera modules by: determining a first disparity map that maps disparity of pixels from the first sub-image to the second sub-image; determining a second disparity map that maps disparity of pixels from the second sub-image to the first sub-image; and for each corresponding pixel location in the overlapping field of view: determining similarity scores between a first pixel of the first sub-image at a first pixel location and second pixels of the second sub-image at second pixel locations, wherein a corresponding distance between the first pixel location and each of the second pixel locations is equal to a different disparity value; determining a highest similarity score from the similarity scores; determining a disparity value associated with the highest similarity score; and assigning the disparity value associated with the highest similarity score to the first pixel location in the first disparity map; generating, by a processor-based computing device programmed to perform the generating, a first virtual camera image associated with the particular time for the first virtual camera from the first set of image frames, the first virtual camera image being generated based on the first set of disparity maps and a position of the first virtual camera relative to the first camera module and the second camera module; and constructing a left panoramic image and a right panoramic image associated with the particular time from the first set of image frames captured by the first set of camera modules and the first virtual camera image of the first virtual camera.