1. A computer-implemented method for generating a stitched image, comprising: accessing, by one or more computing devices, input images of portions of a scene, three dimensional (3D) points associated with the input images, and camera locations associated with the input images; fitting, by the one or more computing devices, a plurality of planes to the 3D points, wherein one or more respective planes are fitted to at least one of one or more foreground objects and one or more background objects in the scene, and wherein distances from the 3D points to the fitted planes are substantially minimized; determining, by the one or more computing devices, a virtual camera to view the scene through the camera locations; selecting, by the one or more computing devices, for respective one or more pixels of a stitched image, one of the input images and one of the fitted planes such that a distance is substantially minimized from the selected one of the fitted planes to a surface corresponding to the respective one or more pixels; and generating, by the one or more computing devices, the stitched image based on the selected one of the fitted planes.