1. A computer-implemented method for generating and storing a compact representation of a light field, comprising: receiving the light field captured as a plurality of images depicting a scene, wherein the light field is one of a three-dimensional (3D) light field and a four-dimensional (4D) light field; receiving depth estimates of points in the scene; determining an error between a reconstruction from the depth estimates and the received images; and storing, in computer storage hardware, the compact representation of the light field which includes the depth estimates and the determined error, wherein, when the light field is the 3D light field, the depth estimates include slopes of lines at pixels of epipolar-plane images (EPIs) generated from the plurality of images, the reconstruction includes one or more EPIs generated based on the slopes, and the error is an error between the EPIs generated from the plurality of images and the one or more reconstructed EPIs, and wherein, when the light field is the 4D light field, the depth estimates include planes passing through pixels of 3D epipolar (plane) volumes generated from the plurality of images, the reconstruction includes one or more 3D epipolar (plane) volumes generated based on the planes, and the error is an error between the 3D epipolar (plane) volumes generated from the plurality of images and the one or more reconstructed 3D epipolar (plane) volumes.