1. A method comprising: receiving, by an image manipulation application executed by a processing device, image data depicting an image space from a plurality of perspectives, wherein the image data comprises an input image having a plurality of super-pixels, each super-pixel comprising image data encoded from the plurality of perspectives; generating, by the image manipulation application, a three-dimensional epipolar data structure from the image data, the three-dimensional epipolar data structure comprising data describing a difference in position of an object between the plurality of perspectives, wherein the three-dimensional epipolar data structure corresponds to a region of the image space, wherein the generating the three-dimensional epipolar data structure comprises: determining first position data from a first subset of pixels of the plurality of super-pixels, wherein the first position data identifies a position of the object within a first plane representing a first dimension and a second dimension of the three-dimensional epipolar data structure, and determining second position data from a second subset of pixels of the plurality of super-pixels, wherein the second position data identifies a position of the object within a second plane representing the second dimension and a third dimension of the three-dimensional epipolar data structure; and generating, by the image manipulation application, a depth map based on the three-dimensional epipolar data structure.