1. An image processing apparatus, comprising: at least one processor; and a memory having instructions stored thereon executed by the at least one processor to perform: generating an output view image by image warping based on a reference view image and binocular disparity information of the reference view image; expanding a hole generated in the output view image by regarding a buffer region adjacent to the hole as the hole, wherein pixels having a distance from an outermost pixel of the hole that is less than a threshold are set as the buffer region; and a neighboring image-based hole restoration processor configured to restore the hole using one or more temporally neighboring images adjacent to the reference view image or the output view image, wherein the restoring comprises: restoring a color value of a pixel in the hole, using color values of pixels included in the one or more temporally neighboring images and corresponding to the pixel in the hole, and binocular disparity values of the pixels included in the one or more temporally neighboring images are less than a threshold; and restoring the color value of the pixel in the hole, using a color value of a first pixel in the buffer region when the pixel is included in the buffer region of the output view image.