1. An image processing apparatus for converting first image data into second image data comprising: setting means for setting a user's viewpoint at a designated position; converting means for converting intermediate data into second image data as viewed from the user's viewpoint, the intermediate data being image data based on pixel values corresponding to light rays incident on pixels of the first image data, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays; and interpolating means for interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value, wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.