1. An image processing apparatus for converting first image data into second image data, the image processing apparatus comprising: setting means for setting a user's viewpoint at a designated position; and converting means for converting the first image data into the second image data as viewed from the user's viewpoint, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays.