1. An image processing apparatus for generating an image of an arbitrary viewpoint using a plurality of input images, comprising: obtaining unit configured to obtain image information of a projection position at which a point along a first straight line passing through the arbitrary viewpoint and a pixel position on the image of the arbitrary viewpoint is projected onto each of the plurality of input images; first defining unit configured to define parallax information of the plurality of input images at the pixel position, using the image information obtained for the point along the first straight line by the obtaining unit; and generating unit configured to generate the image of the arbitrary viewpoint, by defining the image information of the pixel position from the plurality of input images using the parallax information for the pixel position.