1. An image processing apparatus for generating an image of an arbitrary viewpoint using a plurality of input images, wherein the arbitrary viewpoint is virtual viewpoint and different from each viewpoint corresponding to each of the plurality of input images, comprising: obtaining unit configured to obtain image information of a projection position at which a point along a first straight line passing through the arbitrary viewpoint and a pixel position on the image of the arbitrary viewpoint is projected onto each of the plurality of input images; first defining unit configured to define parallax information of the plurality of input images at the pixel position, using the image information obtained for the point along the first straight line by the obtaining unit; and generating unit configured to generate the image of the arbitrary viewpoint, by defining the image information of the pixel position from the plurality of input images using the parallax information for the pixel position, wherein the first defining unit calculates a matching evaluation value for image information corresponding to the point along the first line obtained by the obtaining unit, and wherein the first defining unit calculates the parallax information based on the matching evaluation value and a distance between the point corresponding to the matching evaluation value and the arbitrary viewpoint.