1. An image processing apparatus comprising: an input unit configured to input a plurality of images obtained by capturing a target object from different viewpoints; a first detection unit configured to detect a plurality of line segments inherent to the target object and which forms the target object from each of the plurality of input images; a setting unit configured to set, for each of the plurality of detected line segments, a reference line which intersects with the line segment; a second detection unit configured to detect an edge on the reference line on the basis of a pixel value change pattern on the reference line; an array derivation unit configured to obtain a pattern array by aligning patterns, each of which is a pixel value change pattern of pixels in the vicinity of the edge; and a decision unit configured to decide association of the detected line segments between the plurality of images by comparing the patterns, contained in the obtained pattern array, between the plurality of images.