1. An image processing apparatus comprising: a processor configured to: calculate a degree of similarity between a first pixel in a first image and each of a plurality of second pixels located in a horizontal-direction first range in a second image, extract a second pixel with a highest similarity degree, and calculate a disparity from the first pixel; calculate, only for the extracted second pixel, a degree of similarity between a plurality of third pixels existing in a vertical-direction second range that includes the extracted second pixel and the first pixel, in the second image, and extract a third pixel with a highest similarity degree; record in an amount-of-difference buffer a difference between a vertical-direction position of the extracted third pixel and the first pixel as an amount of difference of the extracted third pixel; determine positions of the second pixels for which a degree of similarity to the first pixel for which a disparity is calculated next is calculated, according to data of the amount-of-difference buffer, and extract pixel data from the second image and input the pixel data for calculating the degree of similarity between the first pixel in the first image and each of the plurality of second pixels located in the horizontal-direction first range in the second image according to the determined positions of the second pixels, wherein for each of the second pixels, the processor is configured to set, based on an amount of difference recorded in the amount-of-difference buffer, a position away from the position of the first pixel by the amount of difference as a vertical-direction position of the second pixel.