1. An image processing apparatus, comprising: a binarization processing unit that performs a binarization process on the basis of existence of at least one of lightness and chroma for a candidate image of template matching; a labeling processing unit that performs a labeling process for the binarized candidate image and thereby extracts a pixel block in the binarized candidate image; an outline determining unit that performs sampling of a position in coordinates on an outermost periphery of the pixel block, and identifies a range from a centroid position of the pixel block and an angle from a predetermined direction as the position in coordinates; a storage device that stores reference outline data obtained by performing the binarization process, the labeling process, and the outline determination process for a reference image of template matching; a correlation determining unit that determines a correlation between (a) the range and the angle of the candidate image and (b) a range and an angle of a position on an outermost periphery of a pixel block in the reference image on the basis of the reference outline data, and excludes the candidate image from a candidate of the template matching in accordance with degree of the determined correlation; and a template matching unit that performs template matching between the reference image and the candidate image that is not excluded by the correlation determining unit and does not perform template matching between the reference image and the candidate image that is excluded by the correlation determining unit.