1. An image collation system, comprising: a three-dimensional data storing unit for storing three-dimensional data on at least a shape and a texture of a reference object; a first direction estimating unit for estimating a first imaging direction of the reference object that matches an imaging direction of a collation target object by comparing global characteristics between an image of the collation target object and the three-dimensional data of the reference object stored in the three-dimensional data storing unit; a second direction estimating unit for generating an image corresponding to the first imaging direction of the reference object based on the three-dimensional data, and estimating a second imaging direction of the reference object that matches the imaging direction of the collation target object by comparing local characteristics between the image of the collation target object and the generated image corresponding to the first imaging direction; and an image conformity determining unit for generating an image corresponding to the second imaging direction of the reference object based on the three-dimensional data, and determining whether the collation target object corresponds to the reference object based on whether the image of the collation target object matches the generated image corresponding to the second imaging direction.