1. An image tracking apparatus, comprising: an imaging device that generates image information by capturing an image formed through an image forming optical system; a setting unit that sets, based upon the image information, first reference image information that includes image information corresponding to a part of an image to be designated as a target in the image information and also sets second reference image information based upon information related to an image assuming a position close to an image corresponding to the first reference image information; and a recognition unit that recognizes the target based upon the first reference image information and the second reference image information; and a focus detection device that detects a focusing condition of the image forming optical system in relation to the target, wherein: the setting unit determines the second reference image information based upon the focusing condition.