1. An image tracking device, comprising: an image pickup device that takes an image formed by an imaging optical system and outputs image information; a focus detector that detects a focusing state of the imaging optical system; a recognizer that stores reference information related to a reference image and recognizes a position of an image corresponding to the reference information, within an image corresponding to the image information; and a controller that performs first focus detection in which the focusing state is detected by the focus detector, before the position is recognized by the recognizer, and second focus detection in which the focusing state is detected for the position that has been recognized by the recognizer.