1. An image tracking device, comprising: an image pickup device that repeatedly takes a photographic subject image which is formed by an optical system and outputs images; a defocus amount calculation device that repeatedly calculates defocus amounts for a plurality of focus detection areas that are set within a photographic scene by the optical system; a tracking subject setting device that sets a photographic subject region as a tracking subject image which is positioned at one of the focus detection areas in a first image output by the image pickup device; a tracking device that detects a position of the tracking subject image in a second image output by the image pickup device by comparing a reference image, which is image information relating to the tracking subject image set by the tracking subject setting device with image information of the second image output by the image pickup device; a defocus amount determination device that determines a second defocus amount as an employed defocus amount, if a first difference, between a first defocus amount for the focus detection area positioned at the tracking subject image which was set by the tracking subject setting device and the second defocus amount for the focus detection area positioned at the tracking subject image within the second image, which was detected by the tracking device, is within a predetermined value or less; and a focus adjustment device that performs a focus adjustment of the optical system based upon the employed defocus amount, wherein if the first difference is greater than or equal to the predetermined value, the defocus amount determination device, selects one defocus amount as a third defocus amount among the defocus amounts for focus detection areas positioned at peripheral areas to the focus detection area for which the second defocus amount was detected, which has the smallest difference from the first defocus amount as a second difference; and determines the third defocus amount as an employed defocus amount if the second difference is within the predetermined value or less.