1. An image tracking apparatus, comprising: an image capturing unit that generates first image information corresponding to an image at a first time by capturing an image formed through an image forming optical system and second image information corresponding to an image at a second time after the first time; a setting unit that sets (1) first tracking image information that includes image information corresponding to a part of an image to be designated as a tracking target, among the first image information, (2) second tracking image information based upon information of an image positioned close to the image corresponding to the first tracking image information, among the first image information, (3) third tracking image information that includes image information corresponding to a part of an image to be designated as a tracking target, among the second image information, and (4) fourth tracking image information based upon information of an image positioned close to the image corresponding to the third tracking image information, among the second image information; and a recognition unit that recognizes the tracking target using the second tracking image information and the fourth tracking image information.