1. An image tracking device, comprising: an image sensor that captures a first image and a second image captured after the first image; a storage unit that stores a partial image corresponding to a tracking target within the first image as a reference image; a setting unit that sets a threshold value that is used to make a decision of similarity between a partial image of the second image and the reference image; and a decision making unit that makes a decision that the partial image and the reference image are similar to each other when a difference between the partial image and the reference image is smaller than the threshold value, and makes a decision that the partial image and the reference image are not similar to each other when the difference between the partial image and the reference image is not smaller than the threshold value, wherein the setting unit sets the threshold value larger when a color indicated in the reference image is a first color than when the color indicated in the reference image is a second color.