1. A subject tracking device comprising: a processor; and a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to: detect at least one subject candidate area in which it is probable that a tracking target subject appears on a first image that is captured by an imaging device, the tracking target subject having been detected in a second image captured by the imaging device before the first image; calculate a degree of motion blur for each of the at least one subject candidate area, wherein each of the calculated degree of motion blur is represented as an index value; determine a tracking target area in which the tracking target subject appears among from the at least one subject candidate area based on a difference of position between the at least one subject candidate area in the first image and a position of the tracking target subject in the second image and based on a moving speed determined in accordance with the calculated degree of motion blur; and decide a movement of the tracking target subject depending on a movement direction from an area in which the tracking target subject appears on the second image to the determined tracking target area.