1. An object tracking device that receives an image signal having a plurality of frame images and tracks a specific object in the image signal, comprising: a small area-setting unit configured to set a predetermined number of small areas in a reference area indicative of an area where an image of the object is formed in a preceding frame image preceding a current frame image; a motion vector detection unit configured to detect a motion vector of the object in each of the small areas; a shape change determination unit configured to determine a change of the object according to the motion vector to thereby obtain shape change information; an area correction unit configured to correct a location and a size of the reference area according to the shape change information to thereby correct the reference area to a corrected reference area; and a tracking unit configured to track the object using the corrected reference area.