1. An object tracking device that tracks an object included in a frame image, the object tracking device comprising: a storage unit configured to store a template image of an object to be a track target; an image input unit configured to receive a frame image in a time series; a search region setting unit configured to set a search region of the object in a frame image at a present point in time, when a zoom scaling factor of the frame image at the present point in time is different from a zoom scaling factor of a frame image at a previous point in time, based on an object region in the frame image at the previous point in time, zoom center coordinates in the frame image at the previous point in time, and a ratio between the zoom scaling factor of the frame image at the previous point in time and the zoom scaling factor of the frame image at the present point in time; a normalizing unit configured to normalize an image of the search region of the object included in the frame image at the present point in time to a fixed size; and a matching unit configured to search the normalized image of the search region for an object region similar to the template image.