1. A subject tracking method, comprising: calculating a similarity factor indicating a level of similarity between an image contained in a search frame at each search frame position and a template image by shifting the search frame within a search target area set in each of individual frames of input images input in time sequence; determining a position of the search frame for which a highest similarity factor value has been calculated, within each input image to be a position (subject position) at which a subject is present; tracking the subject position thus determined through the individual frames of input images; calculating a difference between a highest similarity factor value and a second highest similarity factor value; and setting the search target area for a next frame based upon the calculated difference.