1. A tracking apparatus for tracking a target object designated on an image which is captured by an image sensing element having a plurality of pixels, comprising: a setting unit configured to set a pixel region including a pixel corresponding to a position of the designated target object as a feature color extraction region used to extract a feature color of the target object, and to set colors of respective pixels included in the feature color extraction region as feature candidate colors; a calculation unit configured to calculate, for each of the feature candidate colors, a first area of a pixel group which includes a pixel of a feature candidate color of interest and in which pixels of colors similar to the feature candidate color of interest continuously appear, a second area of pixels of colors similar to the feature candidate color of interest in the plurality of pixels, and a ratio of the first area to the second area; an extraction unit configured to extract a feature candidate color having the smallest first area as a feature color of the target object from feature candidate colors for each of which the ratio of the first area to the second area is higher than a predetermined reference ratio; and a decision unit configured to decide, as the target object, a position corresponding to a pixel of a color similar to the feature color extracted by the extraction unit.