1. An object tracking apparatus that tracks an object on image data captured continuously, comprising: a tracked object designation receiving unit that receives designation of an object to be tracked on image data; an object color setting unit that sets, as an object color, a color of the designated object to be tracked on image data; an object color adjusting unit that calculates tendency of color change in regions on image data and adjusts the set object color based on the calculated tendency of color change to obtain a reference color; wherein the object color adjusting unit removes color change in an unnecessary region from the color change in the regions in the calculation for overcoming appearance change of the object; and a particle filter processing unit that estimates a region corresponding to the designated object to be tracked on image data based on likelihood of each particle calculated by comparing a color around each particle with the obtained reference color, using particles which move on image data according to a predefined rule.