1. A tracking apparatus comprising: a ranging position setting circuit which sets a standard area for tracking to a focus detection area focusing on a tracking object from among a plurality of detection areas in image data including the tracking object; a grouping setting circuit which groups the plurality of focus detection areas set by the ranging position setting circuit in accordance with focus information among the focus detection areas; a tracking feature detection circuit which detects a feature amount of a subject in areas of the groups grouped by the grouping setting circuit, the feature amount including at least one of luminance information, color information, and face information; and a tracking circuit which sets the standard area for tracking to a first candidate tracking position, sets a part having the feature amount detected by the tracking feature detection circuit as a second candidate tracking position, and tracks the tracking object from a plurality of frames of the image data in accordance with the first candidate tracking position or the second candidate tracking position depending on the number of the groups set by the group setting circuit; wherein the tracking circuit is configured to: set the second candidate tracking position to a position of the tracking object when there are no groups set by the grouping setting circuit; set the first candidate tracking position to a position of the tracking object and update the standard area to the first candidate tracking position when one group is set by the grouping setting circuit; set the first candidate tracking position included in the second candidate tracking position to a position of the tracking object and update the standard area to the first candidate tracking position included in the second candidate tracking position when two or more groups are set by the grouping setting circuit and the groups include a group including the second candidate tracking position; and set the second candidate tracking position to a position of the tracking object and update the standard area to the second candidate tracking position when two or more groups are set by the grouping setting circuit and none of the groups include the second candidate tracking position.