1. A tracking apparatus comprising: a grouping setting unit which groups a plurality of focus detection areas in accordance with an in-focus state among the focus detection areas in image data including a tracking target; a tracking feature detection unit which detects a feature amount of the tracking target in areas of the groups grouped by the grouping setting unit; and a tracking unit which sets the focus detection area focused on the tracking target as a first tracking position, sets a part having the feature amount detected by the tracking feature detection unit as a second tracking position, and tracks the tracking target in accordance with the first or second tracking position depending on the number of the set groups.