1. A tracking apparatus comprising: a group setting circuit which groups a plurality of focus detection areas in image data including a tracking target; a feature detection circuit which detects a feature amount of the tracking target in areas of one or more of the plurality groups grouped by the group setting circuit; and a tracking circuit which: sets the focus detection area focused on the tracking target as a first tracking position, sets an area having the feature amount detected by the tracking feature detection unit as a second tracking position, performs a priority tracking position determination to set the first tracking position or the second tracking position as a priority tracking position based on the number of groups set by the group setting circuit, and tracks the tracking target in accordance with the first or second tracking position based on the priority tracking position determination.