1. An autofocus system comprising: an imaging unit that takes a subject image formed by an optical system; an autofocus unit that performs focus adjustment of the optical system so as to bring into focus a subject in a predetermined AF area in a shot image taken by the imaging unit; a pattern registration unit that registers pattern data for specifying a subject; a tracking target detection unit that detects a tracking target subject among subjects in the image taken by the imaging unit, based on the pattern data registered in the pattern registration unit; and a tracking unit that moves the AF area in accordance with movement of the tracking target subject detected by the tracking target detection unit.