1. A focus adjusting apparatus comprising: an image capturing unit which (1) generates an image signal by forming an optical image by an optical system having a focus lens and capturing the formed image in an image capturing range on an image sensor and (2) performs focusing based on a focusing detection signal regarding a focusing detection range specified within the image capturing range to record a portion of the image capturing range in enlargement; a storage unit which stores information regarding image magnification change of the optical system due to movement of the focus lens; and a wobbling drive control unit which controls a wobbling drive of the focus lens based on information regarding a position of the portion of the image capturing range in enlargement and the image magnification change.