1. A focus adjusting device comprising: a detecting unit configured to detect a focusing target from a captured image; a setting unit configured to set, in response to the detection of the focusing target by the detecting unit, a focus detection area for detecting a focusing condition of a focusing lens; and a focus adjusting unit configured to perform focus adjustment by moving the focusing lens based on an image of the focus detection area, wherein the setting unit is operable, when the focusing target is no longer detected after a first focus detection area has been set in response to the detection of the focusing target by the detecting unit, to set at least one second focus detection area in accordance with the position of the first focus detection area, wherein a focused state that is obtained in the focus detection area when the detecting unit is capable of detecting the focusing target is set as a reference value, and the focus adjusting unit is configured to drive the focusing lens on the basis of a comparison between the reference value and a focused state obtained at the first focus detection area and a comparison between the reference value and a focused state obtained at the second focus detection area.