1. An auto focusing apparatus comprising: a detection unit which detects a focus signal indicating a focus state to an object in a focus detection area set within a frame of a photographed image signal; an object information acquisition unit which acquires one or a plurality of pieces of object information within the frame based on the image signal; an extraction unit which extracts a main object as an object to be focused from the object information obtained by the object information acquisition unit; a main area setting unit which sets, in the frame, a main area as one focus detection area within an area where the main object exists; an auxiliary area setting unit which sets, around the main area, auxiliary areas as a plurality of focus detection areas each of which is smaller than the main area; a focus detection area selection unit which selects one or a plurality of focus detection areas from the main area and the auxiliary areas in accordance with an in-focus position of a focus lens based on the focus signals of the main area and the auxiliary areas; and a control unit which controls the focus lens to move to the in-focus position of the focus lens in the focus detection area selected by the focus detection area selection unit.