1. A focus adjusting device comprising: an edge detection unit that detects edges of a subject image for each color component forming an image including the subject image; a distribution detection unit that detects distributions of a focalized state and an unfocused state of the image based on the edges detected by the edge detection unit; and a control unit that moves a lens based on the distributions detected by the distribution detection unit, wherein the subject image is incident from an optical system having the lens, the control unit moves the lens, and thus the subject image is focused on.