1. A method of detecting a defect, comprising: illuminating a sample with a light through an illumination optical system; detecting light from the sample with an image detection section which is illuminated by the light, and forming an image by processing the detected light with an image editing section; extracting a defect candidate with a defect candidate extraction unit by comparing the image of the sample formed in the detecting operation with a reference image to extract a difference in brightness between corresponding portions and determining an inspection condition of a defect detection unit by using images including the image of the sample acquired in the detecting operation, a partial image including the extracted defect candidate and the reference image which corresponds to the partial image including the defect candidate, wherein the extracting and determining operations are effected, at least in part, by at least one of a hardware processor and circuitry.