1. A method of inspecting defects of a circuit pattern formed on a specimen comprising the steps of: taking images of a first area on the specimen on which the circuit pattern has been formed using an electron inspection apparatus and storing the images of the first area, processing the stored images of the first area using an image processing unit and setting inspection conditions and defect classification conditions, taking images successively in a second area wider than the first area on the specimen on which the circuit pattern has been formed and storing the images taken successively in the second area, processing the stored images taken successively in the second area to acquire a processing image including a defect judgment result acquired by taking successive images of the second area by using the inspection conditions and defect classification conditions which have been set, correcting the inspection conditions and the defect classification conditions to improve performance of detecting a defect of interest (DOI) based on the processing image including the defect judgment result, and inspecting successively the circuit patterns formed on the specimen and inspecting the defect of the circuit patterns by using the corrected inspection conditions and the corrected defect classification conditions.