1. An apparatus for inspecting pattern defects, the apparatus comprising: an image acquisition unit which acquires an image of a specimen; a defect candidate extraction unit configured to perform a defect candidate extraction process by comparing a detected image signal with a reference image signal; and a defect detection unit configured to perform a defect detection process and a defect classification process based on a partial image containing a defect candidate that is extracted by said defect candidate extraction unit, wherein said image acquisition unit and said defect candidate extraction unit work asynchronously with each other, and said defect detection unit includes plural defect detectors, each of the plural defect detectors performs the defect detection process and the defect classification process in parallel based on partial images which are different with each other.