1. An apparatus for inspecting pattern defects, the apparatus comprising: an image acquisition means which picks up an image of an inspection target, acquires a detected image signal and a reference image signal, and stores the acquired signals in an image memory; a defect candidate extraction unit which performs a defect candidate extraction process by comparing the detected image signal with the reference image signal, which are read from said image memory; and a defect detection unit which performs a defect detection process and a defect classification process based on a partial image containing a defect candidate that is extracted by said defect candidate extraction unit, wherein the processes performed by said defect candidate extraction unit and/or said defect detection unit is performed asynchronously with an image acquisition process that is performed by said image acquisition means.