1. An apparatus for inspecting pattern defects, the apparatus comprising: a defect candidate extraction unit configured to perform a defect candidate extraction process by comparing a detected image signal with a reference image signal; and a defect detection unit configured to perform a defect detection process and a defect classification process based on a partial image containing a defect candidate that is extracted by said defect candidate extraction unit, wherein the processes performed by said defect candidate extraction unit and/or said defect detection unit are performed asynchronously with an image acquisition process.