1. An apparatus for inspecting pattern defects, the apparatus comprising: an image acquisition unit which acquires an image of a specimen and stores the acquired image in an image memory; a defect candidate extraction unit which performs a defect candidate extraction process by using the acquired image, which is read from the image memory; and a defect detection unit which performs a defect detection process based on a partial image containing a defect candidate that is extracted by the defect candidate extraction unit, wherein the defect detection process performed by the defect detection unit is performed asynchronously with an image acquisition process that is performed by the image acquisition unit.