1. A defect detection apparatus for detecting defects of a pattern formed on a substrate, comprising: an image pickup part for picking up an image of a substrate to acquire a grayscale target image; a defect detector for detecting defects included in a plurality of predetermined inspection regions in said target image; and a defect limiting part for performing limitation of defects to be detected by said defect detector, on the basis of a defect detection condition which is determined for each inspection region on an area or a shape of defect, wherein said defect detector generates a grayscale defect image representing a defect, from said target image, and said defect limiting part comprises a filter circuit for applying a filter to said defect image with positioning the center of said filter almost at each pixel in said defect image, said filter being set in accordance with a defect detection condition for an inspection region including said each pixel; and a binarization circuit for binarizing said defect image to which said filter is applied, to generate a defect region image representing regions of defects after limitation.