1. A defect detection apparatus for detecting defects of a pattern formed on a substrate, comprising: an image pickup part for picking up an image of a substrate to acquire a grayscale target image; a defect detector for detecting defects included in a plurality of predetermined inspection regions in said target image; and a defect limiting part for performing limitation of defects to be detected by said defect detector, on the basis of a defect detection condition which is determined for each inspection region on an area or a shape of defect, wherein said defect detector generates a grayscale defect image representing a defect, from said target image, and said defect limiting part comprises a filter circuit for applying a filter to said defect image with positioning the center of said filter at each pixel in said defect image, element values of elements in said filter being changed in accordance with a defect detection condition for an inspection region including said each pixel; and a binarization circuit for binarizing said defect image to which said filter is applied, to generate a defect region image representing regions of defects after limitation.