1. A pattern defect inspection apparatus comprising: image pickup means for picking up image data of an object to be inspected to generate inspection image data, the object having at least one edge; reference image generation means for generating reference image data representing a reference image for comparing with the inspection image data of the object; edge direction detection means for detecting an edge direction at an image point in the reference image data, the image point being arranged on any point of the reference image; first edge direction differential means for executing differential processing at an image point in the inspection image data comparable to the image point in the reference image data using a differential operator for the edge direction detected by the edge direction detection means and outputting an absolute value of each differential datum obtained by the differential processing; second edge direction differential means for executing differential processing at the image point in the reference image data using the differential operator for the edge direction and outputting an absolute value of each differential datum obtained by the differential processing; differential processing means for executing differential processing at the image point and in pixels surrounding the image point in the reference image data using the differential operator for the edge direction and outputting differential data, and for executing differential processing at the image point in the reference image data using multiple differential operators whose differential directions are the edge direction, the clockwise adjacent direction of the edge direction, and the counterclockwise adjacent direction of the edge direction to provide differential data; means for calculating the maximum absolute value from the differential data of the image point and of pixels surrounding the image point, and calculating a maximum absolute value from the differential data of the image point; and discrimination means for detecting a pattern defect of the object to be inspected on the basis of a difference between differential data obtained by said first edge direction differential means, and differential data obtained by said second edge direction differential means or one of the differential data obtained by said differential processing means.