1. A pattern inspection apparatus comprising: an irradiator configured to irradiate a sample with an electron beam; an electron detector configured to detect an amount of electrons generated on the sample by the irradiation of the electron beam, the sample having a pattern formed thereon; an image processor configured to generate a SEM image of the pattern on the basis of the amount of the electrons; and a controller configured to acquire defect position information on the pattern formed on the sample from an optical defect inspection device, wherein the controller specifies a defect candidate pattern from the SEM image on the basis of the defect position information and judges whether or not a defect in the defect candidate pattern is to be transferred onto a wafer; and wherein the controller determines a view field of the SEM image and sets an average value of areas of a plurality of similar patterns in the view field as a threshold and specifies a pattern having an area least different from the threshold as a reference pattern.