1. A pattern inspecting method for inspecting a defect or defective candidate of patterns on a sample, comprising: a size defining step for measuring geometric distortion in an image of a standard sample by picking up the image of the standard sample while shifting a sampling position on the standard sample, beforehand, and for defining a size for which the measured geometric distortion is neglectable; an image picking-up step for picking up an image of a sample by shifting a sampling position on the sample; an image data obtaining step for obtaining a first image of the sample obtained by the image picking-up step and for obtaining a second image to be compared with the first image; an image dividing step for dividing the first image and the second image into images of a division unit having a size not greater than the size defined by the size defining step; a calculating step for comparing a divided image of the first image which is divided in the dividing step with a divided image of the second image which corresponds to the divided image of the first image, and for calculating a difference in gradation values between both of the divided images of the first and second images; and an extracting step for extracting the defect or the defect candidate of the sample in accordance with the calculated difference in the gradation values of both of the divided images obtained from the calculating step.