1. A pattern inspection method comprising: sequentially imaging plural chips formed on a substrate; selecting at least one of pattern sections of each inspection image obtained by the imaging, which are suitable for calculating position gap between an inspection image of a subject chip and a reference image stored in a memory, while discarding other pattern sections which are not suitable for calculating position gap; calculating position gap between an inspection image of a chip obtained by the imaging and the reference image stored in the memory by using positional information of pattern images included in the inspection image and reference pattern images included in the reference image which are both corresponding to the at least one of pattern sections selected at the selecting; aligning the inspection image and the reference image by using information of the calculated position gap; and comparing the aligned inspection image with the reference image, and extracting a difference between the two images as a defect candidate, wherein in the selecting, the at least one of pattern sections which are suitable for calculating position gap and the other pattern sections which are not suitable for calculating position gap, are determined before firstly starting the calculating of position gap.