1. A pattern inspection apparatus comprising: an optical image acquiring unit configured to acquire an optical image of pattern to be inspected on which a pattern is formed; a reference image generating unit configured to generate a reference image to be compared with the optical image; a normal matrix element calculating unit configured to calculate a plurality of elements of a normal matrix for a least-squares method for calculating a displacement amount displaced from a preliminary alignment position between the optical image and the reference image; a pattern type estimating unit configured to estimate a type of a pattern indicated by the reference image, by using some of the plurality of elements of the normal matrix; a displacement calculating unit configured to calculate the displacement amount displaced from the preliminary alignment position based on the least-squares method, by using a normal matrix obtained by deleting predetermined elements from the plurality of elements of the normal matrix depending upon the type of the pattern which has been estimated; a position correcting unit configured to correct an alignment position between the optical image and the reference image to a position displaced from the preliminary alignment position by the displacement amount; and a comparing unit configured to compare the optical image and the reference image whose alignment position has been corrected.