1. A pattern inspection apparatus comprising: a stage configured to mount thereon a substrate with a pattern formed thereon and to be able to perform two-dimensional movement; a plurality of detectors of a two-dimensional scale, whose height positions are different from each other, configured to perform measurement and arranged at positions on the stage different from a position of the substrate; a scale body of the two-dimensional scale arranged fixed to a position facing the plurality of detectors; a sensor configured to acquire an optical image of the pattern formed on the substrate, in a state where the stage on which the substrate is placed is moving in one direction on a surface for the two-dimensional movement; a calculation processing circuitry configured to calculate an image acquiring position of the optical image by using position information measured by the two-dimensional scale; and a comparison processing circuitry configured to compare, using a reference image corresponding to the image acquiring position of the optical image, the optical image with the reference image for each pixel.