1. A pattern inspection apparatus comprising: an electron scanning unit configured to scan a surface of a sample with an electron beam and to detect secondary electrons emitted from the surface of the sample; a signal processor configured to create a SEM image on the basis of a relation between irradiation position with the amount of the secondary electrons; a storage configured to store distortion amount data representing a magnitude of positional displacement in a scanning direction of the electron beam, the positional displacement being caused by distortion of the SEM image; and a controller configured to match a design data with the SEM image by adjusting at least one of the design data and the SEM image on the basis of the distortion amount data.