1. A pattern measuring method for acquiring an image of a pattern on a specimen and measuring the pattern on the image, the pattern measuring method comprising the steps of: acquiring an image including a measurement pattern and a reference pattern arranged at a position isolated from the measurement pattern on the specimen; executing a pattern matching algorithm which matches the reference pattern on the image with design data of the specimen, as the reference pattern on the image and a part of the design data corresponding to the reference pattern are superimposed; and measuring a difference between the measurement pattern and a part of the design data corresponding to the measurement pattern based on the matched image.