1. A pattern measuring method for acquiring an image of a pattern on a specimen and measuring the pattern on the image, the pattern measuring method comprising the steps of: acquiring an image of a reference pattern arranged at a position isolated from a measuring point on the specimen; and setting a reference position for the measurement based on a position of the reference pattern on the image and based on design data for a portion including the reference pattern and the measuring point.