1. A position measurement system comprising: an image capturing unit that captures first reference points which are three reference points on a plane arranged on a target and whose positional relation with each other is specified, and a second reference point which is a single reference point at a given distance away from the plane and whose positional relation with the first reference points is specified, wherein the second reference point is attached to the target; an identifying unit that identifies images of the first reference points and an image of the second reference point on the basis of the positional relation among the images of the four reference points captured by the image capturing unit; and a calculating unit that calculates a three-dimensional position and triaxial angles of the target on the basis of the positional relation between the images of the first reference points and the image of the second reference point identified by the identifying unit, wherein the calculating unit specifies the plane including the first reference points on the target from the images of the first reference points identified by the identifying unit, and specifies a normal direction of the plane according to the image of the second reference point identified by the identifying unit.