1. A position measuring system comprising: a first marker set having three or more markers that have certain positional relationship and being attached to a second area of an object which contains a first area having an acting portion and the second area continuous with the first area; an image-taking device having a two-dimensional image-taking element that takes a first image including a position of the first marker set and a position of an auxiliary tool in a state that the acting portion is disposed at the position of the auxiliary tool; a processing device that calculates coordinate of the acting portion based on the first image, an origin of the coordinate of the acting portion being a position of first marker set; and a recording device that records the coordinate of the acting portion calculated by the processing device; wherein the image-taking device takes a second image including the first marker in a state that the acting portion is not disposed at a position of the auxiliary tool, and the processing device calculates three-dimensional position of the acting portion by using position and angle of the first marker set determined on the basis of the second image and the coordinate of the acting portion previously recorded in the recording device.