1. A position measuring system comprising: a first marker set having three or more markers that have certain positional relationship and being attached to a second area of an object which contains a first area having an acting portion and the second area continuous with the first area; an image-taking device having a two-dimensional image-taking element that takes an image of the first marker set; a recording device that records coordinate of the acting portion with respect to the first marker set; and a processing device that calculates three-dimensional position of the acting portion by using position and angle of the first marker set determined on the basis of the image of the first marker set taken by the image-taking device and the coordinate of the acting portion with respect to the first marker set that is recorded in the recording device.