1. A position and orientation measurement apparatus comprising: an input unit configured to input an image captured by an imaging apparatus; a storage unit configured to store three-dimensional model data which represents a surface shape of an observation object; a geometric feature extraction unit configured to extract a plurality of geometric features based on geometric information about the observation object by drawing the three-dimensional model data; a reference image storage unit configured to store at least one reference image in which a position and orientation of the imaging apparatus relative to the observation object is known; a geometric feature selection unit configured to select from the plurality of geometric features a geometric feature that corresponds to an image feature in the reference image; and a calculation unit configured to calculate the position and orientation of the imaging apparatus relative to the observation object by associating the geometric feature selected by the geometric feature selection unit with an image of the observation object in the image input by the input unit.