1. A position and orientation measurement apparatus for measuring a position and orientation of an image capturing apparatus, which captures an image of a measurement object, relative to the measurement object, said apparatus comprising: a configuration plane extraction unit configured to extract configuration planes of the measurement object based on held three-dimensional model data of the measurement object; a line segment extraction unit configured to extract measurement line segments used in detection of edges of a captured image from line segments which form the configuration planes of the three-dimensional model data, based on characteristics of adjacent configuration planes; a line segment projection unit configured to project the measurement line segments extracted by said line segment extraction unit onto the captured image, based on an estimated position and orientation of the image capturing apparatus; a selection unit configured to select visible measurement line segments which are visible at the estimated position and orientation from the measurement line segments projected onto the captured image, based on the configuration planes extracted by said configuration plane extraction unit; and a calculation unit configured to calculate a position and orientation of the image capturing apparatus relative to the measurement object based on the measurement line segments selected by said selection unit and corresponding edges of the captured image.