1. An image processing apparatus comprising: a storage unit configured to store defining data used to classify line segments which configure a virtual object corresponding to a physical object into a plurality of groups and to define each of the line segments; an image sensing unit configured to sense an image of the physical object; a selection unit configured to select one group from the plurality of groups one by one without any duplication; a projection unit configured to project line segments which belong to the group selected by said selection unit onto a physical space using the defining data used to define that line segments, and the image of the physical object; and an updating unit configured to update position and orientation information indicating a position and orientation of said image sensing unit based on the line segments projected onto the physical space and a feature in the image of the physical object.