1. A method for recognizing objects with a virtual or augmented reality display system, comprising: identifying map points in a world model that is projected to at least one eye of a user by a projector in a virtual or augmented reality display system, wherein the map points corresponds to one or more real objects and is used to determine a geometric structure of a location of the user; recognizing, using at least one microprocessor, a real object of the one or more real objects in the world model presented by the virtual or augmented reality display system to the user at least by deriving a geometric structure of the real object using an image segmentation technique based in part or in whole upon the map points; and displaying one or more virtual objects to the user relative to the real object, based at least in part on a location of the user.