1. An augmented or virtual reality system, comprising: a set of map points captured in a plurality of existing keyframes from a real world by a set of augmented or virtual reality display systems; a data structure to receive or store the set of map points captured from the real world; a processor communicatively coupled to the data structure to construct or update a geometric map of the real world based at least in part on the set of map points, wherein a node in the geometric map comprises a keyframe that captures at least a subset of map points of the set of map points, and a strength of a connection between two nodes in the geometric map corresponds to a number of map points shared between the two nodes; the processor further configured to generate or update a simplified map corresponding to the geometric map at least by representing the real world with a plurality of point nodes corresponding to the plurality of existing keyframes and connecting the plurality of point nodes with a set of edges; the processor further configured to determine and position a virtual keyframe in relation to a normal direction of at least one existing keyframe of the plurality of existing keyframes, and a head worn or mounted display coupled to the processor, comprising a waveguide, and configured to project light beams to at least one eye of a user based at least in part upon the geometric map, the simplified map, and the virtual keyframe that is determined and positioned in relation to a normal direction of at least one existing keyframe of the plurality of existing keyframes.