TECHNICAL FIELD

This application relates generally to a cross reality system.

BACKGROUND

Computers may control human user interfaces to create an X Reality (XR or cross reality) environment in which some or all of the XR environment, as perceived by the user, is generated by the computer. These XR environments may be virtual reality (VR), augmented reality (AR), and mixed reality (MR) environments, in which some or all of an XR environment may be generated by computers using, in part, data that describes the environment. This data may describe, for example, virtual objects that may be rendered in a way that users' sense or perceive as a part of a physical world and can interact with the virtual objects. The user may experience these virtual objects as a result of the data being rendered and presented through a user interface device, such as, for example, a head-mounted display device. The data may be displayed to the user to see, or may control audio that is played for the user to hear, or may control a tactile (or haptic) interface, enabling the user to experience touch sensations that the user senses or perceives as feeling the virtual object.

XR systems may be useful for many applications, spanning the fields of scientific visualization, medical training, engineering design and prototyping, tele-manipulation and tele-presence, and personal entertainment. AR and MR, in contrast to VR, include one or more virtual objects in relation to real objects of the physical world. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment in using the XR system, and also opens the door for a variety of applications that present realistic and readily understandable information about how the physical world might be altered.

An XR system may represent the physical world around a user of the system as a “mesh.” A mesh may be represented by multiple, interconnected triangles. Each triangle has edges joining points on a surface of an object within the physical world, such that each triangle represents a portion of the surface. Information about the portion of the surface, such as color, texture or other properties may be stored in associate within the triangle. In operation, an XR system may process image information to detect points that and surfaces so as to create or update the mesh.

BRIEF SUMMARY

Aspects of the present application relate to methods and apparatus for providing X reality (cross reality or XR) scenes. Techniques as described herein may be used together, separately, or in any suitable combination.

Some embodiments relate to an XR system including a first XR device that includes a first processor, a first computer-readable medium connected to the first processor, a first origin coordinate frame stored on the first computer-readable medium, a first destination coordinate frame stored on the computer-readable medium, a first data channel to receive data representing local content, a first coordinate frame transformer executable by the first processor to transform a positioning of the local content from the first origin coordinate frame to the first destination coordinate frame, and a first display system adapted to display the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.

Some embodiments relate to a viewing method including storing a first origin coordinate frame, storing a first destination coordinate frame, receiving data representing local content, transforming a positioning of local content from the first origin coordinate frame to the first destination coordinate frame, and displaying the local content to a first user after transforming the positioning of the local content from the first origin coordinate frame to the first destination coordinate frame.

Some embodiments relate to an XR system including a map storing routine to store a first map, being a canonical map, having a plurality of anchors, each anchor of the first map having a set of coordinates, a real object detection device positioned to detect locations of real objects, an anchor identification system connected to the real object detection device to detect, based on the locations of the real objects, anchors of a second map, each anchor of the second map having a set of coordinates and a localization module connected to the canonical map and the second map and executable to localize the second map to the canonical map by matching a first anchor of the second map to a first anchor of the canonical map and matching a second anchor of the second map to a second anchor of the canonical map.

Some embodiments relate to a viewing method including storing a first map, being a canonical map, having a plurality of anchors, each anchor of the canonical map having a set of coordinates, detecting locations of real objects, detecting, based on the locations of the real objects, anchors of a second map, each anchor of the second map having a set of coordinates and localizing the second map to the canonical map by matching a first anchor of the second map to a first anchor of the first map and matching a second anchor of the second map to a second anchor of the canonical map.

Some embodiments relate to an XR system including a server that may have a processor, a computer-readable medium connected to the processor, a plurality of canonical maps on the computer-readable medium, a respective canonical map identifier on the computer-readable medium associated with each respective canonical map, the canonical map identifiers differing from one another to uniquely identify the canonical maps, a position detector on the computer-readable medium and executable by the processor to receive and store a position identifier from an XR device, a first filter on the computer-readable medium and executable by the processor to compare the position identifier with the canonical map identifiers to determine one or more canonical maps that form a first filtered selection, and a map transmitter on the computer-readable medium and executable by the processor to transmit one or more of the canonical maps to the XR device based on the first filtered selection.

Some embodiments relate to a viewing method including storing a plurality of canonical maps on a computer-readable medium, each canonical map having a respective canonical map identifier associated with the respective canonical map, the canonical map identifiers differing from one another to uniquely identify the canonical maps, receiving and storing, with a processor connected to the computer-readable medium, a position identifier from an XR device, comparing, with the processor, the position identifier with the canonical map identifiers to determine one or more canonical maps that form a first filtered selection, and transmitting, with the processor, a plurality of the canonical maps to the XR device based on the first filtered selection.

Some embodiments relate to an XR system including a processor, a computer readable medium connected to the processor, a multilayer perception unit, on the computer readable medium and, executable by the processor, that receives a plurality of features of an image and converts each feature to a respective string of numbers, and a max pool unit, on the computer-readable medium and executable by the processor, that combines a maximum value of each string of numbers into a global feature string representing the image.

Some embodiments relate to a viewing method, including receiving, with a processor, a plurality of features of an image, converting, with the processor, each feature to a respective string of numbers, and combining, with the processor, a maximum value of each string of numbers into a global feature string representing the image.

Some embodiments relate to a method of operating a computing system to identify one or more environment maps stored in a database to merge with a tracking map computed based on sensor data collected by a device worn by a user, wherein the device received signals of access points to computer networks while computing the tracking map, the method including determining at least one area attribute of the tracking map based on characteristics of communications with the access points, determining a geographic location of the tracking map based on the at least one area attribute, identifying a set of environment maps stored in the database corresponding to the determined geographic location, filtering the set of environment maps based on similarity of one or more identifiers of network access points associated with the tracking map and the environment maps of the set of environment maps, filtering the set of environment maps based on similarity of metrics representing contents of the tracking map and the environment maps of the set of environment maps, and filtering the set of environment maps based on degree of match between a portion of the tracking map and portions of the environment maps of the set of environment maps.

Some embodiments relate to a cloud computing environment for an augmented reality system configured for communication with a plurality of user devices comprising sensors, including a user database storing area identities indicating areas that the plurality of user devices were used in, the area identities comprising parameters of wireless networks detected by the user devices when in use, a map database storing a plurality of environment maps constructed from data supplied by the plurality of user devices and associated metadata, the associated metadata comprising area identities derived from area identities of the plurality of user devices that supplied data from which the maps were constructed, the area identities comprising parameters of wireless networks detected by the user devices that supplied data from which the maps were constructed, non-transitory computer storage media storing computer-executable instructions that, when executed by at least one processor in the cloud computing environment, receives messages from the plurality of user devices comprising parameters of wireless networks detected by the user devices, computes area identifiers for the user devices and updates the user database based on the received parameters and/or the computed area identifiers, and receives requests for environment maps from the plurality of user devices, determines area identifiers associated with the user devices requesting environment maps, identifies sets of environment maps from the map database based, at least in part, on the area identifiers, filters the sets of environment maps, and transmits the filtered sets of environment maps to the user devices, wherein filtering a set of environment maps is based on similarity of parameters of wireless networks detected by a user device from which the request for environment maps originated to parameters of wireless networks in the map database for the environment maps in the set of environment maps.

Some embodiments relate to an XR system including a real object detection device to detect a plurality of surfaces of real-world objects, an anchor identification system connected to the real object detection device to generate a map based on the real-world objects, a persistent coordinate frame (PCF) generation system to generate a first PCF based on the map and associate the first PCF with the map, first and second storage mediums on first and second XR devices, respectively, and at least first and second processors of the first and second XR devices, to store the first PCF in first and second storage mediums of the first and second XR devices respectively.

Some embodiments relate to a viewing method including detecting, with at least one detection device a plurality of surfaces of real-world objects, generating, with at least one processor, a map based on the real-world objects, generating, with at least one processor, a first PCF based on the map, associating, with the at least one processor, the first PCF with the map, and storing, with at least first and second processors of first and second XR devices, the first PCF in first and second storage mediums of the first and second XR devices respectively.

Some embodiments relate to an XR system including a first XR device that may include a first XR device processor, a first XR device storage device connected to the first XR device processor, a set of instructions on the first XR device processor, including a download system, executable by the first XR device processor, to download persistent poses from a server, a PCF retriever, executable by the first XR device processor, to retrieve PCF's from the first storage device of the first XR device based on the persistent poses, and a coordinate frame calculator, executable by the first XR device processor, to calculate a coordinate frame based on the PCF's retrieved from the first storage device.

Some embodiments relate to a viewing method including downloading, with a first XR device processor of a first XR device, persistent poses from a server, retrieving, with the first XR device processor, PCF's from the first storage device of the first XR device based on the persistent poses, and calculating, with the first XR device processor, a coordinate frame based on the PCF's retrieved from the first storage device.

Some embodiments relate to a viewing method including a server that may include a server processor, a server storage device connected to the server processor, a map storing routine storing, executable with a server processor of the server, to store the first PCF in association with a map on the server storage device of the server, and a map transmitter, with the server processor, executable with a server processor, to transmit the map and the first PCF to a first XR device. The invention also provides a viewing method including storing, with a server processor of the server, the first PCF in association with a map on a server storage device of the server, and transmitting, with the server processor of the server, the map and the first PCF to a first XR device.

Some embodiments relate to a viewing method including entering, by a processer of a XR device, tracking of head pose by capturing surfaces of an environment with a capture device on a head-mounted frame secured to a head of a user and determining an orientation of the head-mounted frame relative to the surfaces, determining, by the processor, whether head pose is lost due to an inability to determine the orientation of the head-mounted frame relative to the surfaces, and if head pose is lost, then, by the processor, entering pose recovery mode to establish the head pose by determining an orientation of the head-mounted frame relative to the surfaces.

