FIELD OF THE DISCLOSURE

The disclosure relates to systems and methods for displaying content in a spatial 3D environment.

BACKGROUND

A typical way to view content is to open an application that will display the content on a display screen (e.g., a monitor of a computer, smartphone, tablet, etc.) of a display device. A user would navigate the application to view the content. Normally, when the user is looking at the display screen of the display, there is a fixed format as to how the content is displayed within the application and on the display screen of the display device.

With virtual reality (VR), augmented reality (AR), and/or mixed reality (MR) systems (hereinafter collectively referred to as “mixed reality” systems), an application will display content in a spatial three-dimensional (3D) environment. Conventional approaches to display content on a display screen do not work very well when used in a spatial 3D environment. One reason is because, with conventional approaches, a display area of a display device is a 2D medium limited to a screen area of a display screen that the content is displayed on. As a result, conventional approaches are configured to only know how to organize and display the content within that screen area of the display screen. In contrast, the spatial 3D environment is not limited to the strict confines of the screen area of the display screen. Therefore, conventional approaches may perform sub-optimally when used in the spatial 3D environment since conventional approaches do not necessarily have the functionality or capability to take advantage of the spatial 3D environment for displaying the content.

Therefore, there is a need for an improved approach to display content in a spatial 3D environment.

The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem and the understanding of the causes of a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section may merely represent different approaches, which in and of themselves may also be disclosures.

SUMMARY

Embodiments of the disclosure provide an improved system and methods to display information in a spatially organized 3D environment. The method includes receiving content, identifying elements in the content, determining surrounding surfaces, matching the identified elements to the surrounding surfaces, and displaying the elements as virtual content onto the surrounding surfaces. Additional embodiments of present disclosure provide an improved system and methods to push content to a user of a virtual reality or augmented reality system.

In one embodiment, a method includes receiving content. The method also includes identifying one or more elements in the content. The method further includes determining one or more surfaces. Moreover, the method includes matching the one or more elements to the one or more surfaces. Additionally, the method includes displaying the one or more elements as virtual content onto the one or more surfaces.

In one or more embodiments, the content comprises at least one of a pulled content or pushed content. Identifying the one or more elements may include determining one or more attributes for each of the one or more elements. The one or more attributes include at least one of a priority attribute, an orientation attribute, an aspect ratio attribute, a dimension attribute, an area attribute, a relative viewing position attribute, a color attribute, a contrast attribute, a position type attribute, a margin attribute, a type of content attribute, a focus attribute, a readability index attribute, or a type of surface to place attribute. Determining the one or more attributes for each of the one or more elements is based on explicit indications in the content.

In one or more embodiments, determining the one or more attributes for each of the one or more elements is based on placement of the one or more elements within the content. The method further includes storing the one or more elements into one or more logical structures. The one or more logical structures comprise at least one of an ordered array, a hierarchical table, a tree structure, or a logical graph structure. The one or more surfaces comprise at least one of physical surfaces or virtual surfaces. Determining the one or more surfaces comprises parsing an environment to determine at least one of the one or more surfaces.

In one or more embodiments, determining the one or more surfaces includes receiving raw sensor data, simplifying the raw sensor data to produce simplified data, and creating one or more virtual surfaces based on the simplified data. The one or more surfaces comprise the one or more virtual surfaces. Simplifying the raw sensor data includes filtering the raw sensor data to produce filtered data, and grouping the filtered data into one or more groups by point cloud points. The simplified data includes the one or more groups. Creating the one or more virtual surfaces includes iterating through each of the one or more groups to determine one or more real world surfaces, and creating the one or more virtual surfaces based on the one or more real world surfaces.

In one or more embodiments, determining the one or more surfaces comprises determining one or more attributes for each of the one or more surfaces. The one or more attributes include at least one of a priority attribute, an orientation attribute, an aspect ratio attribute, a dimension attribute, an area attribute, a relative viewing position attribute, a color attribute, a contrast attribute, a position type attribute, a margin attribute, a type of content attribute, a focus attribute, a readability index attribute, or a type of surface to place attribute. The method also includes storing the one or more surfaces into one or more logical structures. Matching the one or more elements to the one or more surfaces includes prioritizing the one or more elements, for each element of the one or more elements, comparing one or more attributes of the element to one or more attributes of each of the one or more surfaces, calculating a match score based on the one or more attributes of the element and the one or more attributes of each of the one or more surfaces, and identifying a best matching surface having a highest match score. Additionally, for each of the one or more elements, storing an association between the element and the best matching surface.

In one or more embodiments, one element is matched to one or more surfaces. Furthermore, displaying each surface of the one or more surfaces to a user. In addition, receiving a user selection indicating a winning surface from the one or more surfaces displayed. Yet even furthermore, saving, from the user selection, surface attributes of the winning surface in a user preferences data structure. The content is data streamed from a content provider. The one or more elements are displayed to a user through a mixed reality device.

In one or more embodiments, the method further includes displaying one or more additional surface options for displaying the one or more elements based at least in part on a changed field of view of a user. The displaying of the one or more additional surface options is based at least in part on a time threshold corresponding to the changed field of view. the displaying of the one or more additional surface options is based at least in part on a headpose change threshold.

In one or more embodiments, the method also includes overriding displaying the one or more elements onto the one or more surfaces that were matched. Overriding the displaying of the one or more elements onto the one or more surfaces is based at least in part on historically frequently used surfaces. The method even further includes moving the one or more elements displayed on the one or more surfaces to a different surface based at least in part on a user selecting a particular element displayed at the one or more surfaces to be moved to the different surface. The particular element is moved to the different surface is at least viewable by the user.

In one or more embodiments, the method additionally includes in response to a change to a field of view of a user from a first field of view to a second field of view, lazily moving the displaying of the one or more elements onto new surfaces to follow the change of the user's field of view to the second field of view. The one or more elements may only move directly in front of the user's second field of view upon confirmation from received from the user to move the content directly in front of the user's second field of view. The method includes pausing displaying of the one or more elements onto the one or more surfaces at a first location and resuming displaying of the one or more elements onto one or more other surfaces at a second location is based at least in part on a user moving from the first location to the second location. The pausing of the displaying of the one or more elements is automatic based at least in part on a determination that the user is moving or has moved from the first location to the second location. the resuming of the displaying of the one or more elements is automatic based at least in part on an identification and matching of the one or more other surfaces to the one or more elements at the second location.

In one or more embodiments, determining the one or more surfaces comprises identifying one or more virtual objects for displaying the one or more elements. Identifying the one or more virtual objects is based at least in part on data received from one or more sensors indicating a lack of suitable surfaces. An element of the one or more elements is a TV channel. A user interacts with an element of the one or more elements displayed by making a purchase of one or more items or services displayed to the user.

In one or more embodiments, the method also includes detecting a change of environment from a first location to a second location, determining one or more additional surfaces at the second location, matching the one or more elements currently being displayed at the first location to the one or more additional surfaces, and displaying the one or more elements as virtual content onto the one or more additional surfaces at the second location. Determination of the one or more additional surfaces is initiated after the change of environment exceeds a temporal threshold. A user pauses active content displayed at the first location and resumes the active content to be displayed at the second location, the active content resuming at a same interaction point as where the user paused the active content at the first location.

In one or more embodiments, the method also includes transitioning spatialized audio delivered to a user from a location associated with displayed content at the first location to audio virtual speakers directed to a center of a head of the user as the user leaves the first location, and transitioning from the audio virtual speakers directed to the center of the head of the user to spatialized audio delivered to the user from the one or more additional surfaces displaying the one or more elements at the second location.

In another embodiment, a method for pushing content to a user of a mixed reality system includes receiving one or more available surfaces from an environment of a user. The method also includes identifying one or more contents that match a dimension of one available surface from the one or more available surfaces. The method further includes calculating a score based on comparing one or more constraints of the one or more contents to one or more surface constraints of the one available surface. Moreover, the method includes selecting a content from the one or more contents having the highest score. Yet furthermore, the method includes storing a one-to-one matching of the content selected to the one available surface. Furthermore, displaying on the available surface, to the user, the content selected.

In one or more embodiments, an environment of a user is a personal residence of the user. The one or more available surfaces from the environment of the user is peripheral to a focal view area of the user. The one or more contents are advertisements. The advertisements are targeted to a specific group of users located at a particular environment. The one or more contents are notifications from an application. The application is a social media application. One of the constraints of the one or more constraints of the one or more contents is an orientation. The content selected is a 3D content.

In another embodiment, an augmented reality (AR) display system includes a head-mounted system that includes one or more sensors, and one or more cameras comprising outward facing cameras. The system also includes a processor to execute a set of program code instructions. Moreover, the system includes a memory to hold the set of program code instructions, in which the set of program code instructions comprises program code to perform receiving content. The program code also performs identifying one or more elements in the content. Furthermore, the program code also performs determining one or more surfaces. In addition, the program code also performs matching the one or more elements to the one or more surfaces. Yet even further, the program code also performs displaying the one or more elements as virtual content onto the one or more surfaces.

In one or more embodiments, the content comprises at least one of a pulled content or pushed content. Identifying the one or more elements comprises parsing the content to identify the one or more elements. Identifying the one or more elements comprises determining one or more attributes for each of the one or more elements. In addition, the program code also performs storing the one or more elements into one or more logical structures. The one or more surfaces comprise at least one of physical surfaces or virtual surfaces. Determining the one or more surfaces comprises parsing an environment to determine at least one of the one or more surfaces.

In one or more embodiments, determining the one or more surfaces includes receiving raw sensor data, simplifying the raw sensor data to produce simplified data, and creating one or more virtual surfaces based on the simplified data, wherein the one or more surfaces comprise the one or more virtual surfaces. Determining the one or more surfaces comprises determining one or more attributes for each of the one or more surfaces. In addition, the program code also performs storing the one or more surfaces into one or more logical structures.

In one or more embodiments, matching the one or more elements to the one or more surfaces includes prioritizing the one or more elements, for each element of the one or more elements: comparing one or more attributes of the element to one or more attributes of each of the one or more surfaces, calculating a match score based on the one or more attributes of the element and the one or more attributes of each of the one or more surfaces, and identifying a best matching surface having a highest match score. One element is matched to one or more surfaces. The content is data streamed from a content provider.

In one or more embodiments, the program code also performs displaying one or more surface options for displaying the one or more elements based at least in part on a changed field of view of the user. The program code also performs overriding displaying the one or more elements onto the one or more surfaces that were matched. The program code also performs moving the one or more elements displayed on the one or more surfaces to a different surface based at least in part on a user selecting a particular element displayed at the one or more surfaces to be moved to the different surface. The program code also performs in response to a change to a field of view of a user from a first field of view to a second field of view, lazily moving the displaying of the one or more elements onto new surfaces to follow the change of the user's field of view to the second field of view.

In one or more embodiments, the program code also performs pausing displaying of the one or more elements onto the one or more surfaces at a first location and resuming displaying of the one or more elements onto one or more other surfaces at a second location is based at least in part on a user moving from the first location to the second location. Determining the one or more surfaces comprises identifying one or more virtual objects for displaying the one or more elements. An element of the one or more elements is a TV channel.

In one or more embodiments, the user interacts with an element of the one or more elements displayed by making a purchase of one or more items or services displayed to the user. The program code also performs detecting a change of environment from a first location to a second location, determining one or more additional surfaces at the second location, matching the one or more elements currently being displayed at the first location to the one or more additional surfaces, and displaying the one or more elements as virtual content onto the one or more additional surfaces at the second location.

In another embodiment, an augmented reality (AR) display system includes a head-mounted system includes one or more sensors, and one or more cameras comprising outward facing cameras. The system also includes a processor to execute a set of program code instructions. The system further includes a memory to hold the set of program code instructions, in which the set of program code instructions comprises program code to perform receiving one or more available surfaces from an environment of a user. The program code also performs identifying one or more contents that match a dimension of one available surface from the one or more available surfaces. The program code further performs calculating a score based on comparing one or more constraints of the one or more contents to one or more surface constraints of the one available surface. The program code additionally performs selecting a content from the one or more contents having the highest score. Moreover, the program code performs storing a one-to-one matching of the content selected to the one available surface. The program code also performs displaying on the available surface, to the user, the content selected.

In one or more embodiments, an environment of a user is a personal residence of the user. The one or more available surfaces from the environment of the user is peripheral to a focal view area of the user. The one or more contents are advertisements. The one or more contents are notifications from an application. One of the constraints of the one or more constraints of the one or more contents is an orientation. The content selected is a 3D content.

In another embodiment, a computer-implemented method for deconstructing 2D content includes identifying one or more elements in the content. The method also includes identifying one or more surrounding surfaces. The method further includes mapping the one or more elements to the one or more surrounding surfaces. Moreover, the method includes displaying the one or more elements as virtual content onto the one or more surfaces.

In one or more embodiments, the content is a web page. An element of the one or more elements is a video. The one or more surrounding surfaces comprise physical surfaces within a physical environment or virtual objects not physically located within the physical environment. A virtual object is a multi-stack virtual object. A first set of results of the one or more elements identified and a second set of results of the one or more surrounding surfaces identified are stored in database tables within a storage device. The storage device is a local storage device. A database table storing the results of the one or more surrounding surfaces identified, comprises: a surface id, a width dimension, a height dimension, an orientation description and a position relative to a frame of reference.

In one or more embodiments, identifying the one or more elements in the content includes identifying attributes from tags corresponding to placement of elements, extracting hints from the tags for the one or more elements, and storing the one or more elements. Identifying one or more surrounding surfaces includes identifying user current surrounding surfaces, determining a pose of a user, identifying dimensions of surrounding surfaces, and storing the one or more surrounding surfaces. Mapping the one or more elements to the one or more surrounding surfaces includes looking up pre-defined rules for identifying candidate surrounding surfaces for mapping, and choosing a best fit surface for each of the one or more elements. The displaying of the one or more elements onto the one or more surrounding surfaces is performed by an augmented reality device.

Each of the individual embodiments described and illustrated herein has discrete components and features that may be readily separated from or combined with the components and features of any of the other several embodiments.

Further details of features, objects, and advantages of the disclosure are described below in the detailed description, drawings, and claims. Both the foregoing general description and the following detailed description are exemplary and explanatory, and are not intended to be limiting as to the scope of the disclosure.

