FIELD OF THE INVENTION

The present disclosure relates to mixed reality systems with virtual content warping, and methods for generating a mixed reality experience including warped virtual content using same.

BACKGROUND

Modern computing and display technologies have facilitated the development of “mixed reality” (MR) systems for so called “virtual reality” (VR) or “augmented reality” (AR) experiences, wherein digitally reproduced images, or portions thereof, are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real-world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world visual input.

Various optical systems generate images at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014, the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.

MR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated to take the change in head pose (i.e., the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display device and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion. However, there is an inevitable lag between rendering a scene and displaying/projecting the rendered scene.

Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, a user can see through transparent (or semi-transparent) elements in a display system to view directly the light from real objects in an environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user's view of the real world, where light from the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.

Current optical systems, such as those in MR systems, optically render virtual content. Content is “virtual” in that it does not correspond to real physical objects located in respective positions in space. Instead, virtual content only exist in the brain (e.g., the optical centers) of a user of the head-worn display device when stimulated by light beams directed to the eyes of user.

MR systems attempt to present photo-realistic, immersive MR scenarios. However, lag time between generation of virtual content and display of the generated virtual content combined with head movement during the lag time can result in visual artifacts (e.g., glitches) in MR scenarios. This problem is exacerbated by rapid head movement during the lag time.

In order to address this issue, some optical systems may include a warping software/system that receives source virtual content from a source. The warping system then “warps” (i.e., transforms the frame of reference of) the received source virtual content for display in a frame of reference of the display system/viewer (the “display frame of reference”). Warping or transforming change the frame of reference from which virtual content is presented. This approach takes the originally rendered virtual content, and shifts the way that the virtual content is presented to attempt to display the virtual content from a different perspective.

Some warping software/systems warp the source virtual content in two processing passes. Warping systems warp all of the source subparts forming a 3-D scenario in the source virtual content in a first pass. The warping systems also perform depth testing in this first pass to generate depth data, but the depth testing is performed in the source frame of reference. The warping systems store all the warped subparts resulting from the transformation of the source subparts forming the 3-D scenario and their relative depths in the source frame of reference in that first pass (e.g., in a list).

During warping, two or more different subparts of a 3-D scenario may warp/project into (i.e., be assigned to) the same pixel of a final display image. These subparts are “conflicting,” and the warping system must resolve the conflict to generate a realistic 2-D display image.

After the first pass, some of the warped subparts may be conflicting relative to pixels of the final 2-D display image. The warping systems then perform a second pass through the intermediate warping data stored in the first pass to analyze the depth test data of conflicting warped subparts to identify the warped subparts closest to the viewing location in the output frame of reference. The conflicting warped subpart closest to the viewing location in the output frame of reference is used to generate a final 2-D display image. The remaining conflicting warped subparts are discarded. However, this multi-pass system for warping virtual content from a source may be computationally expensive (resulting in processor/memory related system limitations) and time-consuming (resulting in system latency).

SUMMARY

In one embodiment, a computer implemented method for warping virtual content includes generating warped virtual content by transforming source virtual content. The method also includes determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content. The method further includes storing the warped virtual content in the memory location if the memory location is not occupied by pre-existing virtual content. Moreover, the method includes comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location in the output frame of reference if the memory location is occupied by pre-existing virtual content. In addition, the method includes storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of the warped virtual content is closer to the viewing location than a pre-existing Z location of the pre-existing virtual content in the output frame of reference.

In one or more embodiments, the also includes discarding the warped virtual content if the pre-existing Z of the pre-existing virtual content is closer to the viewing location that the Z location of the warped virtual content in the output frame of reference. Generating the warped virtual content, determining whether the memory location is occupied, comparing respective Z locations if the memory location is occupied, and storing the warped virtual content in the memory may all occur in one pass.

In one or more embodiments, transforming the virtual content includes generate a pixel map by mapping pixels of an image. Transforming may also include dividing the pixel map into a plurality of primitives. Transforming may further include performing a transformation on one of the plurality of primitives into the output frame of reference. The virtual content may be the one of the plurality of primitives. Each of the plurality of primitives may be a quadrilateral, a triangle, and/or a pixel. The transformation may be a Fourier transformation.

In one or more embodiments, the method also includes generating the source virtual content using a source frame of reference different from the output frame of reference. The virtual content may be transformed from the source frame of reference to the output frame of reference. The respective Z locations may be in the output frame of reference different from a source frame of reference used to generate the virtual content.

In one or more embodiments, the virtual content includes image information and a source X, Y location in a source frame of reference. The image information may include a brightness, a color, and/or a Z location in the source frame of reference. The warped virtual content may include an output X, Y location in the output frame of reference.

In another embodiment, a system for warping virtual content includes an input memory to store source virtual content. The system also includes a command processor to divide the source virtual content into a plurality of scheduling units. The system further includes a scheduler to determining a processing sequence. Moreover, the system includes a GPU core, the GPU core having a plurality of shader cores to process one of the plurality of scheduling units to generate warped virtual content. In addition, the system includes a FIFO memory to store output from the GPU core. The system also includes a ROP to rasterize the warped virtual content for display. The system further includes a buffer memory to temporarily store the warped virtual content. When pre-existing virtual content temporality stored in the buffer memory conflicts with the warped virtual content, the ROP compares respective Z locations of the pre-existing virtual content and the warped virtual content.

In one or more embodiments, the source virtual content includes a source X, Y value in a source frame of reference. The warped virtual content may include an output X, Y value in an output frame of reference.

In still another embodiment, a computer program product is embodied in a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for warping virtual content. The method includes generating warped virtual content by transforming source virtual content. The method also includes determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content. The method further includes storing the warped virtual content in the memory location if the memory location is not occupied by pre-existing virtual content. Moreover, the method includes comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location in the output frame of reference if the memory location is occupied by pre-existing virtual content. In addition, the method includes storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of the warped virtual content is closer to the viewing location than a pre-existing Z location of the pre-existing virtual content in the output frame of reference.

In one or more embodiments, the computer programming product is an API.

In yet another embodiment, a system for warping virtual content includes a warping unit to generate warped virtual content. The warping unit includes a pose estimator and a transform unit. The system also includes a compositing unit, including a blending unit. The system further includes a database to temporarily store warped virtual content. When pre-existing virtual content temporality stored in the database conflicts with the warped virtual content, the blending unit of the compositing unit compares respective Z locations of the pre-existing virtual content and the warped virtual content.

Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.

