FIELD OF THE INVENTION

The present disclosure relates to mixed reality systems with virtual content warping, and methods for generating a mixed reality experience including warped virtual content using the same.

BACKGROUND

Modern computing and display technologies have facilitated the development of “mixed reality” (MR) systems for so called “virtual reality” (VR) or “augmented reality” (AR) experiences, wherein digitally reproduced images, or portions thereof, are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real-world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world around the user.

Various optical systems generate images at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014, the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.

MR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated to take the change in head pose (i.e., the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display device and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint (corresponding to a position and/or orientation of the head-worn display device), giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects at different depths, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion. However, there is an inevitable lag between rendering a scene and displaying/projecting the rendered scene.

Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, a user can see through transparent (or semi-transparent) elements in a display system to view directly the light from real objects in a real-world environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user's view of the real world, where light from the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.

MR systems attempt to present photo-realistic, immersive MR scenarios. However, head movement during a lag time between generation of virtual content (“rendered virtual content”) and display of the rendered virtual content may result in visual artifacts (e.g., glitches). This problem may be exacerbated by rapid head movement during the lag time.

In order to address this issue, some optical systems may include a warping software/system/method that receives rendered virtual content from a source. The warping system then “warps” (i.e., transforms the “frame of reference” of) the received rendered virtual content for display in a frame of reference of the display system/viewer (the “display frame of reference”). This approach takes the originally rendered virtual content, and transforms the way that the virtual content is presented to attempt to display the virtual content from a different perspective at the time of display.

Some warping software/systems/methods warp the rendered virtual content to correct for rotational or coarse positional user movements (i.e., of the user's field of view “FOV”). Such systems are not optimized to correct for other types of user movements (e.g., substantially lateral user movements along an axis orthogonal to the user's optical axis). Accordingly, some warping software/systems/methods can result in artifacts with user movement.

SUMMARY

In one embodiment, a computer implemented method for warping virtual content includes receiving rendered virtual content data, the rendered virtual content data including a far depth. The method also includes receiving movement data indicating a user movement in a direction orthogonal to an optical axis. The method further includes generating warped rendered virtual content data based on the rendered virtual content data, the far depth, and the movement data.

In one or more embodiments, the rendered virtual content data corresponds to a near virtual object and a far virtual object, and the warped rendered virtual content data corresponds to a warped near virtual object and a warped far virtual object. A first position shift between the near virtual object and the warped near virtual object may be substantially equal to a second position shift between the far virtual object and the warped far virtual object.

In one or more embodiments, generating the warped rendered virtual content data based on the rendered virtual content data, the far depth, and the movement data reduces a parallax related artifact compared to the rendered virtual content data. The parallax may be related artifact is a disocclusion artifact. The parallax related artifact may be a smearing artifact or a jitter artifact.

In one or more embodiments, the rendered virtual content data corresponds to a first depth segment. The method also includes receiving second rendered virtual content data corresponding to a second depth segment, the second rendered virtual content data including a second far depth. The method further includes generating warped second rendered virtual content data based on the second rendered virtual content data, the second far depth, and the movement data.

In another embodiment, a computer implemented method for warping virtual content includes receiving rendered virtual content data, the rendered virtual content data including a far depth and a near depth. The method also includes receiving movement data indicating a user movement in a direction orthogonal to an optical axis. The method further includes generating warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the movement data.

In one or more embodiments, generating warped rendered virtual content data based on the rendered virtual content data, the far depth, the near depth, and the movement data includes averaging the far depth and the near depth.

In still another embodiment, a computer implemented method for warping virtual content includes receiving rendered virtual content data. The method also includes receiving movement data indicating a user movement in a direction orthogonal to an optical axis. The method further includes receiving gaze data indicating a user gaze. Moreover, the method includes determining a depth based on the gaze data. In addition, the method includes generating warped rendered virtual content data based on the rendered virtual content data, the depth, and the movement data.

In yet another embodiment, a computer implemented method for warping virtual content includes receiving rendered virtual content data. The method also includes receiving movement data indicating a user movement in a direction orthogonal to an optical axis. The method further includes generating mesh data from the rendered virtual content data. Moreover, the method includes determining depth data from the mesh data. In addition, the method includes generating warped rendered virtual content data based on the rendered virtual content data, the mesh data, the depth data, and the movement data.

In one or more embodiments, the method also includes adjusting a mesh size when generating the mesh data. The mesh data may be generated using a grid mesh. The depth data may be determined based on a plurality of depths in the mesh and content of the mesh at each of the plurality of depths.

In another embodiment, a computer implemented method for determining a depth of a virtual object includes re-projecting a first image of the virtual object corresponding to a first eye over a second image of the virtual object corresponding to a second eye at a plurality of depths to generate a plurality of respective combination images. The method also includes measuring the re-projection error in each of the plurality of respective combination images to generate a plurality of re-projection errors. The method further includes identifying a minimum re-projection error in the plurality of re-projection errors corresponding to a combination image of the plurality of respective combination images. Moreover, the method includes identifying a depth corresponding to the combination image as the depth of the virtual object.

In one or more embodiments, the depth of the virtual object is used to warp the virtual object from a first point to view to a second point of view. The plurality of depths may consist of 64 depths. Each of the plurality of depths may correspond to a one pixel difference in re-projecting the first image of the virtual object.

In still another embodiment, a computer implemented method for determining a minimum stabilization depth includes obtaining an amount of point of view (POV) movement. The method also includes determining a relationship between stabilization depth and pixel movement at the stabilization depth based on the amount of POV movement. The method further includes obtaining a maximum unnoticeable pixel movement for a display system. Moreover, the method includes identifying a stabilization depth corresponding to the maximum unnoticeable pixel movement for the display system as the minimum stabilization depth of the display system.

In one or more embodiments, the method also includes comprising warping only content closer than the minimum stabilization depth to a viewer when a point of view changes. The relationship between stabilization depth and pixel movement at the stabilization depth may be an inverse relationship.

Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.

