BACKGROUND

Technical Field

This disclosure relates to a display system, and in particular to a system that generates and displays composite stereoscopic images.

Description of Related Art

Three dimensional (3D) capable electronics and computing hardware devices and real-time computer-generated 3D computer graphics have been a popular area of computer science for the past few decades, with innovations in visual, audio, tactile and biofeedback systems. Much of the research in this area has produced hardware and software products that are specifically designed to generate greater realism and more natural computer-human interfaces. These innovations have significantly enhanced and simplified the end-user's computing experience.

Ever since humans began to communicate through pictures, they faced a dilemma of how to accurately represent the three-dimensional world they lived in. Sculpture was used to successfully depict three-dimensional objects, but was not adequate to communicate spatial relationships between objects and within environments. To do this, early humans attempted to “flatten” what they saw around them onto two-dimensional, vertical planes (e.g., paintings, drawings, tapestries, etc.).

The two dimensional pictures must provide a numbers of cues of the third dimension to the brain to create the illusion of three dimensional images. This effect of third dimension cues can be realistically achievable due to the fact that the brain is quite accustomed to it. The three dimensional real world is always and already converted into two dimensional (e.g., height and width) projected image at the retina, a concave surface at the back of the eye. And from this two dimensional image, the brain, through experience and perception, generates the depth information to form the three dimension visual image from two types of depth cues: monocular (one eye perception) and binocular (two eye perception). In general, binocular depth cues are innate and biological while monocular depth cues are learned and environmental.

A planar stereoscopic display, e.g., a LCD-based or a projection-based display, shows two images with disparity between them on the same planar surface. By temporal and/or spatial multiplexing the stereoscopic images, the display results in the left eye seeing one of the stereoscopic images and the right eye seeing the other one of the stereoscopic images. It is the disparity of the two images that results in viewers feeling that they are viewing three dimensional scenes with depth information.

SUMMARY

This specification describes a computer system that processes an initial image of an environment to generate a composite image of the environment, and displays the composite image on a display device.

In particular, the initial image depicts a physical environment that includes a surface. The surface can include a physically visually-demarked region. That is, a region of the surface is visibly-demarked in the physical environment; for example, the region can be a different color than the rest of the surface, have an outline that demarcates the boundaries of the region from the rest of the surface, and/or be on a platform that is a different elevation than the rest of the surface. As a particular example, the surface can be a table top, and the visually-demarked region a portion of the table top.

The system can determine a portion of the initial image that depicts the visually-demarked region of the surface. For example, the system can process the initial image using one or more machine learning models that have been configured through training to identify visually-demarked regions in images. As another example, a tracking system can track the location and orientation, in a common three-dimensional coordinate system of the physical environment, of i) the visually demarked region and ii) the capture device that captured the initial image. The system can then determine, according to the respective location and orientations, the portion of the initial image that depicts the visually-demarked region.

The system can then process the initial image to generate the composite image, including rendering a virtual object within or proximate to the visually-demarked region. For example, the virtual object can be on top of or below the visually demarked region from the perspective of the composite image.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

In many cases, when generating a composite image for display to a user, e.g., a composite image that that depicts a physical environment with one or more virtual objects in the physical environment, it can be helpful to the user to render the virtual object in relation to real-world physical reference points. For example, it may be disorienting or jarring for the user if the composite images include virtual objects floating in space or seemingly-randomly imposed onto the physical environment of the user. Using techniques described in this specification, a system can render virtual objects to be within or proximate to physical, real-world reference points that are visible to a user, thus creating a more realistic and/or comfortable experience for the user.

Furthermore, using techniques described in this specification, users can use any visually-demarked region of any surface as a reference point for composite images. For example, a user can place a piece of paper or a piece of cardboard onto a surface, e.g., a desk or a table, and identify the piece of paper or cardboard to the system as a reference point according to which the system should render virtual objects. Thus, the user does not need to purchase expensive specialty hardware to act as the reference points in order to enjoy an intuitive virtual experience.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

