BACKGROUND

This disclosure generally relates to displaying augmented reality elements.

When a video capture device, e.g., a video camera on a smartphone, captures video of a scene for use in displaying augmented reality (AR) or mixed reality, AR elements may typically be displayed covering or overlapping the scene. In certain applications, it is desirable to render and display AR elements to appear occluded (concealed behind) or partially occluded by a portion of the scene in the video. For example, in a navigation system that displays an AR view of the scene in front of a vehicle, it may be confusing for a user if an AR element appears in front of objects that are closer than a corresponding distance of the AR element.

For cases where live video footage (e.g., for navigation system), or a relatively contemporaneous video footage, is displayed with AR element, the AR elements must be correctly rendered and/or displayed quickly with little delay. It is challenging to quickly and efficiently determine whether an AR element should appear occluded by portions of a scene in a video when rendering and/or displaying the AR elements.

SUMMARY

In order to occlude AR elements, as displayed in images, a depth map is employed to determine the distance to pixels representing elements of the real world, and the pixel distances are compared with distances of AR elements to determine where in an image occlusion should occur. A depth map is determined for an image frame of a video received from a video capture device. An AR graphical element for overlaying over the image frame is received. An element distance for AR graphical elements relative to a position of a user of the video capture device (e.g., the geographic position of the video capture device) is also received. Based on the depth map for the image frame, a pixel distance is determined for each pixel in the image frame. The pixel distances of the pixels in the image frame are compared to the element distance. In response to a pixel distance for a given pixel being less than the element distance, the given pixel of the image frame is displayed rather than a corresponding pixel of the AR graphical element. Conversely, responsive to the pixel distance for a given pixel being greater than the element distance, a corresponding pixel of the AR graphical element is displayed. The image frames overlaid with AR graphical content may be displayed as a live video feed on a client device of the user (e.g., a mobile computing device). The video capture device may be associated with a mobile computing device (e.g., the video capture device may be a camera integrated into a smartphone). The video capture device may also be associated with a vehicle (e.g., integrated into the vehicle dashboard). In some embodiments, the received geographic location of the video capture device is specified relative to a geographic location of the vehicle.

In some embodiments, the depth map for the image frame is determined using a depth estimation model. The depth estimation model may take the image frame as input and output a depth map as output including a relative depth value (e.g., 0 to 1) for each pixel in the image frame. The depth estimation model may be a self-supervised monocular depth estimation model.

In some embodiments, the pixel distances for pixels in the image frame are determined using a reference distance to a portion of the scene in the image frame. The reference distance may be used to determine a conversion factor from depth values in the depth map to pixel distances. The reference distance may be determined based on the geographic location of the video capture device and the angular orientation of the geographic device.

