BACKGROUND

Depth values of features in a three-dimensional environment may be determined by way of triangulation. Triangulation can involve determining the position of a point in three-dimensional space based on that point's projection onto two or more images. The depth of the point may be determined based on a location of the point's projection in each image and the relative position of the camera(s) when capturing each image. When both images are captured in parallel (e.g., by a stereoscopic camera), the depth of both moving and static features of an environment may be determined. However, when images are captured serially, as in the case of a monoscopic camera, movement of features between successive images can make triangulation difficult to apply. Specifically, for a given point on a moving object, it might be difficult to determine whether the apparent displacement of that point between successive images is due to camera motion or due to movement of the object. Accordingly, determining depth for monoscopic images may be difficult.

SUMMARY

A machine learning (ML) model may be configured to determine depth values for images of a monoscopic video. Since each image may lack explicit depth data, the depth values may be determined by combining the information from (i) a target image for which the depth is to be determined and (ii) a reference image that provides another view of the same scene as the target image. The ML model may be trained to determine the depth values for the target image based on the target image, a static depth image that represents depth values for static features in the target image, and an object mask configured to remove, from the static depth image, moving features represented thereby. The static depth image may be determined based on motion parallax between the reference image and the target image, and may be valid for static features of the target image. The ML model may be configured to determine the depth values for the target image by (i) refining the depth values of the static features in the static depth image and (ii) filling in depth values for the moving features masked by the object mask.

In a first example embodiment, a method includes obtaining, by a processor, a reference image and a target image each representing an environment containing moving features and static features. The reference image has been captured by a camera at a first time and the target image has been captured by the camera at a second time later than the first time. The method also includes determining, by the processor, an object mask configured to (i) mask out the moving features in the target image and (ii) preserve the static features in the target image. The method additionally includes determining, by the processor and based on motion parallax between the reference image and the target image, a static depth image that represents depth values of the static features in the target image. The method further includes generating, by the processor and by way of an ML model, a dynamic depth image that represents depth values of both the static features and the moving features in the target image. The ML model is trained to generate the dynamic depth image by determining depth values of at least the moving features based on (i) the target image, (ii) the object mask, and (iii) the static depth image.

In a second example embodiments, a system is provided that includes a camera configured to generate a reference image and a target image each representing an environment containing moving features and static features. The reference image is generated at a first time, and the target image is generated at a second time later than the first time. The system also includes computing hardware configured to determine an object mask configured to (i) mask out the moving features in the target image and (ii) preserve the static features in the target image. The computing hardware is also configured to determine, based on motion parallax between the reference image and the target image, a static depth image that represents depth values of the static features in the target image. The system further includes an ML model configured to generate a dynamic depth image that represents depth values of both the static features and the moving features in the target image. The ML model is trained to generate the dynamic depth image by determining depth values of at least the moving features based on (i) the target image, (ii) the object mask, and (iii) the static depth image.

In a third example embodiment, a non-transitory computer readable storage medium is provided having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations. The operations include obtaining a reference image and a target image each representing an environment containing moving features and static features. The reference image has been captured by a camera at a first time and the target image has been captured by the camera at a second time later than the first time. The operations also include determining an object mask configured to (i) mask out the moving features in the target image and (ii) preserve the static features in the target image. The operations additionally include determining, based on motion parallax between the reference image and the target image, a static depth image that represents depth values of the static features in the target image. The operations further include generating, by way of an ML model, a dynamic depth image that represents depth values of both the static features and the moving features in the target image. The ML model is trained to generate the dynamic depth image by determining depth values of at least the moving features based on (i) the target image, (ii) the object mask, and (iii) the static depth image.

In a fourth example embodiment, a system is provided that includes means for obtaining a reference image and a target image each representing an environment containing moving features and static features. The reference image has been captured by a camera at a first time and the target image has been captured by the camera at a second time later than the first time. The system also includes means for determining an object mask configured to (i) mask out the moving features in the target image and (ii) preserve the static features in the target image. The system additionally includes means for determining, based on motion parallax between the reference image and the target image, a static depth image that represents depth values of the static features in the target image. The system further includes means for generating, by way of an ML model, a dynamic depth image that represents depth values of both the static features and the moving features in the target image. The ML model is trained to generate the dynamic depth image by determining depth values of at least the moving features based on (i) the target image, (ii) the object mask, and (iii) the static depth image.

These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.

