FIELD

One or more example embodiments of the present disclosure relate to a computer vision system, and a method to train the computer vision system using machine learning.

BACKGROUND

Computer vision applications, for example, such as depth estimation and image/video enhancement, have wide uses in real scenarios. For example, depth estimation may be utilized in foreground/background segmentation for image Bokeh, and image/video enhancement has become a standard feature in many image signal processors (ISP) for most modern cellphone cameras today.

Recently, the field of computer vision has been shifting from statistical methods to deep learning neural network methods. For example, deep learning neural network methods may enable a single model to learn meaning from images to perform a specific computer vision task, obviating the need for a pipeline of specialized, hand-crafted methods. Generally, deep learning neural network methods may utilize a single machine learning technique and corresponding loss function to train for a specific computer vision application, which may limit their accuracy.

The above information disclosed in this Background section is for enhancement of understanding of the background of the present disclosure, and therefore, it may contain information that does not constitute prior art.

SUMMARY

One or more example embodiments of the present disclosure are directed to a computer vision system having a multiple machine learning strategy framework to train for various computer vision applications, and a method for training the computer vision system.

According to one or more example embodiments of the present disclosure, a computer vision (CV) training system, includes: a supervised learning system configured to estimate a supervision output from one or more input images according to a target CV application, and determine a supervised loss according to the supervision output and a ground-truth of the supervision output; an unsupervised learning system configured to determine an unsupervised loss according to the supervision output and the one or more input images; a weakly supervised learning system configured to determine a weakly supervised loss according to the supervision output and a weak label corresponding to the one or more input images; and a joint optimizer configured to optimize the supervised loss, the unsupervised loss, and the weakly supervised loss.

In an example embodiment, the target CV application may correspond to monocular depth estimation, and the supervision output may correspond to a depth estimated from a target image from among the one or more input images.

In an example embodiment, the unsupervised learning system may be configured to convert 2-dimensional (2D) image coordinates corresponding to the one or more input images into 3-dimensional (3D) coordinates according to the estimated depth, and calculate the unsupervised loss based on at least the 3D coordinates.

In an example embodiment, the target CV application may correspond to stereo matching, and the supervision output may correspond to a disparity estimated between a left image and a right image from among the one or more input images.

In an example embodiment, the unsupervised learning system may be configured to shift one of the left image and the right image according to the estimated disparity to generate a warped image, and calculate the unsupervised loss based on at least the warped image.

In an example embodiment, the target CV application may correspond to image/video enhancement, and the supervision output may correspond to high-resolution images estimated from low-resolution input images from among the one or more input images.

In an example embodiment, the unsupervised learning system may be configured to estimate an optical flow corresponding to the estimated high-resolution images, and calculate the unsupervised loss based on at least the estimated optical flow and the estimated high-resolution images.

In an example embodiment, the target CV application may correspond to multi-view depth estimation, and the supervision output may correspond to a depth estimated in each of multiple input frames from among the one or more input images.

In an example embodiment, the unsupervised learning system may be configured to generate a 3D point cloud for each of the multiple input frames according to the estimated depth in each of the multiple input frames, and calculate the unsupervised loss based on at least distances between aligned coordinates of the 3D point clouds.

In an example embodiment, the joint optimizer may be configured to concurrently optimize the supervised loss, the unsupervised loss, and the weakly supervised loss according to a joint loss function including corresponding weights for the supervised loss, the unsupervised loss, and the weakly supervised loss.

In an example embodiment, the system may further include a weak label generator to generate the weak label as a patch-based weak label having a patch-pair corresponding to different cropped regions of the one or more input images, the patch-based weak label defining a relationship corresponding to the supervision output between the cropped regions of the patch-pair.

According to one or more example embodiments of the present disclosure, a method for training a computer vision (CV) system, includes: estimating a supervision output from one or more input images according to a target CV application; determining a supervised loss according to the supervision output and a ground-truth of the supervision output; determining an unsupervised loss according to the supervision output and the one or more input images; determining a weakly supervised loss according to the supervision output and a weak label corresponding to the one or more input images; and optimizing the supervised loss, the unsupervised loss, and the weakly supervised loss.

In an example embodiment, the target CV application may correspond to monocular depth estimation, and the supervision output may correspond to a depth estimated from a target image from among the one or more input images.

In an example embodiment, the method may further include: converting 2-dimensional (2D) image coordinates corresponding to the one or more input images into 3-dimensional (3D) coordinates according to the estimated depth; and calculating the unsupervised loss based on at least the 3D coordinates.

In an example embodiment, the target CV application may correspond to stereo matching, and the supervision output may correspond to a disparity estimated between a left image and a right image from among the one or more input images.

In an example embodiment, method may further include: shifting one of the left image and the right image according to the estimated disparity to generate a warped image; and calculating the unsupervised loss based on at least the warped image.

In an example embodiment, the target CV application may correspond to image/video enhancement, and the supervision output may correspond to high-resolution images estimated from low-resolution images from among the one or more input images.

In an example embodiment, the method may further include: estimating an optical flow corresponding to the estimated high-resolution images; and calculating the unsupervised loss based on at least the estimated optical flow and the estimated high-resolution images.

In an example embodiment, the target CV application may correspond to multi-view depth estimation, and the supervision output may correspond to a depth estimated in each of multiple input frames from among the one or more input images.

In an example embodiment, the method may further include: generating a 3D point cloud for each of the multiple input frames according to the estimated depth in each of the multiple input frames; and calculating the unsupervised loss based on at least distances between aligned coordinates of the 3D point clouds.

In an example embodiment, the optimizing may include: concurrently optimizing the supervised loss, the unsupervised loss, and the weakly supervised loss according to a joint loss function including corresponding weights for the supervised loss, the unsupervised loss, and the weakly supervised loss.

In an example embodiment, the method may further include: generating the weak label as a patch-based weak label having a patch-pair corresponding to different cropped regions of the one or more input images, the patch-based weak label defining a relationship corresponding to the supervision output between the cropped regions of the patch-pair.

