CROSS REFERENCE TO RELATED APPLICATION

This application is a 35 U.S.C. § 371 national stage application of PCT International Application No. PCT/US2018/019385, filed on Feb. 23, 2018, the content of which is incorporated herein by reference in its entirety. The above-referenced PCT International Application was published in the English language as International Publication No. WO 2019/164502 A1 on Aug. 29, 2019.

TECHNICAL FIELD

Embodiments of the inventive concepts relate to methods and devices for image processing and more particularly to generating 3D models.

BACKGROUND

3D Scanning is a process of creating a three-dimensional (3D) model based on a series of captured images. The process of creating the 3D model may be done in real-time while the series of images are captured or may be done after the images are captured. When the 3D model is created in real-time, the process may include user interaction to facilitate the capture of images likely to produce a quality 3D model. Generating the 3D model of the object from the series of captured images may be a time-consuming process utilizing large amounts of computer processing power and computer resources. The processing capabilities may be limited in a real-time process. Accordingly, the quality of the 3D model produced in real-time may be limited.

When the 3D model is created after the images are captured, more processing capability may be available. In some examples, the captured images may be transferred to a remote server for offline processing. In some offline processing solutions, a user may not receive any feedback until after the final 3D model is generated. Some offline processing solutions may provide feedback when taking the pictures in order to maximize the angles covered by the pictures. However, some regions of an object being scanned may include more details than others. Accordingly, more pictures may be desirable from angles that cover the most detailed regions of an object.

SUMMARY

According to some embodiments of the inventive concepts, methods may be provided. A method may include receiving a plurality of two-dimensional (2D) images of a physical object captured from a respective plurality of viewpoints in a three-dimensional (3D) scan of the physical object in a first process. The method may include receiving a first process 3D mesh representation of the physical object and calculating respective second process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images. The method may include generating a second process 3D mesh representation of the physical object using the plurality of 2D images, the second process estimated position and/or orientation information, and the first process 3D mesh representation of the physical object. The method may include generating a 3D model of the physical object by applying surface texture information from the plurality of 2D images to the second process 3D mesh representation of the physical object.

In some embodiments, the calculating the second process estimated position and/or orientation information and generating the second process 3D mesh representation may be iteratively repeated.

In some embodiments, a standard deviation of error of the second process 3D mesh representation with respect to the physical object may be less than a standard deviation of error of the first process 3D mesh representation with respect to the physical object.

In some embodiments, the method may include receiving intrinsic camera parameters corresponding to the plurality of 2D images and calculating updated intrinsic camera parameters based on the plurality of 2D images.

In some embodiments, the generating of the second process 3D mesh representation of the physical object may include determining an initial depth map using the first process 3D mesh representation from a perspective of a reference one of the plurality of 2D images, selecting one or more surrounding ones of the plurality of 2D images based on the second process estimated position and/or orientation information, warping the one or more surrounding ones of the plurality of 2D images to appear from the perspective of the reference one of the plurality of 2D images, and calculating an updated depth map based on a combination of the reference one of the plurality of 2D images and the warped surrounding ones of the plurality of 2D images.

In some embodiments, the method may include calculating surface normal vectors corresponding to positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images. The generating the second process 3D mesh representation of the physical object may be based on the calculated surface normal vectors.

In some embodiments, the method may include receiving respective first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images. The method may include receiving a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images. The calculating of the respective second process estimated position and/or orientation information may include calculating the respective second process estimated position and/or orientation information for each of the respective plurality of viewpoints of the plurality of 2D images using the plurality of 2D images, the first process estimated position and/or orientation information, and the feature point map.

In some embodiments, a standard deviation of error of the second process estimated position and/or orientation information with respect to the physical object may be less than a standard deviation of error of the first process 3D mesh representation with respect to the physical object.

In some embodiments, the calculating of the respective second process estimated position and/or orientation information may include calculating an estimated location of a feature point of the plurality of feature points within a 2D image of the plurality of 2D images based on the first process estimated position and/or orientation information, identifying the feature point within the 2D image to determine an actual location of the feature point within the 2D image, determining a displacement between the estimated location of the feature point within the 2D image and the actual location of the feature point within the 2D image, and calculating a second process estimated position and/or orientation corresponding to the 2D image that yields a smaller displacement than the displacement determined using the estimated location based on the first process estimated position and/or orientation information.

In some embodiments, the calculating of the respective second process estimated position and/or orientation information may further include minimizing a sum of displacements between a plurality of estimated locations of feature points of the plurality of feature points within ones of the plurality of 2D images.

In some embodiments, the first process may include calculating first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images, calculating a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images, and generating the first process 3D mesh representation of the physical object.

In some embodiments, the first process 3D mesh representation of the physical object may be generated while capturing the plurality of 2D images.

In some embodiments, the 3D model may be a second process 3D model and the first process may further include generating a first process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the first process 3D mesh representation of the physical object.

According to some embodiments, computer program products may be provided. A computer program product may include a non-transitory computer readable storage medium having computer readable program code embodied in the medium that when executed by a processor causes the processor to perform a method according to some embodiments of the inventive concepts.

According to some embodiments, devices may be provided. A device may include a processor and a memory coupled to the processor and storing computer readable program code that when executed by the processor causes the processor to perform a method according to some embodiments of the inventive concepts.

According to some embodiments, methods may be provided. A method may include capturing a plurality of two-dimensional (2D) images of a physical object captured from a respective plurality of viewpoints in a three-dimensional (3D) scan of the physical object using at least one camera of an electronic device. The method may include calculating first process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images. The method may include calculating a feature point map comprising positions in a 3D space of a plurality of feature points that are visible in ones of the plurality of 2D images. The method may include generating the first process 3D mesh representation of the physical object. The method may include generating a first process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the first process 3D mesh representation of the physical object. The method may include displaying the first process 3D model on a display of the electronic device and providing an option to a user of the electronic device to further process the first process 3D model. The method may include generating a second process 3D model responsive to receiving instruction from the user to further process the first process 3D model.

In some embodiments, the generating the second process 3D model may include calculating respective second process estimated position and/or orientation information for each one of the respective plurality of viewpoints of the plurality of 2D images, generating a second process 3D mesh representation of the physical object using the plurality of 2D images, the second process estimated position and/or orientation information, and the first process 3D mesh representation of the physical object, and generating the second process 3D model of the physical object by applying surface texture information from the plurality of 2D images to the second process 3D mesh representation of the physical object.

In some embodiments, the second process 3D mesh representation and the second process 3D model may be generated after the first process 3D mesh representation and the first process 3D model are generated. The first process 3D mesh representation, the first process 3D model, the second process 3D mesh representation, and the second process 3D model may be generated by the electronic device.

In some embodiments, the method may include sending the plurality of 2D images and the first process 3D mesh representation from the electronic device to an offline processing device remote from the electronic device. The first process 3D mesh representation and the first process 3D model may be generated by the electronic device and the generating of the second process 3D model may include receiving the second process 3D model from the offline processing device.

According to some embodiments of the inventive concepts, computer program products may be provided. A computer program product may include a non-transitory computer readable storage medium having computer readable program code embodied in the medium. When executed by a processor, the computer readable program code may cause the processor to perform a method according to some embodiments of the inventive concepts.

According to some embodiments of the inventive concepts, electronic devices may be provided. An electronic device may include a processor, a camera coupled to the processor, and a memory coupled to the processor. The memory may store computer readable program code that when executed by the processor causes the processor to perform a method according to some embodiments of the inventive concepts.

