TECHNICAL FIELD

This subject matter of this application relates generally to methods and apparatuses, including computer program products, for multiple maps for 3D object scanning and reconstruction.

BACKGROUND

Generally, current approaches to 3D object reconstruction utilize limited “single map” techniques to provide a variety of high-level 3D model generation algorithms. However, with these techniques, it can be difficult for users to finish scanning objects due to, e.g., unreliable tracking. In “single map” reconstruction, every frame has to determine a location in the single map during scanning. When one frame loses the location information, the subsequent frames are not added into the object reconstruction process—which causes a failure to generate the corresponding 3D model.

SUMMARY

Therefore, what is needed are systems and methods that enable a multi-map-based method for 3D object reconstruction which takes into account both accuracy and efficiency. The techniques described herein advantageously provide for tolerances in tracking loss and also result in substantial improvements for the user experience in scanning objects. For example, during scanning, the systems and methods generate multiple maps instead of only one (as in the prior systems described above). Each of these multiple maps contains its own frames to contribute to the partially scanned object, and each map has exactly the same structure containing 3D model information. Furthermore, each map is able to connect to other map(s) that are generated. This beneficially allows a user to continue scanning the object (and/or scene) without going back to the previous map and re-localizing—even when the tracking is lost. All the user has to do then is to make sure the camera field of view (FoV) has covered the entire area that needs to be scanned. As a result, these innovative techniques improve both speed and robustness of the scanning process.

The invention, in one aspect, features a system for generating multiple maps during object scanning for 3D object reconstruction. The system includes a sensor device that captures one or more RGB images and one or more depth maps of one or more objects in a scene. The system includes a computing device coupled to the sensor device, the computing device comprising a memory that stores computer-executable instructions and a processor that executes the instructions. The computing device receives the one or more RGB images and the one or more depth maps from the sensor device. The computing device creates a first map using at least a portion of the one or more depth maps. The computing device creates a second map using at least a portion of the one or more depth maps. The computing device creates a third map using at least a portion of the one or more depth maps. The computing device finds key point matches among the first map, the second map, and the third map. The computing device performs bundle adjustment on the first map, the second map, and the third map using the matched key points to generate a final map. The computing device generates a 3D mesh of the object using the final map.

The invention, in another aspect, features a computerized method of generating multiple maps during object scanning for 3D object reconstruction. A sensor device captures one or more RGB images and one or more depth maps of one or more objects in a scene. A computing device receives, from the sensor device, the one or more RGB images and the one or more depth maps. The computing device creates a first map using at least a portion of the one or more depth maps. The computing device creates a second map using at least a portion of the one or more depth maps. The computing device creates a third map using at least a portion of the one or more depth maps. The computing device finds key point matches among the first map, the second map, and the third map. The computing device performs bundle adjustment on the first map, the second map, and the third map using the matched key points to generate a final map. The computing device generates a 3D mesh of the object using the final map.

Any of the above aspects can include one or more of the following features. In some embodiments, creating a first map using at least a portion of the one or more depth maps comprises, for each depth map: locating at least one of the one or more objects in the depth map; identifying one or more key points on the located one or more objects; and combining the identified key points to generate the first map. In some embodiments, the portion of the one or more depth maps used to create the first map comprise a view of the one or more objects from a first perspective.

In some embodiments, creating a second map using at least a portion of the one or more depth maps comprises, for each depth map: locating at least one of the one or more objects in the depth map; identifying one or more key points on the located one or more objects; and combining the identified key points to generate the second map. In some embodiments, the portion of the one or more depth maps used to create the second map comprise a view of the one or more objects from a second perspective. In some embodiments, the second perspective is different from the first perspective.

In some embodiments, creating a third map using at least a portion of the one or more depth maps comprises, for each depth map: locating at least one of the one or more objects in the depth map; identifying one or more key points on the located one or more objects; and combining the identified key points to generate the third map. In some embodiments, the portion of the one or more depth maps used to create the third map comprise a view of the one or more objects from a third perspective. In some embodiments, the third perspective is different from the first perspective and the second perspective.

In some embodiments, at least one of the identified key points in the first map overlaps with at least one of the identified key points in the third map. In some embodiments, at least one of the identified key points in the second map overlaps with at least one of the identified key points in the third map. In some embodiments, at least one of the identified key points in the first map overlaps with at least one of the identified key points in the third map.

In some embodiments, finding key point matches among the first map, the second map, and the third map comprises one or more of: determining that a key point from the first map overlaps with a key point from the second map and identifying the overlapping key points as a key point match; determining that a key point from the second map overlaps with a key point from the third map and identifying the overlapping key points as a key point match; and determining that a key point from the first map overlaps with a key point from the third map and identifying the overlapping key points as a key point match. In some embodiments, the final map comprises a plurality of bundle-adjusted matched key points from the first map, the second map, and the third map.

In some embodiments, the 3D mesh is generated using a Marching Cube algorithm applied to at least a portion of the final map. In some embodiments, the one or more objects in the scene are stationary and the sensor device is moving, as the sensor device captures the one or more RGB images and the one or more depth maps. In some embodiments, the one or more objects in the scene are moving and the sensor device is stationary, as the sensor device captures the one or more RGB images and the one or more depth maps. In some embodiments, the one or more objects in the scene are moving and the sensor device is moving, as the sensor device captures the one or more RGB images and the one or more depth maps.

Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating the principles of the invention by way of example only.

