BACKGROUND

Mobile phones incorporate components that make them versatile and practically indispensable to their owners. Most existing smartphones include a camera and various inertial sensors, such as an accelerometer or compass. The smartphones can also include a proximity sensor, magnetometer, or other types of sensors.

Smartphones can be used to capture information with their cameras. Users value a smartphone's ability to take pictures because this feature allows the user to easily capture memorable moments or images of documents, such as might occur when performing bank transactions. However, smartphones are generally used to acquire images of simple scenes, such as a single photograph or a video with a sequence of image frames.

Heretofore, smartphones have not been used to produce output that can be printed to a three-dimensional (3D) printer. Such printers can be used to print a 3D representation of an object. However, a suitable 3D representation of an object has generally been created with specialized hardware and software applications.

SUMMARY

In some aspects, a method is provided for forming a 3D representation of an object with a smartphone or other portable electronic device. Such a representation may be formed by scanning the object to acquire multiple image frames of the object from multiple orientations. Information acquired from these image frames may be combined into a first representation of the object. That first representation may be processed to generate a second representation. In the second representation, the object may be represented by structural information, which may indicate the location of one or more surfaces, and texture information. The second representation may be modified, to remove or change structural information indicating structures that cannot be physically realized with a 3D printer.

In some embodiments, the second representation, and/or the modified second representation, may be a portable document format.

In some aspects, a method is provided for forming a 3D representation of an object with a smartphone or other portable electronic device. Such a representation may be formed by scanning the object to acquire multiple image frames of the object from multiple orientations. Information acquired from these image frames may be combined into a first representation of the object. That first representation may be processed to generate a second representation. In the second representation, the object may be represented by structural information, which may indicate the location of one or more surfaces, and texture information. The second representation may be used to generate a visual display of the object while the image frames are acquired.

In accordance with other aspects, any of the foregoing methods may be embodied as computer-executable instructions embodied in a non-transitory medium.

In accordance with yet other aspects, any of the foregoing methods may be embodied as a portable electronic device in which a processor is configured to perform some or all of the acts comprising the method.

One type of embodiment is directed to a portable electronic device comprising a camera and at least one processor. The at least one processor is configured to form a first representation of an object from a plurality of image frames acquired with the camera from a plurality of directions, the representation comprising locations in a three-dimensional space of features of the object. The at least one processor is further configured to determine, from the first representation, a second representation of the object, the second representation comprising locations of one or more surfaces. The at least one processor is further configured to modify the second representation to remove surfaces that are not printable in three dimensions store the modified second representation as a three-dimensional printable file.

In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of a closed hull. In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of a hull.

In some embodiments, the portable electronic device further comprises one or more inertial sensors. In some embodiments, the at least one processor is further configured to form the first representation based on outputs of the one or more inertial sensors when the plurality of image frames is acquired.

In some embodiments, the portable electronic device further comprises a display. In some embodiments, the at least one processor is further configured to render an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. In some embodiments, the three-dimensional printable file is in a portable document format. In some embodiments, the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure.

One type of embodiments is direct to a method of forming a file for printing on a three-dimensional printer, the file comprising a representation of an object. The method comprises acquiring a plurality of image frames using a camera of a portable electronic device, and while the image frames are being acquired, construct a first three-dimensional representation of the object, determine a second three-dimensional representation of the object, by calculating a convex hull of the object, and display a view of the second three-dimensional representation on a two dimensional screen.

In some embodiments, the method further comprises modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions. In some embodiments, modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. In some embodiments, modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of the convex hull. In some embodiments, modifying the second three-dimensional representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of the convex hull.

In some embodiments, constructing the first representation comprises constructing the first representation based on outputs of one or more inertial sensors of the portable electronic device when the plurality of image frames is acquired. In some embodiments, the method further comprises rendering an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. In some embodiments, the three-dimensional printable file is in a portable document format. In some embodiments, the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure.

One type of embodiment is directed to at least one non-transitory, tangible computer-readable storage medium having computer-executable instructions, that when executed by a processor, perform a method of forming a three dimensional representation of an object from a plurality of image frames captured with a camera of a portable electronic device. The method comprises forming a first representation of the object from the plurality of image frames, the representation comprising locations in a three-dimensional space of features of the object, determining, from the first representation, a second representation of the object, the second representation comprising locations of one or more surfaces, modifying the second representation to remove surfaces that are not printable in three dimensions, and storing the modified second representation as a three-dimensional printable file.

In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not joined to other surfaces to provide a wall thickness above a threshold. In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises removing surfaces that are not a part of a closed hull. In some embodiments, modifying the second representation to remove surfaces that are not printable in three dimensions comprises computing normals to surfaces to remove surfaces having a normal in a direction toward an interior of a hull.

In some embodiments, forming the first representation comprises forming the first representation based on outputs of one or more inertial sensors of the portable electronic device when the plurality of image frames is acquired. In some embodiments, the method further comprises rendering an image of the object based on a first portion of the plurality of image frames while a second portion of the plurality of image frames is being acquired. In some embodiments, the three-dimensional printable file is in a portable document format. In some embodiments, the three-dimensional printable file comprises separate information about structure of the object and visual characteristics of the structure.

