FIELD OF THE INVENTION

The systems and methods relate to creating a three-dimensional virtual image, model, and using a single image capture device, such as a digital camera.

BACKGROUND

Often times in production and design, it is useful to generate small, scaled models of a subject product or building prior to final product fabrication or final build. These models provide a concrete, visual object that pulls all conceptual elements into one object, and therefore make it more reasonable to consider the product or building as a whole. By considering the model as a whole, criticisms can be discovered and rectified before an object, such as the product or building itself and/or a component thereof, is produced, and can therefore increase efficiency and reduce costs during the production process.

Though tremendously invaluable, generating a useful model is a very time-consuming process, and one that requires exceptional skill. Because such skill is required, producing a model typically necessitates the employment of a well-trained artist, which can be very expensive. Because of the highly customized nature of models, each additional model will incur a similar expense, even if models are nearly identical. Further, although they are cheaper to produce than full-scale prototypes, models often require highly specialized building materials. Because of these requirements, models can still be quite expensive to create.

One alternative to traditional modeling is to create a virtual model. A virtual model is a computational representation of a model in which a digital coordinate system (such as a Cartesian system) represents a three-dimensional (3D) space of the real world, and coordinates for many points that are displayed within this system such that an object appears on a monitor, screen, display, or other output device similar to its real world counterpart. The number of points representing the object in question affects its fidelity: more points create a more accurate representation, although greater numbers of points require greater computational power.

Virtual models are a valuable resource in production and design, and have supplemented or even replaced more conventional models and architectural renderings outright in some applications. Virtual models offer designers a way to generate models of objects that can be easily modified; size, shape, color, and many other physical aspects can be modified in a relatively easy manner. Each change can be rendered without the use of any new physical material, and do not require the production of a new physical product, which would be very expensive and time consuming. Virtual models make it possible to visualize the product in question in its intended setting, such as a rendering of a building that does not yet exist in an existing lot. Further, undesirable changes can be quickly and easily undone, and multiple instances of virtual models can be viewed simultaneously for comparison. Virtual, interactive renderings can instead be generated digitally on a computer monitor or other display, and/or sent to a three-dimensional printer for cheap, fast prototyping, making virtual models a tremendously efficient quality control step in the production process, or even a complete alternative to traditional prototyping.

One very large advantage of digital/virtual models is the ability to instantly create exact replicas. Duplicated models can be easily modified for subtle changes and compared for aesthetic or any other preferences. In traditional modeling, each model requires substantial time and resources, and exact replicas are not possible because of subtle variations in the model-making process. In most cases, any substantial changes to a model that are desired must be considered before construction of a new instance of the model in order to be incorporated. This limitation makes it more difficult to affect further modifications that are conceived of during model making, which can therefore be a detriment to the creative process. Changes that can be made to an existing model after construction necessarily replace the original form, making side-by-side comparison impossible. Both of these limitations are easily navigated by the use of virtual models. Changes are easily incorporated at any stage of model development, and multiple, exact replicas can be instantaneously generated and quickly modified for side-by-side comparison or any other purpose.

While virtual representations are an advantageous tool for producers of creative content, the generation of such virtual content is highly specialized. The ability to generate such a useful model is highly dependent upon the skill of the modeler, and upon the modeler's familiarity with software that facilitates such modeling. Such software is highly technical and can take years to learn. Because of its highly specialized application, acquiring this skill is often unreasonable for inventors from varied backgrounds. Furthermore, such software is often prohibitively expensive. In short, the software required to generate virtual content itself, along with the ability to use it, are themselves often a barrier to efficiently generating useful models.

One particularly useful strategy in navigating technical challenges is to generate three-dimensional models based on existing objects in the real, physical world and modify them. Such models provide a detailed substrate for future modification, thus reducing the overall design burden. While useful, generating three-dimensional content based on real world objects is currently also challenging and time consuming for many modelers. This makes it impractical for many designers, despite regularly needing to do so for the purposes of either generating starting forms upon which a new model will be based, or as secondary objects that will interact with the model in some way. Modelers that do attempt to model real world objects are confined to the immobile nature of modeling software. Rather than modeling an object as they can see it in the real world, a modeler must find adequate images of the object they wish to model from many views. Typically, this requires that a blueprint of the object, detailing many views of the object in question, is available to them. Such blueprints are themselves highly technical, time consuming, and often expensive to generate.

One solution to these challenges is automated, imaging-based software capable of analyzing multiple two-dimensional images of the same object and generating three-dimensional coordinates of the object from these images. This would eliminate the technical challenges associated with generating such a model, the cost barrier of expensive modeling software, and the requirement of existing images or blueprints. Once the difficult part of generating a three-dimensional shape has been carried out, a designer can then modify the shape quickly and with relatively little training. Several three-dimensional scanned objects (or parts of objects) could then be assembled into a final model with relative ease.

One example of prior art has attempted to solve this problem using lasers and rotating platters. In this system, an object is placed on the rotating platter, and one or more lasers are aimed at the object. The distance from the laser to the object changes over time as the object rotates and the laser moves across the surface of the object, and these distances are detected and stored as a series of range vectors. These range vectors can be used to reconstruct the surface of the object in virtual space, thus producing a virtual model of the object in question. This prior art requires hardware that is expensive and inaccessible to most designers. Furthermore, such a device is immobile, and limited to objects small enough to fit on the rotating platter

Another example of an existing product is a piece of hardware consisting of a special camera that attaches to a tablet. The hardware is used to project an infrared laser beam in a specific pattern. The camera mounted on the tablet can then be used to “scan” back and forth across an object by way of user movement. An infrared sensor on the device collects and records distortions in a pattern at VGA resolution while scanning. The data collected from this process can then be used to create a depth map of the material being scanned. The tablet's camera is also invoked to record color information. The data can then be exported to software, which reassembles the images into a three-dimensional model.

The prior art requires special hardware and a secondary export to software, which requires additional resources such as a user's time and an expense of additional equipment, such as requiring a user to carry additional hardware. The additional hardware also requires its own battery and must be charged in addition to the tablet. Further, the extra hardware requirement limits the context in which the device is used. Specifically, its use is limited to tablets such as iPads, which requires a user to purchase one if they do not own one, in addition to the costs of specialized hardware. Any future use with anything other than a tablet would require new hardware to be developed in order to interface with existing hardware, such as a satellite, microscope, or mobile phone.

Another example of an existing product uses the camera in a mobile phone to acquire images where a user places an object on a reference surface and walks around the object, taking many static images of the object from many different angles, and revolving the object several times in the process. Although this product does not require special hardware, it is limited with regard to producing the type of models to be used in a production environment. The process is also highly dependent upon the skill of the user in finding appropriate angles and lighting conditions, and in taking enough images. Because the software involves simply image analysis, the software is incapable of distinguishing many objects from one another, and further requires the user to clean up the image by removing elements of the image that were inappropriately included in the three-dimensional model. This product is based on an existing ability to map virtual three-dimensional meshes onto points of light and dark, and is meant to be used in the context of specialized software in order to be useful, which is itself an example of the expensive modeling software mentioned previously.

Another example of an existing product can be used on a standard mobile phone to record images, but must be subsequently processed on far more powerful hardware. This requires a user to purchase very expensive accelerated graphics processors, or to pay a developer for access to their hardware as a service. These additional cost barriers are ostensibly the result of inefficient processing algorithms that require very powerful processors.

A process that does not require expensive hardware, software, or extensive training and experience would be far more desirable, and would substantially increase the accessibility of virtual model making. The increased accessibility and reduction on time and money demands could not only have a tremendous economic effect with the ability to generate prototypes of ideas, but should also help to identify problems and errors more quickly. It would provide designers with a much faster way of navigating the technical, difficult portion of model generation by taking advantage of existing technology to automatically generate a shape that is imaged with a camera, and without requiring the purchase of any additional hardware.

Virtual models of real objects simulated in three-dimensional space rendered in the manner described above have many possible applications beyond production and design. Such applications can range from routine and hobbyist to highly technical, specialized use if the ability to generate and use virtual models is more portable, requires less technical training to use, and more financially accessible to more people. Interior design applications, for example, could be tremendously expedited if one were able to portably record the interior of a room and have the recording translated into a virtual model. Alternatively, scientists studying in the field could record virtual models of specimens for future analysis without disturbing the specimen or its habitat. Applications such as this could not only save tremendously on costs, but would allow work to be carried out in a non-invasive manner.

SUMMARY

It is with respect to the above issues and other problems that the embodiments presented herein were contemplated. That is, in accordance with embodiments of the present disclosure, systems and methods are presented for scanning virtually any real-world object with a single image capture device, such as a digital camera. In accordance with embodiments of the present disclosure, a user, without using multiple cameras, takes one image of an object, rotates the camera at some angle and then takes another image of the object. By merging the two images together and then measuring a rotation, a depth of a scanned object can be determined. A distance as a value may be provided in accordance with embodiments of the present disclosure. For example, a distance from a mobile device with a camera to an object may be provided and later utilized to create and/or modify a virtual three-dimensional model of the object.

Embodiments of the present disclosure may be used in existing mobile technology like smartphones, or any digital camera with some type of rotational measurement device. A scanned object may be used for rapid prototyping and three-dimensional printing. Similar to a flat-bed two-dimensional scanner (e.g., a photocopier), embodiments of the present disclosure may scan any three-dimensional object and display a virtual three-dimensional object on a display of a smartphone and/or three-dimensionally print the scanned object. For example, virtual coordinates of an object may be sent straight to a 3D printer for physical object duplication. The applications of such a technology may be applied in many fields. For instance, if applied in the medical field, three-dimensional body scans may be obtained for surgical or other medical purposes. Such technology might also find tremendous utility in health applications. If, for example, a doctor could quickly generate a virtual model of some region on a patient in need of a medical procedure such as surgery, that doctor may familiarize herself with the region in question and get a sense of what must be done prior to surgery. This reduces physiological stress on the patient by shortening the duration of surgery and reduces the chances of accident by allowing the physician to know exactly what to expect, allowing her to become mentally prepared for the operation. Changes to a virtual model may also be verified by the patient prior to the surgery, further reducing risk of accident.

Similarly, the entertainment industry consisting of movies and games for example, may employ three-dimensional graphics on a regular basis, objects and characters may quickly be scanned and modeled. For example, video game developers or users may scan an object and create their own elements to incorporate into video games. In accordance with embodiments of the present disclosure, the security industry may benefit from conducting body scans of assailants without the assailant's knowledge because, unlike a laser, a camera may be utilized as a passive measuring device, meaning, the scanned object would have no knowledge that it is being scanned. Moreover, security cameras may also reconstruct virtual models of an object and, if retrofitted, could be made to include data outside the visual spectrum of light, such as from X-ray or ultrasound, for example.

In accordance with embodiments of the present disclosure, other applications such as but limited to satellite imaging may find great utility by employing systems, methods, and/or techniques described herein. For example, various sized regions (e.g., small, medium, large) of Earth may be surveyed by satellite utilizing the methods and systems described herein; accordingly, three-dimensional virtual models may be generated quickly and/or efficiently. Such virtual three-dimensional models may be very useful for climate modeling, as one example. Such satellite imaging cameras may be used to reconstruct virtual models of an object and, if retrofitted, could be made to include data outside the visual spectrum of light, such as from X-ray or ultrasound, for example.

Likewise, artists seeking to use a real, physical object in some capacity—such as for demolition in a film sequence—must currently either build a real, physical model of the item in question or generate a virtual one. The latter process requires extensive experience in virtual modeling and access to expensive modeling computer programs, as previously mentioned. The process of generating a virtual model is time consuming for the modeler and thus can be very expensive for the producer, all of which could be bypassed with a better imaging-based solution. Such a technology described herein may open up many possibilities for use with hobbyists who simply like to collect, model, and manipulate virtual counterparts of real world objects and further explore new areas of pastimes using such available technology. Such exploration has potential for generating new uses and new industries. For example, vendors could generate libraries of catalogued items for exchange, charging premiums on rare and high fidelity items. A technology that overcomes the challenges previously mentioned would therefore be tremendously invaluable to society.

In accordance with at least some embodiment of the present disclosure, a mobile device is provided, the mobile device including a display, a processor, and one or more memories including one or more instructions that when executed by the processor, cause the processor to obtain a plurality of two-dimensional images of an object in succession, the plurality of two-dimensional images being captured form different orientations, store said plurality of two-dimensional images into the one or more memories, determine a relative change in position of said plurality of two-dimensional images by comparing two subsequent images of the plurality of two-dimensional images, wherein the relative change is determined by a difference in color intensity values between pixels of one of the two subsequent images and the other image, generate a three-dimensional coordinate based on the relative change in position, generate a computer image including the three-dimensional coordinate, wherein said computer image represents the three-dimensional object, and output to the display, the computer image representing the three-dimensional object.

In accordance with at least some embodiments of the present disclosure, a method of generating a virtual three-dimension model of an object is provided, the method including obtaining a plurality of two-dimensional images of the object in succession, the plurality of two-dimensional images being captured form different orientations, storing said plurality of two-dimensional images into one or more memories, determining a relative change in position of said plurality of two-dimensional images by comparing two subsequent images of the plurality of two-dimensional images, wherein the relative change is determined by a difference in color intensity values between pixels of one of the two subsequent images and the other image, generating a three-dimensional coordinate based on the relative change in position, and generating a computer image including the three-dimensional coordinate, wherein said computer image represents the three-dimensional object.

Further yet, and in accordance with at least some embodiment of the present disclosure, a non-transitory computer readable medium having stored thereon instructions that cause a processor to execute a method is provided, the method comprising one or more instructions to obtain a plurality of two-dimensional images of an object in succession, the plurality of two-dimensional images being captured form different orientations, one or more instructions to store said plurality of two-dimensional images into the one or more memories, one or more instructions to determine a relative change in position of said plurality of two-dimensional images by comparing two subsequent images of the plurality of two-dimensional images, wherein the relative change is determined by a difference in color intensity values between pixels of one of the two subsequent images and the other image, one or more instructions to generate a three-dimensional coordinate based on the relative change in position, one or more instructions to generate a computer image including the three-dimensional coordinate, wherein said computer image represents the three-dimensional object, and one or more instructions to output to the display, the computer image representing the three-dimensional object.

Accordingly, it is these and other advantages that will be apparent from the disclosure.

