BACKGROUND

This description relates to ‘built’ environments and more particularly to representation of locations of items in ‘built’ environments.

Recent advances in video and depth sensor technologies have been used to process sensor data into 3D maps that represent structure of a “built environment.” Such 3D maps can be used in various applications. Building information modelling (BIM) involves producing digital representations of physical and functional characteristics of such environments that are represented in BIM data. Building information modelling often assumes the availability of the location of the devices (assets) as part of the BIM data. Such technological developments are often used in commercial applications for various purposes, such as in developing “as built” documentation.

SUMMARY

This application claims priority to the above mentioned provisional U.S. Patent Application 62/677,219 ('219 application), filed on May 29, 2018, the entire contents of which are hereby incorporated by reference. In the event of disagreement between the subject matter of the '219 application and the present application, the contents of the present application shall control.

According to an aspect, a method includes receiving by a computing device one or more two dimensional digital images of a scene, the image including plural pixels, applying the received image data to scene generator/scene understanding engine that produces from the one or more digital images a metadata output that includes depth prediction data for at least some of the plural pixels in the two dimensional image and that produces metadata for a controlling a three-dimensional computer model engine, and outputting the metadata to a three-dimensional computer model engine to produce a three-dimensional digital computer model of the scene depicted in the two dimensional image.

The above aspect may include amongst other features one or more of the following features:

The method wherein receiving further includes receiving the one or more images and reference measurements associated with the one or more images. The method wherein applying the received image to the scene generator engine/scene understanding service, further comprises the scene generator engine/scene understanding service identifying objects within the image scene; and applying labels to the identified objects. The method wherein identifying objects further includes extracting each labeled object's region and determining and outputting pixel corner coordinates, height and width and confidence values into metadata output to provide specific instructions for a 3D modeling engine to produce the 3D model. The method further includes generating with the metadata statistical information on the identified objects within the image scene.

The method further includes inferring depths of pixels in the image. The method wherein inferring depths of pixels in the image includes transforming the input image by a superpixel segmentation that combines small homogenous regions of pixels into superpixels that are homogenous regions of pixels that combined to function as a single input. The method wherein inferring depths of pixels in the image further includes determining a penalty function for superpixels by determining unary values over each of the superpixels, determining pairwise values over each of the superpixels and determining a combination of the unary and the pairwise values.

The method wherein the unary processing returns a depth value for a single superpixel and the pairwise communicates with neighboring superpixels having similar appearance to produce similar depths for those neighboring superpixels. The method wherein the unary processing for a single superpixel is determined by inputting the single superpixel into a fully convolutional neural net that produces a convolutional map that has been up-sampled to the original image size, applying the up-sampled convolutional map and the superpixel segmentation over the original input image to a superpixel average pooling layer to produce feature vectors; and input the feature vectors to a fully connected output layer to produce a unary output for the superpixel.

The method wherein the pairwise processing for a single superpixel is determined by collecting similar feature vectors are collected from all neighboring superpixel patches adjacent to the single superpixel, cataloguing unique feature vectors of the superpixel and neighboring superpixel patches into collections of similar and unique features, and input the collections into a fully connected layer that outputs a vector of similarities between the neighboring superpixel patches and the single superpixel. The method wherein the unary output is fed into the conditional random fields graph model to produce an output depth map that contains information relating to the distance of surfaces of scene objects from a reference point. The method wherein the depth prediction engine processes input digital pixel data through a pre-trained convolutional neural network to produce a depth map.

According to an aspect, a method includes producing by one or more computer systems a spatial depth prediction of pixels in an image by transforming pixels in an image into a segmented representation, by applying small homogenous regions of pixels to a segmentation of the image to produce superpixel representations of the homogenous regions and processing the segmented representation by determining a pairwise of energy potentials and unary energy potentials to produce a depth map.

The method wherein pairwise potential processing includes matching neighboring super pixels, calculating differences of feature vectors, calculating a fully connected output layer, calculating a conditional random fields graph model loss layer, and producing an output depth map for the input image. The method wherein the unary energy potential processing includes calculating a superpixel segmentation from a super pixel map and an output from a convolutional map of up-sampling data, calculating a superpixel average pool, calculating a fully connected output layer and producing an output depth map for the input image.

According to an aspect, a computer implemented method of producing a three dimensional computer model shell includes producing metadata that instruct a three dimensional modelling engine to produce a spatial shell and to place objects within the spatial shell, with placed objects being placeholders that are swapped for other objects, producing the shell process by producing a scaffold mesh, producing a bounding box, producing a clean final mesh; and producing the three dimensional modelling shell.

The method wherein object placement includes calculating the original coordinate position of bounding boxes, transforming object positions and orientations based on original coordinates, placing the objects into the three dimensional computer model shell.

Other aspects include systems, devices, tablets, smartphones, and/or computer program products embodying computer executable instructions on a non-transitory computer readable medium or media that function according to one or more of the above aspects. Other aspects are disclosed in the detailed description.

One or more aspects include techniques for producing a three-dimensional model of a scene from one or more two dimensional images. The techniques include converting one (or one or more) digital images into a 3-dimensional computer model by receiving by one or more image data translation systems digital image source data, applying the received image data to scene generator engine/scene understanding service, and returning a 3-dimensional digital computer model. A potential use of such technology in residential remodeling. However, for the most part such technology is either too expensive or too complicated for relatively small contractors to use. Other uses exist including in other built in or as built programs.

The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

