BACKGROUND

It is often useful to have a schematic, blueprint or other graphical representation of rooms of a building when either moving furniture, buying new furniture, buying carpets or rugs, remodeling, repainting or otherwise modifying or cataloguing characteristic of rooms or elements therein. Various products exist, which are supposed to assist users in producing such graphical representations. Some such products, which are implemented using software, typically require that a user spend a large amount of time taking manual measurements of rooms and items therein and then manually entering such measurements into a computing device to enable the software running on computing device to generate models based on the manually entered information. More recently, special cameras have been developed that remove some of the manual procedure previously necessary. For example, 360-degree cameras are available that have a field of view (FOV) that covers a full circle in the horizontal plane. Such a camera can be placed in a room (or other region) to obtain a 360-degree image of the room (or other region) from which a virtual tour of the room (or other region) can be generated. One potential problem of using a 360-degree camera is that the person (aka photographer) that is controlling the camera can inadvertently or at least undesirably be captured in the 360-degree image because, unlike when using a more conventional camera having a smaller FOV, the person cannot easily stand outside the FOV of the camera (e.g., by standing behind the camera). One way to overcome this problem is for the person (aka photographer) to place the 360-degree camera on a tripod and then stand in another room and use a remote control to trigger the 360-degree camera. However, such a solution is not optimal as it increases the time and complexity associated with obtaining the 360-degree images of rooms and/or other regions.

After a schematic, blueprint or other graphical representation of a room (or other region) has been generated, it would be useful if certain items of interest in the graphical representation can be tagged and spatial coordinates of such items of interest can be determined. For example, assume a schematic, blueprint or other graphical representation of rooms of a rental unit are being used to specify items of interest that need to be repaired before a tenant either moves into or out of the rental unit. It would be beneficial if the items that need to be repaired can be easily and readily tagged within the schematic, blueprint or other graphical representation in a manner that provides good specificity, e.g., to clearly indicate which one of numerous cabinet pulls in a kitchen needs to be repaired, or to clearly indicate which one of a number of door knobs in a bedroom needs to be repaired.

SUMMARY

Certain embodiments of the present invention can be used to essentially remove a person from an image, or more specifically, to generate an image that is devoid of a person. A method of such an embodiment includes using a camera to obtain a first image (A) of a scene within a FOV of the camera while a person is at a first location within the FOV of the camera, and thus, the person appears in a first portion of the first image (A). The method further includes obtaining a second image (B) of the scene within the FOV of the camera while the person is at a second location within the FOV of the camera that differs from the first location, and thus, the person appears in a second portion of the second image (B) that differs from the first portion of the first image (A). Additionally, the method includes generating, based on the first and second images (A and B), a third image (C) of the scene, such that the third image (C) of the scene is devoid of the person and includes portions of the scene that were blocked by the person in the first and second images (A and B), wherein the generating is performing using one or more processors. While such embodiments are especially useful with a 360-degree camera, such embodiments are also useful with cameras having other FOVs, such as a FOV that is 120 degrees or 180 degrees, but not limited thereto. In certain embodiments, the first and second images (A and B) are captured using a 360-degree camera (or another camera having some other FOV) that is being controlled by a mobile computing device that is in wireless communication camera. Such a mobile computing device that controls the 360-degree camera (or another camera having some other FOV) can be, e.g., a smartphone or a tablet type of mobile computing device, but is not limited thereto. An application installed on such a mobile computing device can be used to control the 360-degree camera (or another camera having some other FOV), as well as to generate the third image (C) of the scene that is device of the person and includes portions of the scene that were blocked by the person in the first and second images (A and B).

In accordance with certain embodiments, the third image (C) of the scene is generate using computer vision to identify the person within each of the first and second images (A and B), and combining a portion of the first image (A) that is devoid of the person with a portion of the second image (B) that is devoid of the person to produce the third image (C) of the scene that is devoid of the person and includes the portions of the scene that were blocked by the person in the first and second images.

In accordance with certain embodiments, the third image (C) of the scene is generated by: identifying first and second portions (A1, A2) of the first image (A) that differ from the second image (B); identifying first and second portions (B1, B2) of the second image (B) that differ from the first image (A); determining a first metric of similarity (a1) indicative of similarity between the first portion (A1) of the first image (A) that differs from the second image (B) and a remaining portion of the first image (A); determining a second metric of similarity (a2) indicative of similarity between the second portion (A2) of the first image (A) that differs from the second image (B) and a remaining portion of the first image (A); determining a third metric of similarity (b1) indicative of similarity between the first portion (B1) of the second image (B) that differs from the first image (A) and a remaining portion of the second image (B); and determining a fourth metric of similarity (b2) indicative of similarity between the second portion (B2) of the second image (B) that differs from the first image (A) and a remaining portion of the first image (A). Further, the third image (C) of the scene is generated by determining, based on the first, second, third, and fourth metrics of similarity (a1, a2, b1, b2), which one of the first portion (A1) of the first image (A) and the first portion (B1) of the second image (B) is to be included the third image (C), and which one of the second portion (A2) of the first image (A) and the second portion (B2) of the second image (B) is to be included the third image (C). More specifically, this may include comparing a sum of the first and fourth metrics (a1+b2) to a sum of the second and third metrics (a2+b3), e.g., to determine whether or not the sum of the first and fourth metrics (a1+b2) is less than the sum of the second and third metrics (a2+b3). Then, based on results of the comparing, there is a determination of which one of the first portion (A1) of the first image (A) and the first portion (B1) of the second image (B) is to be included the third image (C), and which one of the second portion (A2) of the first image (A) and the second portion (B2) of the second image (B) is to be included the third image (C).

In accordance with certain embodiments, for each of the first, second, third, and fourth metrics of similarity (a1, a2, b1, b2), a lower magnitude is indicative of higher similarity, and higher magnitude is indicative of a lower similarity. In such embodiments the comparing comprises determining whether the sum of the first and fourth metrics (a1+b2) is less than or greater than the sum of the second and third metrics (a2+b3). In response to determining that the sum of the first and fourth metrics (a1+b2) is less than the sum of the second and third metrics (a2+b3), there is a determination that the first portion (A1) of the first image (A) and the second portion (B2) of the second image (B) are to be included in the third image (C). On the other hand, in response to determining that the sum of the first and fourth metrics (a1+b2) is greater than the sum of the second and third metrics (a2+b3), there is a determination that the second portion (A2) of the first image (A) and the first portion (B1) of the second image (B) are to be included in the third image (C).

In accordance with other embodiments, for each of the first, second, third, and fourth metrics of similarity (a1, a2, b1, b2), a lower magnitude is indicative of lower similarity, and higher magnitude is indicative of a higher similarity. In such embodiments the comparing comprises determining whether the sum of the first and fourth metrics (a1+b2) is less than or greater than the sum of the second and third metrics (a2+b3). In response to determining that the sum of the first and fourth metrics (a1+b2) is greater than the sum of the second and third metrics (a2+b3), there is a determination that the first portion (A1) of the first image (A) and the second portion (B2) of the second image (B) are to be included in the third image (C). On the other hand, in response to determining that the sum of the first and fourth metrics (a1+b2) is less than the sum of the second and third metrics (a2+b3), there is a determination that the second portion (A2) of the first image (A) and the first portion (B1) of the second image (B) are to be included in the third image (C).

Certain embodiments of the present technology are also directed to one or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform the methods summarized above.

Certain embodiments of the present technology are related to a method for use with a first mobile device comprising a first camera and a second mobile device, wherein the method is for determining spatial information for one or more items of interest within a graphical representation of a region generated based on one or more images of the region captured using the first camera of the first mobile device. Such a method comprises capturing one or more images of the region using the first camera of the first mobile device and generating or otherwise obtaining the graphical representation of the region based on the one or more images of the region captured using the first camera of the first mobile device. The method also includes, for each item of interest, of the one or more items of interest, using the first camera of the first mobile device to capture one or more further images of the region while the second mobile device is placed in close proximity to the item of interest, and thus, the second mobile device appears in the one or more further images. The method further includes, for each item of interest, of the one or more items of interest, determining spatial information for the item of interest based on the one or more further images of the region within which the second mobile device appears.

In accordance with certain embodiments, the second mobile device includes a front side on which is located a display and a front side camera, and a back side on which is located a back side camera, and the method includes: displaying an indicator on the display of the second mobile device, such that the indicator will be shown in the one or more images of the region captured using the first camera of the first mobile device.

In accordance with certain embodiments, determining spatial information, for an item of interest based on the one or more further images of the region within which the second mobile device appears, comprises intersecting a ray from a center of the first camera of the first mobile device to the second mobile device that appears near the item of interest within the graphical representation of the region.

In accordance with certain embodiments, an item of interest within the graphical representation of the region is identified based on the indicator on the display of the second mobile device included in the one or more images of the region captured using the first camera of the first mobile device.

In accordance with certain embodiments the method further comprises, for each item of interest, of the one or more items of interest: capturing a further image that includes the first mobile device, using the front side camera of the second mobile device; and using the further image, captured using the front side camera of the second mobile device, to increase at least one of reliability or accuracy of the spatial information determined for the item of interest.

In accordance with certain embodiments, the first camera of the first mobile device comprises a 360-degree camera. In accordance with certain embodiments, the second mobile device comprises one of a smartphone or a tablet type of mobile computing device.

In accordance with certain embodiments, determining spatial information, for an item of interest based on the one or more further images of the region within which the second mobile device appears, comprises identifying an arm or other body part of a person holding the second mobile device, and intersecting a ray from a center of the first camera of the first mobile device to the identified arm of other body part of the person located near the item of interest within a graphical representation of the region.

Certain embodiments of the present technology are also directed to one or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform the methods summarized above.

A system according to certain embodiments of the present technology comprises: a first mobile device comprising a first camera that is used to capture one or more images of a region, which one or more images are used to generate a graphical representation of the region; and a second mobile device comprising one or more processors. In certain such embodiments, the first mobile device is configured to capture using the first camera thereof, for each item of interest of one or more items of interest, one or more further images of the region while the second mobile device is placed in close proximity to the item of interest, and thus, the second mobile device appears in the one or more further images. At least one of the one or more processors of the second mobile device is configured to determine, for each item of interest of the one or more items of interest, spatial information for the item of interest based on the one or more further images of the region within which the second mobile device appears. The graphical representation of the region can be generated using one or more processors of the first mobile device, of the second mobile device and/or of a server that receives the one or more images of the region captured by the first camera of the first mobile device.

In accordance with certain embodiments, at least one of the one or more processors of the second mobile device is configured to determine, for each item of interest of the one or more items of interest, spatial information for the item of interest by intersecting a ray from a center of the first camera of the first mobile device to the second mobile device that appears near the item of interest within the graphical representation of the region.

In accordance with certain embodiments, the second mobile device includes a front side on which is located a display and a front side camera, and a back side on which is located a back side camera; the second mobile device is configured to display an indicator on the display thereof, such that the indicator will be shown in the one or more images of the region captured using the first camera of the first mobile device; and an item of interest within the graphical representation of the region is identified based on the indicator on the display of the second mobile device included in the one or more images of the region captured using the first camera of the first mobile device.

In accordance with certain embodiments, the second mobile device is configured, for each item of interest of the one or more items of interest, to capture a further image that includes the first mobile device, using the front side camera of the second mobile device; and the second mobile device is configured use the further image, captured using the front side camera of the thereof, to increase at least one of reliability or accuracy of the spatial information determined for at least one item of interest.

In accordance with certain embodiments, at least one of the one or more processors of the second mobile device is configured to determine the spatial information for an item of interest by identifying an arm or other body part of a person holding the second mobile device, and intersecting a ray from a center of the first camera of the first mobile device to the identified arm or other body part of the person located near the item of interest within a graphical representation of the region.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

