CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of Ser. No. 19217058.7, filed 17 Dec. 2019 in Europe and which application is incorporated herein by reference in its entirety. To the extent appropriate, a claim of priority is made to the above disclosed application.

TECHNICAL FIELD

The present disclosure relates to the field of image analysis, in particular, it relates to a method for augmenting a first image using image data from a second image. The disclosure also relates to a device comprising circuitry configured to carry out the method. The disclosure also relates to a computer program product adapted to carry out the method.

BACKGROUND

In image search approaches, it is often desirous to determine and identify which objects are present in an image. Image search approaches and image recognition approaches are common for commercial use, for example to generate product catalogues and product suggestions. It has been desirous to achieve a system where a user can take a photograph of a room, where the image search process can use image data to search product catalogues on the internet to return for example different stores' prices for a given product.

It is often desirous to augment an image such that it can easily be manipulated by a user. This is beneficial for industrial areas where predicting visualizations are can be used such as building construction or indoor design.

Typically, an image can be augmented such that a data model of an object can be placed in the image such that a user can visualize how the scene may look like with the added model. However, these methods are limited, and the final augmented image is a virtual image wherein the realness of the object is missing, since it is constructed from a model.

Therefore, there is room for improvements in the field of image analysis and augmenting of images.

SUMMARY

In view of that stated above, an object of the present disclosure is to provide a method for augmenting an image with image data relating to an object such that the augmented image contains real image data for the object. It is further an object to use image analysis and image manipulation approaches to achieve the above in an automatic or semi-automatic fashion.

To achieve at least one of the above objects and also other objects that will be evident from the following description, a computer implemented method, a device for carrying out the method, and a computer program product for carrying out the method provided according to the present disclosure. Preferred embodiments of the device will be evident from the description.

According to a first aspect, there is provided a computer implemented method for augmenting a first image with image data from a second image, the method comprising:

receiving the first image depicting a first scene;

capturing the second image depicting a second scene using an image capturing device;

identifying an object in the second image;

receiving a 3D model from a database, the database comprising a plurality of 3D models, the 3D model corresponding to the identified object of the second image;

aligning the 3D model with the identified object in the second image;

extracting pixel data from the second image using a contour of the 3D model's projection onto the second image;

inserting the extracted pixel data of the second image data into the first image, thereby rendering an augmented image.

Objects in the second image may be segmented (extracted, distinguished, etc.) using any known algorithm, such as algorithms using one or more of edge features, binary patterns, directional patterns, Gradient features, Spatio-Temporal domain features etc.

The step of identifying an object in the second image may be done by an image search algorithm. By the term “image search algorithm”, should in the context of the present specification, be understood as any known way to search for images (of objects) in a database which are similar to an object of the image and use the outcome (e.g. labels/classification of similar images found in the database) to classify the object. Examples of known commercial image search algorithms at the filing of this disclosure comprises Google images, TinEye and Alibabas Pailitao.

The provided method is an improved method for augmenting an image and keeping the augmented image realistic. By adding image data from the second image, the realistic view of the augmented image may be achieved.

By receiving a 3D model for the identified object, the image segmentation may be improved such that only, or almost only, the image data pertaining to the identified object is extracted, and such that all or almost all image data pertaining to the identified object is extracted. The received 3D model is thus used to decide which part of the second image to extract image data to avoid extracting any surrounding of the object. This may improve the augmented image by improving the realistic view of the image in an efficient manner. A low-complexity process for adding image data from a second image to a first image may thus be achieved.

The provided method allows a user to augment an existing image with image data from another image (e.g. newly taken) to see what an object may look like in the scene of the first image.

According to some embodiments, the method further comprises

determining a first 3D coordinate space of the first image;

determining a second 3D coordinate space of the second image;

determining an object coordinate for the identified object in the second image using the second 3D coordinate space, and

wherein inserting the extracted pixel data of the second image data into the first image comprises inserting the extracted pixel data, at a position coordinate in the first image corresponding to the object coordinate. By determining a first and second 3D coordinate space, a low-complexity model for inserting pixel-data into the first image may be achieved. With the position coordinate corresponding to the object coordinate the predictability for where in the first image the pixel data is inserted may be more precise.

Thereby, placement of the extracted pixel data at a desirous position in the first image is facilitated. Accordingly, the desired position may be easy to manipulate due to the adapted 3D coordinate spaces between the two images.

By way of example, if the identified object in the second scene is a chair at a right bottom corner of the second scene, the extracted pixel data of said chair may be automatically placed at a bottom corner of the first scene.

According to some embodiments, the method further comprises adjusting the position of the extracted pixel data in the augmented image by:

zooming in or out, or translating the first image, thereby manipulating the first 3D coordinate space of the first image,

adjusting the position of the extracted pixel data in the augmented image to correspond to the object coordinate in the manipulated 3D coordinate space of the first image.

By adjusting the position of the pixel data in the augmented image, image manipulation of the augmented image may be facilitated in an efficient manner. The manipulation of the first image allows for manipulation of the augmented image without requiring manipulation of the extracted pixel data, or without requiring that a new image of the object is captured, thus leading to a simplified procedure for augmenting the first image with image data from the second image.

According to some embodiments, the method further comprises

for the 3D model, defining a first value for a scale parameter, a first value for a translation parameter, and a first value for a rotation parameter,

for the identified object in the second image, defining a second value for a scale parameter, a second value for a translation parameter, and a second value for a rotation parameter, and

aligning the 3D model to the determined 3D coordinate space of the second image by manipulating one or more of the first scale parameter, the first translation parameter and the first rotational parameter of the 3D model, such that the first scale parameter corresponds to the second scale parameter, and the first translation parameter corresponds to the second translation parameter, and the first rotation parameter corresponds to the second rotation parameter. The alignment of the 3D model to the identified object in the second image may be performed in an efficient and automatic manner by the rotation, movement or scaling of the 3D model. An image algorithm may be applied to the image and the 3D model to determine when the 3D model has a proper alignment with the identified object.

According to some embodiments the method comprises

requesting an input from a user;

receiving an input from the user, and wherein the step of receiving the 3D model comprises sending data pertaining to the received input to the identified object to the database. By utilizing a user input, the processing power needed to perform the step of receiving a 3D model may be reduced. The user input may pertain to the identified object in the second image, and accordingly, data may be sent to the database requesting a 3D model of such identified object. The user input may in some embodiments pertain to a category or object type of the identified object, accordingly lessen the processing time needed to receive a 3D model corresponding to the identified object.

A user input may allow the method to omit processing steps, leading to a more efficient method for augmenting a first image with image data form a second image. The user input may be requested and received in any known manner such as using a graphical user interface, a voice interface, etc.

According to some embodiments, the method further comprises.

identifying one or more objects in the first image; and

removing at least one of the one or more identified objects in the first image or manipulating at least one of the one or more identified objects by changing a position of at least one of the one or more identified objects in the first image.

By removing, or changing a position of, an object, an empty space in the scene may be created. Consequently, the present embodiment allows for rearranging the scene in the first image as desired to fit the pixel data of the identified object from the second image. Any known method or software for changing a position of, or removing, an object in an image may be employed.

According to some embodiments, the method further comprises

removing the inserted pixel data from the augmented image;

capturing a third image of the second scene using the image capturing device;

re-identifying the previously identified object of the second image in the third image;

calculating a third 3D coordinate space for the third image;

aligning the 3D model with the identified object in third image;

extracting pixel data from the third image of a contour of the 3D model projection onto the third image;

determining a second object coordinate for the identified object in the third image using the third 3D coordinate space;

inserting the extracted pixel data of the third image data into the first image at a position coordinate in the first image corresponding to the second object coordinate.

As such, the augmented image may be manipulated in a simple manner, where the first image is left untouched, and the second image is re-captured as the third image to provide a low-complexity method. By way of example, if a user is unsatisfied with the positioning of the identified object in the augmented image, the user may capture a new image of the second scene comprising the identified object. By the user and the capturing camera device changing position or angle in relation to the identified object, the second object coordinate is manipulated and accordingly, the position coordinate may be manipulated. Advantageously, a low-complexity method may be provided where the realistic view of an augmented image may be manipulated in a simple manner.

According to some embodiments, the step of identifying the object in the second image comprises determining an object type using an image search algorithm. Any suitable image search algorithm may be used to determine the object type. By determining the object type for the object, the method may identify the object in a more efficient manner. By determining the object type, the receiving of the 3D model may be based on the object type. For example, if the object type is deemed to be utensils, the received 3D model may be of the same object type, thus saving time during calculations.

According to some embodiments, the step of identifying the object in the second image comprises scanning a QR code on the object to be identified in the second scene. The QR code may comprise data pertaining to the object type of the identified object. It is to be noted that the QR code may contain a variety of relevant data for identifying the object. The data contained in the QR code may be sent to the database for the receiving of the 3D model. The QR code may in some embodiments comprise data pertaining to a 3D model of the object having said QR code, e.g. an identifier for the 3D model in the database.

According to some embodiments, the step of identifying the object in the second image comprises reading a beacon signal transmitted by the object to be identified in the second scene. By way of example, the beacon signal may determine which object in the image is closest to the camera device, and thereby said object may be the chosen object to be identified. The beacon signal may comprise data identifying the object and such data may thus be transmitted to the database to receive the corresponding 3D model from the database.

According to some embodiments, the step of receiving the 3D model comprises sending image data pertaining to the identified object to the database. As such, mapping the image data to a corresponding 3D model may be performed by e.g. a server hosting the database. Consequently, the computational power required to perform the method of augmenting the first image with image data from the second image may be advantageously distributed between several devices . . . .

According to some embodiments, the step of capturing a second image comprises using AR or VR equipment, and wherein the method further comprises displaying the augmented image using said AR or VR equipment. By using AR or VR equipment the augmented image may be realistically displayed to a user.

It is to be noted that the second image may be captured using any camera device.

According to some embodiments, the method further comprises storing the augmented image in a memory. It is to be noted that the memory may also store the first image. The memory may also store the second image. Data pertaining to the executed steps of the method may be stored in the memory. By storing the augmented image in the memory, the method may be performed iteratively such as to become more and more automatic and quicker in terms of processing. In other words, by the memory storing data pertaining to the images and optionally method steps performed thereto, the memory may generate a compilation of examples of how to for example place an object in the first image. The memory may gather data pertaining to common placements of object types in relation to a specific scene. Such data may be utilised by the memory to recommend manipulations of any one of the images or to recommend a position for placement of the extracted pixel data.

According to a second aspect, there is provided a device configured to augment a first image with image data from a second image, the device comprising circuitry configured to perform the method of the first aspect.

According to a third aspect, there is provided a computer program product comprising a computer-readable storage medium with instructions adapted to carry out the method of the first aspect when executed by a device having processing capabilities.

The second and third aspects may generally have the same features and advantages as the first aspect.

Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a/an/the [element, device, component, means, step, etc]” are to be interpreted openly as referring to at least one instance of said element, device, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

