CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Korean Patent Application No. 10-2018-0164797, filed on Dec. 19, 2018, in the Korean Intellectual Property Office, the disclosure of which is herein incorporated by reference in its entirety.

BACKGROUND

1. Field

Devices and methods consistent the disclosure relate to an electronic apparatus and a controlling method thereof, and more particularly, to an electronic apparatus for identifying an object and a controlling method thereof.

The disclosure also relates to an artificial intelligence (AI) system that simulates functions, such as recognition and judgment, of a human brain using a machine learning algorithm and its application.

2. Description of the Related Art

Recently, artificial intelligence systems that implement human-level artificial intelligence (AI) have been deployed in various fields. An artificial intelligence system is a system embodied in a machine which learns, judges and iteratively improves execution of its function, unlike a conventional rules-based smart system. For example, as use of the artificial intelligence system increases, a recognition rate and the understanding of user's preferences may commensurately increase. Thus, the conventional rule-based smart system has been gradually replaced with a deep-learning based artificial intelligence system.

Artificial intelligence technology consists of machine learning (e.g., deep-learning) and element technologies that implement machine learning.

Machine learning is an algorithmic technology that classifies and/or trains the characteristics of input data. Element technology is a technology that simulates functions, such as recognition and judgment of the human brain, using a machine learning algorithm, such as deep learning, and includes linguistic understanding, visual understanding, reasoning/prediction, knowledge representation, motion control, etc.

Artificial intelligence technology may be applied to various fields, examples of which are described below. Linguistic understanding is a technology for recognizing and applying/processing human language/characters, including natural language processing, machine translation, dialogue system, query response, speech recognition/synthesis, and the like. Visual comprehension is a technology for recognizing and processing an object as if perceived by a human being, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Inference prediction is a technology for judging and logically inferring and predicting information, including knowledge/probability-based reasoning, optimization prediction, preference-bases planning, and recommendations. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling the autonomous movements of a device or object, e.g., travel of a vehicle and the motion of a robot, including motion control (navigation, collision and traveling), operation control (behavior control), and the like.

Recently, various types of devices, such as a glasses device including a camera, have been developed. However, when capturing an object using the camera provided in the glasses device, image blur may occur owing to motion of a user's head or body. For example, referring toFIG. 1, a glasses device may include a camera, and image blur may occur in the image captured by the camera as a result of movement by the user.

Image blur may cause obstacles to augmented reality (AR) services requiring real-time image analysis, and that identification and tracking of objects may be difficult.

Accordingly, a stabilization method for eliminating image blur has been developed. However, setting a parameter for each image frame may be conventionally employed. In case of fast motion in an image, image data can be lost due to excessive crop.

SUMMARY

An aspect of the exemplary embodiments relates to an electronic apparatus for providing an Augmented Reality (AR) function including a motion-stable object by improving an object identification performance from a plurality of consecutive frames, and a controlling method thereof.

According to an embodiment, there is provided an electronic apparatus including a memory, and a processor configured to, obtain a panoramic image by overlapping a partial area of a first image frame stored in the memory with a partial area of at least one second image frame stored in the memory based on pixel information of the first frame and pixel information of the at least one second image frame, identify an area of a predetermined shape of maximum size within the panoramic image, and identify an object from an entire area of the panoramic image or the area in the predetermined shape.

The processor may be further configured to obtain the panoramic image by overlapping an area having a minimum difference between pixel values of adjacent image frames among the first image frame and the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second image frame.

The processor may be further configured to obtain a motion value between adjacent image frames among the first image frame and the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second image frame, and obtain the panoramic image by overlapping the partial area of the first image frame with the partial area of the at least one second image frame based on the motion value.

The processor may be further configured to convert the motion value based on a difference between pixel values in the adjacent image frames and a motion value between the adjacent image frames, and obtain the panoramic image by overlapping the partial area of the first image frame with the partial area of the at least one second image frame based on the converted motion value.

The processor may be further configured to, perform image processing comprising at least one of rotation, position movement, or resizing with respect to each of the first image frame and the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second image frame, and obtain the panoramic image by overlapping a partial area of a frame in which the image processing is performed.

The electronic apparatus may further include a display, wherein the processor is further configured to control the display to display the first image frame, and based on the object identified from the panoramic image, control the display to display the object including at least one of graphical user interface (GUI), character, image, video, or 3D model on an area in which the object is displayed in the first frame in accordance with information on a location at which the object is identified and information on the image processing.

The at least one second frame may be a frame captured before the first frame, wherein the processor is further configured to, based on the panoramic image and pixel information on a third image frame captured after the first image frame, update the panoramic image by overlapping a partial area of the panoramic image with a partial area of the third image frame, identify an area of a predetermined shape of maximum size within the updated panoramic image, and re-identify an object from the updated panoramic image or the area in the predetermined shape within the updated panoramic image.

The third frame may be a frame captured after the first frame, and the processor may be further configured to, based on a ratio between the first image frame with respect to the third image frame and an overlapping area of the third image frame being smaller than a predetermined ratio, re-identify the object from the updated panoramic image, or the area in the predetermined shape within the updated panoramic image.

The processor may be further configured to, based on a plurality of objects being identified in the panoramic image, assign a weighted value to a plurality of respective areas based on at least one of a number of overlapping image frames of a plurality of respective areas of the panoramic image or a capturing time of an image frame in the plurality of respective areas, and identify at least one of the plurality of objects based on a weighted value of the plurality of respective areas.

The electronic apparatus may further include a camera comprising circuitry, wherein the processor is further configured to obtain the plurality of image frames by performing continuous capturing through the camera.

According to an exemplary embodiment, there is provided a method for controlling an electronic apparatus, the method including based on pixel information of a first image frame and pixel information of at least one second image frame among a plurality of frames, obtaining a panoramic image by overlapping a partial area of the first image frame with a partial area of the at least one second image frame, identifying an area of a predetermined shape of maximum size within the panoramic image, and identifying an object from an entire area of the panoramic image or the area in the predetermined shape.

The obtaining may include obtaining the panoramic image by overlapping an area having a minimum difference between pixel values of adjacent image frames among the first image frame and the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second frame.

The obtaining may include obtaining a motion value between adjacent image frames among the first image frame and the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second frame, and obtaining the panoramic image by overlapping the partial area of the first image frame with the partial area of the at least one second image frame based on the motion value.

The obtaining may include converting the motion value based on a difference between pixel values in the adjacent image frames and a motion value between the adjacent image frames, and based on the converted motion value, obtaining the panoramic image by overlapping the partial area of the first image frame with the partial area of the at least one second image frame.

The obtaining may include performing image processing comprising at least one of rotation, positon movement, or resizing with respect to each of the first image frame or the at least one second image frame based on the pixel information of the first image frame and the pixel information of the at least one second image frame, and obtaining the panoramic image by overlapping a partial area of a frame in which the image processing is performed.

The method may further include displaying the first frame, and based on the object identified from the panoramic image, displaying the object including at least one of graphical user interface (GUI), character, image, video, or 3D model on an area in which the object is displayed in the first frame based on information on a location at which the object is identified from the panoramic image and information on the image processing.

The at least one second frame may be a frame captured before the first frame, wherein the method further includes based on the panoramic image and pixel information on a third image frame captured after the first image frame, updating the panoramic image by overlapping a partial image of the panoramic image with a partial area of the third image frame, identifying an area of a predetermined shape of maximum size within the updated panoramic image, and re-identifying an object from an entire area of the updated panoramic image or the area in the predetermined shape within the updated panoramic image.

The third frame may be a frame captured after the first frame, wherein the re-identifying of the object comprises, based on a ratio between the first image frame with respect to the third image frame and an overlapping area of the third image frame being smaller than a predetermined ratio, re-identifying the object from the updated panoramic image, or the area in the predetermined shape within the updated panoramic image.

The identifying of the object may include based on a plurality of objects being identified in the panoramic image, assigning a weighted value to a plurality of respective areas based on at least one of a number of overlapping image frames of a plurality of respective areas of the panoramic image or a capturing time of an image frame in the plurality of respective areas, and identifying at least one of the plurality of objects based on a weighted value of the plurality of respective areas.

The method may further include obtaining the plurality of frames by performing continuous capturing through a camera provided in the electronic apparatus.

According to aspects of the disclosure, an electronic apparatus generates a panoramic image from a plurality of consecutive frames, identifies an object from an area of frames having a smallest motion difference, and improves the accuracy of object identification despite a quick move of a camera, thereby providing Augmented Reality (AR) including a motion-stable object to a user.

