BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a positioning technique for mixing the real world and a virtual world using images.

Description of the Related Art

A mixed reality (MR) technique and an augmented reality (AR) technique have been known as a technique for mixing the real world and a virtual world. These are technologies for mixing a real space and a virtual space created by a computer in a seamless manner. These technologies are expected to be applied to various fields, such as support for assembling work by displaying a work procedure and a wiring state in a superimposed manner during assembling, and support for surgery by displaying a state inside a body in a superimposed manner on a body surface of a patient. In fact, such applications have advanced in the world.

To achieve the MR and the AR, positioning is performed to appropriately place computer graphics (CG) at an arbitrary position in a real environment. As a result of the positioning, even if a user looks around through a display, the user can always visually recognize the CG existing at one point in the real environment, and have an experience as if a physical object existed there.

Examples of a conventional technique for achieving such positioning include a technique of defining a spatial coordinate system using a marker with a characteristic black and white pattern and detecting, through image processing, the marker at a known position by observing the marker using a camera to perform the positioning.

In recent years, a technique has been proposed that achieves the positioning without using such a marker. An example of the technique is a simultaneous localization and mapping (SLAM) method. Andrew J. Davison, “Real-Time Simultaneous Localisation and Mapping with a Single Camera”, http://www.robots.ox.ac.uk/ActiveVision/Papers/davison_iccv2003/davison_iccv2003.p df, discusses the basic principle of SLAM. In SLAM, when an image captured by a camera is input, a position and orientation of the camera is updated by obtaining a relative position and orientation thereof from one or more stationary features in the image. Examples of the features include a point and an edge that form an outline, in particular a corner, where there is a significant color change in the image. For example, when a cube having a different color from that of a desk is placed on the plain desk, each vertex of the cube serves as a feature point, and each outline of the cube serves as a feature edge.

An application of the MR provides an image in which CG are arranged at a specific position in the real environment using a result of positioning.

Accuracy in positioning based on a group of features in an image depends on distribution of features in an image in which the real environment has been captured, in particular, in an image per frame. In the real space, the more the number of portions serving as features is and the more evenly the portions are arranged, the higher accuracy in positioning can be expected. In contrast, an extremely small number of portions serving as features in the real space can result in a portion where accuracy in positioning is insufficient.

In the real space, there may be a case in which a sufficient number of feature portions are arranged in a visual field in one direction which the user faces, while an extremely small number of feature portions are arranged in a visual field in another direction which the user faces. In this manner, if the feature portions are not evenly arranged in the space, accuracy in positioning may be insufficient.

If the number of feature portions or distribution of the feature portions in the space is not desirable, it may not be possible to continue the positioning. However, the user using an application of the MR may not have knowledge about such characteristics of the positioning. Thus, the user realizes that a camera faces a direction not desirable for the positioning for the first time only after experiencing the application to some extent and seeing the behavior of the application showing unstable positioning.

A possible solution to the issue is to guide the camera to face a portion where distribution and the number of feature portions are appropriate in the image.

Japanese Patent Application Laid-Open No. 2013-225245 discusses a mechanism to display a guided object on a display unit presented to the user and notify the user of a recommended observation direction.

According to a conventional technique, when few portions serving as features exist or distribution of the portions is uneven in the real space at the time of positioning using an image obtained by capturing the real space, such as in the positioning used for the MR, the positioning can be performed appropriately in only in part of the real space.

A possible solution to such a situation is that the user adds features to the real space. For example, the user can arrange an object having a corner point serving as a feature in a region where distribution of features is inappropriate in the real space. Alternatively, the user can add the features to the real space, for example, by putting up a poster with a design on a plain wall and carpeting a plain floor.

However, it is not that the features can be added easily to any place where the features are insufficient. For example, it is not easy to fix an object of some kind in midair in a room where a person is having an MR experience. In this case, for example, use of a fixation device may be considered. However, the fixation device may hinder the person's pathway and ruin an experience environment. Thus, there may be a case in which the fixation device cannot be installed.

For this reason, it is important for the user to grasp an appropriate location to add features. However, the user may not have knowledge about the characteristics of the positioning. Accordingly, there may be a case in which appropriate positioning cannot be achieved because the user does not understand where to add the features in the real space and cannot add the features appropriately in the real space.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an information processing apparatus includes an acquisition unit configured to acquire an image captured by an image capturing unit, an estimation unit configured to estimate, based on the image acquired by the acquisition unit, a three-dimensional position in a real space corresponding to each of one or more features included in the image, a first determination unit configured to determine a candidate region being a region on a plane of an object existing in the real space as a candidate for a region to which features are to be added based on the three-dimensional position estimated by the estimation unit, and a second determination unit configured to determine a recommended region to which the features are to be added in the candidate region based on the number of features included in a partial region set in the candidate region determined by the first determination unit.

Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

