BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium.

Description of the Related Art

A method called “dolly shooting” is known as a method for using an image capturing apparatus such as a digital camera to shoot a moving image in which the viewpoint moves in a smooth manner.

Meanwhile, an image capturing apparatus that uses a two-dimensional image sensor, in which a photoelectric conversion unit having a single microlens divided into multiple parts is formed for each pixel, is known (Japanese Patent No. 5917125). The divided photoelectric conversion unit (also called a “divided pixel” hereinafter) is configured so that different partial pupil regions of the exit pupil of a shooting lens receive light through the single microlens, and pupil division is achieved through this configuration. The respective outputs of the pupil-divided photoelectric conversion unit are parallax images corresponding to the partial pupil regions resulting from the division. Repeatedly compositing the parallax images while varying the weighting (compositing ratio) thereof makes it possible to generate a moving image to which is imparted a viewpoint movement effect, with the in-focus plane located on the rotational center of the viewpoint movement.

Additionally, image capturing apparatuses using two-dimensional image sensors are known as plenoptic cameras or light field cameras, and shifting the parallax images in the direction of the parallax makes it possible to carry out refocus processing for moving the in-focus plane after shooting. Japanese Patent Laid-Open No. 2017-5689 discloses a technique in which the rotational center of viewpoint movement in a moving image is changed in the focus direction (the depth direction) by generating the moving image having a viewpoint movement effect after moving the in-focus plane on the basis of a range over which refocusing is possible. Carrying out such processing makes it possible to impart a viewpoint movement effect to a region that was in focus at the time of shooting, to which it has been difficult to add such a viewpoint movement effect thus far.

The baselines between divided pixels are comparatively short in two-dimensional image sensors such as those disclosed in Japanese Patent No. 5917125 and Japanese Patent Laid-Open No. 2017-5689. Thus when a moving image to which a viewpoint movement effect has been added is generated from parallax images obtained from a two-dimensional image sensor, the viewpoint movement effect will be comparatively small. Here, if the moving image having a viewpoint movement effect is generated after moving the in-focus plane as disclosed in Japanese Patent Laid-Open No. 2017-5689, the appearance of a comparatively large viewpoint movement effect can be achieved in the regions of objects that are distant from the in-focus plane after the in-focus plane is moved. However, when the two-dimensional image sensor has a small number of divided pixels, the in-focus plane will only move a small amount when the in-focus plane is moved within the refocusable range, and thus the viewpoint movement effect cannot be increased significantly. Additionally, moving the in-focus plane beyond the refocusable range produces double contours in the image, leading to a drop in image quality. Ultimately, according to the technique of Japanese Patent Laid-Open No. 2017-5689, the in-focus plane remains the rotational center even when the in-focus plane is moved, and thus the range over which the rotational center of the viewpoint movement can be moved is restricted to the range over which the in-focus plane can be moved.

SUMMARY OF THE INVENTION

Having been achieved in light of such circumstances, the present invention provides a technique that makes it possible to generate, from parallax images, a moving image having a viewpoint movement effect in which a position different from the in-focus plane in the depth direction serves as a rotational center.

According to a first aspect of the present invention, there is provided an image processing apparatus comprising at least one processor and/or at least one circuit which functions as: a generating unit configured to generate a plurality of composite images by repeatedly carrying out a compositing process that composites a plurality of parallax images according to a compositing ratio, the parallax images having parallax with respect to each other; a control unit configured to carry out control so that the compositing ratio changes over time as the compositing process is repeated; and a positioning unit configured to position the plurality of composite images so that a rotational center of viewpoint movement arising between the plurality of composite images due to the change over time in the compositing ratio moves in a depth direction.

According to a second aspect of the present invention, there is provided an image processing method executed by an image processing apparatus, comprising: generating a plurality of composite images by repeatedly carrying out a compositing process that composites a plurality of parallax images according to a compositing ratio, the parallax images having parallax with respect to each other; carrying out control so that the compositing ratio changes over time as the compositing process is repeated; and positioning the plurality of composite images so that a rotational center of viewpoint movement arising between the plurality of composite images due to the change over time in the compositing ratio moves in a depth direction.

According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising: generating a plurality of composite images by repeatedly carrying out a compositing process that composites a plurality of parallax images according to a compositing ratio, the parallax images having parallax with respect to each other; carrying out control so that the compositing ratio changes over time as the compositing process is repeated; and positioning the plurality of composite images so that a rotational center of viewpoint movement arising between the plurality of composite images due to the change over time in the compositing ratio moves in a depth direction.

Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

