1. An image capturing apparatus, comprising: a drive device configured to displace at least one of a lens within a lens unit or an image capturing element with respect to an optical axis of the lens unit; a control device configured to displace, based on the drive device, the at least one of the lens or the image capturing element based on a motion of the image capturing apparatus; and change, based on capture of a plurality of images while an image capturing direction is shifted without stopping in order to capture each of the plurality of images to generate a panoramic image, a position of the at least one of the lens or the image capturing element from a first position to a second position, wherein the first position is a position at a start of exposure of each of the plurality of images, wherein the position is changed in a first direction, wherein the first direction is set based on a second direction in which the image capturing direction shifts, so that an optical image of an object that passes through the lens is formed substantially at a center of an image capturing surface of the image capturing element, wherein, based on the shifted image capturing direction, the optical axis at the second position of the at least one of the lens or the image capturing element is parallel to the optical axis at the first position prior to the change in the position of the at least one of the lens or the image capturing element, wherein a third position of the object in the optical image that is formed on the image capturing surface based on the second position of the at least one of the lens or the image capturing element, is substantially same as a fourth position of the object in the optical image that is formed on the image capturing surface at the first position prior to the change in the position of the at least one of the lens or the image capturing element, wherein a deviation, caused by the motion of the image capturing apparatus, in the optical image formed on the image capturing surface of the image capturing element during an exposure period of each of the plurality of images is corrected, wherein the deviation is corrected based on a correction range of deviation correction with respect to the shifted image capturing direction, and wherein the correction range of the deviation correction is greater than a correction operation that is set at a center position at which the optical axis is a center.