1. An image capturing apparatus comprising: a drive unit that displaces at least one of a lens unit and an image capturing element with respect to an optical axis; and a control unit that displaces, using the drive unit, at least one of the lens unit and the image capturing element in accordance with a motion of the image capturing apparatus, wherein, when generating a plurality of captured images while shifting an image capturing direction in order to generate a panoramic image from the plurality of captured images, the control unit changes a position of the lens unit and/or the image capturing element, the position being a position at the start of exposure of each of the captured images, in a direction set according to a direction in which the image capturing direction shifts.