1. A stereoscopic panoramic image synthesis device comprising: an image acquisition unit configured to acquire multiple images continuously captured at a predetermined frame rate while panning an image capturing device including a single imaging optical system and an imaging element; an image selection unit configured to select images for panoramic image synthesis from the acquired multiple images, the image selection unit configured to select images in which a horizontal component of a translation vector of an optical flow between the images is within a predetermined range set in advance; an overlap area detection unit configured to detect an overlap area between temporally adjacent images in the selected images; a trimming unit configured to cut out images of the detected overlap area; a storage unit configured to store the cut images of the overlap area as a left image and a right image; and a panoramic image synthesis unit configured to synthesize a left panoramic image and a right panoramic image based on an image group of the stored left image and an image group of the stored right image.