1. An imaging device, comprising: a single imaging optical system; an imaging unit capable of continuously outputting a pair of viewpoint images made up of a first image and a second image, by pupil-dividing subject images that have passed through first and second different areas in predetermined directions of the imaging optical system, and forming the respective subject images on two photoelectric conversion element groups to photoelectrically convert the respective subject images; a diaphragm unit capable of adjusting an amount of luminous flux that forms images on the photoelectric conversion element groups of the imaging unit; a diaphragm control unit capable of executing first diaphragm control that controls an open amount of the diaphragm unit so as to acquire parallax between corresponding points of the pair of viewpoint images outputted from the imaging unit at one arbitrary first point of time or a plurality of arbitrary first points of time, while executing second diaphragm control that controls the open amount of the diaphragm unit so as to be smaller than the open amount obtained by the first diaphragm control at an arbitrary second point of time that is temporally before or after the first point or points of time; an imaging control unit configured to control the imaging unit so as to output the pair of viewpoint images at the first point or points of time and the second point of time; a parallax information calculation unit configured to calculate parallax information based on the pair of viewpoint images outputted from the imaging unit at the first point or points of time; a plane image generation unit configured to generate a plane image based on the pair of viewpoint images outputted from the imaging unit at the second point of time; and a blurring processing unit configured to determine target pixels in the plane image which are to be subjected to blurring processing, based on the parallax information calculated by the parallax information calculation unit, and to perform blurring processing on the determined target pixels in the plane image, wherein the parallax information includes distance information of respective pixels, and the blurring processing unit determines, as the target pixels of the plane image which are to be subjected to the blurring processing, distant view pixels, distance information of which is larger than a first specified distance and/or close view pixels, distance information of which is smaller than a second specified distance, and performs the blurring processing on the target pixels.