1. An imaging device that includes an imaging element that images a subject through an imaging optical system, the imaging element including a first signal detection unit that detects a signal based on one beam among a pair of beams that passes through different portions in a pupil region of the imaging optical system, and a second signal detection unit that detects a signal based on the other beam among the pair of beams, and the imaging device comprising: a focusing control unit that performs either one of a focusing control based on a phase difference AF method using a result of a correlation operation between a detection signal of the first signal detection unit and a detection signal of the second signal detection unit or a focusing control based on a contrast AF method using contrast of a captured image captured by the imaging element, wherein in a mode where the imaging is consecutively performed by the imaging element a plurality of times, the focusing control unit performs, in a case where a reliability of the focusing control based on the phase difference AF method determined based on the result of the correlation operation between the detection signal of the first signal detection unit and the detection signal of the second signal detection unit read from the imaging element at an arbitrary timing is equal to or lower than a first threshold value, the focusing control based on the contrast AF method as a focusing control in the imaging to be performed after the arbitrary timing, wherein the focusing control unit determines and stores, in a case where the reliability based on the result of the correlation operation is equal to or lower than the first threshold value, a factor of the reliability becoming equal to or lower than the first threshold value using the detection signals which are targets of the correlation operation, and in a case where the reliability is higher than the first threshold value after the focusing control based on the contrast AF method is performed, performs the focusing control based on the phase difference AF method in imaging to be performed next time if the factor is not based on a frequency of a subject image and performs the focusing control based on the contrast AF method in the imaging to be performed next time if the factor is based on the frequency of the subject image.