1. An intermittent photo-taking device comprising: a microscope that is equipped with an objective lens used for observing a sample to be observed; an electric stage that holds the sample to be observed; a driving system that controls variation in relative position between the stage and the objective lens, and driving with respect to an autofocus movement; an imaging device that photographs an image of the sample to be observed through the objective lens; and an intermittent recording device that controls the driving system and the imaging device and captures and processes the photographed image; wherein the intermittent recording device includes a receiving means that receives to setting of photo-taking timing of the intermittent recording, a relative movement means that makes the driving system set a plurality of focus positions of the sample to be observed to be photographed at each photo-taking timing as relative positions with respect to a predetermined reference position, at each photo-taking timing, obtain a focus position as a reference position by the autofocus movement, and makes the objective lens or the stage move relatively to the focus position set to be photographed of the sample to be observed on the basis of the obtained reference position and the relative positions.