1. A defect observation device that compares an inspected image and a reference image and detects a defect area included in the inspected image, the defect observation device comprising: a defect occupation rate determination processing unit that calculates a defect occupation rate to be a ratio of the defect area in the inspected image to an area of the inspected image, determines the magnitude of the defect occupation rate, and determines a single threshold value, wherein it is determined, based on whether the defect occupation rate is higher or lower than the single threshold value, whether an image, which is configured by pixels having an average brightness value of a plurality of pixels included in the inspected image, is generated as the reference image, wherein, when the defect occupation rate is higher than the single threshold value, the image acquisition unit newly images an area assumed as an area where a defect does not exist as the reference image, and when the defect occupation rate is lower than the single threshold value, the reference image generation processing unit generates the image configured by the pixels having the average brightness value of the plurality of pixels included in the inspected image as the reference image.