1. A defect observation device that compares an inspected image and a reference image and detects a defect area included in the inspected image, the defect observation device comprising: a defect occupation rate determination processing unit that calculates a defect occupation rate to be a ratio of the defect area in the inspected image and determines the magnitude of the defect occupation rate and a threshold value, wherein it is determined whether an image configured by pixels having an average brightness value of a plurality of pixels included in the inspected image is generated as the reference image, according to a result of the determination.