1. A defect observation method, comprising: imaging an inspection object using an optical device or a charged particle beam device on the basis of defect information received from an inspection device and obtaining a defect image and a reference image corresponding to the defect image, the reference image including a plurality of reference images, and the plurality of reference images being different images of an area on the inspection object designed to form a same circuit pattern without defects as a pattern intended in an area of the defect image; performing a parameter adjustment process for determining a first parameter to be used in defect extraction by using first feature quantity distribution obtained from the defect image by said imaging and the reference image and a second feature quantity distribution obtained from the plurality of reference images, the second feature quantity distribution being a feature quantity distribution in an area judged as a defect candidate in any of the difference images between the plurality of reference images; and performing an observation process for conducting observation of the inspection object using the optical device or charged particle beam device using the first parameter to generate a plurality of defect candidates occurring in the inspection object, wherein said parameter adjustment process further comprises determining a separating hyperplane based on said first feature quantity distribution and said second feature quantity distribution, and wherein said observation process further comprises discriminating between nuisances and defects among said plurality of defect candidates based on the first parameter and the separating hyperplane.