1. A charged particle beam apparatus that inspects a defect in a pattern of a sample by the irradiation with an electron beam, the charged particle beam apparatus comprising: an electron optical system that detects secondary charged particles by irradiating the sample with the electron beam; an image processor that detects a defect by executing arithmetic processing on an image to be inspected that is obtained from the secondary charged particles; and a display unit that displays an image indicative of the defect, wherein the image processor includes a contour extractor that extracts the contour of the pattern by using a template image obtained on the basis of an image of the sample, an inspection target region setting unit that sets, on the basis of the contour of the pattern, a region to be inspected, a comparison arithmetic unit that compares the image to be inspected with an image of a part corresponding to the image to be inspected, to detect a defect candidate, and a defect determining unit that determines whether or not the defect candidate is included in the region to be inspected and that determines the defect candidate as a defect if the defect candidate is included in the region to be inspected.