1. A method of inspecting a pattern comprising the steps of: radiating and scanning charged particle beam to a region including a first pattern formed on an object to be inspected; detecting secondary charged particles generated from a region including said first pattern by scanning and radiating said charged particle beam to obtain a first charged particles image; radiating and scanning said charged particle beam to a region including a second pattern formed originally to have the same shape as that of said first pattern formed on said object to be inspected; detecting secondary charged particles generated from a region including said second pattern by scanning and radiating said charged particle beam to obtain a second charged particles image; and comparing said first charged particles image with said second charged particles image to detect either defect or a candidate for defect of the pattern formed on said object to be inspected; wherein at the step for detecting either the defect or candidate for defect of said pattern, estimating both a dispersion of amount of image signal in said first charged particles image and a dispersion of amount of image signal in said second charged particles image, calculating an allowable range based on the dispersion of said estimated amount of image signal, calculating a difference between said first charged particles image and said second charged particles image, and determining either a defect or a candidate for defect based on determination whether or not said calculated difference is within said calculated allowable range.