1. A semiconductor inspection system comprising: an inspected image acquisition unit configured to acquire an inspected image by irradiating a sample with a charged particle beam; a first contour line extraction unit configured to extract a first contour line from the inspected image; a second contour line extraction unit configured to extract a second contour line from a reference image corresponding to the inspected image; a distance image creation unit configured to create a reference distance image from the second contour line; a collation unit configured to collate the first contour line with the reference distance image to create a defect size image; a defect candidate detection unit configured to detect a defect candidate from the defect size image; and a candidate size determination unit configured to, upon detection of the defect candidate, calculate a size of the defect candidate.