1. A method of detecting defects of patterns on a semiconductor substrate, comprising: acquiring information on positions of reference defects influencing an operation of a circuit including the patterns when the patterns are formed on the semiconductor substrate; detecting preliminary defects of the patterns formed on the semiconductor substrate; comparing positions of the preliminary defects of the patterns with the previously acquired positions of the reference defects; and setting the preliminary defects having positions substantially the same as the positions of the reference defects from among all of the detected preliminary defects to be the actual defects of the patterns, wherein comparing positions of the preliminary defects of the patterns with the positions of the reference defects comprises overlapping an actual map where the positions of the preliminary defects of the patterns are marked with a reference map where the positions of the reference defects are marked.