1. A semiconductor defect classifying method for a defect classifying apparatus for classifying a defect of a semiconductor device by a defect inspecting apparatus which detects the defect and a review apparatus for observation, the method comprising: the defect classifying apparatus performing pattern matching of a defective image of the defect at a layer to be inspected from the defect inspecting apparatus or the review apparatus with design layout data of the semiconductor device; superimposing the defect on a design layout pattern of at least one of the layer to be inspected, a layer immediately above the layer to be inspected, and a layer immediately below the layer to be inspected; and determining the state of overlap between the defect and the layout pattern of the arbitrarily selected layer and automatically classifying the defect.