1. A semiconductor defect classifying method for a defect classifying apparatus for classifying a defect of a semiconductor device by a defect inspecting apparatus which detects the defect and a review apparatus for observation, the method comprising: the defect classifying apparatus performing pattern matching of a defective image of the defect at a layer to be inspected from the defect inspecting apparatus or the review apparatus with design layout data of the semiconductor device; superimposing the defect on a design layout pattern of at least one of the layer to be inspected, a layer immediately above the layer to be inspected, and a layer immediately below the layer to be inspected; determining the state of overlap between the defect and the layout pattern of the arbitrarily selected layer and automatically classifying the defect; generating a defect shape image through pattern recognition of the defective image; superimposing the defect shape image on the design layout pattern; determining the state of overlap between the defect and the layout pattern of the arbitrarily selected layer and automatically classifying the defect; extracting an outline from the defect shape image; superimposing the outline on the design layout pattern; and determining the state of overlap of the defect and the state of overlap between the defect and the layout pattern of the arbitrarily selected layer and automatically classifying the defect.