1. A defect analysis assistance device that compares observation images with sample patterns of a target to be inspected and design layout data corresponding to the sample patterns, and analyzes a defect of the target to be inspected, the defect analysis assistance device comprising: a layer division unit that divides some or all of analysis target images of the observation images for each hierarchy corresponding to a manufacturing process and generates a plurality of layer division images; a matching processing unit that calculates a degree of matching between each of the layer division images and each design layer of the design layout data; and a design layer specifying section that specifies a design layer with a highest degree of matching of the each design layer as a design layer corresponding to the layer division image.