1. A pattern inspection method comprising: imaging a first pattern formed in a first area of a substrate to acquire a first pattern image; associating the first pattern image with design information for the first pattern to detect a defect by a die-to-database inspection; comparing the number of the detected defect with a prepared first threshold, and when the number of the detected defects is equal to or less than the first threshold, outputting the detected defect as an inspection result; imaging a second pattern to acquire a second pattern image when the number of the detected defect is more than the first threshold, the second pattern being formed in a second area different from the first area within the substrate and being designed to comprise the same shape as the first pattern; and comparing the first pattern image with the second pattern image to further detect a defect by a die-to-die inspection, and outputting the further detected defects as an inspection result, wherein the further detection of defect by the die-to-die inspection comprises: detecting an edge of the first pattern from the first pattern image, and calculating a value of a gradient of a detected edge profile of the first pattern; comparing the value of the gradient with a prepared second threshold; and detecting an edge of the second pattern from the second pattern image and comparing the edge of the second pattern with the edge of the first pattern to detect a defect when the calculated value of the gradient is equal to or more than the second threshold, or calculating a pixel value of the first pattern image and a pixel value of the second pattern image and comparing these pixel values to detect a defect when the calculated value of the gradient is less than the second threshold.