1. A pattern defect inspection apparatus for detecting defects by comparing a detection image obtainable through scanning, by an image sensor, patterns having an identical shape and being continuously laid out on an object to be tested at equal intervals in row and column directions with a reference image obtained by scanning patterns of the identical shape neighboring in the row and column directions thereof, said apparatus comprising: means for generating an average reference image by statistical computation processing from images of identical shape patterns lying next to the detection image including at least four nearest chips on up-and-down and right-and-left sides with the detection image being intermediately situated; and means for detecting a defect by comparing the detection image to the average reference image thus generated.