1. A surface defect inspection system comprising: an image pick-up means for picking up an image by sequentially raster scanning the surface of an object two-dimensionally in abscissa and ordinate directions; threshold means for quantizing an image signal produced from said image pick-up means as a binary signal in accordance with a predetermined threshold and sampling the quantized signal; register means having a storage capacity corresponding to at least one raster scan line for sequentially and temporarily storing the sampled quantized signal delivered from said threshold means; holding means for, when the sampled quantized signal corresponding to a defect pattern is delivered from said threshold means for the first time at a certain abscissa in a certain ordinate corresponding to a certain raster scan line, holding the sampled quantized signal corresponding to the defect pattern at a portion corresponding to the abscissa of the corresponding defect pattern in said register means during succeeding scanning operations in the ordinate direction as long as the sampled quantized signal delivered from said threshold means includes at least one quantized signal corresponding to a defect pattern in an abscissa area having an abscissa of the defect pattern where said register means includes quantized signals corresponding to a defect pattern continuously in the abscissa direction, in each of succeeding raster scan lines; pattern region end decision means for deciding that the defect pattern region has ended in the ordinate direction when the sampled quantized signal delivered from said threshold means includes no quantized signal corresponding to a defect pattern in the abscissa area having an abscissa of the defect pattern in a succeeding raster scan line; pattern feature extracting means for extracting features of the defect pattern in accordance with the sampled quantized signal delivered from said threshold means in synchronism with the scanning in the abscissa direction; memory means having a storage capacity corresponding to at least one raster scan line for temporarily and sequentially storing the result of the calculation delivered from said pattern feature extracting means to a storage area corresponding to the abscissa where the defect pattern exists; reading means for, when said pattern region end decision means decides that the defect pattern region has ended in the ordinate direction, reading the features of the defect pattern from the storage area of said memory means corresponding to the abscissa where the defect pattern exists; and defect decision means for comparing the features of each of the defect patterns separately read out of said reading means with a predetermined reference to thereby decide whether the defect pattern is acceptable or not.