1. A visual inspection device which makes a non-defective/defective determination by use of an inspection image of an inspection object and a determination threshold, the device comprising: a camera capturing the inspection image; a storing unit storing the inspection image acquired by the camera; and a control unit making the non-defective/defective determination by comparing a result of image processing with the inspection image stored in the storing unit with the determination threshold; wherein the control unit includes: an image input accepting unit accepting inputs of a plurality of candidate images to constitute a group of images of items as non-defective items and storing these candidate images in the storing unit; a threshold setting unit setting a defect threshold for determining whether each candidate image has a defective portion based on a plurality of pixel values to be extracted from corresponding regions among the plurality of stored candidate images; a defect amount calculating unit detecting the defective portion with respect to each of the plurality of stored candidate images based on the set defect threshold and calculating a defect amount corresponding to a pixel value or an area of the detected defective portion; an outlier testing unit testing by statistical processing whether or not each of the defect amounts calculated by the defect amount calculating unit is an outlier; and an information displaying unit displaying and outputting outlier information for specifying from among the plurality of stored candidate images a non-candidate image to be eliminated from the group of images of items as non-defective items, the non-candidate image having the defect amount tested to be the outlier by the outlier testing unit; wherein the threshold setting unit resets the defect threshold based on the plurality of stored candidate images except the non-candidate image from the group of images of items as non-defective items, and sets the determination threshold as a standard to be compared with the defect amount calculated based on the reset defect threshold.