1. An image processing unit comprising: a candidate pixel group determiner configured to specify each pixel of an image to be processed as a pixel of interest, extract pixels having a divergence in position in a certain range from the pixel of interest, and determine, from the extracted pixels, pixels having a divergence in pixel value in a certain range from the pixel of interest as a candidate pixel group; an associator configured to associate the pixels of the candidate pixel group with the pixel of interest; and an image processor configured to perform image processing on the pixels of the candidate pixel group associated with the pixel of interest, wherein: the image processor is configured to select, from the pixels of the candidate pixel group, pixels having a divergence in pixel value in a certain range from the pixel of interest as a pixel group to be processed, and smooth the pixel value of the pixel of interest in accordance with the pixel value of each pixel of the candidate pixel group; the determining of the candidate pixel group determiner is executed for each pixel of interest at once; and the image processing of the image processor is executed for each pixel of interest at a plurality of times.