1. An image processing device comprising: an image input part to which a frame image of an imaging area taken with an infrared camera is input; a background model storage part in which a background model is stored with respect to each pixel of the frame image input to the image input part, a frequency of a pixel value of the pixel being modeled in the background model; a background difference image generator that determines whether each pixel of the frame image input to the image input part is a foreground pixel or a background pixel using the background model of the pixel, which is stored in the background model storage part, and generates a background difference image; and a threshold setting part that sets a threshold, the threshold being used by the background difference image generator to determine whether each pixel of the frame image input to the image input part is the foreground pixel or the background pixel, wherein the background difference image generator determines that each pixel of the frame image input to the image input part is the background pixel when a frequency in the background model of the pixel is greater than the threshold set by the threshold setting part, and the background difference image generator determines that the pixel is the foreground pixel when the frequency is less than the threshold, and wherein the threshold setting part calculates and sets the threshold based on a distribution of a pixel value of the pixel that is determined to be the background pixel by the background difference image generator.