1. A background maintenance system for maintaining a background model of an image sequence having a plurality of pixels, comprising: a general-purpose computing device; a computer-readable storage medium having stored and encoded thereon a computer program having program modules containing computer-executable instructions that are executable by the general-purpose computing device, the computer program further comprising: a prediction module that provides at least two predictions of what value a pixel will be in a next frame of the image sequence; a first pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on actual history pixel values; a second pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on predicted history pixel values; a computation module that determines an actual value of the pixel in the next frame of the image sequence; and a declaration module that classifies the pixel as a foreground pixel if the actual value of the pixel in the next frame of the image sequence differs from any one of the first and second pixel value predictions by more than a threshold value.