1. A background maintenance system for maintaining a background model of an image sequence having a plurality of pixels, comprising: a prediction module that provides at least two predictions of what value a pixel will be in a next frame of the image sequence; a first pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on actual history pixel values; and a second pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on predicted history pixel values.