1. An image background maintenance system for processing an image sequence having a plurality of temporally sequential frames, a current frame of the plurality of temporally sequential frames containing a pixel, comprising: a computation module that stores lists of at least two types of past values of the pixel to generate lists of past pixel values for the pixel in previous frames of the plurality of temporally sequential frames, the previous frames being earlier in time than the current frame; a prediction module that determines an actual value of the pixel in the current frame and uses the lists of at least two types of past values to make a first prediction of a subsequent pixel value and a second prediction of a subsequent pixel value for the pixel in the current frame; and a declaration module that classifies the pixel by comparing the at least two predictions of subsequent values to the actual value of the pixel.