1. A method for identifying stale background pixels in a background model of scene depicted in a sequence of video frames, the method comprising: receiving a raw video frame of the sequence of video frames; classifying a first set of one or more pixels of the raw video frame as depicting a portion of a background of the scene; classifying a second set of one or more pixels of the raw video frame as depicting a portion of a foreground of the scene; for at least a first pixel classified as depicting a portion of the foreground of the scene: selecting a background pixel in the background model corresponding to the first pixel and a plurality of pixels in the background model neighboring the background pixel, determining a plurality of changes between color-characteristic values of the plurality of pixels in the background model and respective color-characteristic values of a corresponding plurality of pixels in a video frame previously used to update the color-characteristic values of the background pixel, and upon determining a difference between the change and the plurality of changes is within a specified threshold, classifying the background pixel in the background model as a stale background pixel; and updating the color-characteristic values of the background pixel in the background model using the color-characteristic values of the first pixel.