1. A computer-implemented method of aligning an input video frame from a video sequence with a background model, said background model including a plurality of model blocks, each one of the plurality of model blocks being associated with at least one mode, wherein each mode includes a plurality of frequency domain attributes, said method comprising the steps of: selecting, for each one of the plurality of model blocks, a reference mode from the at least one mode associated with a corresponding model block; generating a background reference image comprising the plurality of selected reference modes; and aligning the input frame with the background model, based on the background reference image.