1. A computer-implemented method of aligning an input video frame from a video sequence with a background model, the background model including a plurality of model blocks, each one of the plurality of model blocks being associated with at least one mode, said method comprising the steps of: selecting, for each one of the plurality of model blocks, a reference mode from the at least one mode associated with a corresponding model block, the selected reference mode being a matched mode with background; generating a background image from the plurality of selected reference modes; estimating misalignment between the input video frame and the generated background image based on the input video frame and the generated background image; and aligning the input video frame with the background model for object detection, based on the estimated misalignment, wherein each of the modes includes a plurality of frequency domain attributes for object detection, and the background image is generated by applying an inverse transformation process to the plurality of selected reference modes.