1. A video surveillance system comprising: a computer system for identifying and tracking moving objects in a video input by performing a method comprising: tracking a set of tracked moving objects in the video input for a first time instant of the video input and generating tracking information corresponding to the set of tracked moving objects based on the tracking, wherein the tracking information includes an appearance model for each moving object in the set of tracked moving objects and a set of motion characteristics for each moving object in the set of tracked moving objects; and detecting a set of moving objects in a subsequent time instant of the video input subsequent to the first time instant, wherein the detecting includes: for a newly detected moving object in the subsequent time instant, initially detecting the newly detected moving object in the video input using background subtraction; and for a moving object in the set of tracked moving objects, subsequently detecting the moving object in the video input using the tracking information, wherein the tracking information is used to enhance the subsequent detection of the moving object against a background of the video input using the appearance model and to directly classify pixel data corresponding to a frame of the video input as one of: the background, the moving object, or an unknown object using the set of motion characteristics for the moving object, wherein the classifying is performed in a single classification step.