1. A method of classifying a visual element in a video frame in a video stream as belonging to a foreground or a background, said frame comprising a plurality of visual elements, said method comprising the steps of: receiving a set of candidate mode models associated with the visual element, each of the candidate mode models representing a time history of visual attributes of said each visual element; determining a set of similarity scores between the input model and each of the set of candidate mode models in the corresponding visual element; selecting matching mode models from the set of candidate mode models based on the determined similarity scores; determining a representative activity count measure and a representative age measure from the matching mode model; establishing a functional relationship between the representative activity count measure and the representative age measure; comparing the functional relationship to a threshold value; and determining that the visual element is to be classified as belonging to the background based on the result of the comparing step.