1. A computer implemented method for tracking an object recorded within a selected frame of a sequence of frames of video data from at least one video camera, using a plurality of layers, at least one object layer of the plurality of layers representing the object, the method comprising the steps of: a) assigning layer ownerships and layer ownership probabilities pixels in a layer of the selected frame using a non-parametric motion model; b) re-assigning layer ownership and layer ownership probabilities for the pixels in the layer by jointly estimating motion, shape and appearance of the layer using a parametric maximization algorithm, the parametric maximization algorithm being based on a parametric motion model, a shape model, and an appearance model, the appearance model being based on at least one of 2D histograms and 3D histograms, the 2D histograms being 2D color histograms of pixels weighted over time and over the spatial extent of the layer as defined by the layer ownerships, the 3D histograms having bins that are populated according to pixel data and a geometric constraint, the geometric constraint being at least one of the distance of a pixel from an edge of a blob within a template and the distance of a pixel from an origin point; and c) tracking the object.