1. A method for tracking a three-dimensional (3D) object using multiple sequences of sets of images, wherein the images in each set are acquired concurrently of the object from different viewpoints, and comprising for each current set of images the steps of: extracting features from each image in the current set, wherein the features are extracted within a window in each image; applying a regression function, subject to motion constraints, to the features to obtain 3D motion parameters; applying the 3D motion parameters to the object to obtain a 3D location of to the object; and projecting a shape of the 3D object at the 3D location onto each image to update a location of the window for each image in a next set of images, wherein the steps are performed in a processor.