1. A method for recognizing heart disease in a cardiac echo video of a heart with an unknown disease using a spatio-temporal disease model derived from a training echo video, the method comprising: generating a plurality of training models for heart diseases, wherein the cardiac echo videos are each derived from a known viewpoint and the heart diseases of the heart are known; generating the plurality of training models for the heart diseases comprises building disease models for the heart diseases in which each of the disease models are for a plurality of viewpoints; analyzing the cardiac echo video of the heart with the unknown disease by fitting a model of shape and motion for each frame and combining the results across the frames, which comprises: applying an outer loop over the heart diseases and an inner loop for the plurality of viewpoints, where an input of sequences is for the cardiac echo video of the heart with the unknown disease; iteratively fitting each of the plurality of viewpoints for a first of the heart diseases during the inner loop until all of the plurality of viewpoints for the first of the heart diseases is finished; taking an overall disease fit for the first of the heart diseases over all of the plurality of viewpoints for the first of the heat diseases; continuing until a last of the heart diseases has been reached for the outer loop; and recognizing a heart disease of the heart diseases that minimizes the fitting after completing the outer loop; and reporting the heart disease using a classification method for choosing among the heart diseases of interest.