1. A method for determining a video summary from a video sequence including a time sequence of video frames, comprising: a) determining a global feature vector representing the entire video sequence; b) selecting a plurality of subsets of the video frames; c) determining a frame feature vector for each video frame in the selected subsets of video frames; d) defining a set of basis functions, wherein each basis function is associated with the frame feature vectors for the video frames in a particular subset of video frames; e) using a data processor to automatically determine a sparse combination of the basis functions representing the global feature vector; f) determining a summary set of video frames responsive to the sparse combination of the basis functions; and g) forming the video summary responsive to the summary set of video frames; wherein the sparse combination of the basis functions is determined by finding a sparse vector of weighting coefficients .alpha. by minimizing D(y,Aα) subject to the constraint ||α|| 0 <m, where y is the global feature vector, A is a matrix representation of the set of basis functions, D(°) is a dissimilarity function which returns a number that represents the dissimilarity between its two arguments, m is the number of non-zero elements in α, ||°|| 0 denotes l 0 norm; wherein the video frames in the summary set of video frames corresponding to the largest weighting coefficients are merged in a manner that preserves the temporal order to form the video summary.