1. A system for extracting video highlights from a video stream comprising: storage, containing said video stream in digital form; and a processor, in communication with said storage, said processor executing software for performing the following functions: i. reading an initial plurality of frames from said video stream and constructing a new dictionary based on said initial plurality of frames, said dictionary containing data summarizing the content of said initial plurality of frames; ii. breaking the remainder of said video stream into a plurality of segments of approximately equal length; iii. reading the next segment from said video stream; iv. detecting interest points in said segment and defining a local spatio-temporal cuboid for each of said detected interest points; v. calculating, for each of said spatio-temporal cuboids, a plurality of feature vectors and concatenating said plurality of feature vectors into a feature representation for each of said spatio-temporal cuboids; vi. determining if a group of feature representations representing all of said spatio-temporal cuboids in said segment can be sparsely reconstructed for a cost not exceeding a predetermined threshold, using existing entries in said dictionary, and, if not, adding said group of feature representations to said dictionary and adding said segment to said video highlights; vii. repeating steps iii.-vi. until all segments in said video stream have been processed.