In order to quickly review and analyze a video, for example a movie, a recorded sporting event or a news broadcast, a summary of the video can be generated. A number of techniques are known for summarizing uncompressed and compressed videos.
The conventional practice is to first segment the video into scenes or ‘shots’, and then to extract low and high level features. The low level features are usually based on syntactic characteristics such as color, motion, and audio components, while the high level features capture semantic information.
The features are then classified, and the shots can be further segmented according to the classified features. The segments can be converted to short image sequences, for example, one or two seconds ‘clips’ or ‘still’ frames, and labeled and indexed. Thus, the reviewer can quickly scan the summary to select portions of the video to playback in detail. Obviously, the problem with such summaries is that the playback can only be based on the features and classifications used to generate the summary.
In order to further assist the review, the segments can be subjectively rank ordered according to a relative importance. Thus, important events in the video, such as climactic scenes, or goal scoring opportunities can be quickly identified, see, Fujiwara et al. “Abstractive Description of Video Using Summary DS,” Point-illustrated Broadband+Mobile Standard MPEG Textbook, ASCII Corp., p. 177 FIGS. 5-24 Feb. 11, 2003, also “ISO/IEC 15938-5:2002 Information technology—Multimedia content description interface—Part 5: Multimedia Description Schemes,” 2002. After an important video segment has been located, the viewer can use fast-forward or fast-reverse capabilities of the playback device to view segments of interest, see “DVR-7000 Instruction Manual,” Pioneer Co., Ltd., p. 49, 2001.
Another technique for summarizing a news video uses motion activity descriptors, see U.S. patent application Ser. No. 09/845,009, titled “Method for Summarizing a Video Using Motion Descriptors,” filed by Divakaran, et al., on Apr. 27, 2001. A technique for generating soccer highlights uses a combination of video and audio features, see U.S. patent application Ser. No. 10/046,790, titled “Summarizing Videos Using Motion Activity Descriptors Correlated with Audio Features,” filed by Cabasson, et al., on Jan. 15, 2002. Audio and video features can also be used to generate highlights for news, soccer, baseball and golf videos, see U.S. patent application Ser. No. 10/374,017, titled “Method and System for Extracting Sports Highlights from Audio Signals,” filed by Xiong, et al., on Feb. 25, 2003. Those techniques extract key segments of notable events from the video, such a scoring opportunity or an introduction to a news story. The original video is thus represented by an abstract that includes the extracted key segments. The key segments can provide entry points into the original content and thus allow flexible and convenient navigation.
There are a number of problems with prior art video recording, summarization and playback. First, the summary is based on some preconceived notion of the extracted features, classifications, and importance, instead of those of the viewer. Second, if importance levels are used, the importance levels are usually quantized to a very small number of levels, for example, five or less. More often, only two levels are used, i.e., the interesting segments that are retained, and the rest of the video that is discarded.
In particular, the hierarchical description proposed in the MPEG-7 standard is very cumbersome if a fine quantization of the importance is used because the number of levels in the hierarchy becomes very large, which in turn requires management of too many levels.
The MPEG-7 description requires editing of the metadata whenever the content is edited. For example, if a segment is cut out of the original content, all the levels affected by the cut need to be modified. That can get cumbersome quickly as the number of editing operations increases.
The importance levels are highly subjective, and highly context dependent. That is, the importance levels for sports videos depend on the particular sports genre, and are totally inapplicable to movies and news programs. Further, the viewer has no control over the length of the summary to be generated.
The small number of subjective levels used by the prior art techniques make it practically impossible for the viewer to edit and combine several different videos based on the summaries to generate a derivate video that reflects the interests of the viewer.
Therefore, there is a need to record and reproduce a video in a manner that can be controlled by the viewer. Furthermore, there is a need for specifying importance levels that are content independent, and not subjective. In addition, there is a need to provide more than a small number of discrete importance levels. Lastly, there is a need to enable the viewer to generate a summary of any length, depending on a viewer-selected level of importance.