Tools for Searching, Annotation and Analysis of Speech, Music, Film and Video A Survey <s> Searching and collecting <s> The amount of digital video being shot, captured, and stored is growing at a rate faster than ever before. The large amount of stored video is not penetrable without efficient video indexing, retrieval, and browsing technology. Most prior work in the field can be roughly categorized into two classes. One class is based on image processing techniques, often called content-based image and video retrieval, in which video frames are indexed and searched for visual content. The other class is based on spoken document retrieval, which relies on automatic speech recognition and text queries. Both approaches have major limitations. In the first approach, semantic queries pose a great challenge, while the second, speech-based approach, does not support efficient video browsing. This paper describes a system where speech is used for efficient searching and visual data for efficient browsing, a combination that takes advantage of both approaches. A fully automatic indexing and retrieval system has been developed and tested. Automated speech recognition and phonetic speech indexing support text-to-speech queries. New browsable views are generated from the original video. A special synchronized browser allows instantaneous, context-preserving switching from one view to another. The system was successfully used to produce searchable-browsable video proceedings for three local conferences. <s> BIB001 </s> Tools for Searching, Annotation and Analysis of Speech, Music, Film and Video A Survey <s> Searching and collecting <s> When explaining the Query-by-humming (QBH) task, it is typical to describe it in terms of a musical question posed to a human expert, such as a music-store clerk. An evaluation of human performance on the task can shed light on how well one can reasonably expect an automated QBH system to perform. This paper describes a simple example experiment comparing three QBH systems to three human listeners. The systems compared depend on either a dynamic-programming implementation of probabilistic string matching, or hidden Markov models. While results are preliminary, they indicate existing string matching and Markov model performance does not currently achieve humanlevel performance. <s> BIB002 </s> Tools for Searching, Annotation and Analysis of Speech, Music, Film and Video A Survey <s> Searching and collecting <s> The overarching goal of the Informedia Digital Video Library project has been to achieve machine understanding of video media, including all aspects of search, retrieval, visualization and summarization in both contemporaneous and archival content collections. The base technology developed by the Informedia project combines speech, image and natural language understanding to automatically transcribe, segment and index broadcast video for intelligent search and image retrieval. While speech processing has been the most influential component in the success of the Informedia project, other modalities can be critical in various situations. Evaluations done in the context of the TRECVID benchmarks show that while some progress has been made, there is still a lot of work ahead. The fundamental “semantic gap” still exists, but there are a number of promising approaches to bridging it. <s> BIB003 </s> Tools for Searching, Annotation and Analysis of Speech, Music, Film and Video A Survey <s> Searching and collecting <s> We present several novel approaches to the Out of Vocabulary (OOV) query problem for spoken audio: indexing based on syllable-like units called particles and query expansion according to acoustic confusability for a word index. We also examine linear and OOV-based combination of indexing schemes. We experiment on 75 h of broadcast news, comparing our techniques to a word index, a phoneme index and a phoneme index queried with phoneme sequences. Our results show that our approaches are superior to both a word index and a phoneme index for OOV words, and have comparable performance to the sequence of phonemes scheme. The particle system has worse performance than the acoustic query expansion scheme. The best system uses word queries for in-vocabulary words and a linear combination of the phoneme sequence scheme and acoustic query expansion for OOV words. Using the best possible weights for linear combination, this system improves the average precision from 0.35 for a word index to 0.40, a result only obtainable if the weights could be learnt on a development query set. The next best system used a word index for in-vocabulary words and the phoneme sequence system otherwise and had average precision of 0.39. <s> BIB004
With vast collections of digital audiovisual material available, actually searching for and finding a resource can be a major barrier to research. Nearly all practical, current, multimedia access depends on good-quality metadata for search. However, content-based search for text has become an everyday tool on the world-wide web, and similar tools are desired for audiovisual media. In this area speech- Free-text searching for speech generally relies on a previous annotation, transcription or content analysis (topics covered in later sections) to derive text from, or associate text with, the spoken word. Systems supporting the free-text querying of textual transcripts are now ubiquitous and similar systems exist for searching speech by querying time-aligned transcripts that can be automatically derived via speech-to-text systems. Surprisingly, with a little engineering ingenuity, the errors and lack of punctuation in these automatically derived transcripts seem to have little impact upon the effectiveness of the search, once the transcription error rate falls below about 40%. (For more details and possible explanations, see  . Important words tend to occur multiple times, and it is unlikely that all occurrences will be mistranscribed.) Such an error rate is readily achievable by speech-to-text systems, at least for clean speech. However, difficulties arise if background noise or music is present, or if several people speak at once. Additionally, speech-to-text systems typically depend upon a dictionary, and this leads to trouble when speech contains words which are not in the dictionary. In such cases, the automatically produced transcription can be missing new, important words until they enter the dictionary (e.g. -email‖ in the 1990s). This can be a problem with dynamically evolving collections such as daily news BIB003 . There has been research into techniques for handling this problem; one technique involves searching a secondary phonetic transcription: a query term which falls outside the dictionary is (hopefully) located by searching for its pronunciation in the phonetic transcription BIB004 BIB001 . In practise, searching for music is done via metadata, but the metadata available is often restricted and sometimes inaccurate. Searches within the Naxos Music Library  , for example, are restricted to fields such as ‗composer' and ‗title', according to the organisation of the Naxos catalogue. The same applies to the Gracenote database (2006), which holds data on CD recordings, often automatically downloaded by media players when playing music from a CD. Problems include titles appearing in different languages from the original composition, and confusion over whether the ‗artist' is the composer or the performer. Audio content-based search for music, often called ‗query by humming', has been a topic of considerable research. While there have been a number of experimental systems, some of them available on the web (e.g., NYU, n.d.), none has reached the stage of a usable tool. There are very significant technical issues to be addressed before this can be achieved and questions have been raised about the degree to which it would ever be a simple-to-use and effective tool BIB002 . Databases that list information about films and television shows are now common on the web. For instance, the Internet Movie Database (IMDb, 2006) provides reviews, plot summaries, much technical production information and sometimes trailers for over 800,000 films and television series (July 2006) . Because volunteers have added so much information about plot summaries and characters to the database, it can be used to find films and television programs by subject, genre, etc. For current television content, new content alert systems based on program schedules provide automatic notification of broadcasts that fit certain criteria (e.g. MeeVee (2006) or Radio Times (2006)). The BBC has announced its commitment to making 1 million hours of television and radio searchable and available online and the BBC Programme Catalogue (BBC, 2006) allows 75 years of broadcasting to be searched. However, these systems do not actually search the content of film, video or broadcast; as in the case of the spoken word resources, they still rely on previous cataloguing, annotation or transcription. One of the earliest projects to provide a search engine for audiovisual media similar to those for text was Speechbot, a general web-deployed tool for audio indexing speech recognition transcriptions. While Speechbot is now unavailable due to the closure of the Compaq Cambridge Research Lab (US), in the past couple of years a number of similar services have emerged. Some tools crawl the web for audio and video made openly available on websites. For example, Podscope offers the ability to search audio blogs and pod casts, as does Blinkx (2006). Other tools support the search of video or audio submitted by users, for example, Podscope  and Google Video Upload  ). Yet other tools index content legitimately provided by media companies and archives. For example, Blinkx has major deals with ITN and Fox News Channel (net imperative, 2006) . Finding relevant sources is only one step in collecting research materials. Many scholars collect large amounts of this material on their own computers and on portable storage media. While professionally curated online archives usually have extensive catalogues and indexes, personal collections of audiovisual materials sometimes suffer from lack of organisation. Software to facilitate organisation of collections of audiovisual material does exist, but it was not found to be commonly used in our interviews with humanities researchers.