This paper presents a new method for a quick similarity-based search through long unlabeled audio streams to detect and locate audio clips provided by users. The method involves feature-dimension reduction based on a piecewise linear representation of a sequential feature trajectory extracted from a long audio stream. Two techniques enable us to obtain a piecewise linear representation: the dynamic segmentation of feature trajectories and the segment-based Karhunen-Loeve (KL) transform. The proposed search method guarantees the same search results as the search method without the proposed feature-dimension reduction method in principle. Experimental results indicate significant improvements in search speed. For example, the proposed method reduced the total search time to approximately 1 12 that of previous methods and detected queries in approximately 0.3 s from a 200-h audio database.
Dynamic segmentation is a generic term that refers to techniques for dividing sequences into segments of various lengths. Dynamic segmentation methods for time-series signals have already been applied to various kinds of applications such as speech coding (e.g. @cite_33 ), the temporal compression of waveform signals @cite_1 , the automatic segmentation of speech signals into phonic units @cite_24 , sinusoidal modeling of audio signals @cite_4 @cite_21 @cite_6 and motion segmentation in video signals @cite_25 . We employ dynamic segmentation to minimize the average dimensionality of high-dimensional feature trajectories.
@cite_4 @cite_33 @cite_21 @cite_1 @cite_6 @cite_24 @cite_25
2030346542 2149410414 2528957674 2065274193
We present an efficient and scalable technique for spatiotemporal segmentation of long video sequences using a hierarchical graph-based algorithm. We begin by over-segmenting a volumetric video graph into space-time regions grouped by appearance. We then construct a “region graph” over the obtained segmentation and iteratively repeat this process over multiple levels to create a tree of spatio-temporal segmentations. This hierarchical approach generates high quality segmentations, which are temporally coherent with stable region boundaries, and allows subsequent applications to choose from varying levels of granularity. We further improve segmentation quality by using dense optical flow to guide temporal connections in the initial graph. We also propose two novel approaches to improve the scalability of our technique: (a) a parallel out-of-core algorithm that can process volumes much larger than an in-core algorithm, and (b) a clip-based processing algorithm that divides the video into overlapping clips in time, and segments them successively while enforcing consistency. We demonstrate hierarchical segmentations on video shots as long as 40 seconds, and even support a streaming mode for arbitrarily long videos, albeit without the ability to process them hierarchically. In this paper, we propose an efficient sinusoidal model of polyphonic audio signals especially good for the application of timescale modification. One of the critical problem of sinusoidal modeling is that the signal is smeared during the synthesis frame, which is a very undesirable effect for transient parts. We solve this problem by introducing multiresolution analysis-synthesis and dynamic segmentation methods. A signal is modeled with a sinusoidal component and a noise component. A multiresolution filter bank is applied to an input signal which splits it into octave-spaced subbands without causing aliasing and then sinusoidal analysis is applied to each subband signal. To alleviate smearing of transients during synthesis, a dynamic segmentation method is applied to the subband signals that determines the optimal analysis-synthesis frame size adaptively to fit its time-frequency characteristics. To extract sinusoidal components and calculate respective parameters, a matching pursuit algorithm is applied to each analysis frame of the subband signal. A psychoacoustic model implementing frequency masking is incorporated with matching pursuit to provide a reasonable stop condition of iteration and reduce the number of sinusoids. The noise component obtained by subtracting the synthesized signal with sinusoidal components from the original signal is modeled by a line-segment model of short time spectrum envelope. For various polyphonic audio signals, the results of simulation shows the proposed sinusoidal modeling can synthesize original signals without loss of perceptual quality and do more robust and high-quality timescale modification for large scale factors. Motion segmentation is an important task for intelligent transportation systems. In this paper, inspired by the fact that a feature point trajectory can be sparsely represented as a combination of several feature point trajectories that share coherent transformations, an efficient and effective motion segmentation method with a sparsity constraint is proposed. Specifically, we first propose an accumulated scheme to efficiently integrate motion information from all the frames of a video sequence to construct a correlation matrix. Then, a sparse affinity matrix is built on the correlation matrix by using information-theoretic principles, where the nonzero elements in the same row of the sparse affinity matrix correspond to the feature point trajectories more likely belonging to the same motion. Thereafter, a segment and merge procedure is proposed to effectively estimate the number of motions via the sparse affinity matrix. Finally, by applying spectral clustering on the sparse affinity matrix, different motions in the video sequence are accurately segmented based on the estimated number of motions. Experimental results on the Hopkins 155 and the 62-clip datasets demonstrate that the proposed method achieves superior performance compared with several state-of-the-art methods. In this paper, we propose a novel method that exploits correlation between audio-visual dynamics of a video to segment and localize objects that are the dominant source of audio. Our approach consists of a two-step spatiotemporal segmentation mechanism that relies on velocity and acceleration of moving objects as visual features. Each frame of the video is segmented into regions based on motion and appearance cues using the QuickShift algorithm, which are then clustered over time using K-means, so as to obtain a spatiotemporal video segmentation. The video is represented by motion features computed over individual segments. The Mel-Frequency Cepstral Coefficients (MFCC) of the audio signal, and their first order derivatives are exploited to represent audio. The proposed framework assumes there is a non-trivial correlation between these audio features and the velocity and acceleration of the moving and sounding objects. The canonical correlation analysis (CCA) is utilized to identify the moving objects which are most correlated to the audio signal. In addition to moving-sounding object identification, the same framework is also exploited to solve the problem of audio-video synchronization, and is used to aid interactive segmentation. We evaluate the performance of our proposed method on challenging videos. Our experiments demonstrate significant increase in performance over the state-of-the-art both qualitatively and quantitatively, and validate the feasibility and superiority of our approach.