In this paper, we tackle the singing voice phoneme segmentation problem in the singing training scenario by using language-independent information -- onset and prior coarse duration. We propose a two-step method. In the first step, we jointly calculate the syllable and phoneme onset detection functions (ODFs) using a convolutional neural network (CNN). In the second step, the syllable and phoneme boundaries and labels are inferred hierarchically by using a duration-informed hidden Markov model (HMM). To achieve the inference, we incorporate the a priori duration model as the transition probabilities and the ODFs as the emission probabilities into the HMM. The proposed method is designed in a language-independent way such that no phoneme class labels are used. For the model training and algorithm evaluation, we collect a new jingju (also known as Beijing or Peking opera) solo singing voice dataset and manually annotate the boundaries and labels at phrase, syllable and phoneme levels. The dataset is publicly available. The proposed method is compared with a baseline method based on hidden semi-Markov model (HSMM) forced alignment. The evaluation results show that the proposed method outperforms the baseline by a large margin regarding both segmentation and onset detection tasks.
The second topic related to our research is the singing voice lyrics-to-audio alignment. Most of these works @cite_2 @cite_3 @cite_13 @cite_15 @cite_33 @cite_11 @cite_9 @cite_12 used the forced alignment method accompanied by music-related techniques. @cite_8 used MFCCs with additional features and also explored specific HMM topologies. @cite_21 used voice accompaniment separation to deal with mixed recording, and vocal detection, fricative detection to increase the alignment performance. Additional musical side information extracted from the musical score is used in many works. @cite_25 used chord information such that each HMM state contains both chord and phoneme labels. @cite_36 constrained the alignment by using musical note length distribution. @cite_11 , Kruspe @cite_39 , Dzhambazov and Serra @cite_26 all used syllable phoneme duration extracted from the musical score and decoded the alignment path by duration-explicit HMM models. @cite_17 introduced an approach based on vowel likelihood models. Chang and Lee @cite_22 used canonical time warping and repetitive vowel patterns to find the alignment for vowel sequence. Some other works achieved the alignment at music structure-level @cite_23 or line-level @cite_37 .
@cite_37 @cite_26 @cite_33 @cite_22 @cite_8 @cite_36 @cite_9 @cite_21 @cite_17 @cite_3 @cite_39 @cite_23 @cite_2 @cite_15 @cite_13 @cite_25 @cite_12 @cite_11
2144827818 182406043 2482558056 2012433269
Aligning lyrics to audio has a wide range of applications such as the automatic generation of karaoke scores, song-browsing by lyrics, and the generation of audio thumbnails. Existing methods are restricted to using only lyrics and match them to phoneme features extracted from the audio (usually mel-frequency cepstral coefficients). Our novel idea is to integrate the textual chord information provided in the paired chords-lyrics format known from song books and Internet sites into the inference procedure. We propose two novel methods that implement this idea: First, assuming that all chords of a song are known, we extend a hidden Markov model (HMM) framework by including chord changes in the Markov chain and an additional audio feature (chroma) in the emission vector; second, for the more realistic case in which some chord information is missing, we present a method that recovers the missing chord information by exploiting repetition in the song. We conducted experiments with five changing parameters and show that with accuracies of 87.5 and 76.7 , respectively, both methods perform better than the baseline with statistical significance. We introduce the new accompaniment interface Song Prompter, which uses the automatically aligned lyrics to guide musicians through a song. It demonstrates that the automatic alignment is accurate enough to be used in a musical performance. This paper proposes an algorithm for aligning singing in pol yphonic music audio with textual lyrics. As preprocessing, t he system uses a voice separation algorithm based on melody transcription and sinusoidal modeling. The alignment is based on a hidden Markov model speech recognizer where the acoustic model is adapted to singing voice. The textual input is preprocess ed to create a language model consisting of a sequence of phonemes, pauses and possible instrumental breaks. Viterbi algorith m is used to align the audio features with the text. On a test set consis ting of 17 commercial recordings, the system achieves an average absolute error of 1.40 seconds in aligning lines of the lyrics. This study addresses the task of aligning lyrics with accompanied singing recordings. With a vowel-only representation of lyric syllables, our approach evaluates likelihood scores of vowel types with glottal pulse shapes and formant frequencies extracted from a small set of singing examples. The proposed vowel likelihood model is used in conjunction with a prior model of frame-wise syllable sequence in determining an optimal evolution of syllabic position. In lyrics alignment experiments, we optimized numerical parameters on two independent development sets and then tested the optimized system on two other datasets. New objective performance measures are introduced in the evaluation to provide further insight into the quality of alignment. Use of glottal pulse shapes and formant frequencies is shown by a controlled experiment to account for a 0.07 difference in average normalized alignment error. Another controlled experiment demonstrates that, with a difference of 0.03, F0-invariant glottal pulse shape gives a lower average normalized alignment error than does F0-invariant spectrum envelope, the latter being assumed by MFCC-based timbre models. Abstract We present a system for the automatic transcription of solo human singing into note sequences. Particular attention is devoted to the modelling of the transitions between consecutive notes of the sung passage. Hidden Markov models (HMMs) are used to represent both individual notes and the transitions between them, in order to capture the variability of the estimated pitch within a statistical framework. A database consisting of annotated passages sung by 26 soprano subjects is compiled and used for the development of the system. Explicit transition models are introduced in order to better identify note boundaries that are otherwise poorly modelled. Context-independent transition models are evaluated first, followed by context-dependent transition models which are shown to improve both the note recognition accuracy and the transition region time-alignment. The final system is found to be able to transcribe sung passages with an overall accuracy of 86.7 .