Automatic Music Transcription: An Overview <s> NMF for AMT <s> Non-negative matrix factorization (NMF) has previously been shown to be a useful decomposition for multivariate data. Two different multiplicative algorithms for NMF are analyzed. They differ only slightly in the multiplicative factor used in the update rules. One algorithm can be shown to minimize the conventional least squares error while the other minimizes the generalized Kullback-Leibler divergence. The monotonic convergence of both algorithms can be proven using an auxiliary function analogous to that used for proving convergence of the Expectation-Maximization algorithm. The algorithms can also be interpreted as diagonally rescaled gradient descent, where the rescaling factor is optimally chosen to ensure convergence. <s> BIB001 </s> Automatic Music Transcription: An Overview <s> NMF for AMT <s> We investigate a data-driven approach to the analysis and transcription of polyphonic music, using a probabilistic model which is able to find sparse linear decompositions of a sequence of short-term Fourier spectra. The resulting system represents each input spectrum as a weighted sum of a small number of "atomic" spectra chosen from a larger dictionary; this dictionary is, in turn, learned from the data in such a way as to represent the given training set in an (information theoretically) efficient way. When exposed to examples of polyphonic music, most of the dictionary elements take on the spectral characteristics of individual notes in the music, so that the sparse decomposition can be used to identify the notes in a polyphonic mixture. Our approach differs from other methods of polyphonic analysis based on spectral decomposition by combining all of the following: a) a formulation in terms of an explicitly given probabilistic model, in which the process estimating which notes are present corresponds naturally with the inference of latent variables in the model; b) a particularly simple generative model, motivated by very general considerations about efficient coding, that makes very few assumptions about the musical origins of the signals being processed; and c) the ability to learn a dictionary of atomic spectra (most of which converge to harmonic spectral profiles associated with specific notes) from polyphonic examples alone-no separate training on monophonic examples is required. <s> BIB002 </s> Automatic Music Transcription: An Overview <s> NMF for AMT <s> Multiple pitch estimation consists of estimating the fundamental frequencies and saliences of pitched sounds over short time frames of an audio signal. This task forms the basis of several applications in the particular context of musical audio. One approach is to decompose the short-term magnitude spectrum of the signal into a sum of basis spectra representing individual pitches scaled by time-varying amplitudes, using algorithms such as nonnegative matrix factorization (NMF). Prior training of the basis spectra is often infeasible due to the wide range of possible musical instruments. Appropriate spectra must then be adaptively estimated from the data, which may result in limited performance due to overfitting issues. In this paper, we model each basis spectrum as a weighted sum of narrowband spectra representing a few adjacent harmonic partials, thus enforcing harmonicity and spectral smoothness while adapting the spectral envelope to each instrument. We derive a NMF-like algorithm to estimate the model parameters and evaluate it on a database of piano recordings, considering several choices for the narrowband spectra. The proposed algorithm performs similarly to supervised NMF using pre-trained piano spectra but improves pitch estimation performance by 6% to 10% compared to alternative unsupervised NMF algorithms. <s> BIB003 </s> Automatic Music Transcription: An Overview <s> NMF for AMT <s> A method for automatic transcription of polyphonic music is proposed in this work that models the temporal evolution of musical tones. The model extends the shift-invariant probabilistic latent component analysis method by supporting the use of spectral templates that correspond to sound states such as attack, sustain, and decay. The order of these templates is controlled using hidden Markov model-based temporal constraints. In addition, the model can exploit multiple templates per pitch and instrument source. The shift-invariant aspect of the model makes it suitable for music signals that exhibit frequency modulations or tuning changes. Pitch-wise hidden Markov models are also utilized in a postprocessing step for note tracking. For training, sound state templates were extracted for various orchestral instruments using isolated note samples. The proposed transcription system was tested on multiple-instrument recordings from various datasets. Experimental results show that the proposed model is superior to a non-temporally constrained model and also outperforms various state-of-the-art transcription systems for the same experiment. <s> BIB004 </s> Automatic Music Transcription: An Overview <s> NMF for AMT <s> Recently, new methods for smart decomposition of time-frequency representations of audio have been proposed in order to address the problem of automatic music transcription. However those techniques are not necessarily suitable for notes having variations of both pitch and spectral envelope over time. The HALCA (Harmonic Adaptive Latent Component Analysis) model presented in this article allows considering those two kinds of variations simultaneously. Each note in a constant-Q transform is locally modeled as a weighted sum of fixed narrowband harmonic spectra, spectrally convolved with some impulse that defines the pitch. All parameters are estimated by means of the expectation-maximization (EM) algorithm, in the framework of Probabilistic Latent Component Analysis. Interesting priors over the parameters are also introduced in order to help the EM algorithm converging towards a meaningful solution. We applied this model for automatic music transcription: the onset time, duration and pitch of each note in an audio file are inferred from the estimated parameters. The system has been evaluated on two different databases and obtains very promising results. <s> BIB005 </s> Automatic Music Transcription: An Overview <s> NMF for AMT <s> Given a musical audio recording, the goal of automatic music transcription is to determine a score-like representation of the piece underlying the recording. Despite significant interest within the research community, several studies have reported on a “glass ceiling” effect, an apparent limit on the transcription accuracy that current methods seem incapable of overcoming. In this paper, we explore how much this effect can be mitigated by focusing on a specific instrument class and making use of additional information on the recording conditions available in studio or home recording scenarios. In particular, exploiting the availability of single note recordings for the instrument in use, we develop a novel signal model employing variable-length spectro-temporal patterns as its central building blocks-tailored for pitched percussive instruments such as the piano. Temporal dependencies between spectral templates are modeled, resembling characteristics of factorial scaled hidden Markov models FS-HMM and other methods combining nonnegative matrix factorization with Markov processes. In contrast to FS-HMMs, our parameter estimation is developed in a global, relaxed form within the extensible alternating direction method of multipliers framework, which enables the systematic combination of basic regularizers propagating sparsity and local stationarity in note activity with more complex regularizers imposing temporal semantics. The proposed method achieves an f -measure of 93-95% for note onsets on pieces recorded on a Yamaha Disklavier MAPS DB. <s> BIB006
The basic idea behind NMF and its variants is to represent a given nonnegative time-frequency representation , e.g., a magnitude spectrogram, as a product of two nonnegative matrices: a diction- Figure 3) . Computationally, the goal is to minimize a distance (or divergence) between V and DA with respect to D and . A As a straightforward approach to solving this minimization problem, multiplicative update rules have been central to the success of NMF. While there is a wide range of applicable methods, AMT has been dominated during the last decade by two algorithmic families: NMF and NNs. specific step size in a gradient (or, rather, coordinate) descentbased minimization of the divergence BIB001 . In an AMT context, both unknown matrices have an intuitive interpretation. The nth column of , V i.e., the spectrum at time point n, is modeled in NMF as a linear combination of the K columns of , D and the corresponding K coefficients are given by the nth column of . A Given this point of view, each column of D is generally referred to as a spectral template and usually represents the expected spectral energy distribution associated with a specific note played on a specific instrument. For each template, the corresponding row in A is referred to as the associated activation and encodes when and how intensely that note is played over time. Given the nonnegativity constraints, NMF yields a purely constructive representation in the sense that the spectral energy modeled by one template cannot be canceled by another. This property is often seen as instrumental in identifying a parts-based and interpretable representation of the input BIB001 . In Figure 3 , an NMF-based decomposition is illustrated. The magnitude spectrogram V shown in Figure 3 (a) is modeled as a product of the dictionary D and activation matrix , A shown in Figure 3 (c) and (d), respectively. The product DA is given in Figure 3(b) . In this case, the templates correspond to individual pitches, with clearly visible fundamental frequencies and harmonics. Additionally, comparing A with the piano-roll representation shown in Figure 1(c) indicates the correlation between NMF activations and the underlying musical score. While Figure 3 illustrates the principles behind NMF, it also indicates why AMT is difficult. Indeed, a regular NMF decomposition would rarely look as clean as in Figure 3 . Compared to speech analysis, sound objects in music are highly correlated. For example, even in a simple piece as in Figure 1 , most pairs of simultaneous notes are separated by musically consonant intervals, which acoustically means that many of their partials overlap [e.g., the A and D notes around 4 s, marked with gray circles in Figure 1(d) , share a high number of partials]. In this case, it can be difficult to disentangle how much energy belongs to which note. The task is further complicated by the fact that the spectrotemporal properties of notes vary considerably between different pitches, playing styles, dynamics, and recording conditions. Furthermore, the stiffness property of the strings affects the travel speed of transverse waves based on their frequency. As a result, the partials of instruments like the piano are not found at perfect integer multiples of the fundamental frequency. Because of this property, called inharmonicity, the positions of partials differ between individual pianos (see Figure 4) . To address these challenges, the basic NMF model has been extended by encouraging additional structure in the dictionary and the activations. For example, an important principle is to enforce sparsity in A to obtain a solution dominated by activations that are few but substantial; the success of sparsity paved the way for a whole range of sparse-coding approaches, in which the dictionary size K can considerably exceed the input dimension M BIB002 . Other extensions focus on the dictionary design. In the case of supervised NMF, the dictionary is precomputed and fixed using additionally available training material. For example, given K recordings, each containing only a single note, the dictionary shown in Figure 3 (b) was constructed by extracting one template from each recording. This way, the templates are guaranteed to be free of interference from other notes and also to have a clear interpretation. As another example, Figure 5 illustrates an extension in which each NMF template is represented as a linear combination of fixed narrow-band subtemplates BIB003 , which enforces a harmonic structure for all NMF templates. This way, a dictionary can be adapted to the recording to be transcribed, while maintaining its clean, interpretable structure. In shift-invariant dictionaries, a single template can be used to represent a range of different fundamental frequencies. In particular, using a logarithmic frequency axis, the distances between individual partials of a harmonic sound are fixed, and thus shifting a template in frequency allows the modeling of sounds of varying pitch. Sharing parameters between different pitches in this way has turned out to be effective for increasing model capacity (see, e.g., BIB004 and BIB005 ). Furthermore, spectrotemporal dictionaries alleviate a specific weakness of NMF models. In NMF, it is difficult to express that notes often have a specific temporal evolution. For example, the beginning of a note (or attack phase) might have entirely different spectral properties than the central part (decay phase). Such relationships are modeled in spectrotemporal dictionaries using a Markov process, which governs the sequencing of templates across frames so that different subsets of templates can be used for the attack and decay parts, respectively BIB004 , BIB006 .