Automatic Music Transcription: An Overview <s> A comparison of NMF and NN models <s> A new method for the estimation of multiple concurrent pitches in piano recordings is presented. It addresses the issue of overlapping overtones by modeling the spectral envelope of the overtones of each note with a smooth autoregressive model. For the background noise, a moving-average model is used and the combination of both tends to eliminate harmonic and sub-harmonic erroneous pitch estimations. This leads to a complete generative spectral model for simultaneous piano notes, which also explicitly includes the typical deviation from exact harmonicity in a piano overtone series. The pitch set which maximizes an approximate likelihood is selected from among a restricted number of possible pitch combinations as the one. Tests have been conducted on a large homemade database called MAPS, composed of piano recordings from a real upright piano and from high-quality samples. <s> BIB001 </s> Automatic Music Transcription: An Overview <s> A comparison of NMF and NN models <s> While recent years have witnessed large progress in the algorithm of automatic music transcription (AMT), the development of general and sizable datasets for AMT evaluation is relatively stagnant, predominantly due to the fact that manually annotating and checking such datasets is labor-intensive and time-consuming. In this paper we propose a novel note-level annotation method for building AMT datasets by utilizing human’s ability in following music in real-time. To test the quality of the annotation, we further propose an efficient method in qualifying an AMT dataset based on the concepts of onset error difference and the tolerance computed from the evaluation result. According to the experiments on five piano solos and four woodwind quintets, we claim that the proposed annotation method is reliable for evaluation of AMT algorithms. <s> BIB002 </s> Automatic Music Transcription: An Overview <s> A comparison of NMF and NN models <s> Given a musical audio recording, the goal of automatic music transcription is to determine a score-like representation of the piece underlying the recording. Despite significant interest within the research community, several studies have reported on a “glass ceiling” effect, an apparent limit on the transcription accuracy that current methods seem incapable of overcoming. In this paper, we explore how much this effect can be mitigated by focusing on a specific instrument class and making use of additional information on the recording conditions available in studio or home recording scenarios. In particular, exploiting the availability of single note recordings for the instrument in use, we develop a novel signal model employing variable-length spectro-temporal patterns as its central building blocks-tailored for pitched percussive instruments such as the piano. Temporal dependencies between spectral templates are modeled, resembling characteristics of factorial scaled hidden Markov models FS-HMM and other methods combining nonnegative matrix factorization with Markov processes. In contrast to FS-HMMs, our parameter estimation is developed in a global, relaxed form within the extensible alternating direction method of multipliers framework, which enables the systematic combination of basic regularizers propagating sparsity and local stationarity in note activity with more complex regularizers imposing temporal semantics. The proposed method achieves an f -measure of 93-95% for note onsets on pieces recorded on a Yamaha Disklavier MAPS DB. <s> BIB003 </s> Automatic Music Transcription: An Overview <s> A comparison of NMF and NN models <s> We consider the problem of transcribing polyphonic piano music with an emphasis on generalizing to unseen instruments. We use deep neural networks and propose a novel approach that predicts onsets and frames using both CNNs and LSTMs. This model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe it correlates better with human musical perception. This technique results in over a 100% relative improvement in note with offset score on the MAPS dataset. <s> BIB004
Given the popularity of NMF and NN-based methods for AMT, it is interesting to discuss their differences. In particular, neglecting the nonnegativity constraints, NMF is a linear, generative model. Given that NMF-based methods are increasingly being replaced by NN-based ones, the question arises in which way linearity could be a limitation for an AMT model. To look into this, assume we are given an NMF dictionary with two spectral templates for each musical pitch. To represent an observed spectrum of a single pitch C4, we can linearly combine the two templates associated with C4. The set (or manifold) of valid spectra for C4 notes, however, is complex, and thus, in most cases, our linear interpolation will not correspond to a real-world recording of a C4. We could increase the number of templates such that their interpolation could potentially get closer to a real C4-but the number of invalid spectra we can represent increases much more quickly compared to the number of valid spectra. Deep networks have shown considerable potential in recent years to implicitly represent such complex manifolds in a robust and comparatively efficient way  . An additional benefit over generative models like NMF is that NNs can be trained in an end-to-end fashion, i.e., note detections can be a direct output of a network without the need for additional postprocessing of model parameters (such as NMF activations). Yet, despite these quite principled limitations, NMF-based methods remain competitive or even exceed the results achieved using NNs. Currently, there are two main challenges for NNbased approaches. First, there are only a few, relatively small annotated data sets available, and these are often subject to severe biases BIB002 . The largest publicly available data set BIB001 contains several hours of piano music-but all recorded on only seven different synthesizer-based and real pianos. While typical data augmentation strategies, such as pitch shifting or simulating different room acoustics, might mitigate some of the effects, there is still a considerable risk that a network overfits the acoustic properties of these specific instruments. For many types of instruments, even small data sets are not available. Other biases include musical style as well as the distribution over central musical concepts, such as key, harmony, tempo, and rhythm. A second considerable challenge is the adaptability to new acoustic conditions. Providing just a few examples of isolated notes of the instrument to be transcribed, considerable improvements are observed in the performance of NMF-based models. There is currently no corresponding, equally effective mechanism to retrain or adapt an NN-based AMT system on a few seconds of audio. Thus, the error rate for nonadapted networks can be an order of magnitude higher than that of an adapted NMF system BIB003 , BIB004 . Overall, as both of these challenges cannot easily be overcome, NMF-based methods are likely to remain relevant in specific use cases. In Figure 7 , we qualitatively illustrate some differences in the behavior of systems based on supervised NMF and NNs. Both systems were specifically trained for transcribing piano recordings, and we expose the approaches to a recording of an organ. Like the piano, the organ is played with a keyboard, but its acoustic properties are quite different. The harmonics of the organ are rich in energy and cover the entire spectrum; the energy of the notes does not decay over time, and the onsets are less pronounced. With this experiment, we want to find out how gracefully the systems fail when they encounter a sound that is outside the piano-sound manifold but still musically valid. Comparing the NMF output in Figure 7 (a) and the NN output in Figure 7(b) with the ground truth, we find that both methods detect additional notes (shown in red), mostly at octaves above and below the correct fundamental. Given the rich energy distribution, this behavior is expected. While we use a simple baseline model for NMF-and thus some errors could be attributed to that choice-the NN fails more gracefully. That is, fewer octave errors and fewer spurious short note detections are observed. (Yet, in terms of recall, the NMF-based approach identifies additional correct notes.) It is difficult to argue why the acoustic model within the network should be better prepared for such a situation. However, the results suggest that the network learned something additional: the LSTM layers as used in the network (compare Figure 6 ) seem to have learned how typical piano notes evolve in time, and thus most note lengths look reasonable and less spurious. Similarly, the bandwidth in which octave errors occur is narrower for the NN, which could potentially indicate that the network models the likelihood of cooccurring notes or, in other words, a simple music language model (MLM). This leads us to our discussion of important remaining challenges in AMT.