Automatic Music Transcription: An Overview <s> NNs for AMT <s> In this paper, we present a connectionist approach to automatic transcription of polyphonic piano music. We first compare the performance of several neural network models on the task of recognizing tones from time-frequency representation of a musical signal. We then propose a new partial tracking technique, based on a combination of an auditory model and adaptive oscillator networks. We show how synchronization of adaptive oscillators can be exploited to track partials in a musical signal. We also present an extension of our technique for tracking individual partials to a method for tracking groups of partials by joining adaptive oscillators into networks. We show that oscillator networks improve the accuracy of transcription with neural networks. We also provide a short overview of our entire transcription system and present its performance on transcriptions of several synthesized and real piano recordings. Results show that our approach represents a viable alternative to existing transcription systems. <s> BIB001 </s> Automatic Music Transcription: An Overview <s> NNs for AMT <s> In this paper a new approach for polyphonic piano note onset transcription is presented. It is based on a recurrent neural network to simultaneously detect the onsets and the pitches of the notes from spectral features. Long Short-Term Memory units are used in a bidirectional neural network to model the context of the notes. The use of a single regression output layer instead of the often used one-versus-all classification approach enables the system to significantly lower the number of erroneous note detections. Evaluation is based on common test sets and shows exceptional temporal precision combined with a significant boost in note transcription performance compared to current state-of-the-art approaches. The system is trained jointly with various synthesized piano instruments and real piano recordings and thus generalizes much better than existing systems. <s> BIB002 </s> Automatic Music Transcription: An Overview <s> NNs for AMT <s> Given a musical audio recording, the goal of automatic music transcription is to determine a score-like representation of the piece underlying the recording. Despite significant interest within the research community, several studies have reported on a “glass ceiling” effect, an apparent limit on the transcription accuracy that current methods seem incapable of overcoming. In this paper, we explore how much this effect can be mitigated by focusing on a specific instrument class and making use of additional information on the recording conditions available in studio or home recording scenarios. In particular, exploiting the availability of single note recordings for the instrument in use, we develop a novel signal model employing variable-length spectro-temporal patterns as its central building blocks-tailored for pitched percussive instruments such as the piano. Temporal dependencies between spectral templates are modeled, resembling characteristics of factorial scaled hidden Markov models FS-HMM and other methods combining nonnegative matrix factorization with Markov processes. In contrast to FS-HMMs, our parameter estimation is developed in a global, relaxed form within the extensible alternating direction method of multipliers framework, which enables the systematic combination of basic regularizers propagating sparsity and local stationarity in note activity with more complex regularizers imposing temporal semantics. The proposed method achieves an f -measure of 93-95% for note onsets on pieces recorded on a Yamaha Disklavier MAPS DB. <s> BIB003 </s> Automatic Music Transcription: An Overview <s> NNs for AMT <s> We present a supervised neural network model for polyphonic piano music transcription. The architecture of the proposed model is analogous to speech recognition systems and comprises an acoustic model and a music language model . The acoustic model is a neural network used for estimating the probabilities of pitches in a frame of audio. The language model is a recurrent neural network that models the correlations between pitch combinations over time. The proposed model is general and can be used to transcribe polyphonic music without imposing any constraints on the polyphony. The acoustic and language model predictions are combined using a probabilistic graphical model. Inference over the output variables is performed using the beam search algorithm. We perform two sets of experiments. We investigate various neural network architectures for the acoustic models and also investigate the effect of combining acoustic and music language model predictions using the proposed architecture. We compare performance of the neural network-based acoustic models with two popular unsupervised acoustic models. Results show that convolutional neural network acoustic models yield the best performance across all evaluation metrics. We also observe improved performance with the application of the music language models. Finally, we present an efficient variant of beam search that improves performance and reduces run-times by an order of magnitude, making the model suitable for real-time applications. <s> BIB004 </s> Automatic Music Transcription: An Overview <s> NNs for AMT <s> In an attempt at exploring the limitations of simple approaches to the task of piano transcription (as usually defined in MIR), we conduct an in-depth analysis of neural network-based framewise transcription. We systematically compare different popular input representations for transcription systems to determine the ones most suitable for use with neural networks. Exploiting recent advances in training techniques and new regularizers, and taking into account hyper-parameter tuning, we show that it is possible, by simple bottom-up frame-wise processing, to obtain a piano transcriber that outperforms the current published state of the art on the publicly available MAPS dataset -- without any complex post-processing steps. Thus, we propose this simple approach as a new baseline for this dataset, for future transcription research to build on and improve. <s> BIB005 </s> Automatic Music Transcription: An Overview <s> NNs for AMT <s> We consider the problem of transcribing polyphonic piano music with an emphasis on generalizing to unseen instruments. We use deep neural networks and propose a novel approach that predicts onsets and frames using both CNNs and LSTMs. This model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe it correlates better with human musical perception. This technique results in over a 100% relative improvement in note with offset score on the MAPS dataset. <s> BIB006
As for many tasks relating to pattern recognition, NNs have, in recent years, had a considerable impact on the problem of music transcription and on music signal processing in general. NNs are able to learn a nonlinear function (or a composition of functions) from input to output via an optimization algorithm, such as stochastic gradient descent  . Compared to other fields, including image processing, progress on NNs for music transcription has been slower, and we will discuss a few of the underlying reasons. CD note played on a piano. The stiffness of the strings causes partials to be shifted from perfect integer multiples of the fundamental frequency (shown as vertical dotted lines). Here, the 23rd partial is at the position where the 24th harmonic would be expected. Note that the fundamental frequency of 34.65 Hz is missing, as piano soundboards typically do not resonate for modes with a frequency smaller than . 50 Hz . One of the earliest approaches based on NNs was Marolt's Sonic system BIB001 . A central component in this approach was the use of time-delay networks, which resemble convolutional networks in the time direction  and were employed to analyze the output of adaptive oscillators to track and group partials in the output of a gammatone filterbank. Although it was initially published in 2001, the approach remains competitive and still appears in comparisons in more recent publications BIB003 . In the context of the more recent revival of NNs, a first successful system was presented by Böck and Schedl BIB002 . One of the core ideas was to use two spectrograms as input to enable the network to exploit both a high time accuracy (when estimating the note onset position) and a high frequency resolution (when disentangling notes in the lower frequency range). This input is processed using one (or more) long short-term memory (LSTM) layers  . The potential benefit of using LSTM layers is twofold. First, the spectral properties of a note evolve across input frames, and LSTM networks have the capability to compactly model such sequences. Second, medium-and long-range dependencies between notes can potentially be captured. For example, based on a popular chord sequence, after hearing C and G major chords followed by A minor, a likely successor is an F major chord. An investigation of whether such long-range dependencies are indeed modeled, however, was not within the scope of this work. Sigtia et al. BIB004 focused on long-range dependencies in music by combining an acoustic front end with a symboliclevel module resembling a language model, as used in speech processing. Using information obtained from MIDI files, a recurrent NN (RNN) was trained to predict the active notes in the next time frame, given those in the past. This approach needed to learn and represent a very large joint probability distribution, i.e., a probability for every possible combination of active and inactive notes across time. Note that, even in a single frame, there are 2 88 possible combinations of notes on a piano. To render the problem of modeling such an enormous probability space tractable, the approach employed a specific NN architecture (the Neural Autoregressive Distribution Estimator, also known as NADE), which represented a large joint probability as a long product of conditional probabilities, an approach quite similar to the idea popularized recently by the well-known WaveNet architecture. Despite the use of a dedicated music language model, which was trained on relatively large MIDI-based data sets, only modest improvements over an HMM baseline could be observed, and thus the question remains open regarding to which degree long-range dependencies were indeed captured. To further disentangle the influence of the acoustic front end from the language model on potential improvement in performance, Kelz et al. BIB005 focused on the acoustic modeling, reporting on the results of a larger-scale hyperparameter search and describing the influence of individual system components. Trained using this careful and extensive procedure, the resulting model outperformed existing models by a reasonable margin. In other words, while in speech processing, language models have led to a drastic improvement in performance, the same effect is still to be demonstrated in an AMT system, a challenge we will discuss in more detail hereafter. The development of NN-based AMT approaches continues. The current state-of-the-art method for general-purpose piano transcription was proposed by Google Brain BIB006 . Combining and extending ideas from existing methods, this approach combines two networks ( Figure 6 ). One detects note onsets, and its output is used to inform a second network, which focuses on perceiving note lengths. This can be interpreted from a probabilistic point of view. Note onsets are rare events compared to framewise note activity detections. The split into two network branches can thus be interpreted as splitting the representation of a relatively complex joint probability distribution over onsets and frame activity into a probability over onsets and a probability over frame activities, conditioned on the onset The resulting template is constrained to represent harmonic sounds by construction. Compared to other fields, including image processing, progress on NNs for music transcription has been slower. distribution. Since the temporal dynamics of onsets and frame activities are quite different, this can lead to improved learning behavior for the entire network when trained jointly.