Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music. Since the beginning of the recent wave of deep learning research, there have been many attempts to create generative models of expressive musical audio de novo. These models would ideally generate audio that is both musically and sonically realistic to the point of being indistinguishable to a listener from music composed and performed by humans.However, modeling music has proven extremely difficult due to dependencies across the wide range of timescales that give rise to the characteristics of pitch and timbre (short-term) as well as those of rhythm (medium-term) and song structure (long-term). On the other hand, much of music has a large hierarchy of discrete structure embedded in its generative process: a composer creates songs, sections, and notes, and a performer realizes those notes with discrete events on their instrument, creating sound. The division between notes and sound is in many ways analogous to the division between symbolic language and utterances in speech.The WaveNet model by BID18 may be the first breakthrough in generating musical audio directly with a neural network. Using an autoregressive architecture, the authors trained a model on audio from piano performances that could then generate new piano audio sample-bysample. However, as opposed to their highly convincing speech examples, which were conditioned on linguistic features, the authors lacked a conditioning signal for their piano model. The result was audio that sounded very realistic at very short time scales (1 or 2 seconds), but that veered off into chaos beyond that. BID4 made great strides towards providing longer term structure to WaveNet synthesis by implicitly modeling the discrete musical structure described above. This was achieved by training a hierarchy of VQ-VAE models at multiple time-scales, ending with a WaveNet decoder to generate piano audio as waveforms. While the results are impressive in their ability to capture long-term structure directly from audio waveforms, the resulting sound suffers from various artifacts at the fine-scale not present in the unconditional WaveNet, clearly distinguishing it from real musical audio. Also, while the model learns a version of discrete structure from the audio, it is not Transcription: onsets & frames (section 4) Synthesis: conditional WaveNet (section 6) Piano roll (MIDI) We have demonstrated the Wave2Midi2Wave system of models for factorized piano music modeling, all enabled by the new MAESTRO dataset. In this paper we have demonstrated all capabilities on the same dataset, but thanks to the new state-of-the-art piano transcription capabilities, any large set of piano recordings could be used, 6 which we plan to do in future work. After transcribing the recordings, the transcriptions could be used to train a WaveNet and a Music Transformer model, and then new compositions could be generated with the Transformer and rendered with the WaveNet. These new compositions would have similar musical characteristics to the music in the original dataset, and the audio renderings would have similar acoustical characteristics to the source piano.The most promising future work would be to extend this approach to other instruments or even multiple simultaneous instruments. Finding a suitable training dataset and achieving sufficient transcription performance will likely be the limiting factors.The new dataset (MIDI, audio, metadata, and train/validation/test split configurations) is available at https://g.co/magenta/maestro-datasetunder a Creative Commons Attribution NonCommercial Share-Alike 4.0 license. The Online Supplement, including audio examples, is available at https://goo.gl/magenta/maestro-examples.
We train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure, enabled by the new MAESTRO dataset.
{"src": "scitldr-AIC_train", "paper_id": "r1lYRjC9F7"}