Synthesizing audio of piano performances from MIDI requires either a huge collection of recordings of individual notes, or a model that simulates an actual piano. These approaches, however, have two main limitations: (i) the “stitching" of individual notes might not optimally capture the various interactions between notes [1]}; and (ii) the quality of the synthesized audio is restricted by the recordings in the sound library or the piano simulator. This further motivates the effort to build realistic piano synthesizers.
In this work, we propose a neural network based synthesizer which takes the onset roll as input,
and generates realistic piano performances in the audio domain.
Our model takes the onset roll instead of the complete piano roll as input, thereby waiving the necessity of fine-grained frame and velocity information, which are often unavailable,
to synthesize expressive piano performances. Without the constraint of frame and velocity information, performance style transfer could also be achieved by interpreting each onset note with the style inferred from a given audio performance.
On top of the onset roll, the generation is further conditioned on variables corresponding to style features, which enable the rendition of expressive piano performances.
In particular, we consider articulation and dynamics, which are two significant features related to expressiveness in piano performances, and base our framework on variational autoencoders (VAEs) [2]} with a Gaussian mixture prior [3]}.
