Towards Context-Aware Neural Performance-Score Synchronisation
Music can be represented in multiple forms, such as in the audio form as a
recording of a performance, in the symbolic form as a computer readable score,
or in the image form as a scan of the sheet music. Music synchronisation
provides a way to navigate among multiple representations of music in a unified
manner by generating an accurate mapping between them, lending itself
applicable to a myriad of domains like music education, performance analysis,
automatic accompaniment and music editing. Traditional synchronisation methods
compute alignment using knowledge-driven and stochastic approaches, typically
employing handcrafted features. These methods are often unable to generalise
well to different instruments, acoustic environments and recording conditions,
and normally assume complete structural agreement between the performances and
the scores. This PhD furthers the development of performance-score
synchronisation research by proposing data-driven, context-aware alignment
approaches, on three fronts: Firstly, I replace the handcrafted features by
employing a metric learning based approach that is adaptable to different
acoustic settings and performs well in data-scarce conditions. Secondly, I
address the handling of structural differences between the performances and
scores, which is a common limitation of standard alignment methods. Finally, I
eschew the reliance on both feature engineering and dynamic programming, and
propose a completely data-driven synchronisation method that computes
alignments using a neural framework, whilst also being robust to structural
differences between the performances and scores.