Like language, music can be represented as a sequence of discrete symbols that form a hierarchical syntax, with notes being roughly like characters and motifs of notes like words.   Unlike text however, music relies heavily on repetition on multiple timescales to build structure and meaning. The Music Transformer has shown compelling results in generating music with structure (Huang et al., 2018).   In this paper, we introduce a tool for visualizing self-attention on polyphonic music with an interactive pianoroll.   We use music transformer as both a descriptive tool and a generative model.   For the former, we use it to analyze existing music to see if the resulting self-attention structure corroborates with the musical structure known from music theory.   For the latter, we inspect the model's self-attention during generation, in order to understand how past notes affect future ones. We also compare and contrast the attention structure of regular attention to that of relative attention (Shaw et al., 2018, Huang et al., 2018), and examine its impact on the resulting generated music.   For example, for the JSB Chorales dataset, a model trained with relative attention is more consistent in attending to all the voices in the preceding timestep and the chords before, and at cadences to the beginning of a phrase, allowing it to create an arc.   We hope that our analyses will offer more evidence for relative self-attention as a powerful inductive bias for modeling music.   We invite the reader to explore our video animations of music attention and to interact with the visualizations at https://storage.googleapis.com/nips-workshop-visualization/index.html. Attention is a cornerstone in neural network architectures. It can be the primary mechanism for constructing a network, such as in the self-attention based Transformer, or serve as a secondary mechanism for connecting parts of a model that would otherwise be far apart or different modalities of varying dimensionalities. Attention also offers us an avenue for visualizing the inner workings of a model, often to illustrate alignments BID3 . For example in machine translation, the Transformer uses attention to build up both context and alignment while in the LSTM-based seq2seq models, attention eases the word alignment between source and target sentences. For both types, attention gives points us to where a model is looking when translating BID6 BID0 . For example in speech recognition, attention aligns different modalities from spectograms to phonemes BID1 .In contrast to the above domains, there is less "groundtruth" in what should be attended to in a creative domain such as music. Moreover , in contrast to encoder-decoder models where attention serves as alignment, in language modeling self-attention serves to build context, to retrieve relevant information from the past to predict the future. Music theory gives us some insight of the motivic, harmonic, temporal dependencies across a piece, and attention could be a lens in showing their relevance in a generative setting, i.e. does the model have to pay attention to this previous motif to generate the new note? Music Transformer , based on self-attention BID6 , has been shown to be effective in modeling music, being able to generate sequences with repetition on multiple timescales (motifs and phrases) with long-term coherence BID2 . In particular, the use of relative attention improved sample quality and allowed the model generalize beyond lengths observed during training time. Why does relative attention help? More generally, how does the attention structure look like on these models?In this paper, we introduce a tool for visualizing self-attention on music with an interactive pianoroll. We use Music Transformer as both a descriptive tool and a generative model. For the former, we use it to analyze existing music to see if the resulting self-attention structure corroborates with musical structure known from music theory. For the latter, we inspect the model's self-attention during generation, in order to understand how past notes affect future ones. We explore music attention on two music datasets, JSB Chorales and Piano-e-Competition. The former are Chorale harmonizations , and we see attention keeping track of the harmonic progression and also voice-leading. The latter are virtuosic classical piano music and attention looks back on previous motifs and gestures. We show for JSB Chorales the heads in multihead-attention distribute and focus on different temporal regions.Moreover, we compare and contrast the attention structure of regular attention to that of relative attention, and examine its impact on the resulting generated music. For example, for the JSB Chorales dataset, a model trained with relative attention is more consistent in attending to all the voices in the preceding timestep and the many chords before, and at cadences to the beginning of a phrase, allowing it to create an arc. In contrast, regular attention often becomes a "local" model only attending to the most recent history, resulting in certain voice repeating the same note for a long duration, perhaps due to overconfidence. We presented a visualization tool for seeing and exploring music self-attention in context of music sequences. We have shown some preliminary observations and we hope this it the beginning to furthering our understanding in how these models learn to generate music.
Visualizing the differences between regular and relative attention for Music Transformer.
{"src": "scitldr-AIC_train", "paper_id": "ryfxVNEajm"}