MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE

Abstract: Transformers and variational autoencoders (VAE) have been extensively employed for symbolic (e.g., MIDI) domain music generation. While the former boast an impressive capability in modeling long sequences, the latter allow users to willingly exert control over different parts (e.g., bars) of the music to be generated. In this paper, we are interested in bringing the two together to construct a single model that exhibits both strengths. The task is split into two steps. First, we equip Transformer decoders with the ability to accept segment-level, time-varying conditions during sequence generation. Subsequently, we combine the developed and tested in-attention decoder with a Transformer encoder, and train the resulting MuseMorphose model with the VAE objective to achieve style transfer of long pop piano pieces, in which users can specify musical attributes including rhythmic intensity and polyphony (i.e., harmonic fullness) they desire, down to the bar level. Experiments show that MuseMorphose outperforms recurrent neural network (RNN) based baselines on numerous widely-used metrics for style transfer tasks.

\section{Introduction}
Music is ubiquitous in our lives.
It conveys various emotions, delightful or sorrowful; intense or relaxed, through the twists and turns of notes that are beyond any language barriers.
It is, therefore, not hard to imagine why automatic music composition has been an active and captivating research topic with endeavors dating back to more than half a century ago \citep{hiller1959experiment}.

In recent years, due to the renaissance of neural networks \citep{lecun2015deep}, we have seen a proliferation of deep learning-based methods being proposed to generate music \citep{hadjeres2017deepbach, dong2018musegan, dhariwal2020jukebox}.
Among the works concerning symbolic domain music generation (i.e., generating sheet music, not audio) \citep{huang19music, donahue2019lakhnes, roberts2018hierarchical}, two deep learning architectures were widely adopted, namely, the Transformer \citep{vaswani2017attention} and the variational autoencoder (VAE) \citep{kingma2014auto}.
Transformers, thanks to the design to aggregate information from the full history of their computation records, are capable of composing coherent music of up to 4 minutes long \citep{payne2019musenet}.
VAEs, on the other hand, give users the opportunity to control the generative process through operations on the learned latent space \citep{roberts2018hierarchical}, which stores the high-level semantics of a musical excerpt.
To the best of our knowledge, though there has been some research on Transformer-based VAEs in the language domain \citep{li2020optimus}, which focused on rather short sequences, no previous effort has been made to bring together the two powerful architectures for music generation.
Hence, it is the goal of this paper to construct such a model to attain both the aforementioned strengths.

In achieving the goal, we split our work into two steps.
First, we devise mechanisms to condition Transformer decoders \citep{radford2019language} with segment-level, time-varying conditioning vectors during long sequence generation.
Three methods, i.e., \textbf{pre-attention}, \textbf{in-attention}, and \textbf{post-attention}, which inject conditions into Transformers at different stages in the architecture, are proposed.
We conduct both objective and subjective studies to show that \textbf{in-attention} most effectively exerts control over the model.
Next, we combine an \textbf{in-attention}-featured Transformer decoder with a Transformer encoder \citep{devlin18bert}, which operates rather on the segment level, to build our ultimate \textbf{MuseMorphose} model for fine-grained music style transfer.
We introduce attribute embeddings \citep{fu2018style} to MuseMorphose, and train the model with the VAE objective, which effectively turns it into a VAE underpinned by Transformers.
Experiments demonstrate that MuseMorphose excels in generating style-transferred versions of pop piano performances (i.e., with expressive timing and dynamics, not just plain sheet music) of 32 bars (or measures) long, in which users can freely control two ordinal musical attributes, i.e.,  \textit{rhythmic intensity} and \textit{polyphony}, in each bar.

Our key contributions can be summarized as:
\begin{itemize}[leftmargin=*]
    \item We devise the \textbf{in-attention} mechanism to firmly harness Transformers' generative process with segment-level, changing conditions.
    \item We pair a Transformer decoder, augmented with \textbf{in-attention}, with a bar-level Transformer encoder to form our \textbf{MuseMorphose} Transformer-based VAE, which marks an advancement over \textit{Optimus} \citep{li2020optimus} in terms of conditioning mechanism, granularity of control, and accepted sequence length.
    \item We employ \textbf{MuseMorphose} for the style transfer of 32-bar-long pop piano performances, on which it surpasses prior art \citep{brunner2018midi, kawai2020attributes} on various metrics without having to use style-directed loss functions.
\end{itemize}

We have set up a companion website,\footnote{Companion website URL: \url{https://slseanwu.github.io/site-musemorphose}}
in which readers may find a high-level techical overview and listen to some of MuseMorphose's generations.

The remainder of this paper is structured as follows.
Section 2 provides a comprehensive walk-through of the technical background.
Sections 3 and 4, which are the main body of our work, focus respectively on the segment-level conditioning for Transformer decoders, and the MuseMorphose Transformer-based VAE model for fine-grained music style transfer.
In each of these two sections, we start from problem formulation; then, we elaborate on our method(s), followed by evaluation procedure and metrics; finally, we present the results and offer some discussion.
Section 5 concludes the paper and provides potential directions to pursue in the future.

\section{Technical Background}
\subsection{Event-based Representation for Music}\label{sec:ev-repr}
To facilitate the modeling of music with neural sequence models, an important pre-processing step is to tokenize a musical piece $X$ into a sequence of events, i.e.,
\begin{equation}
 X = \{x_1, x_2,\dots,x_T\}, \
\end{equation}
where $T$ is the length of the music in the resulting event-based representation.
The event sequence then serves directly as input and output of the sequence model.
Such tokenization process can be easily done with music in the symbolic format of Musical Instrument Digital Interface, or MIDI,\footnote{Check \url{https://midi.org/specifications} for more detailed specifications.} in short.
A MIDI file stores a musical piece's tempo (in beats per minute, or bpm), time signature (e.g., 3/4, 4/4, 6/8, etc.), as well as each note's onset and release timestamps and velocity (i.e., loudness).
Multiple tracks can also be included for pieces with more than one playing instruments.

The first event-based representation scheme for MIDI music, which we call MIDI-like representation hereafter, was formalized in \citep{oore2018time}. It represents a musical piece as a sequence consisting of the following four types of events:
\begin{itemize}[leftmargin=*]
    \item \textsc{Note-On}: marks the onset of a note ($0{\sim}127$, each corresponding to a musical pitch);
    \item \textsc{Note-Off}: marks the release of a note ($0{\sim}127$);
    \item \textsc{Time-Shift}: marks the advance of time ($0{\sim}124$, denoting 8 msec to 1 sec);
    \item \textsc{Set-Velocity}: sets the intended loudness of subsequent notes ($0{\sim}31$ levels).
\end{itemize}
The MIDI-like representation, and its variants, were adopted by several works of music generation, the most prominent ones being \textit{Music Transformer} \citep{huang19music} and \textit{MuseNet} \citep{payne2019musenet}.
The support for multi-track music was added by \citep{donahue2019lakhnes}, using separate sets of \textsc{Note-On}'s and \textsc{Note-Off}'s for different instruments.

Though simple and closely following the MIDI format, the MIDI-like representation has some obvious drawbacks.
First, it does not take into consideration the inherent metric of time in music, given by \textit{beats} and \textit{bars} (i.e., \textit{measures}).
Moreover, the scattered \textsc{Note-On}'s and \textsc{Note-Off}'s render the sequence model prone to generating dangling \textsc{Note-On}'s \citep{huang2020pop}, since there are often numerous events residing between a note's onset and release.
To address these issues, an enhanced event-based representation, Revamped MIDI-derived events (REMI), was proposed in \citep{huang2020pop}.
REMI is beat-based, and improved over MIDI-like representation in the following ways:
\begin{itemize}[leftmargin=*]
    \item \textsc{Bar} and \textsc{Position} ($1{\sim}16$, denoting the time within a bar, in quarter beat increments) events were used instead of \textsc{Time-Shift} to represent the progression of time.
    \item \textsc{Tempo} events were added to explicitly set the pace the music should be played at.
    \item \textsc{Note-On} and \textsc{Note-Duration} (in quarter beat, i.e., sixteenth note, increments) events, which always co-occur at a note's onset, replaced the potentially faraway \textsc{Note-On}/\textsc{Note-Off} pairs in the MIDI-like representation.
\end{itemize}
The authors of REMI also suggested employing simple rule-based chord recognition algorithms to add \textsc{Chord} events, e.g., \textsc{Cmaj}, \textsc{G\footnotesize{7}}, etc., to inform the sequence model of the harmonic characteristic of the music to be generated.
It should be noted that representations above essentially construct for the sequence model a vocabulary much smaller in size ($<$500 unique tokens) compared to those used for text modeling \citep{sennrich16neural, wu2016google}, which usually contain $>$10k unique tokens.

Objective and subjective evaluation in \citep{huang2020pop} have shown that REMI outperformed MIDI-like representation in terms of human preference and the modeling of rhythmic structures.
Hence, in this work, we use slightly modified versions of REMI, which will be explained in detail in Sections 3 and 4.

\subsection{Transformers}
Transformers \citep{vaswani2017attention} are a family of neural sequence models that are generally thought of as the potent successors of recurrent neural betworks (RNN) \citep{hochreiter97lstm, cho-etal-2014-properties}.
They are well known for the outstanding capabilities in language understanding \citep{devlin18bert, yang2019xlnet}, language modeling \citep{radford2019language, brown2020language}, and very recently, image generation \citep{ramesh2021zero, jiang2021transgan, esser2021taming}.

\textbf{Model Architecture.}
In general, a Transformer operates on an input sequence $X = \{x_1,\dots,x_T\}$.\footnote{We note that combinations of Transformers operating on multiple sequences are also often seen.}
It can be either a bidirectional encoder, or an autoregressive decoder of the sequence.
The operations within a typical Transformer are detailed below.
A token $x_t$ in the input sequence first meets an embedding layer to become a token embedding $\bm{x}_t \in \mathbb{R}^{d}$, where $d$ is the model's hidden state dimension.
Then, $\bm{x}_t$ goes through the core of the Transformer---a series of $L$ \textit{(self-)attention modules} sharing the same architecture.
An attention module\footnote{It is also often called a \textit{(self-)attention layer}. However, we use the term \textit{module} for it actually involves multiple layers of operations.} can be further divided into 2 sub-modules: the \textit{multi-head attention} (MHA) \citep{vaswani2017attention} sub-module, and the \textit{position-wise feedforward net} (FFN) sub-module.

Suppose the input token embedding $\bm{x}_t$ has passed $l-1$ such attention modules to produce the hidden state $\bm{h}_t^{l-1} \in \mathbb{R}^{d}$, the operations that take place in the next (i.e., $l^{\text{\,th}}$) attention module can be summarized as follows:
\begin{align}
    \bm{s}_t^l &= \text{LayerNorm}(\bm{h}_t^{l-1} + \text{MHA}(\bm{h}_t^{l-1} \mid H^{l-1})) \\
    \bm{h}_t^l &= \text{LayerNorm}(\bm{s}_t^l + \text{FFN}(\bm{s}_t^l)) \label{eqn:tf-ffn},
\end{align}
where $H^{l-1} = [\bm{h}_1^{l-1},\dots,\bm{h}_T^{l-1}]^\top \in \mathbb{R}^{T \times d}$, and $\bm{s}_t^{l}, \bm{h}_t^{l} \in \mathbb{R}^{d}$. The presence of residual connections \citep{he2016deep} and layer normalization (LayerNorm) \citep{ba2016layer} ensures that gradients back-propagate smoothly into the network.
In the MHA sub-module, the \textit{query}, \textit{key}, and \textit{value} matrices ($Q^l, K^l, V^l \in \mathbb{R}^{T \times d}$) are produced first by three linear transformations, i.e.,
\begin{align}
    Q^l &= H^{l-1}W_Q^l &
    K^l &= H^{l-1}W_K^l &
    V^l &= H^{l-1}W_V^l,
\end{align}
where $W_Q^l, W_K^l, W_V^l \in \mathbb{R}^{d \times d}$ are learnable weight matrices.
In the multi-head case, $Q^l$ is further split into sub-matrices $[Q^{l, 1},\dots,Q^{l, M}]$, where $M$ is the number of heads, and $Q^{l, m} \in \mathbb{R}^{T \times \frac{d}{M}}, \forall m \in \{1,\dots,M\}$; the same applies for $K^l$ and $V^l$.
Then, the \textit{dot-product attention} is performed individually on each $\{Q^{l, m}$, $K^{l, m}$, $V^{l, m}\}$:
\begin{equation}
\begin{split}
    A^{l, m} &= \frac{Q^{l, m}{K^{l, m}}^{\top}}{ \sqrt{d/M}}\label{eqn:dot-prod-attn} \\
    A^{l, m} &= \text{causal{\textunderscore}mask}(A^{l, m}) \; \; \text{(optional)} \\
    S^{l, m} &= \text{row-wise{\textunderscore}softmax}(A^{l, m})V^{l, m},
\end{split}
\end{equation}
where $A^{l, m} \in \mathbb{R}^{T \times T}$ stores the so-called attention scores. Causal (lower-triangular) masking is applied only for autoregressive sequence modeling tasks, where we must guarantee that the query vector at every timestep $t$, namely $\bm{q}_t^l$, has no access to future information stored in $\bm{k}_{>t}^l$.
Finally, the attention results are aggregated and merged with the residual connection to form the final output of MHA, namely $S^l = [\bm{s}_1^l,\dots,\bm{s}_T^l]^\top \in \mathbb{R}^{T \times d}$, through:
\begin{equation}
    S^{l} = \text{LayerNorm}(H^{l-1} + \text{concat}([S^{l, 1},\dots,S^{l, M}])W_O^l)
\end{equation}
where $W_O^l \in \mathbb{R}^{d \times d}$ is another learnable weight matrix.
The use of multiple attention heads, by design, grants the model the freedom to develop specialized heads, each paying attention to different parts of the sequence \citep{clark2019does}.
The resulting $\bm{s}_t^l$'s then pass individually through the FFN sub-module, which is typically a simple two-layer fully-connected network with ReLU activations in between, and merge with another residual connection to become the final output of the attention module, i.e., $\bm{h}_t^l$ (see Eq.~(\ref{eqn:tf-ffn})).
Intuitively speaking, the attention module realizes a \textit{contextualization} process, through which $\bm{h}_t^l$ gathers information from $\bm{h}_t^{l-1}$ and other parts of the sequence (i.e., $H^{l-1}$) in the MHA sub-module, and then has the contextualized information further processed by the FFN sub-module.
In autoregressive sequence modeling (i.e., decoding) tasks, the output of the last attention module, $\bm{h}_t^L$, is then transformed by an output projection and a softmax operation to generate the probability distribution:
\begin{equation}
p(x_{t+1} \mid x_1,\dots,x_t),
\end{equation}
from which the next token can be sampled during inference.

The computations above, however, leave the model unaware of the positions of the input tokens.
That is to say, we would obtain the same results even if we randomly permute the tokens, which is clearly not a desired property for sequence models.
This issue can be addressed by adding a \textit{positional encoding} $\text{pos}(t)$ to the input token embeddings, i.e.,
\begin{equation}
    \bm{x}_t = \bm{x}_t + \text{pos}(t),\label{eq:pos-encode}
\end{equation}
where $\text{pos}(t)$ can either be fixed sinusoidal waves \citep{vaswani2017attention}, or learnable embeddings \citep{devlin18bert}.
More sophisticated methods which inform the model of the relative positions between all pairs of tokens during multi-head attention have also been invented \citep{shaw2018self, dai2019transformer}.
In fact, positional encodings for Transformers are still an active research area, where new mechanisms and evaluation metrics are being continually proposed \citep{ke2021rethinking, wang2021position, liutkus2021relative}.

Due to the quadratic memory complexity arising from Eq.~(\ref{eqn:dot-prod-attn}), it is often impossible for the model to take sequences beyond 2$\sim$3k tokens in length.
To alleviate this, \textit{Transformer-XL} \citep{dai2019transformer} combined relative positional encoding, segment-level recurrence, and cache of hidden states to allow tokens in the current segment to refer to those in the preceding segment, thereby resolving the context fragmentation issue with long input sequences.
Algorithms to estimate the quadratic attention under linear complexity have also been proposed recently \citep{katharopoulos2020transformers, choromanski2021rethinking}.
These linear Transformers can operate on sequences with well over 10k tokens.

Transformers can be trained straightforward with gradient descent-based optimizers such as Adam \citep{kingma2014adam}.
In our work, we extensively employ Transformers, as well as Transformer-XL, as our backbone sequence encoders and decoders, and modify their architecture to achieve controllable conditional music generation.

\textbf{Musical Applications.}
Recent years have seen a plethora of research on leveraging Transformers to generate music in the symbolic domain (e.g., in MIDI format).
\textit{Music Transformer} \citep{huang19music} was the pioneer of such work, which verified for the first time that Transformers are able to generate expressive and coherent piano performances.
\textit{MuseNet} \citep{payne2019musenet} utilized OpenAI's large-scale GPT-2 \citep{radford2019language} language model, trained with large amount of music data and control tokens, to compose music in diverse styles.
Transformer-XL \citep{dai2019transformer} has also been employed to model multi-track music \citep{donahue2019lakhnes}, pop piano performances \citep{huang2020pop}, jazz lead sheets \citep{wu2020jazz}, and even guitar tabs \citep{chen2020automatic} that are readable by human guitarists.
Among those works, \textit{LakhNES} \citep{donahue2019lakhnes} demonstrated that Transformers can benefit from pre-training on large-scale music datasets.
\citep{hsiao21aaai, ren2020popmag} made efforts to compress the event sequences, which are often long in the case of music, by enhancing REMI representation \citep{huang2020pop}, and modifying Transformers' embedding and output projections.
Their methods were shown effective in accelerating the training and inference of Transformer-based music generation models.

\subsection{Variational Autoencoders (VAE)}\label{sec:vae-trick}
Variational Autoencoders (VAE) \citep{kingma2014auto} are a type of deep latent variable model comprising an encoder, a decoder, and a Kullback-Leibler (KL) divergence-regularized latent space in between.
VAEs are aimed simultaneously at representation learning \citep{bengio2013representation} and generative modeling, and have strong applications in image generation \citep{vahdat2020nvae}, text style transfer \citep{john2019disentangled}, and music generation \citep{roberts2018hierarchical}.

\textbf{Model and Training Objective.}
VAEs bear high architectural resemblance to basic Autoencoders (AE) \citep{hinton2006reducing}, which are neural networks primarily used for non-linear dimensionality reduction of data.
An AE uses an encoder to map an instance $\bm{x}$ (e.g., an image, a sentence, a musical piece, etc.) in the input space $\mathcal{X}$ to a vector $\bm{z}$ in the latent space $\mathcal{Z}$---typically, we enforce $\text{dim}(\mathcal{Z}) \ll \text{dim}(\mathcal{X})$---and then uses a decoder to reconstruct $\bm{x}$ from the latent vector $\bm{z}$.
Mathematically, an AE performs:
\begin{equation}
    \bm{z} = \text{enc}(\bm{x}), \;\;
    \bm{\hat{x}} = \text{dec}(\bm{z}); \;\;
    \bm{x}, \bm{\hat{x}} \in \mathcal{X} \; \text{and} \;
    \bm{z} \in \mathcal{Z},
\end{equation}
where $\bm{\hat{x}}$ is the reconstructed input, and minimizes $\delta(\bm{x}, \bm{\hat{x}})$, where $\delta(\cdot, \cdot)$ is some distance metric defined over $\mathcal{X}$.
However, as there are no restrictions put on $\mathcal{Z}$ other than its dimensionality, the mapping $\mathcal{X} \rightarrow \mathcal{Z}$ can be arbitrary, which often results in a poorly-structured latent space unfruitful for generative modeling.

VAEs \citep{kingma2014auto}, on the other hand, are based on the hypothesis that generative processes follow $p(\bm{x}) = p(\bm{x} \vert \bm{z})p(\bm{z})$, where $p(\bm{z})$ is the prior distribution of some intermediate-level latent variable $\bm{z}$.
After a series of mathematical derivations, it can be shown that generative modeling can be achieved through maximizing:
\begin{equation}
    \mathcal{L}_\text{ELBO} =
    \mathbb{E}_{\bm{z}\sim q_\phi(\bm{z}\vert\bm{x})}\log p_\theta(\bm{x}\vert\bm{z}) - D_\text{KL}(q_\phi(\bm{z}\vert\bm{x}) \| p(\bm{z})),\label{eqn:elbo}
\end{equation}
where $D_\text{KL}$ denotes KL divergence, and $q_\phi(\bm{z}\vert\bm{x})$ is the estimated posterior distribution emitted by the encoder (parameterized by $\phi$), while $p_\theta(\bm{x}\vert\bm{z})$ is the likelihood modeled by the decoder (parameterized by $\theta$).
It is often called the \textit{evidence lower bound} (ELBO) objective since it can be proved that:
\begin{equation}
    -\mathcal{L}_\text{ELBO} \leq \log p(\bm{x}).
\end{equation}
$\mathcal{L}_\text{ELBO}$ can be simply understood as adding a latent space KL regularization term to the reconstruction objective (the first term) already present in AEs. Hence, minimal architectural changes are required to turn an AE into a VAE.
The only problem is that sampling from the latent posterior, i.e., $\bm{z}\sim q_\phi(\bm{z}\vert\bm{x})$, is a non-differentiable process, so gradient back-propagation would be hampered.
To resolve this, in practice, we usually restrict $q_\phi(\bm{z}\vert\bm{x})$ to isotropic gaussians $\mathcal{N}(\bm{\mu}, \text{diag}(\bm{\sigma}^2))$, and have the encoder emit the mean and standard deviation vectors, $\bm{\mu}, \bm{\sigma} \in \mathbb{R}^{\text{dim}\mathcal{(Z)}}$, instead (known as \textit{reparameterization trick} \citep{kingma2014auto}), so that we can detach sampling from gradient propagation path with:
\begin{equation}
    \bm{z} = \bm{\mu} + \bm{\sigma} \odot \bm{\epsilon}; \;\; \bm{\epsilon} \sim \mathcal{N}(\bm{0}, I),
\end{equation}
where $\bm{\epsilon}$ is a randomly sampled noise, and $\odot$ denotes element-wise operation.
For simplicity, the prior distribution $p(\bm{z})$ is often set to be isotropic standard gaussian $\mathcal{N}(\bm{0}, I)$.
The construction above implies that independence of latent space dimensions is assumed.
Therefore, the KL divergence can be calculated straightforward with:
\begin{equation}
    D_\text{KL}(q_\phi(\bm{z}\vert\bm{x}) \| p(\bm{z})) =
    \frac{1}{2}\sum_{i=1}^{\text{dim}(\mathcal{Z})} \sigma^2_i + \mu_i^2 - \log(\sigma_i) - 1.
\end{equation}
With the VAE objective, we can explore the regularized, better-structured latent space to generate new instances in $\mathcal{X}$.
One common method is \textit{latent space interpolation}, where we obtain a new latent vector $\bm{z}'$ by interpolating the latent representation of two existing instances $\bm{x_1}, \bm{x_2}$; that is, $\bm{z}' = \gamma\bm{z_1} + (1 - \gamma)\bm{z_2}, \text{where } \gamma \in [0, 1]$, and feed $\bm{z}'$ to the decoder to generate reasonable and creative intermediate versions between $\bm{x_1}, \bm{x_2}$, as done in \citep{roberts2018hierarchical, li2020optimus}.

For practical considerations, $\beta$-VAE \citep{higgins2017beta} introduced an adjustable hyperparameter $\beta$ to the VAE objective, thereby maximizing the function:
\begin{equation}
    \mathcal{L}_{\beta\text{-VAE}} =
    \mathbb{E}_{\bm{z}\sim q_\phi(\bm{z}\vert\bm{x})}\log p_\theta(\bm{x}\vert\bm{z}) - \beta D_\text{KL}(q_\phi(\bm{z}\vert\bm{x}) \| p(\bm{z})),\label{eqn:beta-vae}
\end{equation}
where $\beta$ controls the weight on the KL regularization term.
Though Eq.~(\ref{eqn:beta-vae}) deviates from optimizing ELBO, i.e., Eq.~(\ref{eqn:elbo}), which is more theoretically sound,
this setup grants practitioners additional freedom.
A larger beta ($\beta > 1$) has been shown to promote feature disentanglement in the latent space, while a smaller one trades the smoothness of latent space for better reconstruction.
It is worth mentioning that, in musical applications, $\beta$ is often set to $0.1{\sim}0.2$ \citep{roberts2018hierarchical, brunner2018midi, kawai2020attributes}.

\textbf{Training Techniques.}
In the literature, it has been repeatedly mentioned that VAEs suffer from the \textit{posterior collapse} problem \citep{sonderby2016ladder, bowman2016generating, dieng2019avoiding}, in which case the estimated posterior $q_\phi(\bm{z}\vert\bm{x})$ fully collapses onto the prior $p(\bm{z})$, leading to a zero KL divergence and an information-less latent space.
The problem is especially severe when a powerful decoder is used, or when the decoding is autoregressive, where previous tokens in the sequence reveal strong information.
Numerous methods have been proposed to tackle this problem, either by tweaking the training objective \citep{kingma2016improving, fu2019cyclical} or slightly modifying the model architecture \citep{dieng2019avoiding}.

\citet{kingma2016improving} altered the VAE objective to allow \textit{free bits} in the latent space dimensions, i.e.,
\begin{equation}
    \mathcal{L}_{\text{free-bit}} =
    \mathbb{E}_{\bm{z}\sim q_\phi(\bm{z}\vert\bm{x})}\log p_\theta(\bm{x}\vert\bm{z}) - \sum_{i=1}^{\text{dim}(\mathcal{Z})} \max(\lambda,   D_\text{KL}(q_\phi(z_i\vert\bm{x}) \| p(z_i))),\label{eqn:free-bit-vae}
\end{equation}
where $\lambda$ is a hyperparameter ensuring that it is not advantageous to drive down the $D_\text{KL}$ per dimension further than that value.
From an information-theoretic point of view, the objective above allows each dimension to store $\lambda$ nats\footnote{1 nat $\approx$ 1.44 bits} of information for free.

\textit{Cyclical KL annealing} \citep{fu2019cyclical} dynamically adjusts $\beta$ (i.e., weight on the KL term, see Eq.~(\ref{eqn:beta-vae})) during training.
The technique relies on two hyperparameters: $R$, the cycle duration (in training steps); and, $\beta_{\text{max}}$, the maximum $\beta$ to be used.
In each cycle, $\beta$ is linearly increased from $0$ to $\beta_{\text{max}}$ in the first half, and then fixed at $\beta_{\text{max}}$ in the second half; an identical cycle restarts when $R$ steps have elapsed.
Empirical evidence have shown that this simple technique led to more information-rich latent space, and therefore benefited downstream tasks such as language modeling and text classification.

\textit{Skip-VAE} \citep{dieng2019avoiding} addressed posterior collapse with slight architectural changes to the VAE model.
In Skip-VAEs, the latent condition $\bm{z}$ is fed to all layers and timesteps instead of just the initial ones.
Their theoretical analysis showed that this practice provably increases the mutual information between input $\bm{x}$ and the estimated latent condition $\bm{z}$. This theorem was further supported by empirical evidence.

The three techniques introduced above are easy to implement and compatible with each other.
Hence, in our work, we take advantage of all of them to facilitate the training of our Transformer-based VAE.

\textbf{Integration with Transformers.}
Parameterizing VAEs with Transformers is a rather unexplored direction.
An exemplary work is \textit{Optimus} \citep{li2020optimus}.
\textit{Optimus} successfully linked together a pair of pre-trained Transformer encoder (\textit{BERT}) \citep{devlin18bert} and decoder (\textit{GPT-2}) \citep{radford2019language} with a VAE latent space, and devised mechanisms such that the latent variable can exert control over the Transformer decoder.
Latent space interpolation and editing with Optimus were effective, which led to generated sentences with reasonable semantic transitions.
Another application in story completion \citep{wang2019t} using Transformers and VAE was also proposed.
However, both work were limited to generating short texts (only one sentence), and conditioned the Transformer decoder at the global level.
In our research, we push the frontiers by first figuring out a way to tightly harness Transformers with segment-level, changing conditions, and then applying such technique to conditional long sequence generation via optimizing the VAE objective.

\textbf{Musical Applications.}
\textit{MusicVAE} \citep{roberts2018hierarchical} designed a hierarchical RNN decoder architecture to capture long-term structures in music.
In MusicVAE, a \textit{conductor RNN} is tasked with the bar-level, long-term dependency, while a \textit{note RNN} accepts the outputs of the conductor RNN and generates notes in each bar independently.
This model has been shown capable of encoding and reconstructing melodies and multi-instrument music of up to 16 bars long,
and latent space interpolation could be performed to generate meaningful intermediate versions between two short melodies.
\textit{PianoTree VAE} \citep{wang2020pianotree} tackled polyphonic music\footnote{\textit{Polyphonic} means multiple notes, or even melody lines, are played at a time.} learning with intermediate-level RNNs that handles the abstraction and modeling of simultaneously onset notes, and further lower-level RNNs that determine exact note pitches and duration.
Their work extended latent space interpolation to the polyphonic case.
\citep{hadjeres2017glsr, brunner2018midi, kawai2020attributes} are also VAE-based music generation models, with more emphasis put on the controllability over generated music; these models will be introduced in more detail in the next sub-section.

\subsection{Conditional Music Generation}
\textbf{Conditioning with Control Tokens.}
A native and simple way to realize conditional generation with event-based representations (see Sec.~\ref{sec:ev-repr}) is to augment the vocabulary with a set of \textit{control tokens}.
\textit{MuseNet} \citep{payne2019musenet} has demonstrated that prepending composer and instrumentation tokens to the beginning of the sequences can strongly affect the composing style of the generated music and restrict the instruments used within.
Regarding conditioning at local scale, \citet{huang2020pop, wu2020jazz} have experimented with adding frequently appearing chord markings or structure markings to the event sequences, to guide the generation of musical segments immediately succeeding those tokens.

\textbf{Conditioning with Full Melodies.}
In some use cases, users might have already composed the melody and hence only require a model to generate appropriate accompaniments.
In this scenario, the generation model should be designed to capture the fine-grained information provided in full melodies.
\citet{choi2020encoding} used two Transformer encoders to encode melody and performance style information separately, and then feed their (pooled) outputs to a Transformer decoder for reconstruction.
They showed that by changing the input to the performance style encoder, harmonization of the melody in different styles can be achieved.
Multi-track accompaniment generation was investigated in \textit{PopMAG} \citep{ren2020popmag}.
They utilized a sequence-to-sequence Transformer architecture, in which the decoder has access to every token in the given melody tracks, to generate the content in (missing) accompaniment tracks.
Melody harmonization with a Transformer decoder alone has also been achieved in \citep{hsiao21aaai}, in which sequences of interleaved melody and the corresponding full performance segments were designed for its training.

\textbf{Conditioning with Latent Space Operations.}
Controlling over the style, content, or some observable attributes of the generated music has been shown possible with latent variable models, either through direct manipulation on the latent vector $\bm{z}$ \citep{roberts2018hierarchical, brunner2018midi}, or by introducing additional information to the latent space \citep{chen2020music, kawai2020attributes}.

MusicVAE \citep{roberts2018hierarchical} has demonstrated that vector arithmetic can be performed in the latent space to alter some attributes of the input music.
For example, it is possible to increase the note density of the input music, while largely preserving other content, by feeding to the VAE decoder an edited latent vector $\bm{z}' = \bm{z} + \bm{z}_\text{dense}$, where $\bm{z}_\text{dense}$ is the \textit{difference} between the mean latent vectors of inputs with a high note density and those with a low note density.
\textit{Music FaderNets} \citep{tan2020music} has also achieved attribute control in a similar way, and further showed that the control of a high-level attribute (i.e., the degree of arousal) is achievable with a combination of operations on well-learned low-level representations (i.e., rhythm density and note density).

Similar to \citep{tan2020music}, \textit{GLSR-VAE} \citep{hadjeres2017glsr} and \textit{MIDI-VAE} \citep{brunner2018midi} both used auxiliary losses to force some latent space dimensions to be discriminative of the style or attributes to be controlled.
The former directly linked the value of some latent dimensions to the attributes of the output music (e.g., \# of notes), while the latter imposed a classifier on those latent dimensions for style transfer.
\citet{kawai2020attributes}, on the other hand, introduced a discriminator \citep{goodfellow2014generative} over the latent space to prevent $\bm{z}$ from encoding attribute information, and delegated attribute control to learned embeddings fed directly to the decoder along with $\bm{z}$, which should solely contain content information.
Experiments have shown that high correlation between a given user-specified attribute (e.g., \# of notes, separated into 8 ordinal classes) and the attribute value computed from the outcomes can be attained.
Our work extends this approach to long sequences with Transformers, and verify that such attribute control can be accomplished without a discriminator or any auxiliary losses.

\section{Conditioning Transformers at the Segment Level}
This section sets aside latent variable models first, and is concerned with conditioning an autoregressive Transformer decoder with a series of pre-given, time-varying conditions $c_1,\dots,c_K$, each belonging to a pre-defined, non-overlapped segment of the target sequence.
To approach this problem, three mechanisms, namely, \textbf{pre-attention}, \textbf{in-attention}, and \textbf{post-attention} conditioning, are proposed.
Experiments demonstrate that, in terms of offering tight control, \textbf{in-attention} surpasses the other two, as well as a mechanism slightly tweaked from \citep{li2020optimus}, which serves as our baseline.

\subsection{Problem Formulation}\label{sec:form-ch3}
Under typical settings,
Transformer decoders \citep{vaswani2017attention} are employed for autoregressive generative modeling of sequences:
\begin{equation}
    p(x_t \vert x_{<t}) \,,
    \label{eq:uncond}
\end{equation}
where $x_t$ is the element of a sequence to predict at timestep $t$, and $x_{<t}$ is all the (given) preceding elements of the sequence.
Once a model is trained, the model can generate new sequences autoregressively, i.e., one element at a time based on the previously generated elements, in chronological order of the timesteps \citep{radford2019language}.

The \textit{unconditional} type of generation associated with Eq.~(\ref{eq:uncond}), however, does not offer control mechanisms to guide the generation process of the Transformer.
One alternative is to consider a \textit{conditional} scenario where the model is informed of a \textit{global} condition $c$ for the entire sequence to generate, as used by, for example, the \textit{CTRL} model for text \citep{keskar2019ctrl} or the \textit{MuseNet} model for music \citep{payne2019musenet}:
\begin{equation}
    p(x_t | x_{<t}, c) \,.
    \label{eq:cond_global}
\end{equation}
When the target sequence length is long, it may be beneficial to extend Eq.~(\ref{eq:cond_global}) by using instead a sequence of conditions $c_1, c_2, \dots, c_K$, each belonging to different parts of the sequence, to offer fine-grained control through:
\begin{equation}
    p(x_t | x_{<t}; \, c_1, c_2, \dots, c_K ) \,.
    \label{eq:cond_local_memory}
\end{equation}
This can be implemented, for example, by treating representations of $c_1, c_2, \dots, c_K$ as additional memory for the Transformer decoder to attend to, which can be achieved with minimal modifications to \citep{li2020optimus}.

In this paper, we particularly address the case where the target sequences can be, by nature, divided into multiple meaningful, non-overlapping segments,
such as sentences in a text article, or bars (measures) in a musical piece.
That is to say, for sequences with $K$ segments, each timestep index $t \in [1, T]$ belongs to one of the $K$ sets of indices $I_1, I_2, \dots, I_K$, where $I_k \cap I_{k'} = \varnothing$ for $k \neq k'$ and $\bigcup_{k=1}^K I_k = [1, T]$.
Thus, we can provide the generative model with a condition $c_k$ for each segment during the corresponding time interval $I_k$, leading to the modeling of:
\begin{equation}
    p(x_t | x_{<t}; \, c_k ) \,, \quad \text{for}~ t \in I_k \,.
    \label{eq:cond_local_ours}
\end{equation}
This conditional generation task is different from the one outlined in Eq.~(\ref{eq:cond_local_memory}) in that we know specifically which condition (among the $K$ conditions) to attend to for each time step.
The segment-level condition $c_k$ may manifest itself as a sentence embedding in text, or a bar-level embedding in music.
In our case, we consider the conditions to be bar-level embeddings of a musical piece, i.e., we represent each $c_k$ in the vector form $\bm{c}_k \in \mathbb{R}^{d_c}$, where $d_c$ is the dimensionality of the bar-level embedding.
Collectively, the time-varying conditions $\bm{c}_1, \bm{c}_2, \dots, \bm{c}_K$ offer a \textit{blueprint}, or high-level planning, of the sequence to model.
This is potentially helpful for generating long sequences, especially for generating music, as ancestral sampling from unconditional autoregressive models, which are trained only with long sequences of scattered tokens, may fail to exhibit the high-level flow and twists that are central to a catchy piece of music.

The \textit{segment-level conditional generation} scenario associated with Eq.~(\ref{eq:cond_local_ours}), though seems intuitive for long sequence modeling, has not been much studied in the literature to the best of our knowledge.
Hence, it is our goal to design methods that specifically tackle this task, and perform comprehensive evaluation to find out the most effective way.

\subsection{Method}
\begin{figure*}
    \centering
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/pre-attn-new-converted.pdf}
        \caption{Pre-attention conditioning}
        \label{subfig:pre-attention}
    \end{subfigure} \\\vspace{8mm}
    \centering
    \begin{subfigure}[b]{0.4\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/in-attn-new-converted.pdf}
        \caption{In-attention conditioning}
        \label{subfig:in-attention}
    \end{subfigure}
    \hfill
    \centering
    \begin{subfigure}[b]{0.423\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/post-attn-new-converted.pdf}
        \caption{Post-attention conditioning}
        \label{subfig:post-attention}
    \end{subfigure}
    \caption{The architecture of segment-level conditional Transformer decoders using different conditioning mechanisms.}
    \label{fig:transformers}
\end{figure*}

Here, we elaborate the proposed three mechanisms: \textbf{pre-attention}, \textbf{in-attention}, and \textbf{post-attention} (see Fig.~\ref{fig:transformers} for schematic comparison), for conditioning Transformer decoders at segment level.
For simplicity, it is assumed that the bar-level conditions, i.e., $\bm{c}_k$'s, can be obtained separately beforehand; the method for their extraction are also to be introduced here.
Finally, we explain the data representation and dataset adopted for this task.

\textbf{Pre-attention Conditioning.}
Under pre-attention, the segment-level conditions enter the Transformer decoder only once \textit{before} all the self-attention layers. The segment embedding $\bm{c}_k$ is first transformed to a hidden condition state $\bm{e}_k$ by a matrix $W_{\text{pre}} \in \mathbb{R}^{d_\text{c} \times d_{\text{e}}}$. Then, it is concatenated with the token embedding of each time step in $I_k$ before the first self-attention layer, i.e.,
\begin{equation}
    \bm{h}^0_t = \text{concat}([\bm{x}_t; \, \bm{e}_k]) \,, \quad \bm{x}_t, \bm{e}_k \in \mathbb{R}^{d_{\text{e}}} \, ,
\end{equation}
where $d_e$ is the dimensionality for token embeddings.\footnote{We use a dimensionality for token embeddings ($d_e$) different than that for hidden states ($d$), but we note that this is not commonly the case.}
Note that under this mechanism, the hidden dimension, $d$, for attention modules is implicitly required to be $2d_{\text{e}}$.

The pre-attention conditioning is methodology-wise similar to the use of  segment embeddings in BERT \citep{devlin18bert}, except that BERT's segment embedding is summed directly with the token embedding, while we use concatenation to strengthen the presence of the conditions.

\textbf{In-attention Conditioning.}
The proposed in-attention mechanism more frequently reminds the Transformer decoder of the segment-level conditions \textit{throughout} the self-attention layers.
It projects the segment embedding $\bm{c}_k$ to the same space as the self-attention hidden states via:
\begin{equation}
    \bm{e}^{\top}_k = \bm{c}^{\top}_k W_\text{in}; \;\; W_\text{in} \in \mathbb{R}^{d_\text{c} \times d},
\end{equation}
and then sums the obtained hidden condition state $\bm{e}_k$ with the hidden states of all the self-attention layers but the last one, to form the input to the subsequent layer, i.e.,
\begin{equation}
\begin{split}
    \tilde{\bm{h}}^l_t &= \bm{h}^l_t + \bm{e}_k\,, \quad \forall l \in \{0, \dots, L-1\} \,; \\
    \bm{h}^{l+1}_t &= \text{Transformer\textunderscore self-attention\textunderscore layer}(\tilde{\bm{h}}^l_t) \,.
    \label{eq:in-attn}
\end{split}
\end{equation}
We use summation rather than concatenation here to keep the residual connections intact. We anticipate that by copy-pasting the segment-level condition everywhere, its influence on the Transformer can be further aggrandized.

\textbf{Post-attention Conditioning.}
Unlike the last two mechanisms, in post-attention, the segment embeddings do not interact with the self-attention layers at all. Instead, they are imposed \textit{afterwards} on the final attention outputs (i.e., $\bm{h}^L_t$) via two single-hidden-layer, parametric ReLU-activated feed-forward networks: the \textit{conditioning net} $\mathcal{B}$, and the \textit{gating net} $\mathcal{G}$.
Specifically, the segment embedding $\bm{c}_k$ is transformed by the conditioning net $\mathcal{B}$ to become $\bm{e}_k \in \mathbb{R} ^d$.
Then, a gating mechanism, dictated by the gating net $\mathcal{G}$, determines ``how much'' of $\bm{e}_k$ should be blended with the self-attention output $\bm{h}^L_t$ at every timestep $t$, i.e.,
\begin{equation}
\begin{split}
    \bm{e}_k &= \mathcal{B}(\bm{c}_k) \,; \\
    \alpha_t &= \text{sigmoid}(\mathcal{G}(
        \text{concat}([\bm{h}^L_t; \, \bm{c}_k])
    )) \,;\\
    \tilde{\bm{h}}^L_t &= (1 - \alpha_t) \cdot \bm{h}^L_t + \alpha_t \cdot \bm{e}_k \,.
\end{split}
\end{equation}
Finally, $\tilde{\bm{h}}^L_t$ proceeds to the output projections
to form the probability distribution of $x_{t+1}$.

This mechanism is adapted from the \textit{contextualized vocabulary bias} proposed in the \textit{Insertion Transformer} \citep{stern2019insertion}. It is designed such that the segment-level conditions can directly bias the output event distributions, and that the model can freely decide how much it refers to the conditions for help at different timesteps.

\textbf{Implementation and Training Details.}
\begin{table}
\centering
\caption{Model attributes shared by the implemented Transformer decoders.}\label{tab:model-attr}
\begin{tabular}{l l r}
\toprule
Attr. & Description & Value \\
\midrule
$T_{\text{tgt}}$ & target sequence length & 1,024 \\
$T_{\text{mem}}$ & Transformer-XL memory length & 1,024 \\
$L$ & \# self-attention layers & 12 \\
$n_{\text{head}}$ & \# self-attention heads & 10 \\
$d_{\text{e}}$ & token embedding dimension & 320 \\
$d$ & hidden state dimension & 640 \\
$d_{\text{ff}}$ & feed-forward dimension & 2,048 \\
$d_{\text{c}}$ & condition embedding dimension & 512 \\
\hline
\# params & \multicolumn{2}{l}{58.7 to 62.6 mil.}\\
\bottomrule
\end{tabular}
\end{table}
We adopt Transformer-XL \citep{dai2019transformer} as the backbone sequence model behind all of our conditioning mechanisms. To demonstrate the superiority of using Eq.~(\ref{eq:cond_local_ours}), we involve two baselines in our study, dubbed \textbf{unconditional} and \textbf{memory} hereafter.
The former takes no $\bm{c}_k$'s, thereby modeling Eq.~(\ref{eq:uncond}) exactly. The latter models Eq.~(\ref{eq:cond_local_memory}) with a conditioning mechanism largely resembling the \textit{memory} scheme introduced in \citep{li2020optimus}, differing only in that we have multiple conditions instead of one.
In the \textbf{memory} baseline, the conditions $\{\bm{c}_k\}_{k=1}^{K}$ are each transformed by matrix $W_{\text{mem}} \in \mathbb{R}^{d_c \times Ld}$ to become hidden states $\{\bm{e}_k^l \in \mathbb{R}^{d}\}_{l=1}^{L}$ unique to each layer $l$.
These states are then fed to the decoder before the training sequence, allowing the tokens (i.e., $x_t$'s) to attend to them.
Some common attributes shared among the models (5 in total) are listed in Table \ref{tab:model-attr}.

All the models are trained with the Adam optimizer \citep{kingma2014adam} and teacher forcing, i.e., always feeding in correct inputs rather than those sampled from previous-timestep outputs from the model itself, to minimize negative log-likelihood:
\begin{equation}
    -\sum_{t=1}^T \log p(x_t \, | \, x_{<t}),
\end{equation}
(NLL) of the training sequences.
Due to limited computational resources, we truncate each song to the first 32 bars.
We warm-up the learning rate linearly from $0$ to $2\text{e}{-}4$ in the first 500 training steps, and use cosine learning rate decay (600,000 steps) afterwards.
Trainable parameters are randomly initialized from the gaussian $\mathcal{N}(0, 0.01^2)$. Each model is trained on one NVIDIA Tesla V100 GPU (with 32GB memory) with a batch size of 4 for around 20 epochs, which requires roughly 2 full days.

\textbf{Extraction of Bar-level Embeddings.}
We extract the conditions $c_k$ from the sequences themselves through a separate encoder network $\mathcal{E}$, which is in fact a typical Transformer decoder with 12 self-attention layers and 8 attention heads. The encoder network $\mathcal{E}$ takes a sequence of event tokens corresponding to a musical bar and outputs a $d_\text{c}$-dimensional embedding vector representing the bar.
We use a GPT-2 \citep{radford2019language} like setup and train $\mathcal{E}$
for autoregressive next-token prediction, and average-pool across all  timesteps on the hidden states of a middle layer $l$ of the Transformer to obtain the segment-level condition embedding, namely,
\begin{equation}
    \bm{c}_k =
    \text{avgpool}([\bm{h}^l_1, \dots, \bm{h}^l_{T_k}]) \,,
\end{equation}
where $\bm{h}^l_t \in \mathbb{R}^{d_\mathcal{E}} (=\mathbb{R}^{d_\text{c}})$ is the hidden state at timestep $t$ after $l$ self-attention layers, and $T_k$ is the number of tokens in the bar. According to \citep{chen2020generative}, the hidden states in middle layers are the best contextualized representation of the input, so we set $l = 6$.

The model $\mathcal{E}$ has 39.6 million trainable parameters.
\sloppy We use all the bars associated with our training data (i.e., not limiting to the first 32 bars of the songs) for training $\mathcal{E}$, with teacher forcing and causal self-attention masking to also minimize the NLL, i.e., $- \sum_t \log p(x_t | x_{<t})$, as we do with our segment-level conditional Transformers.
Fig.~\ref{fig:bar-embs} is an illustration of the method introduced above.
\begin{figure}
    \centering
    \includegraphics[width=0.7\textwidth]{figures/bar-emb-enc-revised.png}
    \caption{Illustration of our method for extracting the bar-level embeddings to be used by our conditional Transformer decoders.}
    \label{fig:bar-embs}
\end{figure}

\textbf{Dataset and Data Representation.}
In our experiments, we consider modeling long sequences of symbolic music with up to 32 bars per sequence.
Specifically, our data come from the \textit{LPD-17-cleansed} dataset \citep{bmusegan}, a pop music MIDI dataset containing $>$20K songs with at most 17 instrumental tracks (e.g., piano, strings, and drums) per song.
We take the subset of 10,626 songs in which the piano is playing at least half of the time, and truncate each song to the beginning 32 bars, resulting in a dataset with 650 hours of music. These songs are all with time signature  4/4 (i.e., four beats per bar).
We reserve 4\% of the songs (i.e., 425 songs) as the validation set for objective evaluation, and 1\% (i.e., 107 songs) as the test set for subjective study.

Following REMI \citep{huang2020pop},
we represent music in the form of event tokens, encompassing
\textit{note}-related tokens such as \textsc{Pitch-[trk]}, \textsc{Duration-[trk]}, and \textsc{Velocity-[trk]}, which represent a note played by a specific track (i.e., `\textsc{[trk]}'); and
\textit{metric}-related tokens such as \textsc{Bar}, \textsc{Sub-beat}, and \textsc{Tempo}.
The \textsc{Sub-beat} tokens\footnote{equivalent to \textsc{Position} tokens in REMI} divide a bar into 32 possible locations for the onset (i.e., starting time) of notes, laying an explicit time grid for the Transformer to process the  progression of time.
The \textsc{Bar} token marks the beginning of a new bar, making it easy to associate different bar-level conditions $c_k$ to subsequences belonging to different bars (i.e., different $I_k$'s).
Brief descriptions for all types of tokens in our representation is provided in Table \ref{tab:event-token}.

\begin{table}
\centering
\caption{The vocabulary used to represent songs in \textit{LPD-17-cleansed} dataset.}\label{tab:event-token}
\begin{tabular}{l r r}
\toprule
Event type &  Description  & \# unique tokens \\
\midrule
\textsc{Bar}&  beginning of a new bar & 1 \\
\textsc{Sub-beat}&  position in a bar, in 32nd note steps (\musThirtySecond) & 32 \\
\textsc{Tempo}&  32$\sim$220 bpm, with steps of 3 bpm & 64 \\
\textsc{Pitch}$^*$&  MIDI note numbers (pitch) 0$\sim$127 & 1,757 \\
\textsc{Velocity}$^*$&  MIDI velocities 3$\sim$127 & 544 \\
\textsc{Duration}$^*$&  multiples (1$\sim$64 times) of \musThirtySecond  & 1,042 \\
\midrule
\textbf{All events} & --- & \textbf{3,440} \\
\bottomrule
\multicolumn{3}{l}{\footnotesize{$^*$: unique for each of the 17 tracks}} \\
\end{tabular}
\end{table}

\subsection{Evaluation}\label{sec_eva_obj1}
Besides examining the training and validation losses, for further evaluation, we let our segment-level conditional Transformers generate multi-track MIDI music of 32 bars long, with the bar-level conditions (i.e., $\{\bm{c}_k\}_{k=1}^{32}$) extracted from songs in the validation set (for objective evaluation), or test set (for subjective study).
Therefore, in essence, the models generate \textit{re-creations}, or \textit{covers}, of existing music.
In the following paragraphs, we define and elaborate on the objective and subjective metrics that assess re-creations of music.

\textbf{Objective Metrics for Re-creation Fidelity.}
Three bar-level metrics and a sequence-level metric are defined to quantitatively evaluate whether the re-creations of our models are similar to the original songs from which the segment-level conditions are extracted.
\begin{itemize}[leftmargin=*]
    \item \textbf{Chroma similarity}, or $\mathit{sim}_\text{chr}$, measures the closeness of two bars in tone via:
    \begin{equation}
    \mathit{sim}_\text{chr}(\bm{r}^\text{a}, \bm{r}^\text{b}) =  100~ \frac{\langle \bm{r}^\text{a}, \bm{r}^\text{b} \rangle}{||\bm{r}^\text{a}||\,||\bm{r}^\text{b}||},
    \end{equation}

    where $\langle \cdot,\cdot\rangle$ denotes dot-product, and $\bm{r}\in \mathbb{Z}^{12}$ is the \emph{chroma vector}  \citep{fujishima99} representing the number of onsets for each of the 12 pitch classes (i.e., \textsc{C}, \textsc{C\#}, \dots, \textsc{B}) within a bar, counted across octaves and tracks, with the drums track ignored.
    One of the bars in the pair $(\text{a}, \text{b})$ is from the original song, while the other is from the re-creation.

\item \textbf{Grooving similarity}, or $\mathit{sim}_\text{grv}$,  examines the rhythmic resemblance between two bars with the same formulation as the chroma similarity, measured instead on the \textit{grooving vectors} $\bm{g} \in \mathbb{Z}^{32}$ recording the number of note onsets occurring at each of the 32  sub-beats in a bar, counted across tracks \citep{dixonEtAl04ismir}.

\item \textbf{Instrumentation similarity}, or $\mathit{sim}_\text{ins}$, quantifies whether two bars use similar instruments via:
\begin{equation}
    \mathit{sim}_\text{ins}(\bm{b}^\text{a}, \bm{b}^\text{b}) = 100~ (1 - \frac{1}{17} \sum_{i=1}^{17} \text{XOR}(b_i^\text{a}, b_i^\text{b}) ),
\end{equation}
where
$\bm{b} \in \{0, 1\}^{17}$ is a binary vector indicating
the presence of a track in a bar, i.e., at least one \textsc{Pitch-[trk]} for that \textsc{[trk]} occurs within the bar.

\item \textbf{Self-similarity matrix distance}, or $\mathit{dist_\text{SSM}}$, measures whether two 32-bar sequences have similar overall structure by comparing the mean absolute distance between the \textit{self-similarity matrices} (SSM) \citep{foote1999visualizing} of the two sequences, i.e.,
\begin{equation}
    100\,||S^\text{a}-S^\text{b}||_1,
\end{equation}
where $S \in \mathbb{R}^{N \times N}$ is the SSM.
In doing so, we firstly synthesize each 32-bar sequence into audio with a synthesizer. Then, we use constant-Q transform to extract acoustic chroma vectors indicating the relative strength of each pitch class for each half beat in an audio piece, and then calculate the cosine similarity among such \textit{beat-synchronized}  features; hence, $s_{ij} \in [0, 1]$ for half beats $i$ and $j$ within the same music piece. Since each bar in 4/4 signature has 8 half-beats, we have $N=256$ for our data.
An SSM depicts the self-repetition within a piece.
Unlike the previous metrics, the value of $\mathit{dist_\text{SSM}}$ is the closer to zero the better.
We note that the values of all metrics above, i.e., $\mathit{sim}_\text{chr}, \mathit{sim}_\text{grv}$, $\mathit{sim}_\text{ins}$, and $\mathit{dist_\text{SSM}}$, are all in $[0,100]$.
\end{itemize}

\textbf{Objective Metrics for Re-creation Quality.}
It is worth mentioning that objective evaluation on the quality of generated music remains an open issue \citep{wu2020jazz}.
Nevertheless, we adopt here a recent idea that measures the quality of music by comparing the \textit{distributions} of the generated content and the training data in feature spaces, using KL divergence \citep{yang20evaluation}.

While we employ $\mathit{sim}_\text{chr}$, $\mathit{sim}_\text{grv}$, $\mathit{sim}_\text{ins}$ to measure the similarity between a generated bar and the corresponding reference bar in Sec.~\ref{sec_eva_obj1},
here we instead use them to calculate either the  (self-)similarity among the bars in a \emph{gen}erated 32-bar sequence, or that among the bars in a \emph{real} training sequence. For each sequence, we have $32\cdot(32-1)/2$ such pairs, leading to a distribution.
Since the values of $\mathit{sim}_{\text{chr}}$ and $\mathit{sim}_{\text{grv}}$ are continuous, we discretize their distributions into 50 bins with evenly-spaced boundaries $[0, 2, \dots, 100]$. The KL divergence is calculated on the probability mass functions $p_\text{real}, p_\text{gen} \in \mathbb{R}^{50}$, or $\mathbb{R}^{18}$ for the distribution in terms of $\mathit{sim}_{\text{ins}}$,  and is defined as $\mathit{KL}\, (p_\text{real}\, || \, p_\text{gen})$.
We assume that the values are the closer to zero the better.

\textbf{Subjective Study.}
We also evaluate our three conditional models, namely, \textbf{pre-attention}, \textbf{in-attention}, and \textbf{post-attention}, through a listening test, aiming to find out which one composes the best music as judged by human ears.
The \textbf{unconditional} and \textbf{memory} models are excluded here, to avoid unfairness arising from training on objectives (Eq.~(\ref{eq:uncond}) or (\ref{eq:cond_local_memory})) that differ from our models (Eq.~(\ref{eq:cond_local_ours})).

Test-takers are asked to listen to three independent sets of music randomly drawn from a pool of 21 sets.
Each set contains a synthesized audio of a unique, human-composed song (i.e., the reference song) sampled from the test set; and, three folders that house the compositions of the three conditional models. The order of the three models is randomized for each set and kept secret to test-takers. Within each folder are the synthesized audios of five different compositions of a single model, given the same reference song's bar-level embeddings, i.e., $\{\bm{c}_k\}_{k=1}^{32}$,  as conditions.
After listening to the five pieces in each folder, the test-takers were asked to rate the model, on a five-point Likert scale (the higher rating the better), in the following four aspects:
\begin{itemize}[leftmargin=*]
    \item \textbf{Fidelity (F):} Do the (five) pieces sound like the reference song in general?
    \item \textbf{Humanness (H):} Do they sound like human artwork?
    \item \textbf{Diversity (D):} Do the pieces differ from one another? (This is the reason why we include five compositions for the same condition reference in a folder.)
    \item \textbf{Quality (Q):} Do the pieces sound pleasant in general? We consider this metric as the most important one.
\end{itemize}
The rating was done per folder.
Therefore, from a single test-taker, we would receive three separate separate sets of evaluation scores for each model.

\subsection{Results and Discussion}
\begin{figure}
    \centering
    \begin{subfigure}[b]{0.4\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/training_prgs_ijcai_w_memory.pdf}
        \caption{\# epochs vs. training loss}
        \label{subfig:train-prgs}
    \end{subfigure}
    \hfill
    \centering
    \begin{subfigure}[b]{0.413\linewidth}
        \centering
        \includegraphics[width=\textwidth]{figures/test_valloss_plot_ijcai_w_memory.pdf}
        \caption{Training loss vs. val. loss}
        \label{subfig:valloss}
    \end{subfigure}
    \caption{Training dynamics, on \textit{LPD~-17-cleansed} dataset, of segment-level conditional Transformers  (best viewed in color).}
    \label{fig:train-prgs}

\end{figure}

We begin with presenting the training dynamics of our model (see Fig.~\ref{fig:train-prgs}).
From Fig.~\ref{subfig:train-prgs}, we can see that the model whose loss drops the fastest, \textbf{in-attention}, achieves 0.41 training NLL after 16 epochs, while the \textbf{unconditional} and \textbf{memory} \citep{li2020optimus} models still have a loss of around 0.65 and 0.60 respectively.
This shows
that our segment-level conditioning mechanism
effectively speeds up model training.
We then take also the validation loss into consideration.  Fig.~\ref{subfig:valloss}
shows that our models, especially \textbf{pre-} and \textbf{in-attention}, outperform both \textbf{unconditional} and \textbf{memory} at comparable training NLL.

Table \ref{tab:obj-eval} displays the objective evaluation results, using metrics defined in Sec.~\ref{sec_eva_obj1}.
The scores are calculated on 425 re-creations, each using the bar-level conditions from a unique song in the validation set.
For comparison, we let the \textbf{unconditional} baseline randomly generate 400 32-bar pieces from scratch,
and further include a \textit{random} baseline, whose scores are computed over 400 random pairs of pieces drawn from the training set.

\begin{table*}
\small
\centering
\caption{Objective evaluation results of segment-level conditional Transformers, on re-creating multi-track music ($\downarrow$\,/\,$\uparrow$: the lower/higher the score the better). }\label{tab:obj-eval}
\begin{tabular}{l  llll |ccc}
\toprule
\multirow{2}{*}{\textit{Model}} &
\multicolumn{4}{c|}{\textbf{Fidelity}} &
\multicolumn{3}{c}{\textbf{Quality}}
\\

& $\mathit{sim}_{\text{chr}}\uparrow$ & $\mathit{sim}_{\text{grv}}\uparrow$ & $\mathit{sim}_{\text{ins}}\uparrow$ & $\mathit{dist}_{\text{SSM}}\downarrow$
& $\mathit{KL}_\text{chr}\downarrow$ & $\mathit{KL}_\text{grv}\downarrow$ & $\mathit{KL}_\text{ins}\downarrow$  \\
\midrule
\textit{random}    & 34.6 & 40.5 & 80.4 & 13.5 & --- & --- & ---  \\
\textbf{unconditional}    & --- & --- & --- & --- & .047 & .228 & .219  \\
\hline
\textbf{memory} \citep{li2020optimus}   & 60.7 & 62.7 & 90.7 & 10.3 & .021 & .061 & \textbf{.006}  \\
\textbf{pre-attention}   & 94.5 & 92.9 & 93.0 & 7.10 & .006 & .009 & .034  \\
\textbf{in-attention}  & \textbf{96.3}$^{***}$ & \textbf{95.7}$^{***}$ & \textbf{97.0}$^{***}$ & \textbf{5.83}$^{***}$  & \textbf{.002} & \textbf{.005} & .021 \\
\textbf{post-attention}   & 92.4 & 85.7 & 95.0 & 8.27 & .016 & .054 & .056  \\
\bottomrule
\multicolumn{8}{l}{\scriptsize{$^{***}$: leads all other models with $p < .001$}} \\
\end{tabular}
\end{table*}

Focusing on the \textbf{Fidelity} metrics first, all of our conditional models score quite high on bar-level $\mathit{sim}_\text{chr}, \mathit{sim}_\text{grv}$, and $\mathit{sim}_\text{ins}$, with \textbf{in-attention} consistently outperforming the rest.
\textbf{In-attention} also attains the lowest average  $\mathit{dist}_{\text{SSM}}$,
demonstrating effective and tight control.
Moreover, it is worth noting that the \textbf{memory} model sits right in between ours and the \textit{random} baseline in all metrics, offering limited conditioning ability.

Next, we shift attention to the \textbf{Quality} metrics,
where $\mathit{KL}_\text{chr}$, $\mathit{KL}_\text{grv}$, and $\mathit{KL}_\text{ins}$ represent the KL divergence calculated on the corresponding distributions.
The conditional models clearly outperform the unconditional baseline, suggesting that the Transformer decoders learn better when provided with conditions.
Furthermore, once again, \textbf{in-attention} comes out on top, followed by \textbf{pre-}, \textbf{post-attention}, and then \textbf{memory}, except for metric $\mathit{KL}_\text{ins}$, in which \textbf{memory} performs the best.

\begin{table}
\centering
\caption{Subjective evaluation results of our segment-level conditional Transformer decoders, on re-creating multi-track music (\textbf{F:} Fidelity, \textbf{H:} Humanness, \textbf{D:} Diversity, \textbf{Q:} Quality; all on a five-point scale). \# of test-takers $=17$.}\label{tab:subj-study}
\begin{tabular}{l l l l l}
\toprule
\textit{Model} & \multicolumn{1}{c}{\textbf{F}} &  \multicolumn{1}{c}{\textbf{H}} &  \multicolumn{1}{c}{\textbf{D}} &  \multicolumn{1}{c}{\textbf{Q}} \\
\midrule
\textbf{pre-attention} & 3.43 & 2.76 & 3.08 & 3.12 \\
\textbf{in-attention} & \textbf{3.84}$^{**}$ & \textbf{2.92} & 2.41 & \textbf{3.37}$^{*}$  \\
\textbf{post-attention} & 2.94 & 2.43 & \textbf{3.16}& 2.75 \\
\bottomrule
\multicolumn{5}{l}{\scriptsize{$^{**}$: leads all others with $p < .01$; $^{*}$: with $p < .05$}} \\
\end{tabular}
\end{table}

Table \ref{tab:subj-study} presents the results obtained from our listening test, which include the mean opinion scores (MOS) of 17 anonymous test-takers recruited in our subjective study.
Seven individuals among the test-takers are regarded as \textit{pros} for a self-reported score of 4/5 or 5/5 in the level of prior experience in music composition or performance.
The results largely resonate with those observed on objective metrics. The \textbf{in-attention} mechanism most effectively harnesses the Transformer decoder, scoring the highest in aspects \textbf{F}, \textbf{Q}, and \textbf{H}, but the lowest in \textbf{D}. The behavior of \textbf{post-attention} is the exact opposite, suggesting a trade-off between effective control and diversity.

Paired \textit{Student's $t$-tests} also suggest that, in terms of exerting tight control over the sequences to be generated, the advantage that \textbf{in-attention} mechanism possesses over others is statistically significant.
More specifically, we obtain $p<.001$ on all of the objective \textbf{Fidelity} metrics, and $p < .01$ and $p < .05$ on \textbf{Fidelity (F)} and \textbf{Quality (Q)} scores, respectively, from the subjective study.

From the results presented above, we surmise that the worse performance in general of the \textbf{memory} baseline compared to ours could be due to the advantages of using Eq.~(\ref{eq:cond_local_ours}) over Eq.~(\ref{eq:cond_local_memory}).
Equation (\ref{eq:cond_local_ours}) informs the Transformer decoder of when exactly to exploit each of the conditions, and eliminates the bias caused by the positional encoding fed to Transformers (refer to Eq.~(\ref{eq:pos-encode})), i.e., later tokens in the sequence are made more dissimilar to the conditions provided in the beginning of the sequence, thereby undermining the conditions' effect in the attention process.
Therefore, \textbf{memory} works well only for conditions that remain relatively unchanged throughout a sequence, such as the instrumentation of a song.
Among the three proposed conditioning methods, we conjecture that \textbf{post-attention} works the worst because the conditions do not participate in the attention process. Therefore, the model has less chance to integrate information from them. Moreover, the advantage of \textbf{in-attention} over \textbf{pre-attention} is reasonably due to that, in \textbf{in-attention}, the conditions are fed to all attention layers instead of just once before the first attention layer.
With the comprehensive evaluation conducted, we have sufficient evidence indicating that \textbf{in-attention} works the best in exerting control over Transformer decoders with segment-level, time-varying conditions.

\section{MuseMorphose: Generating Music with Time-Varying User Controls}
In the last section, we have developed the \textbf{in-attention} technique, which can exert firm control over a Transformer decoder's generation with time-varying, predetermined conditioning vectors.
However, that technique alone only enables Transformers to compose re-creations of music
at random;
no freedom has been given to users to interact with such a system to affect the music it generates as one wishes,
thereby limiting its practical value.
This section is aimed at alleviating such a limitation.
We bridge the \textbf{in-attention} Transformer decoder and a jointly learned bar-wise Transformer encoder,
tasked with learning the bar-level latent conditions, with the variational autoencoder (VAE) training objective;
and, introduce \textit{attribute embeddings} \citep{fu2018style, kawai2020attributes}, also working at the bar level, to the decoder to discourage the latent conditions from storing attribute-related information,
and realize fine-grained user controls over sequence generation.
Experiments show that our resulting model, \textit{\textbf{MuseMorphose}}, successfully allows users to harness two easily perceptible musical attributes, \textit{rhythmic intensity} and \textit{polyphony}, while maintaining fluency and high fidelity to the input reference song.
Such combination of abilities is shown unattainable by prior art \citep{brunner2018midi, kawai2020attributes}.

\subsection{Problem Formulation}
For clarity, we model our user-controllable conditional generation task as a style transfer problem without paired data \citep{fu2018style}.
This problem, in its most basic form, deals with an input instance $X$, such as a sentence or a musical excerpt, and an attribute $a$ intrinsic to $X$.
The attribute $a \in \{1,\dots,C\}$ can be either a nominal (e.g., composer style), or an ordinal (e.g., note density) variable with $C$ categories.
As users, we would like to have a model that takes a tuple:
\begin{equation}
    (X, \tilde{a}),
\end{equation}
where $\tilde{a}$ is the user-specified attribute value that may be different from the original $a$, and outputs a style-transferred instance $\tilde{X}$.
A desirable outcome is that the generated $\tilde{X}$ bears the specified attribute value $\tilde{a}$ (i.e., achieving effective \textit{control}), and preserves the content other than the attribute (i.e., exhibiting high \textit{fidelity}).
Being able to generate \textit{diverse} versions of  $\tilde{X}$'s given the same $\tilde{a}$ or to ensure the \textit{fluency} of generations, though not central to this problem, are also preferred characteristics of such models.
The aforementioned framework has been widely adopted by past works in both text \citep{fu2018style, yang2018unsupervised, john2019disentangled} and music \citep{brunner2018midi} domains.

A natural extension to the problem above is to augment the model such that it can process multiple attributes at a time, i.e., taking inputs in the form:
\begin{equation}
    (X, \tilde{a}^1, \tilde{a}^2,\dots,\tilde{a}^J),
\end{equation}
to produce $\tilde{X}$, where ${a}^j \text{, for} \; j\in\{1, J\}$, are the $J$ categorical attributes being considered.
In this case, users have the freedom to alter single or multiple attribute values. Hence, preferably, changing one attribute to $\tilde{a}^j$ should not affect an unaltered attribute $a^{j'}$.
This multi-attribute variant has also been studied in \citep{lample2018multiple, kawai2020attributes}.

In some scenarios, the input sequence $X$ is long and can be partitioned into meaningful continuous segments, e.g., sentences in text or bars in music, through:
\begin{equation}
\begin{split}
    X &= \{X_1, \dots, X_K\}, \text{where} \\
    X_k &= \{x_t \, \vert \, t \in I_k  \}, \; I_k \subset \mathbb{N} \quad \text{for}~ 1\leq k \leq K; \\
    \bigcup_{k=1}^K I_k &= [1, T] \quad \text{and} \quad
    I_k \cap I_{k'} = \varnothing \; \text{for} \; k \neq k' \,, \label{eq:style-trans-part}
\end{split}
\end{equation}
where $T$ is the length of $X$, and $I_1,\dots,I_K$ constitute a partition of $X$ (cf. the paragraph concerning the formulation of Eq.~(\ref{eq:cond_local_ours}) in Sec.~\ref{sec:form-ch3}).
For such cases, it could be a privilege if we can control \textit{each and every} segment individually, since this allows, for example, users to make a musical section a lot more intense, and another much calmer, to achieve dramatic contrast.
Therefore, we formulate here the style transfer problem with \textit{multiple}, \textit{time-varying} attributes.
A model tackling this problem accepts:
\begin{equation}
    (X,
    \{\tilde{a}_k^1, \tilde{a}_k^2,\dots,\tilde{a}_k^J\}_{k=1}^K),
\end{equation}
and then generate a long, style-transferred $\tilde{X}$, where $a_k^1, a_k^2,\dots,a_k^J$ are concerned specifically with the segment $X_k$.
Though Transformers \citep{vaswani2017attention, radford2019language} are impressive in sequence generation, no prior work, to our knowledge, has addressed the problem above with them, reasonably because there has not been a well-known segment-level conditioning mechanism for Transformers.
Thus, given our developed \textbf{in-attention} method, we are indeed in a privileged position to approach this task.

\subsection{Method}\label{sec:attr-comp}
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/muse_optimus_archi.pdf}
    \caption{Architecture of our \textit{MuseMorphose}, a Transformer-based VAE, model for controllable conditional music generation.}
    \label{fig:muse-optimus}
\end{figure}

In what follows, we describe how we pair an \textbf{in-attention} Transformer decoder with a bar-wise Transformer encoder, as well as attribute embeddings, to achieve fine-grained controllable music generation through optimizing the VAE objective.
The musical attributes considered here are: \textit{rhythmic intensity} ($a^\text{rhym}$) and \textit{polyphony} ($a^\text{poly}$; i.e., harmonic fullness),
whose calculation methods will also be detailed.
After that, we present the implementation details and the pop piano dataset, \textit{REMI-pop-1.7k}, used for this task.

\textbf{Model Architecture.}
Figure \ref{fig:muse-optimus} is an illustration of our \textit{MuseMorphose} model,
which consists of a Transformer encoder operating at bar level, a Transformer decoder that accepts segment-level conditions through the \textbf{in-attention} mechanism, and a KL divergence-regularized latent space for the representation of \textit{musical bars} between them.
In MuseMorphose, the input music is partitioned as $X = \{X_1,\dots,X_K\}$, where $X_k$ houses the $k^\text{th}$ bar of the music.
The encoder works on the musical bars, $X_1,\dots,X_K$, in parallel, while the decoder sees the entire piece $X$ at once.
The bar-level attributes, $a_k^\text{rhym}$ and $a_k^\text{poly}$, each consisting of 8 ordinal classes, are first transformed into embedding vectors, $\bm{a}_k^{\text{rhym}}, \bm{a}_k^{\text{poly}} \in \mathbb{R}^{d_{\bm{a}}}$, before entering the decoder's attention layers through \textbf{in-attention}.
On the whole, the operations taking place inside MuseMorphose can be summarized as follows:
\begin{equation}
\begin{split}
    \bm{z}_k &= \textbf{enc}(X_k) \quad \text{for}~ 1\leq k \leq K; \\
    \bm{c}_k &= \text{concat}([\bm{z}_k; \bm{a}_k^{\text{rhym}}; \bm{a}_k^{\text{poly}}]); \\
    \bm{y}_{t} &= \textbf{dec}(x_{<t}; \bm{c}_k), \; t \in I_k \quad \text{for}~ 1\leq k \leq K,
\end{split}
\end{equation}
where $\bm{z}_k \in \mathbb{R}^{d_{\bm{z}}}$ is the latent condition for the $k^\text{th}$ bar;
$\bm{a}_k^{\text{rhym}}, \bm{a}_k^{\text{poly}}$ are the bar's attribute embeddings,
$I_k$ stores the timestep indices (see Eq.~(\ref{eq:style-trans-part})) of the bar; and,
$\bm{y}_{t}$ is the predicted probability distribution for $x_t$.
Note that though the input is split into bars on the encoder side, the decoder still deals with the entire sequence, thanks to both {\textbf{in-attention}} segment-level conditioning and $\bigcup_{k=1}^K I_k = [1, T]$.
This asymmetric architecture is likely advantageous due to that it enables fine-grained conditions, or controls, to be given at the bar level, and also promotes the coherence of long generations.

Since we adopt the VAE framework here, we now elaborate more on how we construct the latent space for $\bm{z}_k$'s.
First, we treat the encoder's attention output at the first timestep (corresponding to the \textsc{Bar} token), i.e., $\bm{h}^{L_{\textbf{enc}}}_{k, 1}$, as the contextualized representation of the bar.\footnote{Since after $L_{\textbf{enc}}$ attention layers, $\bm{h}^{L_{\textbf{enc}}}_{k, 1}$ should have gathered information from the entire bar.}
Then, following the conventional VAE setting \citep{kingma2014auto}, it is projected by two separate learnable weights $W_{\bm{\mu}}, W_{\bm{\sigma}}$, to the mean and std vectors:
\begin{equation}
    \bm{\mu}_k = {\bm{h}^{L_{\textbf{enc}}}_{k, 1}}^\top W_{\bm{\mu}} \quad \quad
    \bm{\sigma}_k = {\bm{h}^{L_{\textbf{enc}}}_{k, 1}}^\top W_{\bm{\sigma}}\,,
\end{equation}
where  $W_{\bm{\mu}}, W_{\bm{\sigma}} \in \mathbb{R}^{d \times d_{\bm{z}}}$.
Afterwards, we may sample the latent condition to be fed to the decoder from the isotropic gaussian defined by $\bm{\mu}_k$ and $\bm{\sigma}_k$:
\begin{equation}
   \bm{z}_k \sim \mathcal{N}(\bm{\mu}_k, \text{diag}(\bm{\sigma}^2_k)).
\end{equation}
For simplicity, the prior of latent conditions, i.e., $p(\bm{z}_k)$, is set to the standard gaussian $\mathcal{N}(\bm{0}, I_{d_{\bm{z}}})$, which has been typically used for VAEs.

It is worthwhile to mention that the subsequent copy-paste of the $\bm{z}_k$'s, done by the \textbf{in-attention} decoder, to every attention layer and every timestep in $I_k$ coincides with the design of \textit{Skip-VAE} \citep{dieng2019avoiding},
which provably mitigates the posterior collapse problem, i.e., the case where the decoder completely ignores $\bm{z}_k$'s and degenerates into autoregressive sequence modeling.
Also, our work makes a notable advance over \textit{Optimus} \citep{li2020optimus} in that we equip Transformer-based VAEs with the capability to model long sequences (with length in the order of ${\geq}10^3$), under fine-grained changing conditions from the latent vector (i.e., $\bm{z}_k$)  and user controls (i.e., $\tilde{a}_k^\text{rhym}$ and $\tilde{a}_k^\text{poly}$).

\textbf{Training Objective.}
MuseMorphose is trained with the $\beta$-VAE objective \citep{higgins2017beta} with free bits \citep{kingma2016improving}. It minimizes the loss, $\mathcal{L}_{\text{MuseMorphose}}$, which is written as:
\begin{equation}
\begin{split}
    \mathcal{L}_{\text{MuseMorphose}} &= \mathcal{L}_{\text{NLL}} + \beta \mathcal{L}_{\text{KL}} \,, \quad \text{where} \\
    \mathcal{L}_{\text{NLL}} &= -\log p_{\textbf{dec}}(X \, \vert \, \bm{c}_1,\dots,\bm{c}_K)\,; \\
    \mathcal{L}_{\text{KL}} &= \frac{1}{K} \sum_{k=1}^K  \sum_{i=1}^{d_{\bm{z}}} \max(\lambda,   D_\text{KL}(q_{\textbf{enc}}(z_{k, i} \vert X_k) \, \| \, p(z_{k, i})))\,.\label{eq:mopt-loss}
\end{split}
\end{equation}
The first term, $\mathcal{L}_{\text{NLL}}$, is the conditional negative log-likelihood (NLL) for the decoder to generate input $X$ given the conditions $\bm{c}_1,\dots,\bm{c}_K$,\footnote{They are concatenations of latent condition, $\bm{z}_k$, and attribute embeddings, $\bm{a}_k^{\text{rhym}}$ and $\bm{a}_k^{\text{poly}}$.}
hence referred to as \textit{reconstruction NLL} hereafter.
The second term, $\mathcal{L}_{\text{KL}}$, is the KL divergence between the posterior distributions of $\bm{z_k}$'s estimated by the encoder (i.e., $q_{\textbf{enc}}(\bm{z}_{k} \vert X_k)$) and the prior $p(\bm{z}_k)$.
$\beta$ and $\lambda$ are simply hyperparameters to be tuned.
We refer readers to Fig.~\ref{fig:muse-optimus} for a big picture of where the two loss terms act on the model.

Previous works that employ autoencoders for style transfer tasks \citep{john2019disentangled, kawai2020attributes, dai2019style} often suggested adding adversarial losses \citep{goodfellow2014generative} on the latent space, so as to discourage it from storing style-related, or attribute-related, information.
However, a potential downside of this practice is that it introduces additional complications to the training of our Transformer-based network, which is already very complex itself.
We instead demonstrate that by using suitable $\beta$ and $\lambda$ to control the size of latent information bottleneck, both strong style transfer and good content preservation of input $X$ can be accomplished without auxiliary training losses.

\textbf{Calculation of Musical Attributes.}
The attributes chosen for our task, \textit{rhythmic intensity} ($a^\text{rhym}$) and \textit{polyphony} ($a^\text{poly}$), are able to be perceived easily by people without intensive training in music. Meanwhile,
they are also important determining factors of musical emotion.
To obtain the ordinal classes $a^\text{rhym}$ and $a^\text{poly}$ of each bar, we first compute the raw scores $s^\text{rhym}$ and $s^\text{poly}$.
\begin{itemize}[leftmargin=*]
    \item \textbf{Rhythmic intensity score}, or $s^\text{rhym}$, simply measures the percentange of sub-beats with at least one note onset, that is:
    \begin{equation}
        s^\text{rhym} = \frac{1}{B} \sum_{b=1}^B \bm{1}( n_{\text{onset}, b} \geq 1 ) \,,
    \end{equation}
    where $B$ is the number of sub-beats in a bar and $\bm{1}(\cdot)$ is the indicator function.
    \item \textbf{Polyphony score}, or $s^\text{poly}$, is bit more implicit, and is defined as the average number of notes being \textit{hit} (onset) or \textit{held} (not yet released) in a sub-beat, i.e.,
    \begin{equation}
        s^\text{poly} = \frac{1}{B} \sum_{b=1}^B ( n_{\text{onset}, b} + n_{\text{hold}, b} ) \,.
    \end{equation}
    This makes sense since if there are more notes pressed simultaneously, the music would feel \textit{harmonically fuller}.
\end{itemize}
After we collect all bar-wise raw scores from the dataset, we divide them into 8 bins with roughly equally many samples, resulting in the 8 classes of $a^\text{rhym}$ and $a^\text{poly}$.
For example, in our implementation, the cut-off $s^\text{rhym}$'s between the classes of $a^\text{rhym}$ are: $[.250, .297, .344, .391, .453, .500, .563]$, while the cut-offs for $a^\text{poly}$ are: $[2.72, 3.05, 3.36, 3.86, 4.58, 5.31, 6.12]$.

\begin{table}
\centering
\caption{Main characteristics of our \textit{MuseMorphose} model.}\label{tab:model-attr-muse-optimus}
\begin{tabular}{l l r}
\toprule
Attr. & Description & Value \\
\midrule
$T$ & target sequence length & 1,280 \\
$L$ & \# self-attention layers & 24 \\
$L_\textbf{enc}$ & \# encoder self-attention layers & 12 \\
$L_\textbf{dec}$ & \# decoder self-attention layers & 12 \\
$n_{\text{head}}$ & \# self-attention heads & 8 \\
$d_{\text{e}}$ & token embedding dimension & 512 \\
$d$ & hidden state dimension & 512 \\
$d_{\text{ff}}$ & feed-forward dimension & 2,048 \\
$d_{\bm{z}}$ & latent condition dimension & 128 \\
$d_{\bm{a}}$ & attribute embedding dimension (each) & 64 \\
\hline
\# params & --- & 79.4 M\\
\bottomrule
\end{tabular}
\end{table}

\textbf{Implementation and Training Details.}
Table \ref{tab:model-attr-muse-optimus} lists the important architectural details of our MuseMorphose model.
For better training outcomes, we introduce both \textit{cyclical KL annealing} \citep{fu2019cyclical} and \textit{free bits} \citep{kingma2016improving} (both have been introduced in Sec.~\ref{sec:vae-trick}) to our training objective (see Eq.~(\ref{eq:mopt-loss})).
After some trial and error, we set $\beta_{\text{max}} = 1$, and anneal $\beta$, i.e., the weight on the $\mathcal{L}_{\text{KL}}$ term, in cycles of 5,000 training steps.
The free bit for each dimension in $\bm{z}_k$ is set to $\lambda = 0.25\,$.
This setup is referred to as \textit{preferred settings} hereafter, for they strike a good balance between content preservation (i.e., fidelity) and diversity.\footnote{Note that this is however judged by our ears.}
We also train MuseMorphose with the pure \textit{AE objective} (i.e., $\beta=0$, constant), and \textit{VAE objective} (i.e., $\beta=1$, constant; $\lambda=0$), to examine how the model behaves on the two extremes.
However, across all three settings, we do not add $\mathcal{L}_{\text{KL}}$ in the first 10,000 steps, to make it easier for the decoder to exploit the latent space in the beginning.

Furthermore, we set $K=16$ during training, which means that we feed to the model a random crop of 16-bar excerpt, truncated to maximum length $T=1{,}280$, from each musical piece in every epoch.
This is done mainly to save memory and speed up training. During inference, the model can still generate music over 16 bars long with a sliding window-like mechanism.
Data augmentation on pitches is also employed.
The key of each sample is transposed randomly in the range of $\pm6$ half notes in every epoch.

The models are trained with Adam optimizer \citep{kingma2014adam} and teacher forcing.
We use linear warm-up to increase the learning rate from $0$ to $1{e}{-}4$ in the first 200 steps, followed by a 200,000-step cosine decay down to $5{e}{-}6$.
Model parameters are initialized from the gaussian $\mathcal{N}(0, 0.01^2)$.
Using a batch size of 4, we can fit our MuseMorphose into a single NVIDIA Tesla V100 GPU with 32GB memory.
For all three loss variants (\textit{preferred settings}, \textit{AE objective}, \textit{VAE objective}) alike, the training converges in less than 2 full days. At inference time, we perform \textit{nucleus sampling} \citep{holtzman2019curious} to sample from the output distribution at each timestep, using a softmax temperature $\tau=1.2$ and truncating the distribution at cumulative probability $p=0.9$.

Due to the affinities in training objective and application, we implement \textit{MIDI-VAE} \citep{brunner2018midi} and \textit{Attr-Aware VAE} \citep{kawai2020attributes} as baselines.
Their RNN-based decoders, however, operates only on single bars (i.e., $X_k$'s) of music. Therefore, during inference, each bar is generated independently and then concatenated to form the full piece.
We deem this acceptable since the bar-level latent conditions $\bm{z}_k$'s should store sufficient information to link the piece together.
We follow their specifications closely, add all auxiliary losses as required, and increase their number of trainable parameters by enlarging the RNN hidden state for fair comparison with MuseMorphose.
The resulting number of parameters in our implementations of \textit{MIDI-VAE} and \textit{Attr-Aware VAE}, respectively, are 58.2 million and 60.0 million.
For fair comparison, in our implementation the two baseline models are trained under the \textit{preferred settings} for MuseMorphose, with teacher forcing as well.

\begin{table}
\centering
\caption{The vocabulary used to represent songs in \textit{REMI-pop-1.7K} dataset.}\label{tab:event-token-remi}
\begin{tabular}{l r r}
\toprule
Event type &  Description  & \# unique tokens \\
\midrule
\textsc{Bar}&  beginning of a new bar & 1 \\
\textsc{Sub-beat}&  position in a bar, in 16th note steps (\musSixteenth) & 16 \\
\textsc{Tempo}&  32$\sim$224 bpm, with steps of 3$\sim$6 bpm & 54 \\
\textsc{Pitch}&  MIDI note numbers (pitch) 22$\sim$107 & 86 \\
\textsc{Velocity}&  MIDI velocities 40$\sim$86 & 24 \\
\textsc{Duration}&  multiples (1$\sim$16 times) of \musSixteenth  & 16 \\
\textsc{Chord}&  chord markings (root \& quality, e.g., \textsc{Cmaj\footnotesize{7}})  & 133 \\
\midrule
\textbf{All events} & --- & \textbf{330} \\
\bottomrule
\end{tabular}
\end{table}

\textbf{Dataset and Data Representation.}
For this task, we consider generating expressive pop piano performances.
The pop piano MIDI dataset used is \textit{REMI-pop-1.7K}, which was released in \citep{hsiao21aaai}.
According to \citet{hsiao21aaai}, the piano performances in the dataset are originally collected from the Internet
in the MP3 (audio) format. They further employed \textit{Onsets and Frames} piano transcription \citep{hawthorne2018onsets}, \texttt{madmom} beat tracking tool \citep{bock2016madmom}, and \texttt{chorder} rule-based chord detection\footnote{\url{https://github.com/joshuachang2311/chorder}} to transcribe the audio into MIDI format with tempo, beat, and chord information.
\textit{REMI-pop-1.7K} encompasses 1,747 songs, or 108 hours of music, in total.
We hold out 5\% of the data (i.e., 87 songs) each for the validation and test sets.
We utilize the validation set to monitor the training process, and the test set to generate style-transferred music for further evaluation.

The data representation adopted here is largely identical to REMI \citep{huang2020pop} and the one used in Sec.~3.
Differences include: (1) an extended set of \textsc{Chord} tokens are used to mark the harmonic settings of each beat; (2) only a single piano track is present; and, (3) each bar contains 16 \textsc{Sub-beat}'s, rather than 32.
Table \ref{tab:event-token-remi} presents descriptions of the event tokens used to represent pop piano songs in \textit{REMI-pop-1.7K}.

\subsection{Evaluation}
To evaluate the trained models, we ask them to generate style-transferred musical pieces, i.e., $\tilde{X}$'s, based on 32-bar-long excerpts, i.e., $X$'s, drawn from the test set.
Following the convention in text style transfer tasks \citep{john2019disentangled, dai2019style}, the generated $\tilde{X}$'s are evaluated according to: (1) their \textit{fidelity} w.r.t.~input $X$;
(2) the strength of attribute \textit{control} given specified attributes $\tilde{a}_k^\text{rhym}$ and $\tilde{a}_k^\text{poly}$; and, (3) their \textit{fluency}.
In addition, we include a (4) \textit{diversity} criterion, which is measured across samples generated under the same inputs
$X$,  $\tilde{a}_k^\text{rhym}$ and $\tilde{a}_k^\text{poly}$.

The experiments are, therefore, conducted under the following two settings:
\begin{itemize}[leftmargin=*]
    \item \textbf{Setting \#1:} We randomly draw 20 excerpts from the test set, each being 32 bars long, and randomly assign to them \textbf{5 sets of different attribute inputs}, i.e., $\{\tilde{a}_k^\text{rhym}, \tilde{a}_k^\text{poly}\}_{k=1}^{32}$, for a model to generate $\tilde{X}$.
    This would result in $20\cdot5=100$ samples on which we may compute the \textit{fidelity}, \textit{control}, and \textit{fluency} metrics.
    \item \textbf{Setting \#2:} We draw 20 excerpts as in setting \#1; however, here, we assign \textbf{only 1 set of attribute inputs} to them each, and have the model compose \textbf{5 different versions} of $\tilde{X}$ with exactly the same inputs.
    By doing so, we would obtain $20 \cdot \binom{5}{2}= 200$ pairs of generations on which we may compute the metrics on \textit{diversity}.
\end{itemize}

Concerning the metrics, for \textit{fidelity} and \textit{diversity}, we directly employ the bar-wise chroma similarity and grooving similarity (i.e., $\mathit{sim}_\text{chr}$ and $\mathit{sim}_\text{grv}$) defined in Sec.~\ref{sec_eva_obj1}. We prefer them to be higher in the case of \textit{fidelity}, but lower in the case of \textit{diversity}.
On the other hand, the metrics for \textit{control} and \textit{fluency} are defined in the following paragraphs.

\textbf{Evaluation Metrics for Attribute Control.}
Since our attributes are ordinal in nature, following \citep{kawai2020attributes}, we may directly compute the Pearson's correlation, $r$, between a specified input attribute class $a$ and the attribute raw score $s$ (see the definitions in Sec.~\ref{sec:attr-comp}, Calculation of Musical Attributes) computed from the resulting generation, to see if they are \textit{strongly} and \textit{positively} correlated.
Taking rhythmic intensity as an example, we define:
\begin{equation}
    r_{\text{rhym}} = \text{Pearson{\textunderscore}corr}(\tilde{a}^\text{rhym}, s^\text{rhym}),
\end{equation}
where $\tilde{a}^\text{rhym}$'s are user-specified inputs, and $s^\text{rhym}$'s are computed from model generations given the $\tilde{a}^\text{rhym}$'s; the definition of $r_{\text{poly}}$ is similar.

Additionally, in the multi-attribute scenario, we would like to avoid \textit{side effects} when tuning one attribute but not the others.
More specifically, the model should be able to transfer an attribute $a$ without affecting other attributes $a'$.
To evaluate this, we define another set of correlations, $r_{\text{poly|rhym}}$ and $r_{\text{rhym|poly}}$, as:
\begin{equation}
    r_{\text{poly|rhym}} = \text{Pearson{\textunderscore}corr}(\tilde{a}^\text{rhym}, s^\text{poly}),
\end{equation}
and similarly for $r_{\text{rhym|poly}}$.
We prefer these correlations to be close to zero (i.e.,
$|r_{a'|a}|$ as small as possible), which suggest more independent controls of each attribute.

\textbf{Evaluation Metric for Fluency.}
In line with text style transfer research \citep{dai2019style, john2019disentangled}, we evaluate the fluency of the generations by examining their \textbf{perplexity} (PPL), i.e., the exponentiation of entropy, which measures the degree of uncertainty involved during their generation.
A PPL value represents, on average, from how many possibilities we are choosing uniformly to produce each token in a sequence.
Therefore, a low PPL means such a sequence is rather likely to be seen in a language. In our case here, the language is pop piano music.
However, it is impossible to know the true perplexity of a language, so our best effort is to train a language model (LM) to check instead the perplexity for that LM to generate the sequence.
This estimated perplexity was proven to upper-bound the true perplexity \citep{brown1992estimate}, and hence can be considered a valid metric.

To this end, we train a stand-alone 24-layer Transformer decoder LM on the training set of \textit{REMI-pop-1.7k}, and compute the PPL of each (32-bar-long) generation $\tilde{X}$ with:
\begin{equation}
    \text{PPL}_{\tilde{X}} = \exp( - \sum_{t=1}^{T} \log p_{\text{LM}}(\tilde{x}_t \, \vert \, \tilde{x}_{<t}) ) ,
\end{equation}
where $T$ is the length of $\tilde{X}$, and $p_{\text{LM}}(\cdot)$ is the probability given by the stand-alone LM.
Note that PPL is the only \textit{piece-level} metric used for our evaluation. All other metrics are measured at the bar level.

\subsection{Results and Discussion}
We commence with examining the models' losses, which are displayed in Table \ref{tab:losses-vae}.
Comparing MuseMorphose (\textit{preferred settings}) and the RNN-based prior art, which are all trained under the same latent space KL constraints (i.e., free bits and cyclical scheduling on $\beta$),
MuseMorphose attains lower validation reconstruction NLL and KL divergence at the same time.
This could be due to both the advantage of looking at the entire sequence, a trait that RNNs do not have; and, that a Transformer decoder can more efficiently exploit the information from latent space when paired with our \textbf{in-attention} conditioning.
Next, turning attention to the three settings of MuseMorphose, we can observe clearly an inverse relationship between the amount of latent space constraint and reconstruction performance.
When the latent space is completely free (i.e., the \textit{AE objective} variant), the NLL can be reduced easily to a very low level.
Yet, even when the strict \textit{VAE objective} is used, MuseMorphose can still reach an NLL  around those of the baselines.

\begin{table*}
\small
\centering
\caption{Training and validation losses of models trained on \textit{REMI-pop-17K} dataset. The best checkpoint is chosen to be the one with the lowest val.~reconstruction NLL.}\label{tab:losses-vae}
\begin{tabular}{l r | cc | cc}
\toprule
\multirow{2}{*}{\textit{Model}}  & \multirow{2}{*}{\shortstack{Best ckpt.\\at step}} &
\multicolumn{2}{c|}{\textbf{Recons. NLL}} &
\multicolumn{2}{c}{\textbf{KL divergence}} \\
& & \textit{Train} & \textit{Val.} & \textit{Train} & \textit{Val.} \\
\midrule
\textbf{MIDI-VAE} \citep{brunner2018midi}  & 65K & 0.676 & 0.894 & 0.697 & 0.682  \\
\textbf{Attr-Aware VAE} \citep{kawai2020attributes}   & 190K & 0.470 & 0.688 & 0.710 & 0.697   \\
\hline
\textbf{MuseMorphose} (Ours), AE objective   & 90K & 0.130 & 0.139 & 6.927 & 6.928   \\
\textbf{MuseMorphose} (Ours), VAE objective   & 75K & 0.697 & 0.765 & 0.485 & 0.484   \\
\textbf{MuseMorphose} (Ours), preferred settings  & 140K & 0.457 & \textbf{0.584} & 0.636 & \textbf{0.636}   \\
\bottomrule
\multicolumn{6}{l}{\footnotesize{\textbf{bold}: leads prior art under the same latent space constraints}} \\
\end{tabular}
\end{table*}

\begin{table*}
\small
\centering
\caption{Evaluation results of various models, on the style transfer of pop piano performances ($\downarrow$\,/\,$\uparrow$: the lower/higher the score the better). }\label{tab:obj-eval-vae-gen}
\begin{tabular}{l cc | cc | c | cc}
\toprule
\multirow{2}{*}{\textit{Model}}  &
\multicolumn{2}{c|}{\textbf{Fidelity}} &
\multicolumn{2}{c|}{\textbf{Control}} &
\multicolumn{1}{c|}{\textbf{Fluency}} &
\multicolumn{2}{c}{\textbf{Diversity}} \\
 & $\mathit{sim}_{\text{chr}}\uparrow$ & $\mathit{sim}_{\text{grv}}\uparrow$ & $r_{\text{rhym}}\uparrow$ & $r_{\text{poly}}\uparrow$
& PPL$\downarrow$ & $\mathit{sim}_{\text{chr}}\downarrow$ & $\mathit{sim}_{\text{grv}}\downarrow$  \\
\midrule
\citet{brunner2018midi}   & 75.6 & 85.4 & .656 & .119 & 8.84 & 74.8 & 86.5  \\
\citet{kawai2020attributes} & 85.0 & 76.8 & \textbf{.961} & .708 & 10.7 & 86.5 & \textbf{84.7}  \\
\hline
\textbf{Ours}, AE objective  & \textbf{98.5} & \textbf{95.7} & .189 & .154 & \textbf{6.10} & 97.9 & 95.4  \\
\textbf{Ours}, VAE objective & 78.6 & 80.7 & .894 & \textbf{.852} & 7.89 & \textbf{73.2} & 84.9  \\
\textbf{Ours}, preferred settings &  91.2 & 84.5 & .909 & \textbf{.852}  & 7.39 & 87.1 & 87.6 \\
\bottomrule
\end{tabular}
\end{table*}

Table \ref{tab:obj-eval-vae-gen} shows the evaluation results on our style transfer task.
\textbf{Fidelity}-wise, our model mostly surpasses the baselines under the \textit{preferred settings}, except in $\mathit{sim}_{\text{grv}}$, where it scores a little lower than \textit{MIDI-VAE}.
Moreover, our \textit{AE objective} variant sticks almost perfectly to the inputs, while \textit{VAE objective} gets a much lower $\mathit{sim}_{\text{chr}}$. Perceptually, we find that $\mathit{sim}_{\text{chr}} < 80$ is a level where a generation often does not feel like the same piece as the input anymore.
Attribute \textbf{Control} is the aspect where \textit{AE objective} fails without question; meanwhile,
MuseMorphose under \textit{preferred settings} and \textit{VAE objective} exhibit equally strong attribute control with $r_{\text{rhym}} \approx .90$ and $r_{\text{poly}} \approx .85$\,.
This suggests that, in the absence of style control-targeted losses, having a narrow enough latent space bottleneck is a key to successful style control.
Comparing with the baselines, our model is the only one capable of tightly controlling both attributes,
despite the closer competitor, \textit{Attr-Aware VAE}, achieving a really high $r_{\text{rhym}}$.
We surmise that rhythmic intensity is an easier attribute for the models to capture, since it only involves counting the number of \textsc{Sub-beat}'s appearing in a bar.
On \textbf{Fluency}, MuseMorphose significantly ($p < .001$, with 100 samples) beats prior art under all settings.
\textbf{Diversity}-wise, though our model does not seem to have an edge over the baselines, we note that there exists an inherent trade-off between diversity and fidelity.
Nevertheless, if we compare our \textit{preferred settings} with Attr-Aware VAE, the closer competitor, we can notice that the margins by which MuseMorphose wins over Attr-Aware VAE on \textbf{Fidelity} ($6.2$ for $\mathit{sim}_{\text{chr}}$; $7.7$ for  $\mathit{sim}_{\text{grv}}$) is somewhat larger than those by which it loses on \textbf{Diversity} ($0.6$ for $\mathit{sim}_{\text{chr}}$; $2.9$ for  $\mathit{sim}_{\text{grv}}$).

It is worth mentioning that manipulating the fidelity-diversity trade-off is possible with a single model, by either forcibly enlarging/shrinking the std vectors (i.e., $\bm{\sigma}_k$'s) from the VAE encoder, or tuning the hyperparameters used during sampling from decoder outputs, e.g., temperature or the truncation point on probability distribution.
Their respective effects, however, are beyond the scope of this study and left for further investigation.

\begin{figure}
    \centering
    \begin{subfigure}[b]{\linewidth}
        \centering
        \includegraphics[width=0.3\textwidth]{figures/midi-vae-ctrl-rhym.pdf}
        \hspace{1cm}
        \includegraphics[width=0.3\textwidth]{figures/attr-aware-ctrl-rhym.pdf} \\
        \vspace{4mm}
        \centering
        \includegraphics[width=0.3\textwidth]{figures/mmph-noKL-rhym-new.pdf}
        \hfill
        \includegraphics[width=0.3\textwidth]{figures/mmph-constKL-rhym-new.pdf}
        \hfill
        \includegraphics[width=0.3\textwidth]{figures/mmph-rhym-new.pdf}
        \caption{On controlling \textbf{rhythmic intensity} (\textit{x-axis}: user-specified $\tilde{a}^{\text{rhym}}$; \textit{y-axis}: $s^{\text{rhym}}$ (left), $s^{\text{poly}}$ (right) computed from the resulting generations; error bands indicate $\pm1$ std from the mean).}
        \label{subfig:rhym-ctrl}
    \end{subfigure}
    \\ \vspace{12mm}
    \centering
    \begin{subfigure}[b]{\linewidth}
        \centering
        \includegraphics[width=0.3\textwidth]{figures/midi-vae-ctrl-poly.pdf}
        \hspace{1cm}
        \includegraphics[width=0.3\textwidth]{figures/attr-aware-ctrl-poly.pdf} \\
        \vspace{4mm}
        \centering
        \includegraphics[width=0.3\textwidth]{figures/mmph-noKL-poly-new.pdf}
        \hfill
        \includegraphics[width=0.3\textwidth]{figures/mmph-constKL-poly-new.pdf}
        \hfill
        \includegraphics[width=0.3\textwidth]{figures/mmph-poly-new.pdf}
        \caption{On controlling \textbf{polyphony} (\textit{x-axis}: user-specified $\tilde{a}^{\text{poly}}$; \textit{y-axis}: $s^{\text{rhym}}$ (left), $s^{\text{poly}}$ (right) computed from the resulting generations; error bands indicate $\pm1$ std from the mean).}
        \label{subfig:poly-ctrl}
    \end{subfigure}
    \vspace{1mm}
    \caption{Comparison of the models on attribute controllability. We desire both a high correlation $r_{a}$ and a low $| r_{a'|a} |$, where $a$ is the attribute in question, while $a'$ is not.}
    \label{fig:eval-attr-ctrl}

\end{figure}

The attribute controllability of the models can be more thoroughly analyzed using Figure \ref{fig:eval-attr-ctrl}.
Focusing on \textit{rhythmic intensity} first (Fig.~\ref{subfig:rhym-ctrl}), although the RNN-based methods have satisfactory control of this attribute, the resulting polyphony scores are more or less undesirably affected ($|r_{\text{poly|rhym}}| \approx .15$ or $.25)$.
This is reasonable since the $|r_{\text{poly|rhym}}|$, and $|r_{\text{rhym|poly}}|$ alike, computed on the training set of \textit{REMI-pop-1.7K} are around $.3$\,, meaning that the two musical attributes are moderately correlated by nature.
MuseMorphose (\textit{VAE} and \textit{preferred settings}), on the other hand, is able to maintain $|r_{\text{poly|rhym}}| \leq .05$, revealing our model's yet another strength in \textbf{independent attribute control}.
However, we may also notice that MuseMorphose is not that great at differentiating classes $0$ and $1$. We suspect that this is due to the higher diversity in terms of rhythm intensity score of the training data in class $0$.
Moving on to \textit{polyphony} (Fig.~\ref{subfig:poly-ctrl}), the more challenging attribute, it is clear that only MuseMorphose manages to have a firm grip on it, while once again maintaining good independence.

To summarize, our MuseMorphose model, underpinned by Transformers and the \textbf{in-attention} conditioning mechanism, outperforms both baselines and ticks all the boxes in our controllable music generation task---in particular, it accomplishes high \textit{fidelity}, strong and independent \textit{attribute control}, good sequence-level \textit{fluency}, and acceptable \textit{diversity}, all at once.
Nevertheless, we also emphasize that using a combination of VAE training techniques, i.e., cyclical KL annealing and free bits, and picking suitable values for the hyperparameters $\beta$ and $\lambda$, are indispensable to the success of MuseMorphose.

\subsection{Qualitative Study with a Demo Web App}

In promotion of our powerful MuseMorphose model,
we implement a web application in which users can compose with the model in real-time using a graphical interface.
A birds-eye view of our web application can be found in Figure \ref{fig:user-interface}.
Upon entering the application, users will be given an existing human-composed 8-bar musical excerpt, shown in the piano roll format.
They can choose to play the music first, in which case the notes will dynamically light up when they sound.
Then, they may use the attribute tuners down below to select the attribute classes (i.e., $\tilde{a}_k^\text{rhym}$ and $\tilde{a}_k^\text{poly}$) they desire for each bar.
After they are done with the tuning, they can send a request to a backend GPU server, on which MuseMorphose runs, for the model to compose given the music and the specified attribute classes.
The style-transferred generation will be returned in around 10 seconds.
We also make it easy to navigate between original music and the style-transferred version for comparison, as well as to re-compose with different attribute classes or other songs.

Using the web application, we conduct a further qualitative study.
Figure \ref{fig:quali-samples} displays a human-composed musical excerpt and two different style-transferred generations from it.
First, comparing both generations (b)(c) with original music (a), we can see that the skeleton of (a) is largely preserved in both (b) and (c).
Next, focusing on the bar-wise differences in (a) vs.~(b) and (a) vs.~(c), we may observe that attribute controls clearly take effect.
Finally, examining the notes around the 3$^\text{rd}$ and 4$^\text{th}$ bars of (b) and (c),
it is evident that the model is capable of responding to drastic changes in attribute classes.
The analysis above demonstrates once again the strengths of our MuseMorphose model.
Although the web app has not been made publicly accessible, we encourage readers to check our companion website\footnote{ \url{https://slseanwu.github.io/site-musemorphose}} for listening samples composed by MuseMorphose.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/user_interface.jpg}

    \caption{User interface for our \textit{MuseMorphose} demo web app. Each column of attribute tuners correspond to the bar above in the piano roll (\textit{x-axis}: time; \textit{y-axis}: note pitch).}
    \label{fig:user-interface}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{\linewidth}
        \centering
        \includegraphics[width=0.82\textwidth]{figures/app_orig.png}
        \caption{Original (human-composed) music and its attribute classes.}
        \label{subfig:quali-orig}
    \end{subfigure}
    \\
    \vspace{4mm}
    \begin{subfigure}[b]{\linewidth}
        \centering
        \includegraphics[width=0.82\textwidth]{figures/app_high_version.png}
        \caption{Generation \#1: \textbf{Low} attribute settings for Bars 3 \& 4, \textbf{high} settings elsewhere.}
        \label{subfig:quali-high}
    \end{subfigure}
    \vspace{4mm}
    \\
    \centering
    \begin{subfigure}[b]{\linewidth}
        \centering
        \includegraphics[width=0.82\textwidth]{figures/app_low_version.png}
        \caption{Generation \#2: \textbf{High} attribute settings for Bars 3 \& 4, \textbf{low} settings elsewhere.}
        \label{subfig:quali-low}
    \end{subfigure}
    \vspace{2mm}
    \caption{Visualizations of an original music excerpt and its style-transferred generations by \textit{MuseMorphose}. The digits in the \textbf{upper} and \textbf{lower} rows represent the bar-wise \textbf{rhythmic intensity} and \textbf{polyphony} classes respectively.}
    \label{fig:quali-samples}

\end{figure}

\section{Conclusion and Future Directions}
In this paper, we have developed \textbf{MuseMorphose}, a sequence variational autoencoder with Transformers as its backbone, which delivered a stellar performance on the controllable conditional generation of pop piano performances.
In achieving so, we set off from defining a novel segment-level conditioning problem for generative Transformers, and devised three mechanisms, namely, \textbf{pre-attention}, \textbf{in-attention}, and \textbf{post-attention} conditioning, to approach it.
Conducted objective and subjective evaluations demonstrated that \textbf{in-attention} came out on top in terms of offering firm control with time-varying conditioning vectors.
Subsequently, we leveraged \textbf{in-attention} to bring together a Transformer encoder and Transformer decoder, forming the foundation of \textbf{MuseMorphose} model.
Experiments have shown that, when trained with a carefully tuned VAE objective, MuseMorphose emerged as a solid all-rounder on the music style transfer task with long inputs,
where we considered controlling two musical attributes, i.e., \textit{rhythmic intensity} and \textit{polyphony}, at the bar level.
Our model outperformed two previous methods \citep{brunner2018midi, kawai2020attributes} on commonly accepted style transfer evaluation metrics, without using any auxiliary objectives tailored for style control.

Our research can be extended in the following directions:
\begin{itemize}[leftmargin=*]
    \item \textbf{Applications in Natural Language Processing (NLP):} The framework of MuseMorphose is easily generalizable to texts by, for example, treating \textit{sentences} as segments (i.e., $X_k$'s) and an \textit{article} as the full sequence (i.e., $X$). As for attributes, possible options include \textit{sentiments} and \textit{text difficulty}.
    Success in controlling the latter, for example, is of high practical value since it would enable us to rewrite articles or literary works to cater to the needs of pupils across all stages of education.
    \item \textbf{Music generation with long-range structures:}
    Being inspired by the two astounding works of high-resolution image generation \citep{ramesh2021zero, esser2021taming}, in which Transformers are used to generate the high-level latent semantics of an image (i.e., $\bm{z}_k$'s, the sequence of latent conditions),
    we conjecture that long range musical structures, e.g., motivic development, recurrence of themes, contrasting sections, etc., may be better generated in a similar fashion.
    To this end, it is likely that we need to learn a \textit{vector-quantized} latent space \citep{van2017neural} instead, so that the latent conditions can be represented in token form for another Transformer to model.
\end{itemize}

To conclude, our work has not only bridged Transformers and VAEs for controllable sequence modeling, where potentials for further applications exist, but has also laid the solid foundation for a pathway to long sequence generation never explored before.

\section{Acknowledgement}
We wish to express our sincere gratitude to \textbf{Fa-Hsun Chen} (Shih-Lun's colleague at Dept.~of CSIE, National Taiwan University) for taking care of our web app's frontend design and implementation.

Sound (cs.SD); Machine Learning (cs.LG); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)