Assisted Sound Sample Generation with Musical Conditioning in Adversarial Auto-Encoders

Abstract: Generative models have thrived in computer vision, enabling unprecedented image processes. Yet the results in audio remain less advanced. Our project targets real-time sound synthesis from a reduced set of high-level parameters, including semantic controls that can be adapted to different sound libraries and specific tags. These generative variables should allow expressive modulations of target musical qualities and continuously mix into new styles. To this extent we train AEs on an orchestral database of individual note samples, along with their intrinsic attributes: note class, timbre domain and extended playing techniques. We condition the decoder for control over the rendered note attributes and use latent adversarial training for learning expressive style parameters that can ultimately be mixed. We evaluate both generative performances and latent representation. Our ablation study demonstrates the effectiveness of the musical conditioning mechanisms. The proposed model generates notes as magnitude spectrograms from any probabilistic latent code samples, with expressive control of orchestral timbres and playing styles. Its training data subsets can directly be visualized in the 3D latent representation. Waveform rendering can be done offline with GLA. In order to allow real-time interactions, we fine-tune the decoder with a pretrained MCNN and embed the full waveform generation pipeline in a plugin. Moreover the encoder could be used to process new input samples, after manipulating their latent attribute representation, the decoder can generate sample variations as an audio effect would. Our solution remains rather fast to train, it can directly be applied to other sound domains, including an user's libraries with custom sound tags that could be mapped to specific generative controls. As a result, it fosters creativity and intuitive audio style experimentations.

\section{Introduction}

Modern music production techniques rely on large and heterogeneous sound sample libraries along with diverse digital instruments and effects. It opens to a great variety of sound design possibilities and limitless contents to compose with, however principled interactions and scaled visualisations are still lacking in order to efficiently explore such potential and use it to generate \textit{target sound qualities}.
\\
\textit{Unsupervised generative models} learn an underlying data distribution solely based on the observation of examples, in order to consistently generate novel content. They have been successfully applied to complex computer vision tasks such as processing facial expressions, landscapes, visual styles and paintings. Some solutions to audio emerged more recently, including pioneer musical systems such as \textit{NSynth} (Neural Synthesizer \cite{nsynth}) for real-time high-quality sound synthesis. However, the heavy model architecture and prohibitive training time restrict its dissemination. The learned internal representation remains mostly uninformative and its many generative parameters are still too little correlated to explicit semantic qualities.
\\
In this paper, we develop a high-level sound synthesis system with meaningful data visualisations and explicit musical controls. It is a lighter \textit{non-autoregressive model} that can be trained fast on small datasets, including an user's personal libraries. Our goal is to learn expressive style variables from any sound tags, so that the model fosters creativity and \textit{assists digital interactions} in music production. Considering note samples of orchestral instruments, we could for instance synthesise novel timbres or \textit{playing style hybrids}.
\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth\center]{dafx_gen_sample2.pdf}
\caption{High-level note sample generation from the latent representation and musical conditioning in the decoder with FiLM. Intermediate features are modulated by the note targets and expressive style controls in order to synthesize new timbres and effects.}
\end{figure}
\\
We train \textit{Wasserstein Auto-Encoders} (WAEs \cite{WAE}\cite{infoVAE}) on Mel- spectrogram magnitudes to organise a generative latent representation of individual note samples spanning the \textit{tessitura} of 12 orchestral instruments. The considered database has intrinsic attributes: note classes, playing styles and timbres (each instrument subset), that we wish to control when generating new notes from the latent space. Thus we extend the WAE model with musical conditioning in the decoder and \textit{Adaptive Instance Normalization} (AdaIN \cite{AdaIN}). Using \textit{Feature Wise Linear Modulation} (FiLM \cite{FiLM}) and adversarial training with a \textit{Fader latent discriminator} \cite{fader}, our WAE-Fader model effectively learns these generative controls along with expressive style variables that can be mixed continuously.
\\
We evaluate these features in terms of generative performances and representation. We perform an \textit{ablation study} and show that the model can sustain a good test reconstruction quality while achieving an accurate attribute-conditional generation. The success of the method relies on an attribute-free latent representation so that the decoder is pushed to learn the conditioning. These distributions can be visualized directly in the 3-dimensional latent space where clusters denote an undesired attribute encoding. We measure it with inter-class statistics and latent post-classification. The experiment validates correlations between low attribute encoding and effective conditioning.
\\
We obtain an expressive note sample generator with \textit{3-dimensional} representations of the training sound domains, decoding \textit{probabilistic latent samples} with explicit control over the rendered note qualities. The learned style variables of the orchestra can ultimately be mixed continuously, as \textit{faders} do, in order to intuitively explore new musical effects. Generated spectrogram magnitudes can approximately be inverted to waveform with the \textit{Griffin-Lim} iterative algorithm (GLA \cite{GLA}). Ultimately we fine-tune the decoder with a pretrained inversion network \cite{MCNN} for \textit{real-time waveform synthesis}. We embed the resulting generative system in a \textit{plugin} allowing for \textit{MIDI} mapping, live exploration and \textit{Digital Audio Workstation} (DAW) integration.

\section{State-of-art}

\subsection{Generative models and regularized auto-encoders}

\textit{Generative models} aim to find the underlying probability distribution of the data $p(\mathbf{x})$ based on a set of examples in $\mathbf{x}\in\mathbb{R}^{d_{x}}$. To do so, we consider \emph{latent variables} defined in a lower-dimensional space $\mathbf{z}\in\mathbb{R}^{d_{z}}$ ($d_{z} \ll d_{x}$), a higher-level representation that could have led to generate any given example. The latent variable generative model is defined by the joint probability distribution $p(\mathbf{x}, \mathbf{z}) = p(\mathbf{x} \vert \mathbf{z})p(\mathbf{z})$, where the \textit{prior} $p(\mathbf{z})$ is usually modelled with simpler distributions such as Gaussian or uniform while a complex conditional distribution $p(\mathbf{x} \vert \mathbf{z})$ maps latent codes to the data space. The model could be evaluated with the maximum marginal likelihood over the considered dataset. However for complex distributions that could model real-world data, integration cannot be computed in closed form.
\\
Regularized auto-encoders have been used to reformulate the problem as an optimization by jointly learning the generative mapping $p_\theta (\mathbf{x|z}) \in \mathcal G$ and an encoding distribution $q_{\phi}(\mathbf{z}\vert\mathbf{x})\in\mathcal{Q}$ from families $\mathcal G,\mathcal{Q}$ of \textit{approximate densities} both parameterized with neural networks. This was initially proposed through \textit{Variational Inference} in the \textit{Variational Auto-Encoder} (VAE \cite{VAE}) that maximizes a lower bound of the data log-likelihood:
\begin{equation}
\resizebox{.9\hsize}{!}{$
\mathbb{E}_{q_\phi(\mathbf{z|x})} \big[ \log{ p_\theta (\mathbf{x|z}) } \big] - D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel p_\theta(\mathbf{z}) \big] \leq \log{ p_\theta (\mathbf{x})}
$}
\end{equation}
This amounts to optimizing the \textit{Evidence Lower Bound Objective} (ELBO) that can be interpreted as follow, the first term is the Negative Log-Likelihood (NLL) data reconstruction cost and the second is the Kullback-Leibler Divergence (KLD) that quantifies the error made by using the approximate $q_\phi(\mathbf{z|x})$ rather than the true $p_\theta(\mathbf{z})$. This latent regularization pushes the encoder to remain close to the prior latent density and can be weighted with a $\beta$ parameter that balances these two objectives.
\begin{equation}
\resizebox{.9\hsize}{!}{$
\mathcal{L}_{\theta, \phi}^{\text{ELBO}} = - \mathbb{E}_{q_\phi(\mathbf{z|x})} \big[ \log{ p_\theta (\mathbf{x|z}) } \big] + \beta \cdot D_{KL} \big[ q_\phi(\mathbf{z|x}) \parallel p_\theta(\mathbf{z}) \big]
$}
\end{equation}
The VAE is implemented with a \textit{stochastic encoder} that parameterises an isotropic Gaussian latent distribution $q_\phi(\mathbf{z|x})$\\$\sim \mathcal{N}(\mu_\phi(\mathbf{x}),\sigma_\phi(\mathbf{x}))$ regularized against an unit variance prior. These assumptions allow analytical KLD computation and differentiable latent sampling for direct optimization of the ELBO.
\\
The KLD forces each \textit{individual} latent code to resemble the prior, which implicitly matches the whole encoded distribution. However a fitted ELBO value does not always result in an \textit{effective inference}. Since the latent codes of different inputs are individually regularized, the KLD may prevent the encoder from learning any useful features (\textit{posterior collapse} \cite{VAEfail}) while the decoder only produces $p_\theta (\mathbf{x})$ regardless of the encoded information. Other conflicting solutions of the ELBO lead to undesired solutions and known limitations of VAEs such as \textit{blurriness} of generated samples or \textit{uninformative latent dimensions} (\cite{VAEcoll}).
\\
With justifications stemming both from \textit{Likelihood-free Optimization} (InfoVAE \cite{infoVAE}) and the theory of \textit{Optimal Transport} (WAE \cite{WAE}), a more general framework for training regularized auto-encoders was recently proposed and that we call Wasserstein Auto-Encoders (WAEs). Considering a \textit{deterministic decoder} $G_\theta:\mathbf{z}\rightarrow \mathbf{x}$ and any family of conditional encoder distribution $Q_\phi(\mathbf{z|x})\in\mathcal{Q}$, it is sufficient that the marginal $Q_Z(\mathbf{z}) := \mathbb{E}_X\big[Q(\mathbf{z|x})\big]$ matches any prior $P_Z$. In comparisons with VAEs, WAEs can optimize any non-negative cost function $C$ and \textit{any divergence measure} $D_Z$ between latent distributions, without requiring a stochastic encoder nor restricting the latent model to Gaussian prior:
\begin{equation}
\resizebox{.9\hsize}{!}{$
\mathcal{L}_\text{WAE} := \inf_{Q(\mathbf{z|x})\in\mathcal{Q}} \mathbb{E}_X \mathbb{E}_{Q(\mathbf{z|x})}\big[C(\mathbf{x},G(\mathbf{z}))\big]+\beta\cdot D_Z(Q_Z(\mathbf{z}),P_Z)
$}
\end{equation}
Thus we set our experiment in the more flexible WAE framework. These regularized auto-encoders are powerful unsupervised representation learning models, rather light-weight and fast to train, performing both inference (encoder) and generation (decoder). They are effective on small datasets (hundreds of training examples), learning a structured latent representation with disentangling capacities encouraged when $\beta > 1$. Once trained, probabilistic samples of the latent prior are consistently decoded into new samples and latent interpolations map to smooth data variations.

\subsection{Maximum Mean Discrepancy Regularization}

As shown for VAEs, the choice of \textit{latent divergence} heavily impacts the resulting model performances. Since the point-wise KLD has strong intrinsic limitations, a more flexible regularization is required for WAEs. Such \textit{differentiable} divergence on latent distributions was developed in the \textit{Reproducing Kernel Hilbert Space} (RKHS) as a distance between probabilistic moments $\mu_{p,q}$ computed with a non-parametric kernel $k$:
\begin{align}
|| \mu_p - \mu_q||^2_H  &= \langle \mu_p - \mu_q, \mu_p - \mu_q \rangle \\
&= \mathbb{E}_{p,p}k(x,x') + \mathbb{E}_{q,q}k(y,y') - 2\mathbb{E}_{p,q}k(x,y)\nonumber
\end{align}
It defines the \textit{Maximum Mean Discrepancy} (MMD \cite{MMD}) between two distributions $x\sim p(x)$ and $y\sim q(y)$, where $\mathbb{E}_{p,q}$ is the expectation that can be evaluated with the \textit{Radial Basic Function} (RBF) kernel of free parameter $\Sigma$:
\begin{align}
k_{\text{(RBF)}} (x,y) &= \exp \left( \frac{ ||x - y||^2}{- 2 \Sigma^2} \right) \qquad \qquad
\end{align}
To the extent of latent regularization, MMD can be computed between every deterministic mini-batch encoding $\mathbf{z}_\text{encoder} = Q(\mathbf{x})$ and random samples from any latent prior $\mathbf{z}_\text{prior} \sim P_Z$. Throughout the model optimization, MMD is thus matching the aggregated encoder posterior to the prior rather than regularizing each latent point individually. In comparisons with KLD, WAE-MMD allows for less constrained inference and richer latent representations. For instance, increasing $\beta$ to two orders of magnitude above the reconstruction cost does not impede the decoder training. Since the WAE objective does not optimize the bounded NLL, the overall generative performances can be improved.
\\
Other kernel functions can be used, which may be more discriminating at the expense of heavier computations. Alternatively to MMD, the WAE-GAN uses an adversarial latent discriminator to assess the divergence, thus optimizing a parametric function that could match even closer the encoder to the prior. However, since we consider a low-dimensional latent space of only 3 dimensions and remain with a simple isotropic Gaussian prior, MMD-RBF is sufficient yet light and stable to train on.

\subsection{Conditioning and feature normalizations}

Regularization in auto-encoders encourages disentanglement of independent generative factors onto separate latent dimensions that would in turn control the corresponding decoded data variations. However this is only partly achieved on toy datasets ($\beta$-VAE \cite{betaVAE}) and in most cases the unsupervised latent dimensions are hardly related to explicit generative parameters. An additional supervision signal may be applied to the generative neural network in order to control and render specific attributes of the data. Thus we consider observations $\mathbf{x}$ paired with attribute annotations $\mathbf{y}$, and condition the decoder as $G:\{\mathbf{z,y}\}\rightarrow \mathbf{x}$.
\\
The simplest conditioning for categorical attributes is to encode them into one-hot vectors that are concatenated to the latent codes before being processed by the decoder. However more advanced conditioning techniques have been developed as for visual style transfer, using full images as conditions (conditional style transfer \cite{styleFiLM}). In the \textit{Feature-wise Linear Modulation} (FiLM \cite{FiLM}) approach, a separate generator learns a mapping from any style inputs to adaptive biases $\beta_\text{FiLM} (\mathbf{y})$ and scales $\gamma_\text{FiLM} (\mathbf{y})$ applied to the conditional network computations. This modulation may be placed anywhere within the architecture and proved to be particularly suited to \textit{Adaptive Instance Normalization} (AdaIN \cite{AdaIN}). Considering the \textit{l-th} hidden layer output activations $\mathbf{h}^l = g^l(\mathbf{h}^{l-1})$ of a generative neural network, the conditional modulation is thus be computed as:
\begin{equation}
\text{AdaIN}(\mathbf{h}^l,\mathbf{y}) = \gamma^l_\text{FiLM} (\mathbf{y})\Bigg[ \frac{\mathbf{h}^l-\mu(\mathbf{h}^l)}{\sigma(\mathbf{h}^l)} \Bigg] + \beta^l_\text{FiLM} (\mathbf{y})
\end{equation}
in which mean and standard deviation \{$\mu$,$\sigma$\} are computed across features, independently for each channel and each sample. In the context of style transfer, it can be interpreted as aligning the mean and variance of the content features with those of the style condition. It is a versatile conditioning technique, requiring little additional computations (particularly when applied channel-wise in convolution layers). It also suits well to handling multiple conditions that may more efficiently be mapped throughout the network rather than arbitrarily concatenated to the input. Thus we will use FiLM and AdaIN for conditioning the decoder on both note and style classes. However, such normalization is not suited to classification tasks since content features are individually normalized. In order to preserve its inference power, we will use \textit{Batch Normalization} (BN) on the encoder's hidden activations.

\subsection{Adversarial latent training}

Adversarial regularization was proposed as an alternative to MMD in the WAE-GAN. For simple low-dimensional latent distributions, the expense of an additional parametric adversarial regularizer is not required. Nonetheless, adversarial latent training remains relevant for expressive conditioning. As detailed in the previous section, adaptive conditioning techniques paired with specific feature normalizations substantially improved feed-forward style transfer. However, in an auto-encoder setting, if the latent space implicitly encodes the attributes of interest, the decoder bypasses the conditioning and does not learn any effective generative controls. This problem was tackled in image generation with the introduction of an adversarial \textit{Fader} latent discriminator $\mathcal{F}$ (Fader Networks \cite{fader}) that competes with the non-conditional encoder in order to prevent correlations between attributes and latent distributions. As for the conditional models, we consider annotated data samples $\{\mathbf{x,y}\}$ and for simplicity, a categorical one-hot representation $\mathbf{y}\in\{0,1\}^n$ with a single $y_i=1$ and its opposite $\Bar{\mathbf{y}}:=\mathbf{1}^n-\mathbf{y}$. Such attribute-free latent representation is implemented in two separate optimization steps, first latent classification of the true attribute $\mathcal{F}:\mathbf{z}\rightarrow \hat{\mathbf{y}} \sim p_\psi(\mathbf{y}|Q(\mathbf{x}))$, then adversarial confusion of the latent classifier at predicting the opposite:
\begin{equation}
\begin{split}
\mathcal{L}^\text{class.}(\psi|\phi) = - \sum_\mathbf{x,y} \log(p_\psi(\mathbf{y}|Q(\mathbf{x})))\\
\mathcal{L}^\text{adv.}(\phi|\psi) = - \sum_\mathbf{x,y} \log(p_\psi(\Bar{\mathbf{y}}|Q(\mathbf{x})))
\end{split}
\end{equation}
As the encoder is pushed to remain invariant to attributes, the decoder is forced to learn the conditioning in order to reconstruct every input samples along with their source attributes. Thus it replaces adversarial training in the high-dimensional pixel space with latent attribute confusion in the low-dimensional latent space in order to efficiently learn style transfer variables. Applied to facial expressions, these \textit{Fader} variables can continuously modulate complex visual features such as gender (female $ \leftrightarrow $ male) or age (younger $ \leftrightarrow $ older). Moreover, in mixing several attributes, one could generate new style qualities.

\subsection{Audio synthesis}

Neural networks can be trained on spectrogram magnitudes (and other spectral features) for audio analysis purpose. It eases the subsequent modelling task, often involving pattern detection, from a pre-processed structured sound representation. However, for generative purpose, an inversion from magnitude to waveform is required since the complex phase information was discarded. It is commonly done offline with GLA \cite{GLA}. Further advances in generative neural networks for audio have targeted raw waveform modelling through specific architecture design. \textit{Wavenet} \cite{wavenet}\cite{nsynth} is amongst them the most popular solution. It uses several stacks of dilated causal convolutions in order to aggregate multiple temporal granularities and structure long-term dependencies, which is challenging at the high audio sample rate. The output is a single auto-regressive sample prediction given all the previous sample context $p(x_t|x_1..x_{t-1})$. It results in high-quality real-time audio synthesis. However this sample level modelling requires long training times, heavy architectures that offer little knowledge over their learned features.
\\
The \textit{Multi-head Convolutional Neural Network} (MCNN \cite{MCNN}), a recent alternative for audio waveform modelling, was designed as a feed-forward real-time magnitude spectrogram inversion system that is not restricted to linear frequency scale. It proved to outperform GLA quality for speech. The use of differentiable GPU-based STFT computations enables a faster optimization onto spectral losses, rather than \textit{auto-regressive} sample predictions:
\begin{equation}
\begin{split}
\mathbf{x}\xrightarrow{|\text{STFT}|}|\mathbf{S}|\xrightarrow{\text{MCNN}}\hat{\mathbf{x}}\xrightarrow{\text{STFT}}\hat{\mathbf{S}} \Longrightarrow \mathcal{L}^\text{MCNN}(\mathbf{S},\hat{\mathbf{S}}) \\
\mathcal{L}^\text{MCNN} = \lambda_0\cdot\mathcal{L}^\text{SC} + \lambda_1\cdot\mathcal{L}^\text{logSC} + \lambda_2\cdot\mathcal{L}^\text{IF} + \lambda_3\cdot\mathcal{L}^\text{WP}\\
\end{split}
\end{equation}
where $|\mathbf{S}|$ can be any spectrogram magnitude (including Mel-scaled frequencies). The model is well tailored to audio with multiple heads of 1-dimensional temporal up-sample convolutions. These heads focus on different spectral components and sum into waveform. It remains light-weight and could be adapted in an end-to-end waveform auto-encoder. Four objectives were originally proposed, using the complex STFT for the \textit{Instantaneous Frequency} (IF) and \textit{Weighted Phase} (WP) losses, that we could not optimize successfully. Hence we will only use the \textit{Spectral Convergence} (SC) and log-scale magnitude (logSC) losses:
\begin{equation*}
\mathcal{L}^\text{SC}(\mathbf{S},\hat{\mathbf{S}}) = \| |\mathbf{S}| - |\hat{\mathbf{S}}| \|_F / \| |\mathbf{S}| \|_F \text{ with }\|.\|_F \text{ the Frobenius norm}
\end{equation*}
\begin{equation}\label{eq9}
\mathcal{L}^\text{logSC}(\mathbf{S},\hat{\mathbf{S}}) =  \| \log(|\mathbf{S}|+\epsilon) - \log(|\hat{\mathbf{S}}|+\epsilon) \|_1
\end{equation}

\section{Method}

Our experiment begins with the WAE-MMD, isotropic unit variance Gaussian prior $\mathbf{z}_{\text{prior}} \sim \mathcal{N}(0,1)$, RBF kernel and BN in both encoder and decoder in order to structure a 3-dimensional generative latent sound representation. Given a magnitude spectrogram $|\mathbf{S}|$ and a corresponding set of annotated attributes $\mathbf{y}$, we are learning $Q:|\mathbf{S}|\rightarrow \mathbf{z}$ and $G:\mathbf{z} \rightarrow |\hat{\mathbf{S}}|$ such as $|\mathbf{S}| \approx |\hat{\mathbf{S}}| = G(Q(|\mathbf{S}|))$ with \textit{Binary Cross-Entropy} (BCE) reconstruction cost:
\begin{equation}
\begin{split}
\mathcal L_{\text{WAE}} = \text{BCE}(|\mathbf{S}|,|\hat{\mathbf{S}}|) + \beta\cdot\text{MMD}_\text{RBF}(\mathbf{z},\mathbf{z}_{\text{prior}})\\
\text{BCE}(x,\hat{x}) = -\big[x\log\hat{x}+ (1-x)\log(1-\hat{x})\big]\text{ ; }|x|<1
\end{split}
\end{equation}
We can sample random codes from the latent prior and consistently decode new magnitude samples, however there is no control on the output features.
\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth\center]{dafx_model_flow2.pdf}
\caption{How information flows in the adversarial optimization of the WAE-Fader}
\end{figure}
For the orchestra, we consider $\mathbf{y}=\{\mathbf{y}_{\text{note}},\mathbf{y}_{\text{style}}\}$ with $\mathbf{y}_{\text{note}}=\{\text{semitone, octave}\}$. We define the timbre attribute as the class of an instrument subset, which comprises the \textit{Ordinario} mode as well as diverse extended playing techniques such as \textit{Staccato}, \textit{Flatterzunge} or \textit{Pizzicato}. When considering a single subset, we thus aim at controlling the playing techniques of the considered instrument as $\mathbf{y}_{\text{style}}$. When considering multiple instruments, instead we aim at controlling the different timbres, either in \textit{Ordinario} or with mixed playing styles within each instrument subset.
\\
For explicit controls over the rendered attributes, we condition the decoder as $G:\{\mathbf{z,y}\} \rightarrow |\hat{\mathbf{S}}|$ using AdaIN. An additional FiLM generator is fed with concatenated one-hot vectors of the three attribute classes (semitone, octave and style). It learns an adaptive mapping to biases $\beta_\text{FiLM} (\mathbf{y})$ and scales $\gamma_\text{FiLM} (\mathbf{y})$ that are used to modulate the normalized decoder activations. In order to effectively learn the style conditioning and expressively modulate timbres or playing techniques, we use adversarial training with a Fader latent discriminator $\mathcal{F}:\mathbf{z}\rightarrow\hat{\mathbf{y}}_{\text{style}}\sim p_\text{class.}(\mathbf{y}_{\text{style}}|Q(|\mathbf{S}|))$ that competes with the non-conditional encoder in classifying the considered styles from latent codes:
\begin{equation}
\begin{split}
\mathcal L_{\text{class.}} = -\log p_\text{class.}(\mathbf{y}_{\text{style}}|Q(|\mathbf{S}|))\\
\mathcal L_{\text{WAE-Fader}} = \mathcal L_{\text{WAE}} -\alpha\cdot\log p_\text{class.}(\bar{\mathbf{y}}_{\text{style}}|Q(|\mathbf{S}|))
\end{split}
\end{equation}
with $\bar{\mathbf{y}}_{\text{style}}:=\textbf{1}-\mathbf{y}_{\text{style}}$ and $\alpha$ that weights the adversarial loss in the encoder. Classification is optimized on the NLL with \textit{Softmax} probilities. The resulting attribute confusion prevents the latent space from implicitly encoding the style distributions, thus the decoder is forced to use the conditioning to reconstruct the source features from the attribute-free code. Ultimately these learned style variables can continuously be mixed, as actual \textit{faders} do. We refer to this final model as WAE-Fader, that still uses MMD regularization. It allows for controlling the strength of each rendered attribute and intuitively exploring hybrid sound effects from any custom tags, here either chosen from extended playing techniques or from diverse orchestral timbre domains.
\\
The resulting generative system maps any latent coordinate $\mathbf{z}\sim\mathcal{N}(0,1)\in\mathbb{R}^3$ to target note spectrograms with expressive musical style controls. Inversion from spectrogram magnitudes to audio waveforms can be done offline with GLA. Alternatively, we pretrain a MCNN on a larger corpus of musical note samples to allow real-time rendering. In order to improve the final audio quality, we fine-tune the full generative model by freezing the encoder parameters and jointly optimizing the learned decoder with the pretrained MCNN as:
\begin{equation}\label{eq12}
\resizebox{.89\hsize}{!}{$
\mathbf{x}\xrightarrow{\text{STFT}}\mathbf{S}\xrightarrow{|\text{Mels}|}|\mathbf{S}|\xrightarrow{Q}\mathbf{z}\xrightarrow{G\circ\text{MCNN}}\hat{\mathbf{x}}\xrightarrow{\text{STFT}}\hat{\mathbf{S}} \Longrightarrow \mathcal{L}^\text{MCNN}(\mathbf{S},\hat{\mathbf{S}})
$}
\end{equation}
This waveform pipeline $\{G\circ\text{MCNN}\}$ is embed in a plugin for live interactions and DAW integration. Using a MIDI interface, we can for instance trigger target note classes $\mathbf{y}_{\text{note}}$ with keys and map the continuous generative parameters to faders. These are the latent dimensions $\mathbf{z}$, that can also be randomly sampled, and most interestingly the adversarially learned style variables $\mathbf{y}_{\text{style}}$ that can be mixed to explore new sound effects.

\section{Experiment}

\subsection{Dataset}

We use the Studio-On-Line (SOL \cite{SOL}) library of around 15000 individual note samples, across the tessitura of 12 orchestral instruments grouped in 4 families and with many extended playing techniques, that may be specific or shared across instrument families. These are \textit{Wind} (Alto-Saxophone, Bassoon, Clarinet, Flute, Oboe), \textit{Brass} (English-Horn, French-Horn, Tenor-Trombone, Trumpet), \textit{String} (Cello, Violin) and \textit{Keyboard} (Piano). Notes are consistently tagged with the intrinsic attributes of the dataset: note classes (12 semitones across 9 octaves), several dynamics and playing styles of every instrument. We define two style experiments for the orchestra. If training on a single instrument, we aim for expressive synthesis of its playing styles. If training on multiple instruments, we aim for timbre control. Each instrument subset defines a timbre domain, either in \textit{Ordinario} (its common mode) or with all styles mixed.
\\
Audio files are down-sampled to 22050Hz and pre-processed into Mel-spectrograms with a FFT size of 2048, hop size of 256 and 500 bins ranging the full spectrum. As we consider a generator of \textit{individual} notes, we set a common audio length of 34560 samples ($\sim$1.6s) from the attack which amounts to 128 STFT frames. We choose this duration as a trade-off between input and latent dimensionality, limiting the amount of silence after shorter playing modes (eg. \textit{Pizzicato}) while keeping some sustain for longer notes (from which some sustain and decay may have been cropped). Magnitudes are floored to 1e-3 and log-scaled in [0,1] according to the BCE range. Each playing style subset of each instrument is split into 80\% training, 10\% validation and 10\% test notes. In average each instrument has 10 playing styles and 100 to 200 notes for each.

\subsection{Implementation details}

\textbf{Architecture of the WAE-Fader:} Our experiments have been implemented in the \textit{PyTorch} environment and our codes will be shared with this dependency. All convolution layers use 2-d. square kernels, an input zero-padding of half the kernel size and are followed by 2-dimensional feature normalization. All fully-connected linear layers are followed by 1-dimensional feature normalization. The non-linear activation used after every normalization is CELU. The deterministic encoder has 5 convolution layers with [12, 24, 48, 96, 128] output channels, kernel size 5 and stride 2, that down-sample the input spectrograms into 128 output maps that are flattened into an intermediate feature vector of size 8192. It is followed with a bottleneck of 3 linear layers of output sizes [1024, 512, 3] mapping to the latent space. For input Mel-spectrograms of size (500,128), it amounts to a dimensionality reduction of more that 5 orders of magnitude. All normalizations are BN. The decoder mirrors this structure with 3 linear layers of output sizes [512, 1024, 8192]. This vector is then reshaped into 128 maps. To avoid the known \textit{checkerboard artifacts} \cite{deconvolution} of the transposed convolution, we use \textit{nearest neighbor} up-sampling followed with convolution of stride 1. These maps are processed with 4 up-sampling of ratios 3, the last one directly mapping to the input dimensionality of (500,128), and 5 convolutions with [96, 48, 24, 12, 1] output channels and kernel sizes [5, 5, 7, 9, 7]. All normalizations are AdaIN and the decoder output activation is sigmoid, bounded in [0,1] according to the BCE range. The FiLM conditioning is applied feature-wise at the output of the first two linear layers and channel-wise after. It amounts to 3688 modulation weights computed by an additional FiLM generator of 3 linear layers of output sizes [512, 1024, 3688]. Its output is split into biases and scales of sizes [512, 1024, 128, 96, 48, 24, 12]. The Fader latent discriminator has 3 linear layers of output sizes [1024, 1024, $\text{n}_\text{style}$] with \textit{LeakyReLU} activations and a dropout ratio of 0.3, mapping latent codes to probabilities of the $\text{n}_\text{style}$ classes.
\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth\center]{dafx_enc_fader2.pdf}
\caption{The Fader latent discriminator tries to infer the true style attribute while the encoder adversarially aims at fooling it. It encourages attribute invariance in the latent representation and learning of continuous generative controls in the decoder.}
\end{figure}
\\
\\
\textbf{Training parameters:} We train our models with the Adam optimizer, an initial learning rate of 5e-4 and a batch size of 90. All model weights are initialized with Xavier uniform distribution. Depending on the considered data subset size, between 200 and 800 epochs are needed. A single instrument (1000-1500 notes) can be modelled in less than 2 hours on one NVIDIA TITAN Xp GPU. Training over all instruments and styles at once (around 11000 notes) takes less than 12 hours. In the first part of the training (30 to 100 epochs), we only optimize the reconstruction and classification objectives. Then we gradually introduce the MMD regularization ($\beta$-warmup) and the adversarial feedback in the encoder ($\alpha$-warmup) until the first half of training epochs. The rest of the training jointly optimizes all training objectives at their target strengths $\beta$ = 40 and $\alpha$ = 4. These value were estimated in order to approximately balance the gradient magnitudes back-propagated by each loss. However, for the baseline WAE-MMD models we warmup $\beta$ to 500 since it does not prevent from optimizing the reconstruction cost.
\\
\\
\textbf{Signal reconstruction:} The above described model trains on inputs with 128 frames of Mel-spectrogram, which amount to 34560 waveform samples according to our STFT settings. The generated Mel-magnitudes can be approximated back to the linear frequency scale and iteratively inverted with GLA for 100 to 300 iterations. To allow real-time rendering and a possibly improved audio quality, we reproduce the original MCNN architecture for Mel-spectrogram magnitudes inversion. We use 8 heads, $\lambda_0$ = 1 and $\lambda_1$ = 6. We could not successfully optimize the complex losses, however, we compute these magnitude losses on both the linear and Mel frequency scales. We pretrain this model on a larger dataset of around 50 hours audio comprising SOL and subsets of the \textit{Vienna Symphonic Library} (VSL). Ultimately, we fine-tune the trained decoder with this pretrained MCNN. To do so, we freeze the encoder weights and optimize $G\circ\text{MCNN}$ on the model train set. In equation (\ref{eq12}), the auto-encoder pair $G,Q$ maps to Mel-spectrogram magnitudes $|\mathbf{S}|$ which are inverted to signals by the MCNN. However, the loss computation $\mathcal{L}^\text{MCNN}(\mathbf{S},\hat{\mathbf{S}})$ is not necessarily restricted to this frequency scale. Thus we evaluate and sum $\mathcal{L}^\text{SC}$ $\mathcal{L}^\text{logSC}$ from equation (\ref{eq9}) on both linear and Mel frequency scaled magnitudes.

\begin{table}[ht!]
\begin{center}
\begin{tabular}{ |c|c|c|c| }
\hline
classified attribute ($\text{n}_\text{style}$) & train set & validation & test \\
\hline
\hline
Semitone (12) & 1.00 & 0.99 & 0.99 \\
\hline
Octave (9) & 1.00 & 0.99 & 1.00 \\
\hline
Ordinario timbres (12) & 1.00 & 1.00 & 1.00 \\
\hline
Extended timbres (12) & 1.00 & 1.00 & 1.00 \\
\hline
Violin playing styles (10) & 1.00 & 0.97 & 0.95 \\
\hline
Clarinet playing styles (10) & 1.00 & 0.96 & 0.94 \\
\hline
Piano playing styles (10) & 1.00 & 0.92 & 0.95 \\
\hline
Trumpet playing styles (10) & 1.00 & 0.92 & 0.94 \\
\hline
Alto-Saxophone pl. styles (10) & 1.00 & 0.98 & 1.00 \\
\hline
Tenor-Trombone pl. styles (11) & 1.00 & 0.90 & 0.90 \\
\hline
\end{tabular}
\caption{Reference \textbf{F1-scores} of the pretrained data classifiers used for the evaluation of conditional note generations\label{tab:classif_ref}}
\end{center}
\end{table}

\subsection{Evaluations}

\textbf{Generative performances:} First, we evaluate the ability of our models to produce accurate spectrograms by computing the reconstruction scores on the test set with \textit{Root-Mean Squared Error} (RMSE) and \textit{Log-Spectral Distance} \resizebox{.39\hsize}{!}{$\text{LSD} = \sqrt{\sum \big[10\log_{10}(|\mathbf{S}|/|\hat{\mathbf{S}}|) \big]^2 }$}. Regarding the conditioning aspects, we first pretrain data classifiers to reliably discriminate the different attribute classes and report their performances in Table~\ref{tab:classif_ref}. These classifiers share the same architecture as the encoder but map to the $\text{n}_\text{style}$ classes of interest. We use them as references to evaluate the effectiveness of the conditioning. Then, we sample an evaluation batch of 1000 random latent points from the prior, along with random semitone and octave targets. This evaluation batch is decoded to each attribute of the model (either playing styles or timbres) and classified with the corresponding reference classifier. A high accuracy means an effective conditioning for the task of musical note generation. We report the average accuracy for all the target conditions, with random octaves both in [0-8] (full orchestral range) or in [3-4] where models train on the overlap of every instrument tessitura.
\\
\\
\textbf{Latent space structure:} The effectiveness of the conditioning relies on learning an attribute-free latent representation of the data. If the attribute distributions are clustered, the decoder may learn their correlations with latent dimensions and bypass the conditioning signal. This phenomenon is alleviated with adversarial training of the non-conditional encoder against a Fader latent discriminator. As we map to 3-dimensional spaces, we can directly visualize this latent organization. We also propose two evaluations of the attribute representations. First, we compute the average inter-class latent statistics with MMD. In this case, low values mean that the attribute distributions blend in the final representation. Second, we also perform a post-classification task by training classifiers at predicting the attributes from the learned latent representation. These models use the same architecture as the Fader discriminator, and we report their final accuracy. In this case, low scores mean that the latent representation did not encode the attributes.

\section{Results}

\subsection{Ablation study}

\begin{table}[t]
\begin{center}
\begin{adjustbox}{width=\columnwidth,center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\hline
model & \multicolumn{2}{c|}{test rec.} & \multicolumn{4}{c|}{note cond. acc.} & \multicolumn{2}{c|}{style cond. acc.}\tabularnewline
\hline
 & MSE & LSD & st.$_{34}$ & oct.$_{34}$ & st.$_{08}$ & oct.$_{08}$ & style$_{34}$ & style$_{08}$\tabularnewline
\hline
\hline
\multicolumn{9}{|c|}{Violin playing styles (n$_{style}$=10) 1475 training note samples}\tabularnewline
\hline
WAE-MMD & 0.76 & 68.2 & NA & NA & NA & NA & NA & NA\tabularnewline
\hline
WAE-note & 0.69 & 55.4 & 0.73 & 0.72 & 0.47 & 0.43 & NA & NA\tabularnewline
\hline
WAE-style & 0.74 & 59.6 & 0.47 & 0.39 & 0.30 & 0.22 & 0.20 & 0.17\tabularnewline
\hline
WAE-Fader & 0.80 & 91.1 & \textbf{0.96} & \textbf{0.77} & 0.97 & 0.48 & \textbf{0.88} & \textbf{0.93}\tabularnewline
\hline
\hline
\multicolumn{9}{|c|}{Ordinario timbres (n$_{style}$=12) 1784 training note samples}\tabularnewline
\hline
WAE-MMD & 1.04 & 88.3 & NA & NA & NA & NA & NA & NA\tabularnewline
\hline
WAE-note & 0.84 & 71.6 & 0.99 & 0.96 & 0.62 & 0.53 & NA & NA\tabularnewline
\hline
WAE-style & 0.80 & 65.7 & 0.64 & 0.58 & 0.30 & 0.24 & 0.33 & 0.19\tabularnewline
\hline
WAE-Fader & 1.01 & 105 & \textbf{1.00} & \textbf{1.00} & 0.94 & 0.68 & \textbf{0.95} & \textbf{0.70}\tabularnewline
\hline
\hline
\multicolumn{9}{|c|}{Extended timbres (n$_{style}$=12) >11000 training note samples}\tabularnewline
\hline
WAE-MMD & 0.93 & 175 & NA & NA & NA & NA & NA & NA\tabularnewline
\hline
WAE-note & 0.69 & 173 & 0.99 & 0.98 & 0.72 & 0.64 & NA & NA\tabularnewline
\hline
WAE-style & 0.65 & 172 & 0.84 & 0.83 & 0.44 & 0.39 & 0.61 & 0.34\tabularnewline
\hline
WAE-Fader & 1.32 & 182 & \textbf{1.00} & \textbf{1.00} & 0.90 & 0.71 & \textbf{0.95} & \textbf{0.64}\tabularnewline
\hline
\end{tabular}
\end{adjustbox}
\caption{The ablation study confirms the effectiveness of the \textbf{WAE-Fader} conditioning, both on target notes and playing styles or timbres. The conditional latent sampling is either performed with random octaves in [3,4] (the overlap of every tessitura) and [0-8] (the full orchestra range), we report the accuracy of the conditioning with respect to the targets $\text{note}_{34,08}$ (st. is semitone classe and oct. is octave classe) and $\text{style}_{34,08}$.\label{tab:abla_gen}}
\end{center}
\end{table}
We defined both generative and representation evaluations to assess the effectiveness of our proposed musical conditioning. To study the benefits and compromises of each model feature, we train the base WAE-MMD and compare it with ablations of the WAE-Fader. The incremental model comparisons are WAE-MMD (no conditioning), WAE-note (semitone and octave conditioning), WAE-style (note and style conditioning) and WAE-Fader. In order to simplify the notation, we do not specify the MMD but this regularization is used for all models. We performed this ablation study on the violin subset that has the following annotated playing styles: \textit{Ordinario, Sustained, Short, Non-vibrato, Staccato, Pizzicato-secco, Medium-vibrato-short, Tremolo, Medium-vibrato-sustained} and \textit{Pizzicato-l-vib}. We also compare the WAE-Fader on instrument timbres, either in ordinario or for all extended techniques mixed per instrument subset. Table~\ref{tab:abla_gen} reports the successive generative performances of the models. Table~\ref{tab:abla_lat} reports the latent evaluations, showing how the conditioning tasks are reflected in the learned representations. It confirms the effectiveness of the expressive conditioning when the attribute-invariance assumption is achieved.
\\
As we can see, conditioning WAE-note on the semitone and octave classes shows that the WAE-MMD model can partly learn the note controls with FiLM conditioning. Accordingly, the latent space structure does not exhibit strong correlations with the note classes anymore but with the style attributes that become the main unsupervised data feature. We also notice that this additional supervision improves the reconstruction quality. However, when adding the style conditioning in WAE-style, it seems that most performances drop. Indeed, the overall conditioning becomes little effective, both for the target note and style conditions. The final results show that the adversarial latent training enables the WAE-Fader model to effectively learn the complete conditioning, at the expense of a possible drop in its reconstruction accuracy.
\\
It also seems that the task of modelling the playing styles when learning on a single instrument is more challenging than changing the timbres across multiple instruments. This can be seen in the lower performance of the WAE-style model applied to the violin. This may also be explained by the reduced size of the training data when the learning is restricted to single instrument subsets. These observations are supported by the resulting audio outputs of the conditional note generations. Indeed, it appears that meaningful and expressive variations when switching to any attribute conditions are only achieved with our proposed WAE-Fader model. This is successful for conditioning applied on both timbre attributes or playing styles.
\begin{table}[t]
\begin{center}
\begin{adjustbox}{width=\columnwidth,center}
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
model & \multicolumn{3}{c|}{inter-class MMD} & \multicolumn{3}{c|}{post-class. acc.}\tabularnewline
\hline
 & st. & oct. & style & st. & oct. & style\tabularnewline
\hline
\hline
\multicolumn{7}{|c|}{Violin playing styles (n$_{style}$=10) 1475 training note samples}\tabularnewline
\hline
WAE-MMD & 0.25 & 0.26 & 0.30 & 0.92 & 0.94 & 0.56\tabularnewline
\hline
WAE-note & 0.03 & 0.10 & 0.50 & 0.04 & 0.49 & 0.82\tabularnewline
\hline
WAE-style & 0.12 & 0.16 & 0.35 & 0.25 & 0.55 & 0.59\tabularnewline
\hline
WAE-Fader & 0.02 & 0.01 & 0.46 & 0.08 & 0.34 & 0.64\tabularnewline
\hline
\hline
\multicolumn{7}{|c|}{Ordinario timbres (n$_{style}$=12) 1784 training note samples}\tabularnewline
\hline
WAE-MMD & 0.12 & 0.38 & 0.28 & 0.75 & 0.87 & 0.59\tabularnewline
\hline
WAE-note & 0.02 & 0.28 & 0.51 & 0.33 & 0.57 & 0.83\tabularnewline
\hline
WAE-style & 0.04 & 0.40 & 0.35 & 0.13 & 0.60 & 0.63\tabularnewline
\hline
WAE-Fader & 0.33 & 0.08 & 0.03 & 0.17 & 0.25 & 0.23\tabularnewline
\hline
\hline
\multicolumn{7}{|c|}{Extended timbres (n$_{style}$=12) >11000 training note samples}\tabularnewline
\hline
WAE-MMD & 0.02 & 0.41 & 0.12 & 0.71 & 0.89 & 0.48\tabularnewline
\hline
WAE-note & 5e-3 & 0.30 & 0.22 & 0.07 & 0.49 & 0.71\tabularnewline
\hline
WAE-style & 4e-3 & 0.32 & 0.18 & 0.07 & 0.56 & 0.55\tabularnewline
\hline
WAE-Fader & 3e-3 & 0.20 & 0.11 & 0.11 & 0.46 & 0.43\tabularnewline
\hline
\end{tabular}
\end{adjustbox}
\caption{The ablation study allows to monitor the latent organization in the different models and throughout their training, as shown in Figure~\ref{fig:lat_plot}. We use both inter-class statistics and latent post-classification to estimate the final attribute invariance in the learned representation.\label{tab:abla_lat}}
\end{center}
\end{table}

\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth\center]{latent_plot_fader_allinst_ord.pdf}
\caption{Latent organization as the WAE-Fader model trains on the ordinario timbres, each instrument domain being represented by a separate color. In \textbf{\textit{A}}, at epoch 24, the encoder does not optimize the adversarial loss yet. Its unsupervised representation exhibits the attribute classes. In \textbf{\textit{B}}, at epoch 674, the $\alpha$-warmup is finished and the adversarial latent training had blended the attribute distributions. The 2-dimensional projections are performed with t-Distributed Stochastic Neighbor Embedding (TSNE).\label{fig:lat_plot}}
\end{figure}

\subsection{Expressive note sample generations}

\begin{table*}[t]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
model & \multicolumn{2}{c|}{test rec.} & \multicolumn{4}{c|}{note cond. acc.} & \multicolumn{2}{c|}{style cond. acc.} & \multicolumn{3}{c|}{inter-class MMD} & \multicolumn{3}{c|}{post-class. acc.}\tabularnewline
\hline
 & MSE & LSD & st.$_{34}$ & oct.$_{34}$ & st.$_{08}$ & oct.$_{08}$ & style$_{34}$ & style$_{08}$ & st. & oct. & style & st. & oct. & style\tabularnewline
\hline
\hline
Clarinet & 0.87 & 116 & 0.96 & 0.99 & 0.98 & 0.45 & 0.97 & 0.92 & 0.05 & 0.58 & 0.12 & 0.15 & 0.76 & 0.41\tabularnewline
\hline
Piano & 0.99 & 113 & 0.53 & 0.91 & 0.47 & 0.72 & 0.72 & 0.64 & 0.03 & 0.03 & 0.08 & 0.16 & 0.20 & 0.43\tabularnewline
\hline
Trumpet & 0.90 & 107 & 0.91 & 0.93 & 0.96 & 0.37 & 0.90 & 0.87 & 0.60 & 0.11 & 0.02 & 0.42 & 0.50 & 0.29\tabularnewline
\hline
Alto-Sax. & 1.22 & 131 & 0.96 & 0.99 & 0.98 & 0.40 & 0.76 & 0.71 & 0.14 & 0.09 & 0.50 & 0.08 & 0.48 & 0.48\tabularnewline
\hline
T. Trombone & 0.96 & 100 & 1.00 & 1.00 & 0.92 & 0.41 & 0.83 & 0.77 & 0.04 & 0.14 & 0.34 & 0.06 & 0.55 & 0.47\tabularnewline
\hline
\end{tabular}

\caption{Additional WAE-Fader results on the playing techniques of instruments in other orchestral families\label{tab:addFader}}
\end{center}
\end{table*}
In this section, we report additional experiments on the WAE-Fader models when conditioned on the playing styles of different instruments and families. As shown in Table~\ref{tab:addFader}, our model seems to train successfully on playing styles in every instrument families, as well as across the 12 instrument timbres of the orchestra as shown in the previous ablation study. This amounts to a great variety of sound qualities spanning extended modes of the orchestra, and let us hypothesize that the model could be applied to other sound domains as long as the tags are consistent with the data. Furthermore, the style variables learned with the Fader latent discriminator are continuous independent controls that can be mixed. Hence, this can allow our system to modulate the strength of rendered styles and create new effects by combining multiple attributes. Our model can also be used for sample modifications, akin to traditional audio effects, by encoding a given sample and manipulating the attribute conditions in order to decode different sample transformations.

\subsection{Audio outputs and plugin development}

As discussed previously, our proposed models can generate magnitude spectrograms, while controling their expressive qualities. These spectrograms can be either inverted to waveform offline with GLA or real-time if paired with MCNN. When fine-tuning the learned decoders with the pretrained MCNN on magnitude losses, we obtain a quality almost equivalent to the GLA approximation. We provide audio examples of test set reconstructions and conditional note generations inverted with both GLA and MCNN for individual listening evaluation on the companion webpage. While the audio quality of these results can still be improved, we can already confirm the ability of the model to provide semantic controls. As the learned style variables of WAE-Fader can be mixed continuously, we also provide some sound examples that were generated when modifying multiple orchestral attributes.
\\
Our proposal provides intuitive sound synthesis of target sound qualities with learned style variables that can be modulated and combined. The unsupervised latent dimensions organize remaining data features, which can be directly visualized in a 3-d. space, in order to perform sampling or explicit control. These features allow to generate timbres, playing styles and hybrid effects across multiple attribute combinations through intuitive interactions. We provide a real-time implementation of our models by relying on the fine-tuned $\{{G\circ\text{MCNN}}\}$ generation. This implementation relies on the \textit{LibTorch} C++ API, which converts trained \textit{PyTorch} models, that we further embed in a \textit{PureData} external. This plugin can be mapped to a MIDI controller or integrated in a DAW for composition and musical performance. This allows to play notes with a keyboard, while using continuous faders to control latent coordinates and mix style conditions.

\section{Conclusion}

We developed an expressive musical conditioning of the Wasserstein Auto-Encoders able to model a collection of orchestral note samples. The model learns effective target semitone and octave controls as well as continuous style variables. We considered extended playing techniques and timbre subsets as attributes, and used adversarial latent training to encourage an attribute-invariant representation in the WAE-Fader. Our ablation study validates the effectiveness of style conditioning when this invariance condition is obtained.
\\
We fine-tuned the decoders with a Mel magnitude spectrogram inversion network that allows real-time waveform rendering and are currently working on refining the audio quality. This results in a note sample generator with meaningful data visualizations and intuitive controls of audio styles. These parameters can be mixed, as faders, in order to explore hybrid sound effects. Our final generative model is embed in a plugin for MIDI mapping and live interactions. This system provides assisted music production and fosters creative sound experimentations. We provide sound examples from our orchestral models, either inverted offline with GLA or with the fine-tuned waveform generation pipeline. These sounds allow for subjective evaluation of both semantic and audio qualities of our solution.
\\
Although we used clearly defined metadata attributes pertaining to instrumental playing styles, the model can potentially be applied to any sound domain. For instance, a user library with custom tags could be mapped to sound synthesis parameters. Furthermore, as the architecture is rather light and scales to small datasets, it could be trained on user libraries. Future experiments will target the quality of the waveform modelling systems for variable note lengths and real-time synthesis. Ultimately, our models could be implemented as a standalone instrument with physical controls that can be mapped to pretrained style variables. This would allow an intuitive and creative exploration across a vast amount of sound variations with a reduced set of adaptive parameters.

\section{Acknowledgements}
This work was supported by the MAKIMOno project 17-CE38-0015-01 funded by the French ANR and the Canadian NSERC (STPG 507004-17), the ACTOR Partnership funded by the Canadian SSHRC (895-2018-1023) and an NVIDIA GPU Grant. We acknowledge and thank the initial contributions of Jean-Baptiste Dakeyo and Geoffroy Thibault, as well as the further collaboration of Antoine Caillon and Martin Fouilleul.

\bibliographystyle{IEEEbib}

Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)


Open-charm meson elliptic flow measurement in Pb-Pb collisions at sqrt(sNN) = 2.76 TeV with ALICE at the LHC

Abstract: A Large Ion Collider Experiment (ALICE) is one of the four large experiments at the Large Hadron Collider (LHC), and it is dedicated to the study of ultra-relativistic heavy-ion collisions, with the goal of investigating the properties of the high-density state of QCD matter produced in these collisions.The study of D meson production azimuthal anisotropy and the measurement of their elliptic flow (v2) can provide insight on the degree of thermalisation of charm quarks in the medium and on the charm hadronization mechanism.We present the measurement of the D+ and D0 meson v2 in Pb-Pb collisions at sqrt(sNN)=2.76 TeV at the LHC with ALICE. We discuss the details of the analysis and we show the results obtained from data samples collected in 2011.

\section{Introduction}
\label{sec:intro}
At high energy densities, Quantum--Chromo--Dynamics predicts the formation of a state of matter in which partons are not confined anymore inside hadrons. This state of matter is called Quark Gluon Plasma (QGP). The ALICE experiment \cite{alice} is one of the four large experiments at the Large Hadron Collider (LHC) and its main experimental goal is the study and characterisation of the QGP created in the high energy Pb-Pb collisions at $\sqrt{s_{NN}} = 2.76$~TeV delivered by the LHC.
In non-central Pb--Pb collisions the fireball shows azimuthal anisotropy with respect to the reaction plane (the plane defined by the beam direction and the impact parameter). Due to collective effects, this spatial anisotropy is translated into a momentum anisotropy driven by different pressure gradients in the in-plane and out-of-plane regions. This anisotropy can be quantified expanding the particles momentum azimuthal distribution (with respect to the event plane) in a Fourier series. At mid-rapidity the main coefficient is the second coefficient of the Fourier expansion $v_2$, also called \textit{elliptic flow} \cite{v2}.
The measurement of heavy--flavour particles $v_2$ is particularly interesting as it can probe the degree of thermalisation of the medium, the interaction of heavy quarks with the QGP and the heavy flavours hadronization mechanism \cite{rhicflowRapp,Molnar:2004ph}. The charm $v_2$ has been measured at RHIC by the PHENIX collaboration via non-photonic electrons \cite{phenixcraa}. In ALICE, Vertexing and tracking of charged particle at central rapidity ($|\eta|<0.8$) relies on the six layers of high resolution silicon detectors of the Inner Tracking System (ITS), with the two innermost layers equipped with Silicon Pixel Detecetors (SPD), a large volume Time Projection Chamber (TPC), and a high granularity Transition-Radiation Detector (TRD). Thanks to its vertex reconstruction, tracking and particle identification capabilities, ALICE can perform a direct charm $v_2$  measurement through fully reconstructed D meson hadronic decays. In these proceedings we present the result on the measurement of D meson $v_2$ from $\sim9.5\cdot10^6$ ($\sim7.1\cdot10^6$) Pb-Pb collisions in the centrality class 30-50\% (15-30\%).

\section{Event selection and event plane determination}
The data sample used for the D meson $v_2$ analysis has been collected with a minimum bias trigger given by the coincidence of signal in the VZERO scintillators and the SPD, and a semi-central trigger tuned to select events with centrality 0-50\%.
The centrality selection of the events is done on the distribution of signal in the VZERO scintillators. A Glauber fit is performed on MonteCarlo simulations to extract the total cross section and normalise \cite{alberica}. The event plane is measured from the distribution of charged tracks in the $0<\eta<0.8$ region using the equation
\begin{equation}
\Psi =\frac{1}{2} \tan^{-1}\left(\frac{\sum_{i=0}^{N} w_i \sin 2\phi_i}{\sum_{i=0}^{N} w_i \cos 2\phi_i}\right)
\end{equation}
where $\Psi$ is the second harmonic event plane and $\phi_i$ is the angle of the $i^\text{th}$ track in the ALICE reference frame \cite{poskEP}. Weights are applied to the tracks depending on their angle in the ALICE reference frame to account for different efficiencies among TPC sectors. To compute the event plane resolution each event is splitted in 2 sub-events with each track in the $0<\eta<0.8$ range randomly assigned to one of the sub-events and the sub-event plane angle $\Psi_{a/b}$ is computed. From the two sub-events plane angles the resolution is computed following the prescriptions in \cite{poskEP}. The resulting event plane is flat to about 2\% level and its resolution is 0.86 in the centrality class 30-50\% and 0.9 in 15-30\% centrality.

\section{D meson reconstruction and $v_2$ extraction}
Open charm hadrons reconstruction at ALICE is based on the invariant mass analysis of fully reconstructed hadronic decay topologies. The D meson $v_2$ has been measured for the decay channels $\text{D}^+\rightarrow K^-\pi^+\pi^+$, $\text{D}^0\rightarrow K^-\pi^+$ and the $\text{D}^{*+}\rightarrow \text{D}^0\pi^+$ analysis is ongoing. As the mean proper decay length c$\tau$ of these particles is of the order of $\sim150$ (D\textsuperscript{0})$-300$ (D\textsuperscript{+})$\mu m$, the strategy to reduce the large combinatorial background is based on the selection of displaced-vertex topologies, i.e. with
large separation between the primary and the secondary vertices and on the good alignment between the reconstructed D meson momentum and the flight line.
Particle identification is provided by time of flight measurements in the Time Of Flight (TOF) detector and by specific energy deposit in the TPC. The PID selection strategy was defined with the aim of preserving all the signal. Further analysis details can be found in \cite{Draa}.
The invariant mass distribution of the reconstructed D meson candidates is splitted in the in-plane and out-of-plane regions defined as the regions $\left[0<\Delta\phi<\frac{\pi}{4}\right)\bigcup \left[\frac{3\pi}{4}<\Delta\phi<\pi\right)$ and $\left[\frac{\pi}{4}<\Delta\phi<\frac{3\pi}{4}\right)$ respectively,
where $\Delta\phi$ is the azimuthal angle of the reconstructed candidate with respect to the event plane.
An invariant mass analysis is used to extract the amount of signal candidates measured in the two regions ($N_\text{IN}$, $N_\text{OUT}$). A Gaussian distribution is assumed for the signal shape. In each $p_\text{T}$ bin the width of the Gaussian is fixed to the value obtained fitting the $\Delta\phi$ integrated invariant mass distribution. From the values of $N_\text{IN}$ and $N_\text{OUT}$ it is possible to directly compute the elliptic flow as
\begin{equation}
 v_2=\dfrac{\pi}{4}\dfrac{N_{IN} - N_{OUT}}{N_{IN} + N_{OUT}}.
\label{eq:v2}
\end{equation}
Other methods to extract $v_2$ based on 2-particles correlations \cite{qcumulants} have been implemented. Their results have been found in agreement with the $v_2$ obtained from equation \ref{eq:v2}.
\section{Results}
\begin{figure}[htb]
\begin{minipage}[t]{.5\textwidth}
\centering
\captionsetup{margin={.05\textwidth, 3pt}}
\includegraphics[width=\textwidth]{2012-May-18-v2D0Dplus-centr3050-EP-withHadrons.eps}\\
\caption{$\text{D}^0$ (blue circles) and $\text{D}^+$ (grey crosses) $v_2$ measured by ALICE in the 30-50\% centrality class. The empty boxes show the systematic uncertainties from data, the shaded area the uncertainty on the B feed-down. Black points are charged hadron $v_2$.}\label{fig:d0dplus}
\end{minipage}
\begin{minipage}[t]{.5\textwidth}
\centering
\captionsetup{margin={3pt, .05\textwidth}}
\includegraphics[width=\textwidth]{2012-May-18-v2D0-centr3050vs1530-EP.eps}\\
\caption{$\text{D}^0$ $v_2$ measured by ALICE in the 30-50\% (blue circles) and 15-30\% (red squares) centrality classes. The empty boxes show the systematic uncertainties from data, the shaded area the uncertainty on the B feed-down.}\label{fig:centdep}
\end{minipage}
\end{figure}
\subsection{Systematic uncertainties}
Several sources of systematic uncertainties have been considered in the analysis. The main contributions to the total systematic uncertainty come from uncertainties on the yield extraction and from the topological cut selection. The systematic uncertainty on the yield extraction is estimated using different background functions in the fitting procedure and allowing the Gaussian width to vary between in-plane and out-plane invariant mass distributions. The analysis was also repeated using three different sets of topological cuts. Yield extraction and cut variation gave a combined total systematic uncertainty of absolute value 0.02-0.05 each (depending on the channel and on $p_\text{T}$) for the $\text{D}^0$ analysis. Other minor sources of systematic uncertainties are the centrality dependence of the resolution in the centrality range considered, that introduces a systematic uncertainty of $\sim$3\% and a 7\% asymmetric systematic uncertainty towards larger values of $v_2$ that comes from the definition of the sub-events.
\subsection{B feed-down subtraction}
The signal contains a fraction of D meson coming from B decays. Thus the measured elliptic flow $v_2^\text{obs}$ is a combination of the elliptic flows of prompt D ($v_2^\text{prompt}$) and D from B feed-down ($v_2^\text{feed-down}$) mesons. It can be expressed as $v_2^\text{obs}=f_\text{prompt}v_2^\text{prompt}+(1-f_\text{prompt})v_2^\text{feed-down}$, where $f_\text{prompt}$ is the fraction of prompt D meson in the sample. From MonteCarlo simulations and FONLL \cite{FONLL98} predictions we estimated $f_\text{prompt}$ to be in the range 0.7-0.95 depending on channel and $p_\text{T}$, as well as on the topological cuts applied. The relative $R_{\text{AA}}(p_{\text{t}})=\frac{1}{<T_{\text{AA}}>}\frac{dN_{\text{AA}}/dp_{\text{t}}}{d\sigma_{pp}/dp_{\text{t}}}$ suppression of feed-down and prompt candidates also affects the value of $f_\text{prompt}$. The effect was estimated by computing the prompt fraction varying this ratio in the range $0.5<R_{AA}^\text{prompt}/R_{AA}^\text{feed-down}<2$. Most of the
models predict $0\leq v_2^\text{feed-down}\leq v_2^\text{prompt}$ due to the larger mass of the b quark. For our measurement we make the assumption that $v_2^\text{feed-down}=v_2^\text{prompt}$. This assumption introduces an asymmetric systematic uncertainty on our measurement towards higher $v_2$ values. The maximum value of this uncertainty is given by $v_2^\text{prompt}/f_\text{prompt}$, that corresponds to the limit in which $v_2^\text{feed-down}=0$.
\subsection{D meson elliptic flow}
The D meson elliptic flow was measured at ALICE for the $\text{D}^0\rightarrow K^-\pi^+$ and $\text{D}^+\rightarrow K^-\pi^+\pi^+$ channels in the centrality class 30-50\%. The results are shown in figure \ref{fig:d0dplus} for several $p_\text{T}$ bins in the range $2<p_\text{T}<16$~GeV/$c$. In the range where also $\text{D}^+$ measurement is available ($3<p_\text{T}<8$~GeV/$c$), the two measurements are compatible within statistical uncertainties and both are compatible with the ALICE measurement of charged hadron $v_2$ \cite{alicev2} in the same rapidity region. The $\text{D}^0$ $v_2$ measurement was also carried out in the centrality class 15-30\% for $2<p_\text{T}<16$~GeV/$c$. The result is shown in figure \ref{fig:centdep} compared to the result in 30-50\%. Our measurement has been compared to theoretical predictions \cite{aichelin,beraudo,whdg,bamps}, shown in figure \ref{fig:models} (left). The same models have been compared with the D meson $R_{AA}$ measurement from ALICE (figure \ref{fig:models},
right).
\begin{figure}[htb]
\begin{minipage}[b]{.5\textwidth}
\centering
\captionsetup{margin={.05\textwidth, 3pt}}
\includegraphics[width=\textwidth]{2012-Jul-09-v2D0-centr3050-EP-withModel.eps}\\
\end{minipage}
\begin{minipage}[b]{.5\textwidth}
\centering
\captionsetup{margin={3pt, .05\textwidth}}
\includegraphics[width=.9\textwidth]{2012-Jun-01-RAADwithModels_cmpv2_4models.eps}\\
\end{minipage}
\caption{Left: $\text{D}^0$ $v_2$ measured by ALICE in the 30-50\% centrality class (blue circles). The empty boxes show the systematic uncertainties from data, the shaded area the uncertainty on the B feed-down contribution. Right: D meson $R_{AA}$ measurement \cite{Draa} (black). The two measurements are compared to theoretical models \cite{aichelin,beraudo,whdg,bamps}.}\label{fig:models}
\end{figure}

\section{Conclusions}
In this paper we presented a measurement of the D meson $v_2$ in Pb-Pb collisions at $\sqrt{s_\text{NN}} = 2.76$~TeV collected by the ALICE experiment. The results indicate a non-zero $v_2$ with 3$\sigma$ significance in the $p_\text{T}$ range $2<p_\text{T}<6$~GeV/$c$. The charm $v_2$ is also compatible, within uncertainties, with the $v_2$ of charged hadrons in the same $p_\text{T}$ and rapidity region.
The data collected in the centrality classes 15-30\% and 30-50\% suggest a decrease of the charm $v_2$ with the collision centrality.
Models of heavy quark transport in the medium can describe the data. However, a simultaneous description of D meson $v_2$ and $R_{AA}$ is still lacking.

\bibliographystyle{elsarticle-num}


A Simple Guide to S3 Methods

Abstract: Writing functions in R is an important skill for anyone using R. S3 methods allow for functions to be generalised across different classes and are easy to implement. Whilst many R users are be adept at creating their own functions, it seems that there is room for many more to take advantage of R's S3 methods. This paper provides a simple and targeted guide to explain what S3 methods are, why people should them, and how they can do it.

\section{Introduction}\label{introduction}

A standard principle of programming is DRY - Don't Repeat Yourself.
Under this axiom, the copying and pasting of the same or similar code
(copypasta), is avoided and instead replaced with a function, macro, or
similar. Having one function to replace several of the same or similar
coded sections simplifies code maintenance as it means that only one
section of code needs to be maintained, instead of several. This means
that if the code breaks, then one simply needs to update the function,
rather than finding all of the coded sections that are now broken.

S3 methods in the R programming language are a way of writing functions
in R that do different things for objects of different classes. S3
methods are so named as the methods shipped with the release of the
third version of the ``S'' programming language, which R was heavily
based upon (Chambers and Hastie 1992, Team (2016), R Core Team (2012)).
Hence, methods for S 3.0 = S3 Methods.

The function \texttt{summary()} is an S3 method. When applied to an
object of class \texttt{data.frame}, \texttt{summary} shows descriptive
statistics (Mean, SD, etc.) for each variable. For example,
\texttt{iris} is of class \texttt{data.frame}:

\begin{Schunk}
\begin{Sinput}
class(iris)
\end{Sinput}
\begin{Soutput}
#> [1] "data.frame"
\end{Soutput}
\end{Schunk}

So applying \texttt{summary} to \texttt{iris} gives us summary
information relevant to a dataframe

\begin{Schunk}
\begin{Sinput}
summary(iris)
\end{Sinput}
\begin{Soutput}
#>   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width
#>  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100
#>  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300
#>  Median :5.800   Median :3.000   Median :4.350   Median :1.300
#>  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199
#>  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800
#>  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500
#>        Species
#>  setosa    :50
#>  versicolor:50
#>  virginica :50
#>
#>
#>
\end{Soutput}
\end{Schunk}

\texttt{summary} also performs differently when applied to different
object. In fact, you can find all the classes that work with an S3
method by typing the following:

\begin{Schunk}
\begin{Sinput}
methods(summary)
\end{Sinput}
\begin{Soutput}
#>  [1] summary.aov                    summary.aovlist*
#>  [3] summary.aspell*                summary.check_packages_in_dir*
#>  [5] summary.connection             summary.data.frame
#>  [7] summary.Date                   summary.default
#>  [9] summary.ecdf*                  summary.factor
#> [11] summary.glm                    summary.infl*
#> [13] summary.lm                     summary.loess*
#> [15] summary.manova                 summary.matrix
#> [17] summary.mlm*                   summary.nls*
#> [19] summary.packageStatus*         summary.PDF_Dictionary*
#> [21] summary.PDF_Stream*            summary.POSIXct
#> [23] summary.POSIXlt                summary.ppr*
#> [25] summary.prcomp*                summary.princomp*
#> [27] summary.proc_time              summary.srcfile
#> [29] summary.srcref                 summary.stepfun
#> [31] summary.stl*                   summary.table
#> [33] summary.tukeysmooth*
#> see '?methods' for accessing help and source code
\end{Soutput}
\end{Schunk}

There's over 30 different methods!

We can use summary on a linear model, for example:

\begin{Schunk}
\begin{Sinput}
lm_iris <- lm(Sepal.Length ~ Sepal.Width, data = iris)

summary(lm_iris)
\end{Sinput}
\begin{Soutput}
#>
#> Call:
#> lm(formula = Sepal.Length ~ Sepal.Width, data = iris)
#>
#> Residuals:
#>     Min      1Q  Median      3Q     Max
#> -1.5561 -0.6333 -0.1120  0.5579  2.2226
#>
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)
#> (Intercept)   6.5262     0.4789   13.63   <2e-16 ***
#> Sepal.Width  -0.2234     0.1551   -1.44    0.152
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.8251 on 148 degrees of freedom
#> Multiple R-squared:  0.01382,    Adjusted R-squared:  0.007159
#> F-statistic: 2.074 on 1 and 148 DF,  p-value: 0.1519
\end{Soutput}
\end{Schunk}

\texttt{summary} produces a description of the linear model, describing
how it was called (\texttt{call}), as well as the \texttt{residuals},
\texttt{coefficients}, \texttt{t-values}, \texttt{p-values}, \(R^2\),
and more. This output is \textbf{completely} different to the
information output from \texttt{summary} used for the \texttt{iris}
dataframe.

So how does the same function, \texttt{summary} perform differently for
different objects? The answer is that R is helpful, and \emph{hides}
this information. There are in fact, many different \texttt{summary}
functions. For example:

\begin{itemize}
\tightlist
\item
  \texttt{summary.lm}
\item
  \texttt{summary.data.frame}
\item
  \texttt{summary.Date}
\item
  \texttt{summary.matrix}
\end{itemize}

Being an S3 method, \texttt{summary} calls the appropriate function
based upon the class of the object it operates on. So using
\texttt{summary} on an object of class ``Date'' will evoke the function,
\texttt{summary.Date}. \textbf{But all you need to do is type
\texttt{summary}}, and the S3 method does the rest. By abstracting away
this detail (the object class), the intent becomes clearer.

To further illustrate, using \texttt{summary} on the \texttt{iris} data
will actually call the function \texttt{summary.data.frame}, since
\texttt{iris} is of class \texttt{data.frame}. We can find the class of
an object using \texttt{class}

\begin{Schunk}
\begin{Sinput}
class(iris)
\end{Sinput}
\begin{Soutput}
#> [1] "data.frame"
\end{Soutput}
\end{Schunk}

\begin{Schunk}
\begin{Sinput}
summary.data.frame(iris)
\end{Sinput}
\begin{Soutput}
#>   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width
#>  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100
#>  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300
#>  Median :5.800   Median :3.000   Median :4.350   Median :1.300
#>  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199
#>  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800
#>  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500
#>        Species
#>  setosa    :50
#>  versicolor:50
#>  virginica :50
#>
#>
#>
\end{Soutput}
\end{Schunk}

which is the same as \texttt{summary(iris)}

\begin{Schunk}
\begin{Sinput}
sum1_df <- summary.data.frame(iris)

sum2_df <- summary(iris)

all.equal(sum1_df, sum2_df)
\end{Sinput}
\begin{Soutput}
#> [1] TRUE
\end{Soutput}
\end{Schunk}

And using summary on the linear model object, \texttt{lm\_iris}
performs:

\begin{Schunk}
\begin{Sinput}
summary.lm(lm_iris)
\end{Sinput}
\begin{Soutput}
#>
#> Call:
#> lm(formula = Sepal.Length ~ Sepal.Width, data = iris)
#>
#> Residuals:
#>     Min      1Q  Median      3Q     Max
#> -1.5561 -0.6333 -0.1120  0.5579  2.2226
#>
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)
#> (Intercept)   6.5262     0.4789   13.63   <2e-16 ***
#> Sepal.Width  -0.2234     0.1551   -1.44    0.152
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.8251 on 148 degrees of freedom
#> Multiple R-squared:  0.01382,    Adjusted R-squared:  0.007159
#> F-statistic: 2.074 on 1 and 148 DF,  p-value: 0.1519
\end{Soutput}
\end{Schunk}

the same as \texttt{summary(lm\_iris)}

\begin{Schunk}
\begin{Sinput}
sum1_lm <- summary.lm(lm_iris)

sum2_lm <- summary(lm_iris)

all.equal(sum1_lm, sum2_lm)
\end{Sinput}
\begin{Soutput}
#> [1] TRUE
\end{Soutput}
\end{Schunk}

One could coerce a different method upon a different class, for example
using \texttt{summary.data.frame} on an ``lm'' object:

\begin{Schunk}
\begin{Sinput}
summary.data.frame(lm_iris)
\end{Sinput}
\begin{Soutput}
#>   coefficients       residuals          effects               rank
#>  Min.   :-0.2234   Min.   :-1.5561   Min.   :-71.56593   Min.   :2
#>  1st Qu.: 1.4640   1st Qu.:-0.6333   1st Qu.: -0.65192   1st Qu.:2
#>  Median : 3.1514   Median :-0.1120   Median : -0.00897   Median :2
#>  Mean   : 3.1514   Mean   : 0.0000   Mean   : -0.42040   Mean   :2
#>  3rd Qu.: 4.8388   3rd Qu.: 0.5579   3rd Qu.:  0.61051   3rd Qu.:2
#>  Max.   : 6.5262   Max.   : 2.2225   Max.   :  2.15225   Max.   :2
#>  fitted.values       assign     qr.Length  qr.Class  qr.Mode  df.residual
#>  Min.   :5.543   Min.   :0.00   300      -none-   numeric    Min.   :148
#>  1st Qu.:5.789   1st Qu.:0.25     2      -none-   numeric    1st Qu.:148
#>  Median :5.856   Median :0.50     2      -none-   numeric    Median :148
#>  Mean   :5.843   Mean   :0.50     1      -none-   numeric    Mean   :148
#>  3rd Qu.:5.901   3rd Qu.:0.75     1      -none-   numeric    3rd Qu.:148
#>  Max.   :6.080   Max.   :1.00                                Max.   :148
#>    xlevels         call         terms
#>  Length:0      Length:3      Length:3
#>  Class :list   Class :call   Class1:terms
#>  Mode  :list   Mode  :call   Class2:formula
#>                              Mode  :call
#>
#>
#>  model.Sepal.Length  model.Sepal.Width
#>  Min.   :4.300000    Min.   :2.000000
#>  1st Qu.:5.100000    1st Qu.:2.800000
#>  Median :5.800000    Median :3.000000
#>  Mean   :5.843333    Mean   :3.057333
#>  3rd Qu.:6.400000    3rd Qu.:3.300000
#>  Max.   :7.900000    Max.   :4.400000
\end{Soutput}
\end{Schunk}

However the output may be a bit confusing.

To summarize, the important feature of S3 methods worth noting is that
only the \textbf{first part}, \texttt{summary}, is required to be used
on these objects of different classes.

\section{Why hide the text?}\label{why-hide-the-text}

Hiding the trailing text after the \texttt{.} avoids the need to use a
different \texttt{summary} function for every class. This means that one
does not need to remember to use \texttt{summary.lm} for linear models,
or \texttt{summary.data.frame} for data frames, or
\texttt{summary.aProposterousClassOfObject}. By using S3 methods,
cognitive load is reduced - you don't have to think as much to remember
what class an object is - and the commands are more intuitive. To get a
summary of most objects, use \texttt{summary}, to plot most objects, use
\texttt{plot}. Perhaps the most nifty feature of all is that a user can
create their own S3 methods using the same functions such as
\texttt{summary} and \texttt{plot}. This means a user can create their
own special class of object

\begin{Schunk}
\begin{Sinput}
test_class <- 1:10

class(test_class) <- "myclass"

class(test_class)
\end{Sinput}
\begin{Soutput}
#> [1] "myclass"
\end{Soutput}
\end{Schunk}

and then write their own S3 method for it - e.g.,
\texttt{summary.myclass} or \texttt{plot.myclass}, each proiding
appropriate summary information, or nice plots, for that object.

\section{How to make your own S3
method?}\label{how-to-make-your-own-s3-method}

Creating your own S3 method is not particularly difficult and is usually
highly practical. A use case scenario for creating an S3 method is now
discussed.

The Residual Sums of Squares (RSS), \(\sum(Y_i - \hat{Y})^2\) is a
useful metric for determining model accuracy for continuous outcomes.
For example, for a Classification and Regression Tree

\begin{Schunk}
\begin{Sinput}
library(rpart)

fit.rpart <- rpart(Sepal.Width ~ Sepal.Length + Petal.Length + Petal.Width + Species, data = iris)
\end{Sinput}
\end{Schunk}

The RSS is calculated as

\begin{Schunk}
\begin{Sinput}
print_rss <- sum(residuals(fit.rpart)**2)
\end{Sinput}
\end{Schunk}

One might be inclined to write a function to perform this task

\begin{Schunk}
\begin{Sinput}
rss <- function(x){

  sum(residuals(x)**2)

}

rss(fit.rpart)
\end{Sinput}
\begin{Soutput}
#> [1] 10.17245
\end{Soutput}
\end{Schunk}

However, there are many different decision tree models that one would
like to compare, say boosted regression trees (BRT), and random forests
(RF). The same code will not work:

\begin{Schunk}
\begin{Sinput}
library(randomForest)
set.seed(71)
fit.rf <- randomForest(Sepal.Length ~ ., data=iris, importance=TRUE,
                        proximity=TRUE)

rss(fit.rf)
\end{Sinput}
\begin{Soutput}
#> [1] 0
\end{Soutput}
\end{Schunk}

In this case, one could write three functions, one for each decision
tree method: ``rss\_rpart'', ``rss\_brt'', and ``rss\_rf''. But to avoid
having three functions and instead use just one, one could place all
three functions inside of one function, using an if-then-else clause to
direct the object of the appropriate class to the appropriate method.
This shall be referred to as a ``Poor man's S3 method''.

\begin{verbatim}
dt_rss <- function (x){

if ("rpart"

  result <- sum((residuals(x)**2))

  return(result)

}

else if ("gbm"

  result <- sum(x$residuals**n2)

  return(result)

}

else if ("randomForest"

  temp <- x$y - x$predicted

  result <- sum(temp**2)

  return(result)

}

else warning(paste(class(x), "is not of an rpart, gbm, or randomForest object"))
}
\end{verbatim}

Here it is in action

\begin{verbatim}
dt_rss(fit.rpart)

#> [1] 10.17245
\end{verbatim}

The RSS method works, and if it is applied to a class that is not known,
a special message is provided

\begin{Schunk}
\begin{Sinput}
fit.lm <- lm(Sepal.Width ~ Species, data = iris)

dt_rss(fit.lm)
\end{Sinput}
\begin{Soutput}
#> Warning in dt_rss(fit.lm): lm is not of an rpart, gbm, or randomForest
#> object
\end{Soutput}
\end{Schunk}

The ``poor man's S3 method'' does what it needs to do. However, one must
ask how sustainable this would be for an entire programming language?
Imagine if a colleague creates a new tree method that needs it's own
\texttt{rss()}. He will need to convince the maintainer to add his class
into your ifelse() chain. Failing this, he could just overwrite the
function \texttt{rss()}, with predictably disastrous results. In
reality, it's probably better to do all of these things with one method.
R's S3 methods mean that R developers can utilise a common interface
without having to update it when new classes come along. It also means
overloading clashes are less likely.

So let us create an S3 method to demonstrate.

First define the S3 method with \texttt{UseMethod()}

\begin{Schunk}
\begin{Sinput}
rss <- function(x) UseMethod("rss")
\end{Sinput}
\end{Schunk}

This creates the building block of an S3 method, the ``root'', if you
will.

Here we have specified that our method will be called \texttt{rss}. Now
we need to create the special cases of rss - the methods
\texttt{rss.rpart}, \texttt{rss.gbm}, and \texttt{rss.randomForest},
where the sections of code after \texttt{rss.} are the classes of object
we want them to work on.

\begin{verbatim}

rss.rpart <- function(x){

  sum((residuals(x)**2))

}

rss.gbm <- function(x){

  sum(x$residuals**2)

}

rss.randomForest <- function(x){

  res <- x$y - x$predicted

  sum(res**2)

}
\end{verbatim}

A default method can also be created - \texttt{rss.default} - which, as
the name suggests, is the default method when the argument \texttt{x} is
not a class that has a specific version of the method defined.

\begin{Schunk}
\begin{Sinput}
rss.default <- function(x, ...){

  warning(paste("RSS does not know how to handle object of class ",
                class(x),
                "and can only be used on classes rpart, gbm, and randomForest"))

          }
\end{Sinput}
\end{Schunk}

In this case a warning is issued, to let the user know that the object
class they were using was not appropriate.

We can now apply the \texttt{rss} method to an \texttt{rpart} model

\begin{Schunk}
\begin{Sinput}
rss(fit.rpart)
\end{Sinput}
\begin{Soutput}
#> [1] 10.17245
\end{Soutput}
\end{Schunk}

Also observe what happens when the object used is not of the decision
tree classes

\begin{Schunk}
\begin{Sinput}
rss(lm.fit)
\end{Sinput}
\begin{Soutput}
#> Warning in rss.default(lm.fit): RSS does not know how to handle object of
#> class function and can only be used on classes rpart, gbm, and randomForest
\end{Soutput}
\end{Schunk}

This guide to S3 methods was written to provide R users with the minimal
amount of information to start building their own S3 methods. For a more
complete treatment on S3 methods, see Advanced-R (Wickham 2014), R
Packages (Wickham, Hadley 2015), and the official S3 documentation (Team
2016, R Core Team (2012)).

\section*{References}\label{references}
\addcontentsline{toc}{section}{References}

\hypertarget{refs}{}
\hypertarget{ref-chambers1992}{}
Chambers, John M, and Trevor J Hastie. 1992. \emph{Statistical Models in
S}. CRC Press, Inc.

\hypertarget{ref-R}{}
R Core Team. 2012. \emph{R: A Language and Environment for Statistical
Computing}. Vienna, Austria: R Foundation for Statistical Computing.
\url{http://www.R-project.org/}.

\hypertarget{ref-rclassmethods}{}
Team, R Core. 2016. \emph{R: Class Methods}. R Foundation for
Statistical Computing.
\url{https://stat.ethz.ch/R-manual/R-devel/library/base/html/UseMethod.html}.

\hypertarget{ref-wickham2014}{}
Wickham, Hadley. 2014. \emph{Advanced R}. CRC Press.

\hypertarget{ref-wickham2015}{}
Wickham, Hadley. 2015. \emph{R Packages}. O'Reilly Media, Inc.

\address{%
Nicholas Tierney\\
Queensland University of Technology\\
Level 8, Y Block, Main Drive, QUT, Brisbane, Australia\\
}
\href{mailto:nicholas.tierney@gmail.com}{\nolinkurl{nicholas.tierney@gmail.com}}

A Simple Guide to S3 Methods