Deep Neural Baselines for Computational Paralinguistics

Abstract: Detecting sleepiness from spoken language is an ambitious task, which is addressed by the Interspeech 2019 Computational Paralinguistics Challenge (ComParE). We propose an end-to-end deep learning approach to detect and classify patterns reflecting sleepiness in the human voice. Our approach is based solely on a moderately complex deep neural network architecture. It may be applied directly on the audio data without requiring any specific feature engineering, thus remaining transferable to other audio classification tasks. Nevertheless, our approach performs similar to state-of-the-art machine learning models.

\section{Introduction}
\label{sec:intro}
\emph{Computational paralinguistics} describe a research field that has been established in the past twenty years through remarkable research progress, e.g. in the area of automated affect recognition or determining illnesses from speech~\cite{schuller2013computational,grishman1986computational,kiss2016language}.
More generally, emphasis is put on the analysis of the human voice to design~\emph{affective computers} that possess empathic competencies such as recognizing, expressing, modeling, or communicating emotions~\cite{picard2003affective}.
In most paralinguistic problems, task-specific feature engineering and model tuning allows to build state-of-the-art statistical learning models, e.g. machine learning (ML) classifiers~\cite{schuller2013paralinguistics}.
Powerful software packages such as the open-source toolkits \emph{OpenSMILE}~\cite{eyben2010opensmile}, \emph{OpenXBOW}~\cite{schmitt2017openxbow} and \emph{AuDeep}~\cite{freitag2017audeep} were created to extract these relevant features from raw audio data.
However, more general approaches ease the transfer of models to familiar problem domains without prior expert knowledge about audio signal processing, while still providing adequate predictive performance.
Recent work in deep learning focuses on end-to-end modeling of affect recognition problems, i.e. training models on raw signal data without extensive data pre-processing~\cite{graves2014towards}.
Even though deep learning has revolutionized artificial intelligence (AI) research areas, e.g. computer vision and machine translation, the full potential of Deep Neural Networks (DNNs) can usually only be utilized with datasets of sufficient quality and quantity~\cite{rouast2019deep}.
The availability of public databases for affect recognition problems has especially improved through annual competitions and challenges.\\
This paper proposes an end-to-end deep learning approach for the Interspeech 2019 Computational Paralinguistics Challenge (ComParE)~\cite{schullerinterspeech}.
While avoiding task-specific feature engineering and providing an agnostic approach for modeling paralinguistic problems, it addresses the aforementioned growing demand for general approaches in computational paralinguistics and affective computing.
Our main contribution is a moderately complex depp neural network (DNN) architecture that is able to detect and classify sleepiness in human voice from the SLEEP corpus.
We show that its performance is comparable to the provided baseline which fuses three support vector machines (SVMs) trained on more than 6,000 extracted audio features from the toolkits \emph{OpenSMILE}, \emph{OpenXBOW} and \emph{AuDeep}.
Also, we perform a sanity check on its transferability.
\section{Deep Learning in Computational Paralinguistics}
\label{sec:deep-learning-in-paralinguistics}
Throughout recent years there has been a shift in affective computing from classical ML approaches towards deep learning~\cite{rouast2019deep}.
Among other areas, this also concerned computational paralinguistics.
The inherent absence of task-specific, manual feature extraction and selection in deep learning allows researchers to design complex, non-linear models.
Those either reveal useful (latent) feature embeddings (e.g., unsupervised learning with deep auto encoders~\cite{freitag2017audeep}) or learn feature representations directly from unstructured data for predictive modeling (e.g., classification of images with CNNs~\cite{krizhevsky2012imagenet}).
Generally, DNNs allow end-to-end problem modeling, where raw data (e.g. audio signal data) is fed to the model and a prediction, e.g. a class label or a continuous value, is returned.
To summarize, three major causes have enabled this shift, namely (i) increased computing capabilities and (ii) learning capacity and precision of deep learning models through wider and deeper neural networks, as well as (iii) an ever growing amount of free-to-use, labeled datasets publicly available~\cite{rouast2019deep}.\\
DNNs can be considered the state-of-the-art technology to human affect recognition software~\cite{cummins2018speech}.
To tackle the problem of diverse data representations, DNNs can be altered accordingly in order to enhance their performance.
Within the field of computational paralinguistics CNNs, as well as Recurrent Neural Networks (RNNs), are amongst the best known network architectures.
This work emphasizes the former, which are foremost established and applied numerously~\cite{krizhevsky2012imagenet,liang2015recurrent,lawrence1997face} in the field of image processing.\\
Moreover, CNNs are able to learn features representations of different abstraction levels, ranging from low level features, such as edges, to high level features, such as eyes or noses, natively~\cite{rouast2019deep,cummins2018speech}.
Those deep spatial features outperform handcrafted spatial features in most cases~\cite{rouast2019deep,cummins2018speech}.
Another major advantage of CNNs is the low amount of parameters, compared to other network types like RNNs, and hence faster computing times during model training.
CNNs can be applied in multiple dimensionalities, such as 3D, mostly used for video analysis, 2D for the aforementioned image processing tasks, as well as 1D for one dimensional raw data inputs.
Therefore, CNNs have been an attractive option to researchers in computational paralinguistics, starting to be applied as early as 2011~\cite{jaitly2011learning}.
\section{Related Work}
\label{sec:related-work}
Greeley et al. investigate the effect of sleepiness on war fighters and civilian pilots and how machines could non-intrusively detect fatigue in very noisy environments~\cite{greeley2006detecting}.
The authors report that changes in single discrete voice parameters are not sufficient to detect whether the speaker suffers from fatigue.
Therefore, they propose a more holistic approach, combining the coefficients of the cepstral transformation with an automatic speech recognition system.
This correlation-based voice metric achieved on-par results with state-of-the-art fatigue measurements.
Krajewski et al. follow a somewhat classical ML procedure towards measuring fatigue in audio recordings~\cite{krajewski2009acoustic}.
Their results are based on speech characteristics such as prosody, articulation, and speech-quality-related and reach a classification accuracy of 86.1\% on data collected in a sleep deprivation study.
Their top result was achieved utilizing an SVM.
Within the ComparE challenge 2010, Marie-José Caraty and Claide Montacié address the problem of detecting vocal fatigue based on the statement that it highly affects the work in some professions~\cite{caraty2014vocal}.
The authors build their investigations upon three experiments, being prosodic analysis, a two-class SVM classifier, and a combination of multiple phoneme-based comparison functions.
The two-class SVM model reached an unweighted accuracy of 68.2\%.
The authors state that this suggests the feasibility of vocal fatigue detection.
Recently, Cummins et al. revisited approaches and results of past years of paralinguistics challenges and outline a noticeable shift from classical ML techniques towards deep learning models~\cite{cummins2018speech}.
Accordingly, the majority of submissions made use of some sort of deep learning approach.
Participants utilized DNNs for feature representation learning, classification or the combination of both, which underlines the growing interest in DNNs.
In 2011 there has been a ComParE sub-challenge, asking participants to distinguish between \emph{sleepy} and \emph{not-sleepy}.
The best solution at the time can be considered a classical ML solution, and has not been challenged through an end-to-end deep learning approach as of 2017.
This leads to the assumption that there is a gap in research approaches concerning sleepiness detection and investigating more general, non task-specific approaches is reasonable.
\section{Paralinguistic Problem Modeling}
\label{sec:problem}
The datasets provided for the 2019 ComParE challenges -- in their original form -- are not suited specifically for deep learning due to the limited number of samples.
Thus, we experimented with pre-processing as well as augmentation techniques and tested different CNN architectures.
These steps are outlined in the following subsections and evaluated in section~\ref{sec:results}.
\subsection{SLEEP Corpus}
\label{sec:problem:sub:sleep}
The SLEEP corpus consists of $16,462$ audio samples, each with a duration of about four seconds, which are split into \emph{training}, \emph{development} and \emph{test} set.
Its audio files have sampling rates of 16 kHz with a quantization of 16 bit.
The audio samples were gathered from $915$ subjects performing different speaking tasks such as reading out given text passages.
Recordings were carried out between 6 p.m. and midnight.
Afterwards, participants and post-hoc observers had to report sleepiness on the Karolinska Sleepiness Scale (KSS), ranging from 1 (extremely alert) to 9 (very sleepy).
These ratings were averaged to build the final label per recording.
The dataset was created at the Institute of Psychophysiology, Duesseldorf, Germany and the Institute of Safety Technology, University of Wuppertal, Germany.
More detailed statistics about the dataset and the label distribution are provided in Table~\ref{table:dataset_stats} and Figure~\ref{fig:class_distribution}, respectively.
The task of the challenge is to build a regression model that is able to predict the KSS rating for an audio recording~\cite{schullerinterspeech}.
\begin{table}[th]
\caption{Number of samples for training, development and test set as well as the respective mean ($\mu$), standard deviation ($\sigma$), minimum and maximum duration of the samples (in seconds).}
\label{table:dataset_stats}
\begin{tabular}{l|l|l|l|l|l}
Dataset     & Samples    & $\mu$ & $\sigma$ & min. & max.  \\ \hline
Training    & 5,564      & 3.87 & 0.64    & 1.56 & 5.00 \\
Development & 5,328      & 3.87 & 0.65    & 1.57 & 5.00 \\
Test        & 5,570      & 3.86 & 0.63    & 1.59 & 5.00 \\
\end{tabular}
\end{table}
\begin{figure}
	\includegraphics[width=\columnwidth]{class_distribution_train_devel}
	\caption{KSS rating distribution of the training and development set.}
	\label{fig:class_distribution}
	\vspace{-3mm}
\end{figure}
\subsection{Data Pre-processing and Augmentation}
\label{sec:problem:sub:data}
In classical ML problem modeling, feature engineering and feature selection are integral parts of the data preparation process.
Hereby, the available data is reduced to relevant pieces of information, e.g. by excluding irrelevant data or modeling specific features.
Our approach omits the extraction of problem-specific features, but relies on the DNN models being capable of learning relevant feature representations themselves.
Nevertheless it is necessary to synthetically increase the training data volume to model the regression task as a deep learning problem.
The following subsections delineate pre-processing steps and augmentation techniques.
\subsubsection{Audio Pre-processing}
\label{sec:problem:sub:data:subsub:audio}
We experimented with different down-sampling rates for the provided audio files.
This is inspired by narrowband telephony only transmitting frequencies up to 4 kHz but still retaining the majority of information.
With respect to the Nyquist-Shannon sampling theorem, we choose a down-sampling to at least 8 kHz.
As a side effect, the down-sampling reduces the amount of data thus increases processing speed.
\subsubsection{Sliding Windows}
\label{sec:problem:sub:data:subsub:window}
As mentioned previously, the small volume of training data ($5,564$ samples) needs to be increased by magnitudes to suit a deep learning set up.
Therefore, we slice windows from a single audio recording in a sliding window manner.
We experimented with varying window sizes, as well as different strides, i.e. step sizes.
Consequently, we do not obtain one data point per audio sample, but get $(L-w) / s$ overlapping windows for a sample of length $L$ with window size $w$ and stride $s$.
The best results were achieved with $w=1.5s$, and $s=100ms$, that, given an exemplary sample of length $L=4s$, results in $(4s-1.5s) / 100ms = 25$ samples of the same label.
The total amount of data samples extracted from the training dataset is $134,395$ and $128,808$ from the development dataset.
\subsubsection{Data Up- and Downsampling}
\label{sec:problem:sub:data:subsub:upsampling}
As described in section~\ref{sec:problem:sub:data}, the distribution of samples is imbalanced at the extrema of the KSS.
To prevent the models from overfitting on the large corpus of samples labelled between 3 and 8, we performed up- and down-sampling of under- and overrepresented samples, respectively.
Consequently, the extracted sample windows labelled 1 or 9 (extremely alert and very sleepy) were included multiple times into the dataset, i.e. up-sampled, whereas sample windows labelled between 3 and 8 were only partly included, i.e. down-sampled.
\subsubsection{Data Augmentation}
\label{sec:problem:sub:data:subsub:augmentation}
To further increase the volume of available training data, we tested the following augmentation techniques:\\
\textbf{Reversing samples:} We flipped each sample window and included both the reversed and the  original sample in the training set.
The hypothesis was that the relevant patterns might be independent of the exact sequential structure of the sample.\\
\textbf{Background overlay:} To make the models more robust, we overlayed the training samples with background noise manually extracted from parts of training recordings where voice was not present.
Again, the original as well as the modified samples were included for training.
This is a common approach and not limited to the available training data since background noise can be recorded separately for different background settings.\\
\textbf{Noisy labelling}: With previously mentioned sliding window approach, all windows extracted from a single sample were labelled with the same KSS rating.
As the actual self-reported degree of sleepiness may not be occurrent during the whole sample, we applied noise randomly taken from a normal distribution onto the KSS ratings.
The hypothesis was that this bias might lead to more robust models, as the KSS ratings could have been ambiguous at the time of labelling (i.e. participants' self-reports compared to the post-hoc observers').
We included both the samples with the actual and those with noisy labels in the training dataset.
\subsection{Model Architecture}
\label{sec:problem:sub:model}
The following describes our DNN architecture and the mechanism necessary both aggregate the predicted labels of the sliding windows and map them back onto the original audio samples.
\subsubsection{Prediction Aggregation Mechanism}
\label{sec:problem:sub:model:subsub:pred}
In section~\ref{sec:problem:sub:data:subsub:window} we outlined the sliding window approach to increase the volume of the dataset.
As this approach leaves us with $n$ samples per audio file (depending on the file's length $L$, window size $w$, and stride $s$), a mechanism for merging the predicted labels (i.e. KSS ratings) during the inference to finally generate one label for the original audio sample is needed.
For example, if an audio file $a$ leads to $25$ windows, which were fed to the model during inference (i.e. prediction), $25$ predicted labels, one for each window, have to be merged and mapped back onto the original audio sample $a$.
We aggregated the predicted labels in two ways, by taking the mean and the median of the predictions.
Ultimately, we clipped the resulting prediction for the original audio sample into the KSS rating from 1 to 9 and performed a typecast to an integer number.
\subsubsection{Convolutional Neural Network}
\label{sec:problem:sub:model:subsub:cnn}
Figure~\ref{fig:dnn_architecture} depicts the proposed DNN architecture that is capable of learning spatial -- thus short sequential -- feature representations from raw audio data.
The DNN consists of two 1D convolutional layers, each with four filters and kernel size tree, that are connected through batch normalization and max pooling layers, each having the size of the vector.
After these convolutional blocks (Conv-Blocks), one fully connected layer with 32 neurons leads to a final dense layer with a single linear activation unit.
The regression model was trained using the mean squared error (MSE) loss and the Adam optimization algorithm with a learning rate of $0.001$ without decay.
Except for the final dense layer, ReLU activations were used.
To prevent overfitting during training, a dropout was applied after the convolutional (drop rate 0.1) and dense layers (drop rate 0.5)
Evaluated on the development dataset, the described model resulted in the best model.
However, we discuss further (hyper-) parameters in section~\ref{sec:results:sub:devel}.
\begin{figure}
	\includegraphics[width=\columnwidth]{network}
	\caption{Proposed DNN Architecture}
	\label{fig:dnn_architecture}
	\vspace{-3mm}
\end{figure}
\section{Results}
\label{sec:results}
The baselines' results are reported with Spearman’s rank correlation coefficient $ \rho $~\cite{schullerinterspeech}.
Since the actual labels of the test set are not available ex-ante, we trained our models on the training set and tested on the development set.
We then submitted the best model to the challenge and reported the score on the test set.
For the sake of interpretation, we include our loss metric MSE as well as the mean absolute error (MAE).
\subsection{Development Dataset}
\label{sec:results:sub:devel}
\begin{table}[th]
\vspace{-2mm}
\caption{Evaluation scores for MSE, MAE, and $\rho$ on the development set with different parameter combinations regarding (i) sample rate, (ii) window size, (iii) amount of convolutional blocks.
The proposed approach is varied by each one of the three parameters while the others remain fixed.
Note that the stride was fixed at $0.1 s$.
Models denoted with * were submitted to the challenge.
}
\label{tab:dev_results}
\centering
\begin{tabular}{l|r|r|r}
                                                    & MSE  & MAE  &$\rho$\\ \hline
\multicolumn{1}{l|}{Strongest Baseline}             & --   & --   & 0.27 \\ \hline
\multicolumn{1}{l|}{\makecell[l]{{Proposed Approach*}\\
(16 kHz, 1.5 s, 2 Conv-Blocks)}}                    & 4.44 & 1.72 & 0.29 \\ \hline
\multicolumn{1}{l|}{Smaller Window Size (1 s)}      & 3.96 & 1.67 & 0.28 \\
\multicolumn{1}{l|}{Smaller Sampling Rate (8 kHz)*}  & 3.89 & 1.65 & 0.26 \\
\multicolumn{1}{l|}{More Conv-Blocks (3)}           & 4.12 & 1.70 & 0.24 \\
\end{tabular}
\end{table}
To allow proper comparison of models with (hyper-) parameters, we set the batch size to $64$ and the amount of epochs to $8$ for training.
This allowed testing multiple parameter combinations without large computational and time-consuming overhead.
Table~\ref{tab:dev_results} shows the evaluation scores for MSE, MAE, and $\rho$ on the development set for different parameters and models.
The results imply that our models perform slightly better ($\rho \approx 0.29 \pm 0.03$) than the strongest baseline models ($\rho \approx 0.26$) on the development set.
Generally, we found that during training, the loss was monotonically decreasing on the training and the development set.
However, most models tend to overfit on the training set after $6$ epochs and we therefore saved the best models during the entire training process based on their loss on the development set.
The different pre-processing and augmentation techniques described in section~\ref{sec:problem:sub:data:subsub:augmentation} did not produce better results regarding $\rho$ (see figure~\ref{fig:augmentation}).
However, a combination of techniques is beyond the scope of this work as we aim to provide a generic approach rather than a tailored solution for this specific problem.
\begin{figure}
	\includegraphics[width=\columnwidth]{augmentation}
	\caption{Overview of Spearman's rank correlation coefficient ($ \rho $) of applied pre-processing and augmentation techniques compared to the best model without further processing (\textbf{None}) on the development set.
	Models denoted with * were submitted to the challenge.}
	\label{fig:augmentation}
	\vspace{-3mm}
\end{figure}
\subsection{Test Dataset}
\label{sec:results:sub:test}
After experimenting with different model (hyper-) parameters, we selected the best performing model, trained it again on the entire development set, and submitted the resulting model to the ComparE challenge.
As we only received $\rho$, MSE and MAE cannot be reported.
Note that the distribution of predicted labels is very similar to the actual distribution of labels in the development and training set (underrepresentation of 1, 2, 8, and 9).
The best provided baseline, an ensemble of three SVMs, achieved $\rho=0.343$.
To test different parameter configurations and augmentation techniques, three of the described models were submitted to the challenge and performed as follows:
\begin{enumerate}
	\item The proposed approach with smaller sampling rate (8 kHz) achieved $\rho=0.28$ on the test set.
	\item The proposed approach with noisy labels (see figure~\ref{fig:augmentation}) achieved $\rho=0.302$ on the test set.
	\item The proposed approach with default sampling rate (16 kHz) and without augmentation achieved $\rho=0.335$ on the test set.
\end{enumerate}
The two left submissions out of five possible submissions were reserved for experimentation with other approaches within the challenge.
\subsection{Transferability}
\label{sec:results:sub:transferability}
In order to evaluate the transferability of our DNN architecture, a similar model was trained to classify Styrian Dialects within the respective ComParE sub-challenge.
However, instead of solving a regression problem, the final single linear activation unit was replaced with a three-unit softmax activated dense layer.
On the development set, our best model achieved an Unweighted Average Recall (UAR) of $44.00\%$.
The same model scored an UAR of $38.28\%$ on the test set in our submission to the challenge.
Compared the three strongest baseline models with an average of $40.00\%$ on the test set, we claim that transferring our DNN architecture to other audio classification tasks is feasible.
\section{Conclusion}
\label{sec:conclusion}
This paper presented an end-to-end deep learning approach to detect and classify sleepiness in the human voice.
The proposed 1D convolutional DNN is capable of learning spatio-temporal feature representations from raw audio data.
Its performance is comparable with models trained on current audio feature extraction toolkits.
Moreover, we performed a sanity check on the transferability of our approach.
The results indicate that our DNN architecture may be used as a problem-agnostic and straightforward baseline in addition to classical ML approaches.
The end-to-end method emphasizes generalizability and transferability to other domains, e.g. in computational paralinguistics, contrary to problem-specific feature engineering.
Our proposed architecture is especially suitable for context aware multimedia recommendation systems.
In a possible use-case, the system could recommend e.g. radio stations or songs dependent on the fatigue level, detected in the user's voice.
Future work could deepen the extent of architecture search and parameter tuning as other automatic ML approaches suggest, to ultimately further democratize AI research~\cite{zoph2018learning}.

\paragraph*{Acknowledgements.}
The HRADIO project and thus this work was funded by H2020, the EU Framework Programme for Research and Innovation.

\bibliographystyle{IEEEtran}

\newpage

Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)


Local Music Event Recommendation with Long Tail Artists

Abstract: In this paper, we explore the task of local music event recommendation. Many local artists tend to be obscure long-tail artists with a small digital footprint. That is, it can be hard to find social tag and artist similarity information for many of the artists who are playing shows in the local music community. To address this problem, we explore using Latent Semantic Analysis (LSA) to embed artists and tags into a latent feature space and examine how well artists with small digital footprints are represented in this space. We find that only a relatively small digital footprint is needed to effectively model artist similarity. We also introduce the concept of a Music Event Graph as a data structure that makes it easy and efficient to recommend events based on user-selected genre tags and popular artists. Finally, we conduct a small user study to explore the feasibility of our proposed system for event recommendation.

\section{Introduction}\label{sec:introduction}

Suppose that you move to a new city and are interested in exploring the local music scene. Typically, you might pick up the arts section of the local newspaper or go online to find a community notice board. Either way you would likely come to a long listing of music events where each event description would provide a small amount of contextual information: the names of the artists, the name and location of the venue, the date and start time of the event, the price of the tickets, and perhaps a few genre labels or a sentence fragment that reflects the kind of music you would expert to hear at the event.

While this ``public list of events'' model has been successful at getting fans to music events for many decades, we can use modern recommender systems to make music event discovery more efficient and effective. For example, companies like BandsInTown\footnote{https://www.bandsintown.com} and SongKick\footnote{https://www.songkick.com/} help users \emph{track} artists so that that the user can be notified when a favorite artist will be playing nearby. They also recommend upcoming events with artists who are similar to one or more of the artists that the user has selected to track. These services have been successful in growing both the number of users and in the number of artists and events covered by their service. For example, BandsInTown claims to have 38 million users and lists events for over 430,000 artists\footnote{According to https://en.wikipedia.org/wiki/Bandsintown on March 28, 2018.}. Event listings are added by aggregating information of ticket sellers (e.g., Ticketmaster\footnote{https://www.ticketmaster.com/}, TicketFly\footnote{https://www.ticketfly.com/}) and by artist managers and booking agents who have the ability to directly upload tour dates for their touring artists to these services.

While this coverage is impressive, a large percentage of the events found in local newspapers are not listed on these commercial music event recommendation services. Many talented artists play at small venues (e.g., neighborhood pubs, coffee shops, and DIY shows) and are often not represented by (pro-active, tech-savvy) managers. Yet many music fans enjoy the intimacy of a small venue and a personal connection with local artists and may have a hard time discovering these events.

As such, our goal is to develop a locally-focused music event recommendation system to help foster music discovery within a local music community. Here we define \emph{local} as all music events within a small geographic region (e.g., 10 square miles). This includes national and regional touring acts who may pass through town but it also includes non-touring artists (e.g.,  a high school punk band, a barber shop quartet, a jazz trio from the nearby music conservatory, or a neighborhood hip hop collective.)

What makes this problem technically challenging is that a large percentage of our local artists have a small \emph{digital footprint} or no digital footprint at all. That is, we may not be able to find these artists on sites that typically provide useful music information \cite{turnbull2008five} (e.g., Spotify\footnote{https://developer.spotify.com/}, Last.fm\footnote{https://www.last.fm/api}, AllMusic\footnote{https://www.allmusic.com/}). Similarly, we often do not have music recordings from these artists so we will not be able to make use of content-based methods for automatic tagging \cite{turnbull2008semantic} or acoustic similarity\cite{mcfee2012learning}. Rather, we will rely the small amount of contextual information that can be scraped from the event listings in the local newspaper or community notice board.

We will first introduce the concept of a \emph{Music Event Graph} as a 4-partite graph that connects genre tags to popular artists to event artists to events. We then use latent semantic analysis (LSA) \cite{deerwester1990indexing} to embedding tags and artists into a latent feature space. We show that LSA is particularly advantageous when considering new or not well-known (long-tail) artists who have small digital footprints. This approach also allows us to independently control the \emph{popularity bias}\cite{turnbull2008five} of our event recommendation algorithm so that events with popular artists are no more or less likely to be recommended than events featuring more obscure local artists.

\section{Related Work}\label{sec:related-work}

We have been unable to find previous research on the specific task of music event recommendation though there is a significant amount of work on both music recommendation \cite{celma2010, schedl2017} (i.e., recommending songs and artists) and event recommendation \cite{macedo15,dooms11, minkov2010} (i.e., events posted on social networking sites.) It both cases, it is common to explore content-based (i.e., the substance of an item), collaborative filtering-based (e.g., usage patterns from a large group of users), and hybrid approaches. We  consider our approach to be a hybrid approach since we make use of both social tags (content) and artist similarity (collaborative filtering\footnote{While the details of the Last.fm algorithm for computing artist similarity remain a corporate trade secret, it would be reasonable to expect that these scores are computed using some form of collaborative filtering based on the large quantities of user listening histories that they collect \cite{barrington2009}.}) scores from Last.fm.

As with many successful recommender systems, we make use of matrix factorization to embed data into a low dimensional space \cite{koren2009}. In particular, we use Latent Semantic Analysis (LSA) \cite{deerwester1990indexing} which is a common approach used in both text information retrieval \cite{manning2008} and music information retrieval systems (e.g., \cite{levy2007semantic, laurier2009, oramas2015semantic}). LSA is relatively easy to implement\footnote{For example, see http://scikit-learn.org/stable/modules/generated/ sklearn.decomposition.TruncatedSVD.html}, can improve recommendation accuracy, provides a compact representation of the data, works well with sparse input data, and can help alleviate problems caused by synonymy and polysemy \cite{manning2008}. We should note that other embedding techniques, such as probabilistic LSA \cite{hofmann1999probabilistic} and latent Dirichlet allocation (LDA) \cite{blei2003latent}, could also be used as an alternative to LSA.

\section{Event Recommendation}

When developing an event recommendation system, we will consider an interactive experience with three steps:

\begin{enumerate}
\item \textbf{User selects genre tags}: Ask the user to select one or more tags from a list of board genres (``rock'', ``hip hop'', ``reggae'') based on the most common genres of the artists who are playing at upcoming local events.
\item  \textbf{User selects preferred popular artists}: Ask the user to select one or more artists from a list of recognizable mainstream artists (The Beatles, Jay-Z, Bob Marley) based on the selected genres and related to the artists who are playing an upcoming event.
\item \textbf{Display of recommended event list}: Show recommended events (with justification) to the user based on the the selected genre tag and popular artist preferences.
\end{enumerate}
This is a common \emph{onboarding} process for both commercial music event services (e.g., BandsInTown) and music streaming services (e.g., Apple Music) since it quickly gives recommender systems a small but sufficient amount of music preference information for new users. After onboarding, a user can drill down into specific artists or events, as well as  listen to related music, explore a map of venues, etc.

In this section, we describe the concept of a Music Event Graph and show how we can use it to efficiently recommend local music events based on the music preference information that is collected during user onboarding.

\subsection{Music Event Graphs}

When considering event recommendation, there are two phases that we need to consider: offline computation of relevance information for all upcoming events and real-time personalized event recommendation. We will use a Music Event Graph to help us structure our event recommendation system. The music event graph is a k-partite graph with $k=4$ levels. Our four levels represent common genre tags, popular artists, event artists, and events as is shown in Figure \ref{fig:event-graph-creation}

\begin{figure}
\centering
  \includegraphics[width=3.3in]{CreatingTheEventGraph.png}
  \caption{Construction a Music Event Graph:  First we collect \emph{events} and the \emph{event artists} who will be performing at these events (right nodes and edges). Then, we select \emph{genre tags} and related \emph{popular artists} (left nodes and edges). Finally, we connect event artists to popular artists based on our artist similarity calculation (middle edges).}
\label{fig:event-graph-creation}
\end{figure}

To construct the graph, we follow the following steps:
\begin{enumerate}
\item Collect a set of upcoming local \emph{events}
\item Construct the set of \emph{event artists} from all of the local events
\item  Find the most frequently used \emph{genre tags} (e.g., ``rock'', ``jazz'', ``hip hop'') associated with the event artists.
\item Using the genre tags, create a set of \emph{popular artists} by selecting the most well-known artists that are strongly associated with each genre.
\item For each event artist, find the most similar artists from the set of popular artists.
\end{enumerate}
In Section \ref{sec:artist-similarity}, we will describe how we use harvested tags and artist similarity information to compute similarity between pairs of artists, as well as between artists and tags. These similarities are represented as real-valued weights, and as such, the event graph contains weighted edges.

Based on the interactive design described above, we can efficiently recommend events using a Music Event Graph. The user selects one or more preferred genres and then a set of relevant popular artists. Next our algorithm selects the event artists and their related events that are connected to the user's selected genres and popular artists. This graph traversal algorithm is depicted in Figure \ref{fig:event-graph-recommendation}.
\begin{figure}
\centering
  \includegraphics[width=3.3in]{EventGraphRecommendation.png}
  \caption{Construction the Music Event Graph: a user selects genre tags $\{t_1,t_3\}$. She is then shown popular artists $\{pa_1, pa_2, pa_5, pa_6\}$ and selects $\{pa_2, pa_6\}$. We then use the graph to  strongly recommend event $e_4$ with artists $ea_3$ and $ea_5$ based on multiple connections. We would also recommend events $e_1$ and $e_2$ based on their connections through $ea_1$ and $ea_3$, respectively. }
\label{fig:event-graph-recommendation}
\end{figure}

We note that our algorithm uses weighted edges to compute a user-specific relevance scores for each event as we move from left to right in the graph structure. In addition, we can use the graph structure to provide \emph{recommendation transparency} \cite{sinha2002role} by keeping track of the paths that are used to get from the user genre and popular artist selections to the recommend event artists and events.

\section{Artist Similarity and Tag Affinity}\label{sec:artist-similarity}

At the core of the event recommendation system, we use Latent Semantic Analysis (LSA) when calculating artist similarity and artist-tag affinity. That is, we use truncated single value decomposition (SVD) to transform a large, sparse data matrix of artist similarity and tag information into a lower dimensional matrix such that each artist and tag is embedded into a dense, k-dimensional \emph{latent} feature space. Note that $k$ is a hyperparameter that is set based on empirical evaluation. We can then calculate artist-artist or artist-tag similarity using the cosine distance between pairs of vectors in this latent space.

Before we describe LSA, we will start with some useful notation for our problem setup:
\begin{itemize}

\item [$\mathcal{A}$]: set of artists.
\item [$\mathcal{T}$]: set of tags. Tags are any free text token that can be used to describe music. This may include genres, emotions, instruments, usages, etc.
\item [$\mathcal{T}^G$]: a small subset of \emph{genre} tags (e.g., ``rock'', ``country'', ``blues'') that are frequently used to categorize music.
\item [$\mathcal{A}^P$]: set of \emph{popular} artists where each artist in the set is none of the most recognizable artists associated with at least one of the genre tags $\mathcal{T}^G$.
\item [$\mathcal{E}$]: set of local music events
\item [$\mathcal{A}^{\mathcal{E}}$]: set of \emph{event} artists where each artist has one or more upcoming events in $\mathcal{E}$
\item [$\mathcal{F}$]: set of features where $\mathcal{F}= \mathcal{A} \cup \mathcal{T}$. That is, we will describe each artist $a$ as a (sparse) feature vector of artist similarity and tag affinity values in $\mathbb{R}^{|\mathcal{A}|+|\mathcal{T}|}$
\item [$X$]: (sparse) raw data matrix. The dimension of $X \in \mathbb{R}^{|\mathcal{A}| \times |\mathcal{F}|}$ where the $x_{i,j} \in [0,1]$ represents the affinity between the $i$-th artist (a row) and $j$-th feature (column). A value of 0 represents either no affinity or unknown affinity.  Note that all artists are self-similar so that $x_{i,i} = 1$. In terms of practical implementation, we can construct $X$ by stacking our $|\mathcal{A}| \times |\mathcal{A}|$ artist similarity matrix next to our$ |\mathcal{A}| \times |\mathcal{T}|$ artist-tag affinity matrix.
\end{itemize}
LSA uses the truncated SVD algorithm to decompose the raw data matrix $X$ as follows:
\begin{equation}
X \approx X_k = U_k \Sigma_k V_k^{T}
\end{equation}
such that the matrix $X_k$ is a rank-$k$ approximation of $X$, $U_k$ is an $|\mathcal{A}| \times k$ matrix, $\Sigma_k$ is a diagonal $k \times k$ matrix of singular values, and $V_k^{T}$ is a $k \times |\mathcal{F}|$ matrix.
We will then project each artist and tag in a $k$ dimensional latent feature space:
\begin{equation}
X_{SVD} = \Sigma_k V_k^{T}
\end{equation}
where $X_{SVD} \in \mathbb{R}^{k \times |\mathcal{F}|}$ or equivalently $\mathbb{R}^{k \times (|\mathcal{A}|+|\mathcal{T}|)}$ by construction. That is, the first $|\mathcal{A}|$ columns of $X_{SVD}$ represent artists and the last $|\mathcal{T}|$ columns represent tags all embedded into the same $k$ dimensional space.
We can also embed a new artist with raw feature vector $\mathbf{x} \in \mathbb{R}^{1 \times |\mathcal{F}|}$ by computing
\begin{equation}
\mathbf{x}_{SVD} = \mathbf{x} V_k \Sigma_k^{-1}.
\end{equation}
so that $\mathbf{x}_{SVD} \in \mathbb{R}^{1 \times k}$ is projected in the same latent feature space.

Finally, we can compute artist-artist, artist-tag, or tag-tag similarity in the embedded space by comparing their respective (column) vectors in $X_{SVD}$. For example, if we have two latent feature vectors $p$ and $q$, we can compute their cosine similarity:
\begin{equation}
\cos(p, q) = \frac{p \cdot q}{||p||~||q||}
\end{equation}
where $p$ and $q$ are $k$-dimensional vectors and $||x|| = \sqrt[]{\sum_{i=1}^{k} x_i^2}$ is the l2-norm of a vector $x$. One nice property of cosine similarity, is that it tends to remove popularity bias. That is, we normalize the feature vectors by their length (l2-norm) such that each artist (and tag) vector is the same length. Without length normalization, popular artists which tend to have a bigger digital footprint (resulting in a denser raw feature vector with a bigger l2-norm) tend to produce larger similarity scores on average than if we did not normalize by length.

\section{Event and Artist Data}\label{sec:data}

The data for our experiments is constructed by scraping local events from both TicketFly\footnote{https://www.ticketfly.com scraped February 15, 2018.} and the web-based public event calendar from a local newspaper\footnote{Details omitted during anonymous review process.}. We collected a total of $|\mathcal{E}| = 96$ events with 66 events from TicketFly, 36 events from the local newspaper, and 6 overlapping events between both websites. These events produced a set $|\mathcal{A}^{\mathcal{E}}| = 154$ event artists. We are also able to download short biographies of almost all of the event artists for events obtained from Ticketfly. The local newspaper only provides us with 1 to 3 genre tags for about half of the events we obtained from their site.

We then used the Last.fm API \footnote{https://www.last.fm/api} to collect music information (popularity, biography text, artist similarity scores, and tags affinity scores) for each of our event artists. We then use snowball sampling on the similar artists and obtain this same Last.fm music information. We continue sampling these non-event artists until we have a set of 10,000 artists (i.e., $|\mathcal{A}| = 10,000$.)

We define our set of tags $\mathcal{T}$ as the 1585 tags which are associated with 20 or more artists. Our set of genre tags $\mathcal{T}^{\mathcal{G}}$ are the top 20 tags which are most frequently associated with our event artists $\mathcal{A}^{\mathcal{E}}$. These include tags like ``rock'', ``jazz'',  and ``reggae''. However, we manually prune tags which are obviously not genres like ``seen live'' and  ``favorites''.  Finally, for each artist, we concatenate all available biographies (Last.fm, TicketFly, local newspaper) and attempt to find each of our tags in the combined biography text. If a tag is found, we label the artist with that tag. This is especially important since otherwise, many of our event artists would not be labeled with  any tags. In the end, we have 977,270 artist similarities and 456,867 artist-tag affinities.

\section{Exploring Artist Similarity in the Long Tail}

The core of our local event recommendation algorithm is our artist similarity calculation based on Latent Semantic Analysis (LSA). In this section, we show that most local event artists are relatively obscure \emph{long-tail} artists and that they tend to have small digital footprints. We also explore the relationship between digital footprint size and the accuracy of our artist similarity calculation.

\subsection{Long-tail Event Artists}

\begin{figure}
\centering
  \includegraphics[width=3.3in]{LongTail.png}
  \caption{(Top) Plot of Last.fm Listener Count vs. popularity rank (divided by total number of artists) of 10,000 artists with most. (Bottom) Histogram of popularity for 154 event artists placed into 10 deciles of overall artist popularity. Note that most event artists reside in the long-tail of this popularity distribution.}
\label{fig:long-tail}
\end{figure}

In the top plot of Figure \ref{fig:long-tail}, we rank all 10,000 of our artists by their Last.fm listener counts. This shows a typical long-tail (power-law) distribution where a small number of popular artists in the short-head (left) receive much more attention than the vast majority of other artists in the long tail (right) \cite{celma2010,anderson2004long}.
For example, 16.3\% of the most popular artists represent 80\% of the listener counts.
In the bottom plot,  we show a histogram of the event artists' Last.fm listener counts broken down into deciles. We note that a disproportionate number of local event artists reside in the long-tale of this popularity distribution. In particular, 99 of the 154 event artists (64.2\%) are in the lowest three deciles of the ranking.

\subsection{The Digital Footprint of Event Artists }

\begin{figure}
\centering
  \includegraphics[width=3.3in]{FootprintDistribution.png}
  \caption{Cumulative distributions of digital footprint size (i.e., the number of nonzero artist similarity and tag affinity scores for each artists) for event artists and all artists. }
\label{fig:footprint-dist}
\end{figure}

As we discussed in the Introduction, obscure artists tend to have small digital footprints. To show this, we will consider the digital footprint of an artist to be the number of artist similarities plus the number of tag affinities for that artist. Equivalently, it is the number of nonzero values in the row of our raw data matrix $X$ that is associated with the artist. We note that digital footprint size is correlated with popularity rank ($r = -0.56$) such that popular artists tend to have a larger digital footprint.

In Figure \ref{fig:footprint-dist}, we plot the empirical cumulative distribution for both event artists and all artists as a function of the digital footprint size. We see that about 27.2\% of the event artists have 15  or fewer digital footprints whereas only 2.8\% of all artists have so few digital footprints. This suggests that it will be important for us to design an artist similarity algorithm that works well in this \emph{small digital footprint} setting.

\subsection{Artist Similarity with Latent Semantic Analysis}

\begin{figure}
\centering
  \includegraphics[width=3.3in]{ReducedDigitalFootprint.png}
  \caption{Plot of artist similarity ranking performance as a function of the  (artificially reduced) digital footprint size for various LSA embeddings with ranks of 32, 64, 128, and 256 dimensions. The \emph{Raw} approach represents computing cosine distances without first applying LSA to the raw artist data vectors. }
\label{fig:reduced-footprint}
\end{figure}

In Section \ref{sec:artist-similarity}, we introduced LSA as a algorithm for computing artist similarity. However, as we observed in the previous subsection, we are particularly interested in the case where an artist is represented by a small number of artist similarities and tag affinities (i.e., a small digital footprint.) To explore this, we will artificially reduce the digital footprint of artists to a fixed sized and see how well LSA is able to accurately compute artist similarity.

To do this, we randomly split our data set of $\mathcal{A} = 10,000$ artists into a training set with $\mathcal{A}_{train} = 9,000$ artists and a test set of $\mathcal{A}_{test} = 1,000$ artists. Note that this involves removing 1000 rows \emph{and} 1000 columns from our raw data matrix $X$ since artists are also features. The training data will be used to to calculate our matrix decompositions $\mathcal{A}_{train} \approx U_k \Sigma_k V_k^{T}$ for a given embedding dimension $k$.

Before projecting the $\mathcal{A}_{test}$ into the latent feature space, we limit the digital footprint size of each artists by randomly selecting artist similarity and tag affinity features to zero out.  We can then project $\mathcal{A}_{test}$ into the latent feature space and calculate the cosine distance between each pair of test set artists. Finally, we can calculate the Area Under the ROC Curve (AUC) \cite{manning2008} for each artist where the original artist similarities serve as the ground truth.

Figure \ref{fig:reduced-footprint} shows a plot of artificially reduced digital footprint size verses average AUC over the 1,000 test set artists for various LSA embedding dimensions. We also plot the curve for when we compute cosine distances between the \emph{raw} test artist vectors without projecting into a latent feature space. Here we note that LSA shows a improvement over raw cosine distance in small footprint setting of between 1 and 16 nonzero features. Once the digital footprint is larger than 128 nonzero features, the raw cosine approach slightly out performs LSA-based approach. However, the compactness of representing each artist with 32 or 64 floating point numbers may be advantageous in terms of storage size and computation time when we consider a much larger set of artists and tags. As such, we will use 64-dimensional LSA embeddings for the remaining experiments in this paper.

\section{Exploring Event Recommendation}

To explore the performance of event recommendation using event graphs and LSA-based artist similarity, we conducted a small user study with a short 2-phase survey. We recruited 51 participants who were very familiar with the local music scene and attend live events in the area on a weekly basis. In the first phase of our survey, we asked participants to select between 1 and 3 genres from a set of 20 common genres. For each selected genre, the test subject was then asked to select between 1 and 3  artists from  a set of 16 popular artists that were representative of the genre (i.e., having a high cosine similarity score between the 64-dimensional latent feature vectors of the genre and the artist.)  In the second stage, participants were shown a list of the 154 event artists in our data set. They were asked to select all artists that they would like to see at a live event in the local area and were required to select 5 or more event artists.

To evaluate our system, we use each test subject's selected genres or popular artists from phase 1 of the survey to rank order the 154 event artists using one of the approaches described below. In all cases, we embedded artists and tags into a 64-dimensional latent feature space using LSA with the data set that is described in Section \ref{sec:data}.  We then calculate the area under the ROC curve (AUC) for each user where ground truth relevance is determined from phase 2 of the survey.

Each test subject provides multiple genre and multiple popular artist preferences. We explore a number of ways to combine these preferences to produce one ranking of the event artists for each test subject. We consider \emph{early fusion} and \emph{late fusion} steps for a number of approaches. In early fusion, we start with a set of latent feature vectors where each vector is associated with one of the users genre or artist preferences. We consider three approaches:
\begin{itemize}
\item \textbf{average} the latent feature vectors into one vector
\item \textbf{cluster} the latent feature vectors and use the $k$ centroid vectors
\item \textbf{none} use all of the latent feature vectors
\end{itemize}
When clustering, we use the $k$-means clustering algorithm\footnote{http://scikit-learn.org/stable/modules/generated/ sklearn.cluster.KMeans.html} with the number of clusters ($k$) equal to the rounded natural log of the number of user preferences.

For \emph{late fusion}, we must output one ranking of the event artists for each user. We consider three approaches
\begin{itemize}
\item \textbf{average cosine} ranks event artists by the average of the cosine similarity scores between the event artist vector and each vector in the set of user preference vectors.
\item \textbf{average rank} creates one ranking of event artists for each user preference vector, calculates the average rank for each event artist over this set of rankings, and then ranks them by this average rank.
\item \textbf{interleave} creates a set of rankings of the event artists for each user preference vector, and then constructs a final ranking by alternating between these ranking lists and picking top remaining artists that have not already been added to the final ranking.
\end{itemize}

\begin{table}[]
\centering
\caption{Event artist recommendation performance. The mean and standard deviation of AUC for our 8 expert test subjects when considering popular artist preferences, genre preferences, and both preferences together. See text for details on the seven approaches and the two baselines. }
\vspace{3mm}
\label{tab:event-artist-recommend}
\begin{tabular}{|l|c|c|c|}
\hline
\multicolumn{1}{|c|}{Approach} & \multicolumn{3}{|c|}{ User Preferences } \\
\hline
\multicolumn{1}{|c|}{Early / Late Fusion}   & Artists & Genres & Both \\
\hline
none / avg. cosine    & \textbf{.79} (.09) & .69 (.16) & .74 (.12) \\
none / avg. rank      & .75 (.11) & .66 (.15) & .76 (.09) \\
none / interleave     & \textbf{.79} (.11) & .69 (.15) & .71 (.15) \\
average / cosine      & \textbf{.79} (.09) & .69 (.16) & .74 (.12) \\
cluster / avg. cosine & .78 (.09) & .69 (.18) & .74 (.11) \\
cluster / avg. rank   & .74 (.13) & .66 (.20) & .68 (.17) \\
cluster / interleave  & .78 (.10) & .69 (.16) & .75 (.09) \\
\hline
\multicolumn{1}{|c|}{Baseline} & \multicolumn{3}{|c|}{~} \\
\hline
random & \multicolumn{3}{|c|}{.50 (.12)}\\
popularity & \multicolumn{3}{|c|}{.53 (-)}\\
\hline
\end{tabular}
\end{table}

Table \ref{tab:event-artist-recommend} shows average AUCs (and standard deviations) for our seven early/late fusion approaches when we use each user's popular artist preferences, genre tag preferences, and both sets of preferences together. We also include a popularity baseline that ranks all event artists by their Last.fm listener count as well as a random shuffle baseline.  We observe that artist preferences alone result in the best performance and a number of our proposed early/late fusion approaches produce similar results.

We should also mention that we collected survey data from individuals who attended local shows on a less frequent (monthly) basis. The results for these test subjects was significantly lower (average AUC of 0.61) and more variable (AUC standard deviation of 0.15) for our best performing approach (Genre Preferences / None / Interleave.)  Having done error analysis on many of these less regular attendees, we often found that they selected a very eclectic set of event artists which did not match their preferences. As such, it would have been difficult for any recommender system to make accurate recommendations for many of these test subjects. This suggests that test subjects need to have a high level of familiarity with the local music community in order to provide useful ground truth for our experiment.

\section{Discussion}

In this paper, we explored the understudied task of local music event recommendation. This is an exciting task for the research community because it involves many interesting problems: long-tail recommendation,  the new user \& new artist cold start problems, multiple types of music information (artist similarity, tags), and user preference modeling. It is also an interesting problem outside of the academic research community since music event recommender systems can be used to help grow and support the local arts community. By promoting the work of talented local musicians, such systems can help fans discover new artists and help musicians reach new audiences. These audiences in turn attend more events which help sustain concert venues, music festivals, and other (local) businesses who benefit from direct ticket sales and other forms of indirect support (e.g., food, drinks, merchandise.)

While we were able to evaluate our system using a survey of local music experts, a more natural way to evaluate music event recommendation would be to build an interactive application thats collects user feedback over a longer period of time. We plan to develop such an app in the coming months and hope that it will be useful for expanding on the research that is presented in this paper.

Information Retrieval (cs.IR); Multimedia (cs.MM)


Metal Abundances in the ICM as a Diagnostics of the Cluster History

Abstract: Galaxy clusters with a dense cooling core exhibit a central increase in the metallicity of the intracluster medium. Recent XMM-Newton studies with detailed results on the relative abundances of several heavy elements show that the high central abundances are mostly due to the contribution from supernovae type Ia. The dominant source is the stellar population of the central cluster galaxy. With this identification of the origin of heavy elements and the observed rates of SN Ia in elliptical galaxies, the central abundance peak can be used as a diagnostic for the history of the cluster core region. We find for four nearby cooling core clusters that the enrichment times for the central peaks are larger than 6 - 10 Gyrs even for a higher SN Ia rate in the past. This points to an old age and a relatively quiet history of these cluster core regions. A detailed analysis of the element abundance ratios provides evidence that the SN Ia yields in the central cluster galaxies are more rich in intermediate mass elements, like Si and S, compared to the SN Ia models used to explain the heavy element enrichment in our Galaxy.

\section{Introduction}
\label{}

Our understanding of the complex physics prevailing in cooling cores in
clusters of galaxies is mostly driven by observation rather than by theory.
It was for example the large amount of X-ray radiation detected from
cluster cores that led to the scenario of cooling flows
(e.g. Fabian \& Nulsen 1977, Fabian 1994).
And it was later again observations by XMM-Newton showing that no steadily
cooling gas is observed and by Chandra displaying interaction effects
of central AGN with the cooling core plasma which lead to major revision
of the physical model of cluster cooling cores with much smaller mass deposition
rates than inferred previously (e.g. Peterson et al. 2001,  Matsushita et al. 2002,
McNamara et al. 2000, Forman et al. 2004,  Churazov et al. 2000, 2001,
B\"ohringer et al. 2002).
To understand in detail the interaction effects of the central AGN with the
ICM and how the ICM is heated in a globally homogeneous and fine tuned way is
even more complicated. We have to hope again to exploit further observational
effects to guide our interpretation. In this context the entropy distribution
and the distribution of heavy elements in the ICM provide another important
diagnostic tool to learn more about the energy recycling and transport
processes in the central ICM. This conference contribution will concentrate on the
abundance distributions as diagnostics.

\begin{figure}
\centerline{\includegraphics[clip=,width=0.70\textwidth]{BoehringerFig1.ps}}
\caption{Relative abundance of Fe in the ICM as a function of cluster
radius in four nearby clusters (identified by the four different symbols)
as deduced from XMM-Newton observations (B\"ohringer et al. 2004).
The dashed line shows the Fe abundance value of 0.2 solar, which is observed
at large radii in clusters.}
\label{Fig1}
\end{figure}

To model the observed abundance distribution of heavy
elements we need to know three things: the sources, the time scales, and the
transport processes that redistribute the ICM.

\section{The origin of the observed abundance distributions}
\label{}

\begin{figure}
\centerline{\includegraphics[clip=,width=0.98\textwidth]{BoehringerFig2.ps}}
\caption{Abundance profiles of O and Si in the X-ray halo of M87 in the Virgo
  cluster as deduced from XMM-Newton observations (Matsushita et al. 2003). The upper
curves for the two-temperature model give the more realistic results.}
\label{Fig2}
\end{figure}

We know two sources for the enrichment of the ICM with heavy elements:
core collapse supernovae, type II, which produce a broad spectrum of element
masses with a bias towards the lighter elements like O and Mg, and type Ia
supernovae, thermonuclear explosions of white dwarfs, which dominantly
yield Fe group elements and lighter elements like Si and S but very little O
and Mg. One of the characteristics of cooling core clusters is the central
abundance peak of Fe and the heavier elements (e.g. DeGrandi \& Molendi 2001)
as shown in Fig. 1. A more detailed inspection which elements follow this
abundance peak shows that it is traced by the heavier SN Ia products but not by
the lighter elements like O and Mg (Fig.2, Matsushita et al. 2003). This
is consistent with the picture that SN II activity happens in the early
history of cluster formation when the stellar populations of the cluster
galaxies are still young. These elements have time to mix well in the ICM and
show a more homogeneous distribution. SN Ia are still occuring and observed in
the present day cluster ellipticals and in the central cD galaxies. The more
recent yields obviously lead to more local enrichments. In particular the
massive stellar population of the cD galaxies dominating the centers of cooling
core clusters are responsible for the central enrichment.

\begin{figure}
\centerline{\includegraphics[clip=,width=0.70\textwidth]{BoehringerFig3.ps}}
\caption{Relatve abundance of O and Mg in the central ICM of the
Centaurus cluster observed with XMM-Newton (Matsushita et al. 2004). The dashed line
shows the stellar Mg abundance for comparison.}
\label{Fig3}
\end{figure}

An even more careful inspection of the central abundances reveals also slight
increases in the relative abundances of O and Mg as shown in Fig. 3 for the
case of the Centaurus cluster (Matsushita et al. 2004). The ICM abundances of
these elements are comparable to the stellar abundances which implies that the
ICM in the very central region most probably reflects the stellar mass loss of
the central galaxy. Consideration of the budget of the central ICM for example
for the case of M87 shows that
the inner 10 kpc region of the ICM, roughly the region of the scale radius
of cD galaxies (or approximately half light radius), contains a gas mass of
about $2\cdot 10^9$ M$_{\odot}$ and an iron mass of about
$6\cdot 10^6$ M$_{\odot}$. Both can be replenished by stellar mass loss from
M87 in about 2 - 3 Gyrs
if we adopt a stellar mass loss rate of $2.5 \cdot 10^{-11} \times L_B$
(Ciotti et al. 1991) and a Fe production with a supernova rate of 0.15 SNU
(Cappelaro et al. 1999). It is important, however, for this enrichment to
accumulate that the gas is not cooling and condensing with the
classically calculated cooling flow rates, since in this case the central ICM
produced by stellar mass loss would disappear faster than it can be
replenished. Both, the evidence described in the previous section and the
observed abundances of O and Mg support the notion that the central ICM is not
condensing at the high rates inferred previously within cooling flow models.

\begin{figure}
\centerline{\includegraphics[clip=,width=0.98\textwidth]{BoehringerFig4.ps}}
\caption{Enrichment ages of the central iron abundance peak
in the four galaxy clusters, M87/Virgo, Centaurus,
Perseus, and A1795. The enrichment times were calculated
on the basis of a SN Ia rate of 0.15 SNU
(Capellaro et al. 1999) and an additional contribution by stellar mass loss.
{\bf Left panel:} with constant SN Ia rate, {\bf Right panel:} with increasing
SN Ia rate in the past described by a power law with time
exponents $s=-2$ (thick lines) and $s=-1.1$. (thin lines).}
\label{Fig3}
\end{figure}

\section{Enrichment times}
The central abundance peak extends much further than the central 10 kpc region
inside the central galaxy. Since the central galaxy dominates the stellar
population out to radii of the order of 100 kpc and is therefore responsible
for most of the secular heavy elements in this zone, this implies some transport
of the centrally enriched ICM to larger radii and larger enrichment times than
inferred above for the 10 kpc region. In Fig. 4 we show the enrichment times
for the Fe abundance peak as a function of radius. The enrichment times were
calculated from the total amount of excess Fe inside a given radius divided by the
iron yield inferred for the integrated stellar light of the cD galaxy with
the above quoted Ciotto et al. rate for stellar mass loss and the Cappelaro et al.
rate supernovae type Ia with a Fe mass yield per supernove of 0.7 M$_{\odot}$
(for more details of these calculations see B\"ohringer et al. 2004).
The excess Fe abundance is thereby determined by subtracting an ubiquitous Fe
abundance of  0.15 solar believed to originate from early SII enrichment.
Alternatively we were also considering the likely case that the type Ia supernova
rate was higher in the past and varied with time in a power law behavior.
Even for this case, where the enrichment times are shortened, we find quite large
enrichment times of $5 - 9$ Gyrs for radii around 50 kpc and $7 - 12$ Gyrs
for 100 kpc.

If these estimates are correct the results imply that the central regions of
the clusters and their ICM can not have suffered major disturbances during
these times and that the ICM experienced only mild turbulent redistribution
as signified by the abundance peak which is broader than the light distribution
of the central galaxy. The central regions of cooling flow clusters must be
old with ages of the order of 10 Gyrs! Again, this scenario does not allow
the inclusion of cooling flows with the large previously inferred mass
deposition rates, since this would make the accumulation of the large observed
central Fe mass excess of the order of $10^9$ M$_{\odot}$ impossible.
See also De Grandi et al. (2004) for a complementary discussion of the
origin of the central Fe abundance from the SN yields integrated over the
entire past history.

\section{Supernova yields}
Adopting this scenario for the enrichment of the central heavy element
abundance peak, we can now study the element abundance ratio in more detail
and compare the results with predictions of the supernova nucleosynthesis
model calculations. The best measured abundances for the different
atomic mass groups are those of Fe, Si
and O. Thus we will concentrate here on the use
of these elements.

\begin{figure}
\centerline{\includegraphics[clip=,width=0.84\textwidth]{BoehringerFig5.ps}}
\caption{Comparison of the Fe/O and Fe/Si abundance ratios for different
radial regions (with $r \le 100$ kpc, except for the left light green data point
for low temperature clusters for $r = 50 - 200$ kpc)
in different galaxy clusters with data taken from Matsushita et al.
(2003 - M87; red), Matsushita et al. (2004 - Centaurus, blue), Sun et al. (2003 - NGC1550, green),
Xue et al. (204 - RGH80, green), Buote et al. (2003 - NGC5044, orange thick), Tamura et al. (2004 -
several cool clusters observed with XMM, light green thin), and Dupke \& White (2000 - A496
observed with ASCA, blue hidden behind N5044 data points).
The solid line gives the W7 model (Nomoto et al. 1984) which
accounts for the abundance of our sun. The three dashed lines give the
delayed deflagration models WDD1, WDD2, and WDD3 by Iwamoto et al.(1999).
The lower asterisk indicates the SN II yields, the upper asterisk the composition
of the Sun. The box indicates the composition of low metallicity halo stars in
our galaxy (from Clementi et al. 1999).
The range of observational values is well bracketed by the
different theoretical SN Ia yields.}
\label{Fig3}
\end{figure}

In Fig. 5 we compare the abundance ratios of Fe/O and Fe/Si in different regions
in different groups and clusters of galaxies. In this diagram
the Fe/O ratio is a diagnostics of the ratio
of Fe yields from SN II and SNIa and the Fe/Si ratio can then be used to test
various models of SN Ia yields (Matsushita et al. 2003).
In the plot we have compiled the available set of
observational results from XMM and Chandra (and one ASCA result) from the literature
and compare them to various models with different deflagration speeds of the
SN Ia explosion from Nomoto et al. (1984) and Iwamoto et al. (1999).
The models lines were calculated by mixing an increasing amount of SN Ia
material with the early composition taken to be similar to the average
of the low metallicity galactic halo stars (Clementini et al. 1999) which is also
close to the expected SN II yields as shown in the figure.
The results show that the bulk of the data lay below the curve of the W7 model
which fits the solar abundance and is successfully used to describe the
chemical history of our galaxy (Edvardsson et al. 1993). The WDD1 and WDD2 models
provide yields with larger Si mass fractions compared to iron
than supplied by the W7 model. The data points for M87 are
the most extreme in requiring a relatively large Si yield (Matsushita et
al. 2003). The possible reason for this is a more incomplete nuclear burning
in SN Ia explosions in older stellar populations (Finoguenov et al. 2001)
which is also consistent with the observed statistically lower luminosity of
SN Ia lightcurves observed in supernovae type Ia in elliptical galaxies
compared to spirals (Ivanov et al. 2000).

\section{Conclusions} The results presented from the analysis of a few nearby
clusters studied with XMM-Newton and CHANDRA provide new clues on the
dynamical history of the central regions of clusters, the transport processes
in the ICM, and the nucleosynthesis yields of the two main types of
supernovae. The hopefully long life of the XMM and CHANDRA observatories will
allow us to extend these studies systematically to a well selected larger
sample of groups and clusters of galaxies, which will finally show if the
conclusions drawn in this contribution can be generalized and established with
higher precision.

\section{Acknowledgements}

The majority of the results have been obtained with
XMM-Newton, an ESA mission with instruments and contributions directly funded
by ESA member States and the USA (NASA). The XMM-Newton Project is supported
by the Bundesministerium f\"ur Bildung und Forschung, the Max-Planck-Society,
and the Haidenhain Stiftung.