Personalized Recommendation of PoIs to People with Autism

Abstract: The suggestion of Points of Interest to people with Autism Spectrum Disorder (ASD) challenges recommender systems research because these users' perception of places is influenced by idiosyncratic sensory aversions which can mine their experience by causing stress and anxiety. Therefore, managing individual preferences is not enough to provide these people with suitable recommendations. In order to address this issue, we propose a Top-N recommendation model that combines the user's idiosyncratic aversions with her/his preferences in a personalized way to suggest the most compatible and likable Points of Interest for her/him. We are interested in finding a user-specific balance of compatibility and interest within a recommendation model that integrates heterogeneous evaluation criteria to appropriately take these aspects into account. We tested our model on both ASD and "neurotypical" people. The evaluation results show that, on both groups, our model outperforms in accuracy and ranking capability the recommender systems based on item compatibility, on user preferences, or which integrate these two aspects by means of a uniform evaluation model.

\section{Introduction}
\label{sec:introduction}
The personalized suggestion of Points of Interest (PoIs) challenges the research on recommender systems \cite{Ricci-etal:11} because, in order to provide truly inclusive services, different factors have to be taken into account, which go farther than modeling user interests.
Specifically, when suggesting PoIs to people with Autism Spectrum Disorder (ASD), we must take into account at least two aspects:

\begin{enumerate}
    \item
The recommender system must work under data scarcity. There is a low number of users who can be analyzed to learn their interests: research indicates that Autism Spectrum Disorder affects around 1 in 100 people in EU \cite{elsabbagh2012global}. Moreover, ASD people are hard to be contacted because they have interaction problems and a tendency to avoid new experiences \cite{hobson1995}. Finally, their attention problems cause difficulties in providing detailed feedback about items \cite{murray2005attention}.
\item
User preference management is not enough to generate useful suggestions. In fact, ASD people have idiosyncratic sensory aversions that influence the way they perceive items, especially places \cite{tavassoli2014sensory,robertson2017sensory, american2013diagnostic}. Therefore, traditional data about interests, used in recommenders for “neurotypical” individuals (i.e., not belonging to the autism spectrum), should be combined with these aversions because what bothers autistic people has great importance in their daily choices and can determine a high level of stress and anxiety \cite{gillott2007levels}.
In order to take spatial needs into account in PoI recommendation, idiosyncratic aversions to noise, brightness and other sensory features have to be analyzed to recommend places that the user can perceive as safe and thus serenely experience. For ``safe PoIs'' we mean places which present ``safe'' characteristics from the sensory point of view; e.g., being quiet, scarcely crowded, or with smooth lights.
Notice that the inclusive recommendation of items generally goes beyond user preferences management: for example, in technology-enhanced learning, it is necessary to consider specific user features, such as learning capability, and corresponding specific item features, e.g., readability level \cite{pera2014automating}.
\end{enumerate}
Starting from Multi Criteria Decision Analysis \cite{vonWinterfeldt:86}, which provides techniques for the evaluation of multiple dimensions of items, and on match-making models based on user-to-item similarity \cite{Bridge-etal:05,Lops-etal:11}, most recommender systems, including collaborative multi-criteria ones \cite{Adomavicius-Kwon:07,Jannach-etal:14}, assume that the attributes of an item contribute to its utility to the user in an additive way. However, we notice that, depending on a person's idiosyncrasies and on their strength, problematic features might make items unsuitable for her/him even though they meet her/his preferences. Moreover, the impact of compatibility on users' choices varies individually and it cannot be separately managed with respect to preferences; e.g., some autistic people are determined to visit noisy and crowded places if they like them very much. Therefore, the models of item evaluation must reflect individual evaluation criteria by balancing feature compatibility and preference satisfaction.
In this work, we investigate the role of these two aspects in rating estimation, considering both ASD and neurotypical people. Specifically, we pose the following research questions:
\begin{itemize}
    \item
    {\em RQ1: in PoI recommendation, does a customized model of item evaluation, which balances feature compatibility and preference satisfaction in a personalized way, outperform recommender systems that manage only one of these aspects?}
    \item
    {\em RQ2: in PoI recommendation, does a customized model of item evaluation, which balances feature compatibility and preference satisfaction in a personalized way, outperform recommender systems which deal with both aspects but uniformly manage them?}
\end{itemize}
In order to answer these questions, we propose a novel Top-N recommender system that applies heterogeneous evaluation criteria to take user preferences and compatibility requirements into account by exploiting feature-based user profiles for the specification of individual needs.
Our recommender is focused on PoI suggestion to people with autism; however, it might be interesting to investigate its adaptation to other needs, e.g., related to motor disabilities, by extending the type of features which influence item compatibility. Our work has the following key aspects:
\begin{itemize}
    \item
    We acquire data about people's aversion to sensory features in terms of disturbance caused by low or high feature values; e.g., darkness or strong light in physical places.
    For this purpose we use a questionnaire derived from \cite{tavassoli_sensoriality} but composed of a lower number of questions than that work.
    \item
    As our questionnaire provides data about users' aversion to a subset of the values that each feature can take, we also define general functions to estimate aversion to the whole range of possible values of the feature. Then, we derive feature compatibility with the user as the complement of aversion.
    \item
    For the estimation of item ratings, we distinguish user preferences for broad item categories from idiosyncratic sensory aversions. Moreover, as users might balance differently these aspects in item evaluation, we combine preferences and features compatibility by applying user-specific weights, which we acquire by analyzing users' ratings in conjunction with their declared preferences and idiosyncrasies.
\end{itemize}
We tested our model on 20 adults with autism and 128 neurotypical ones\footnote{We had no mean to know whether the subjects of this second group belong to the spectrum or not.  However, we can reasonably expect that the sample respects the proportion of the entire population; thus, it might include 1 or 2 ASD persons at most (over 128). Henceforth, for simplicity, we refer to this sample as neurotypical people.}. On both groups of people, our model outperforms in accuracy and ranking capability a set of baseline recommender systems which singularly take item compatibility, or user preferences into account, and baselines that uniformly manage compatibility and preference information without differentiating their contribution.
In summary, we provide the following novel contributions:
\begin{itemize}
    \item
    A Top-N PoI recommender system that fuses compatibility and preference data in rating estimation by balancing the impact of these aspects in a user-specific way.
    \item
    A validation of the recommender on autistic and neurotypical people aimed at evaluating the performance of our model on both groups of users.
\end{itemize}
The approach presented in this paper is part of a wider ongoing project, PIUMA (Personalized Interactive Urban Maps for Autism)\footnote{PIUMA project involves a collaboration among the Computer Science and the Psychology Departments of the University of Torino and the Adult Autism Center of the city of Torino, Italy.}, which has the aim to develop novel digital solutions for helping people with ASD in their everyday movements \cite{DBLP:conf/chi/RappCBACBTKCB17, DBLP:conf/mhci/RappCMBSKB19}. The final result of PIUMA will be a mobile app showing maps customized to ASD users.
By means of the present work, we aim at adding a personalized selection of PoIs.

In the following, Section \ref{sec:spatialNeeds} discusses the spatial needs of people with autism. Section \ref{sec:related} presents the related work.
Sections \ref{sec:knowledge} outlines how we retrieve information about PoIs, users' preferences and idiosyncrasies. Section \ref{sec:model} presents our model. Section \ref{sec:validation} describes the validation methodology we applied and Section \ref{sec:results} discusses the evaluation results. Section \ref{sec:conclusions} concludes the paper.

\section{Spatial needs of people with autism}
\label{sec:spatialNeeds}
Symptoms of autism span from severe language and intellectual disabilities in individuals with low- or mid-functioning autism, to no disabilities and an Intelligence Quotient (IQ) above the average in persons with high-functioning autism and Asperger’s syndrome.
Autism entails an atypical social functioning, which often results in avoiding everyday interactions \cite{hobson1995}.
Questionnaire-based studies suggest atypical sensory perception in over 90\% of individuals with autism spectrum conditions  \cite{robertson2017sensory, golan2010enhancing, robertson2013relationship}, which means that individuals with autism appear  to react differently to sensory stimulations: a majority of them may become overwhelmed by environmental features that are easily managed by neurotypical people. For ASD individuals, the brain seems unable to appropriately balance the senses \cite{robertson2013relationship}.
At least in part because of these characteristics, persons with ASD tend to have a reduced range of activities and interests, often preferring mechanical, deterministic situations, having the need to find reassurance by sticking to rigid, repetitious routines \cite{simm2016anxiety}.  It seems that they are less likely to explore new environments, and more likely to revisit well-known locations than neurotypical individuals \cite{smith2015spatial}. These peculiarities may entail idiosyncratic modes of perceiving space \cite{rapp2018designing}.

Several studies  \cite{golan2010enhancing, robertson2013relationship, rapp2019spatial} reported that people with autism actively avoid places that may negatively impact on their senses.
Sight, smell and hearing are relevant in reference to mobility in urban environments and high sensory stimulation negatively influences individuals in their movements.  Further relevant environmental dimensions that could impact their sense of safeness are the temperature, openness, and crowding of a place. Such idiosyncratic sensory aversions may result in anxiety, fatigue, disgust, sense of oppression or distraction \cite{rapp2019spatial}.

This should result in technological supports able to satisfy autistic people's idiosyncratic spatial needs, focusing on aversions derived from their high sensitivity to sensory stimulation. Moreover, there is a high need to personalize solutions because sensory sensitivity seems highly idiosyncratic; thus there are no features of places that may reassure the entire autistic population, and specific characteristics of each person should be considered \cite{putnam2019interactive}.

\section{Related work}
\label{sec:related}
 Technology is widely used to support people with autism in managing specific problems because they commonly exhibit an affinity with it \cite{putnam2008software,ramdoss2012computer}. In general, the research on autism tends to pay more attention to children \cite{goldsmith2004use} and it overlooks adults’ needs. This might be a consequence of the “medical model”, which promotes intervention toward school-aged individuals. Moreover, specifically the HCI community prefers to address social interaction problems \cite{putnam2019interactive,kientz2013interactive,grynszpan2014innovative}, likely because these are seen as the core characteristics of autism from a clinical point of view: therefore, that research focuses, e.g., on face-to-face conversation \cite{boyd2016saywat} and emotion management \cite{simm2016anxiety}, ignoring spatial difficulties.

Most of the applications investigating the adoption of personalization strategies for ASD people regard the educational domain.
For example, Judy et al. \cite{judy2012} present a personalized e-learning system that provides learning paths of different difficulty based on the user's past results.  They use ontologies to describe learning materials, annotation schemas and service ontologies, and they use a genetic algorithm as an optimization technique, representing a set of learning objects as chromosomes. No evaluation is provided.

Garc{\'{\i}}a et al. \cite{DBLP:conf/ht/GarciaSFBFP16} propose an adaptive web-based application that helps students with autism overcome the challenges they may face when going to university. The adaptation consists of how the information site presents itself to autistic and non-autistic students but  the  information is the same for everyone. The adaptive functionality is based on learning styles (visual vs. verbal, global vs. analytical, active vs. reflective) and user history. For example, if the user is more visual than verbal, the video version of the content will be shown at the top of the learning object. Otherwise it will be moved to the bottom. No evaluation with ASD users is presented.

Hong et al. \cite{hong2012designing} propose to provide autistic users with suggestions within a social network aimed at supporting the independence of young adults. However, they focus on the organization of the social network, by relying on peers' suggestions, instead of automatically generating recommendations to users.

Differently, Costa et al. \cite{costa2017task} develop a task recommendation system that uses a case-based reasoning machine learning technique to supplement the child’s regular therapy.  The recommender suggests the daily activity to be performed (related to eating, keeping clean, getting dressed, etc.) based on  age, gender and time of day. It does not consider  the child's preferences, while the level of difficulty of the activities is manually set by the therapist. No evaluation with ASD people is described.
Moreover, in \cite{Ng-Pera:18} Ng and Pera propose a hybrid game recommender for adult people with autism, based on collaborative and graph-based recommendation techniques, but they only carry out a preliminary test on neurotypical people.
Finally, in \cite{premasundari2019food} Premasundari and Yamini propose a system that recommends food and therapy for autistic children based on their symptoms. The system uses K-means algorithm for grouping the symptoms based on their type and association rule mining for the recommendation of food and therapies. However, the final target are the parents and caregivers, not people with autism. The system has been evaluated only from a usability point-of-view.

Our work differs from the above ones for several reasons. Firstly, we focus on a different domain, i.e., spatial support. Secondly, we target adult people, who are the final users of the system and whose preferences and requirements have to be considered to succeed in item recommendation. Thirdly, we evaluated the approach with ASD people: this has rarely, if ever, been done in the related research.
Fourthly, our approach employs personal preferences for item categories and aversions to sensory features to steer recommendation in a context where a limited amount of feedback about items can practically be collected from users. This is different from the situation of other recommender systems in the health domain, which are targeted to users who are more willing to provide their preferences as needed, e.g., food recommendation \cite{Freyne-Berkovsky:10}.

Our work also differs from general content-based recommender systems \cite{Lops-etal:11}, feature-based \cite{Han-etal:05}, collaborative multi-criteria \cite{Adomavicius-Kwon:07,Jannach-etal:14,Zheng:17} and hybrid ones \cite{Burke:02,Gemmel-etal:12,Cantador-etal:11} because we treat sensory features of items as sources of discomfort for users rather than liking or disliking factors. We separately model the influence of idiosyncratic sensory features, which determine the compatibility of items with the user, from her/his preferences for types of items. Notice that this separation makes our model different from recommender systems that deal with negative preferences as well, e.g., \cite{Musto-etal:11}, because we support the management of heterogeneous criteria to deal with user preferences and sensory idiosyncrasies.

Previously, the INTRIGUE \cite{Ardissono-etal:03} tourist guide introduced the notion of compatibility requirements in PoI recommendation but it did not investigate their different meaning and impact in the evaluation of items with respect to preferences.

It is worth mentioning that, while constraint-based recommender systems \cite{Zanker-etal:10,Felfernig-etal:11,Wibowo-etal:18,Dragone-etal:18} are too knowledge intensive for our purposes (we are not suggesting item bundles with constraint satisfaction requirements), the optimization of soft constraints for path-finding under suitability criteria is relevant to our future work, in order to extend PoI recommendation with instructions for reaching the target places. This type of technique has been explored in recommender systems for routing; e.g.,
see \cite{Quercia-etal:14} and
\cite{Verma-etal:18}.

\section{Preliminary study setup}
\label{sec:knowledge}
In this section we present how we gather data about users and POIS to run our experiments, as well as the sample's features.
\subsection{Data}
In order to gather data about autistic users' preferences and sensory aversions, we decided to bootstrap the user profiles by explicitly eliciting this type of information from people. Thus, we defined a questionnaire that will also be proposed to the users in the registration phase of the PIUMA app (not yet fully developed) in order to integrate an initial preference acquisition with the possibility to rate PoIs while the user is visiting them.
The information about sensory aversions is hard to obtain \cite{tavassoli2014sensory}: usually, very long and complex questionnaires have to be completed for this purpose \cite{robertson2017sensory}.
Moreover, asking people with ASD for such data is challenging because they have difficulties in social interactions and they tend to avoid new experiences \cite{schopler1986social}.
Given our users' attention problems \cite{bara2001, murray2005attention} and considering the application context, which is not a clinical setting, we decided to avoid long and detailed surveys. Consequently, we carefully prepared with psychologists a smaller questionnaire able to capture such information in a shorter way.
Users filled in the questionnaire, possibly in the presence of an operator (when needed), and they answered to the question using the [1, 5] Likert scale. The questionnaire is composed of two sections:
\begin{itemize}
\item
In the first one we elicit user preferences about categories of PoIs such as restaurants, parks, and so forth; see the left column of Table \ref{tab:questions1}.
\item
In the second one we gather information about users' aversions to sensory features of PoIs (right column).
We defined the questions about aversions by adapting the Sensory Perception Quotient (SPQ) test by Tavassoli et al. \cite{tavassoli_sensoriality}, a standard sensory questionnaire for adults with and without autism that assesses basic sensory hyper- and hyposensitivity.\footnote{The standard test is part of the battery of assessment tests that the people  compile when they become patients of the Autistic Adult center. It is made of 92 items, that is too much to be proposed to users in the context of a recommender system. In our questionnaire, we extrapolated the questions from Tavassoli's test. Specifically, we adapted the questions based on what users said in previous interviews carried out during a participatory design session for the design of the crowdsourcing system \cite{DBLP:conf/mhci/RappCMBSKB19}. The adaptation of the test is out of the scope of this paper.}
For some features (brightness and space), we asked the user to evaluate two extreme conditions, i.e., low or high levels, assuming that the middle ones are less problematic than the other ones.
In other cases (crowding, noise and smell) we only asked about her/his annoyance in relation to the highest level because usually the low levels of these features are neutral.
\end{itemize}

After the questionnaire, we asked the users to evaluate 50 specific PoIs located in Torino city center (e.g., \textit{How much do you like Castle Square?}) in order to collect a dataset of ratings to test our model.  We used the same [1, 5] Likert scale as above, but the ``I don't know the place'' option was available as well.

Those PoIs have been taken from Maps4All \cite{Maps4All} and they are representatives of all the categories of places it defines. Notice that we used an ad hoc
crowdsourcing platform, designed to gather sensory features of places \cite{DBLP:conf/mhci/RappCMBSKB19}, because open data made available by geo web sites like OpenStreetMap \cite{OpenStreetMap} do not contain the specific sensory information we need. Maps4all allows the collection of such data: for each place, the user can rate in the [1, 5] scale its sensory features, and in particular its level of i) brightness, ii) crowding, iii) noise, iv) smell, v) openness, and vi) temperature.
These sensory features have been defined on the basis of an authors' user study findings \cite{rapp2019spatial} and state-of-art research \cite{robertson2013relationship, robertson2017sensory}. The user can also provide a global evaluation of the place. For each datum, the system returns the mean evaluation it collected.
Currently, the Maps4All platform has been populated by means of two experimental crowdsourcing sessions during two lessons at the Master degree in Social Innovation and ICT at University of Torino, in May and December 2019. About 120 students have been involved in the crowdsourcing tasks, and they have been asked to provide evaluations for at least three PoIs each in Torino city centre.
In total, during the two crowdsourcing sessions we collected the evaluations of 282 PoIs. Specifically, the 50 PoIs we considered for our study have been evaluated by at least three people each.

\begin{table}[t]
\centering
\caption{Short questionnaire to elicit information about preferences and sensory idiosyncrasies (translated from Italian).}
\vspace{-10pt}
\resizebox{\columnwidth}{!}{%
\setlength{\tabcolsep}{6pt}
\begin{tabular}{p{0.5\columnwidth}p{0.5\columnwidth}}
\toprule

 \textit{From 1 (not at all) to 5 (very much), how much do you like doing the following activities?}
\begin{itemize}
    \item be in nature, go to parks, gardens, green areas, \dots
    \item visit museums, exhibitions, cultural events
    \item go to the cinema, theater, concerts
    \item go to comic shops
    \item go to clothing stores
    \item go to malls and markets
    \item go to the library
    \item go to the bookshop
    \item play sport
    \item go to pubs, cafe
    \item go to the restaurant
    \item go to the ice cream shop
    \item stay in squares
    \item go to the railway stations
\end{itemize}
 &
 \textit{In a place, how much does it bother you:}
\begin{itemize}
    \item too much light
    \item very low light
    \item a lot of people
    \item a lot of noise
    \item strong smells
    \item cramped places (narrow, small)
    \item large places
    \end{itemize}
                      \\
\bottomrule
\end{tabular}}
\label{tab:questions1}
\end{table}

\subsection{Sample}
For our study we involved two groups of users\footnote{All participants signed a privacy consensus according to GDPR. University research ethical committee approval was obtained for the study}:
\begin{itemize}
\item
20 ASD adults (from 22 to 40 years-old, mean age: 26,3, median 28; 11 men, 9 women) patients of the Autistic Adult center, medium- and high-functioning.
\item
128 neurotypical subjects (from 19 to 71 years-old, mean age: 28,1, median 23; 63 men, 65 women) from university students and authors' contacts.
\end{itemize}
Given the 50 PoIs selected, the mean number of PoI evaluations we obtained is 31 fpr ASD participants and 39 for neurotypical ones.

\section{Recommendation Model}
\label{sec:model}
As previously discussed, we assume that both user preferences and item compatibility should be taken into account to identify the most relevant items that an individual user can safely experience and like. However, evaluation criteria might be personal and users could weight these aspects differently in their decision-making processes. For instance, in contrast to the tendency of people with ASD to visit places in which they feel comfortable, during our participatory design interviews sessions, we had the chance to  interact with a guy with autism who frequently visits bowling halls, regardless of the negative impact of noise and crowd on his senses, because he likes bowling so much that he does not want to give it up. We thus propose a recommendation model which, based on the observed item evaluations, can weight the contribution of compatibility and preferences in rating prediction on a user-specific basis. For clarity purposes we split the description of our model as follows:
\begin{enumerate}
\item
input data for recommendation (Section \ref{sec:input});
\item
estimation of the compatibility of the individual features of an item with the user (Section \ref{sec:featureCompatibility});
\item
estimation of the overall compatibility of the item with the user (Section \ref{sec:itemCompatibility});
\item
preference-based item evaluation (Section \ref{sec:preferences});
\item
integration of compatibility and preference-based evaluation to predict the user's rating of the item (Section \ref{sec:overallItemEvaluation}).
\end{enumerate}
\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\linewidth]{figures/aversion.pdf}
    \caption{Representation of a user's aversion to a feature.}
    \label{fig:aversion}
\end{figure}
Before describing our model we introduce the notation we use:
\begin{itemize}
    \item
    $U$ is the set of users and $I$ the set of items of the domain.
    \item
    $C$ is the set of item categories; e.g., shops, cinemas, etc..
    \item
    $L$ is a Likert scale in $[1, v_{max}]$; in this work, $v_{max}=5$.
    \item
    $F = F^\uparrow \cup F^V$ is the set of sensory features defined in our domain. We assume that each feature $f \in F$ takes values in $L$. Specifically:
    \begin{itemize}
        \item
        $ F^\uparrow$ is the set of features $f$ such that, the higher the value of $f$, the stronger its negative impact on the user; e.g., noise.
        \item
        $F^V$ denotes features whose extreme values make users uncomfortable, while the middle ones are less problematic; e.g., brightness.
    \end{itemize}
    In our domain there are no features such that people are expected to feel comfortable with high values and uncomfortable with low ones; therefore, we omit this class.
\end{itemize}
For each user $u \in U$ and item $i \in I$, we estimate $u$'s evaluation of $i$ (denoted as $\hat{r}_{ui}$) as a decimal number in the [1, $v_{max}$] interval, by taking $u$'s previous ratings, preferences for item categories and idiosyncrasies into account.

\subsection{Input Data}
\label{sec:input}
Our model takes two types of information as input:
\begin{itemize}
    \item
    The profile of $u \in U$, extracted from the questionnaire
    data, which specifies:
    \begin{itemize}
        \item
        The ratings $r_j$ in $L$ (s)he provided for a set of items $j \in I$.
        \item
        Her/his declared preferences for the categories $c \in C$, each one expressed in the $L$ scale.
        \item
        Her/his declared sensory aversion to specific values of item features, expressed in $L$. We denote $u$'s aversion to a value $v$ of a feature $f \in F$ as $a_{ufv}$. For example, $a_{uf5} = 4$ means that $u$ is fairly disturbed by items having $f=5$.
        \begin{itemize}
            \item
            For each feature $f \in  F^\uparrow$ we assume by default that $a_{uf1} = 1$. Therefore, the user profile stores a single value, $a_{ufv_{max}}$, which specifies $u$'s aversion to the maximum value of $f$; i.e., $v_{max}$.
            \item
            For each feature $f$ in $F^V$, the user profile stores two values which express $u$'s aversion to the minimum and maximum values of $f$ respectively: e.g., \{$a_{uf1} = 3$, $a_{ufv_{max}} = 4$\}.
        \end{itemize}
    \end{itemize}
    \item
    Item $i$'s profile, extracted from the crowdsourced data, which specifies:
    \begin{itemize}
    \item
    The category $c \in C$ of $i$.
    \item
    A vector $\vec{\bf {i}}$ storing, for each feature $f \in F$, the value of $f$ in item $i$ (denoted as  $\vec{{\bf i}}_f^{\,}$) retrieved by querying Maps4All; $\vec{{\bf i}}_f^{\,}$ takes values in the [1, $v_{max}$] interval.
    \end{itemize}
\end{itemize}

\subsection{Compatibility of Individual Features with the User}
\label{sec:featureCompatibility}
We can define compatibility as the opposite of aversion in the range of values that features can take. However, user profiles only include one or two aversion values declared by users for each feature and the missing ones have to be interpolated.
We thus define two patterns to approximate users' idiosyncratic aversions to item features starting from the values stored in the user profile:
\begin{itemize}
    \item
    For each $f \in F^\uparrow$ we approximate aversion as a linearly increasing function. If we represent feature values in the $X$ axis and user aversion in the $Y$ axis of a plane, we can define this function as a line which connects point (1, 1) to point ($v_{max}$, $a_{ufv_{max}}$), as in the left graph in Figure \ref{fig:aversion}:
        \begin{equation}
        line^\uparrow(x) = 1 + \frac{(a_{ufv_{max}}-1)(x-1)}{v_{max}-1}
        \label{eq:crescente}
    \end{equation}
    We thus estimate $u$'s aversion to $f$ in $i$ ($ea_{ufi}$) as follows:
    \begin{equation}
        ea_{ufi} = line^\uparrow(\vec{{\bf i}}_f^{\,})
        \label{eq:upLine}
    \end{equation}
    \item
    As far as $F^V$ is concerned, and given \{$a_{uf1}, a_{ufv_{max}}$\} in $u$'s profile, we approximate aversion by means of a concave function on the range of $f$. The aversion function has a "V" shape which we approximate by drawing two lines, as the right graph of Figure \ref{fig:aversion}:
    \begin{itemize}
        \item
        The first line ($line^\uparrow$) connects points (1, 1) and ($v_{max}$, $a_{ufv_{max}}$) to represent the increment of aversion towards the maximum value of $f$.
        \item
        The second line ($line_\downarrow$) connects points (1, $a_{uf1}$) and ($v_{max}$, 1) to represent the decrease in aversion while $f$ takes higher values than its minimum:
        \begin{equation}
        line_\downarrow(x) = 1+\frac{(x-v_{max})(1 - a_{ufv_{min}})}{v_{max}-1}
        \label{eq:lineDown}
    \end{equation}
    \end{itemize}
    Therefore, we estimate $u$'s aversion to $f$ in $i$ by selecting the maximum values of the two lines, i.e.:
    \begin{equation}
        ea_{ufi} = \max (line^\uparrow(\vec{{\bf i}}_f^{\,}), line_\downarrow(\vec{{\bf i}}_f^{\,}))
        \label{eq:v}
    \end{equation}
    This corresponds to the broken thick line in Figure \ref{fig:aversion}.
\end{itemize}
Notice that $ea_{ufi}$ takes values in the [1, $v_{max}$] interval and higher values of this measure mean that the feature generates more discomfort to $u$.
Given $ea_{ufi}$, the compatibility of $f$ with $u$ in $i$, denoted as $comp_{fui}$ (in [1, $v_{max}$]), can thus be defined as:
\begin{equation}
    comp_{fui} = v_{max} + 1 - ea_{ufi}
\end{equation}

\subsection{Overall Item Compatibility: Aggregation Measures}
\label{sec:itemCompatibility}
We propose alternative aggregation measures to compute the overall compatibility of an item $i$ with a user $u$ ($comp_{iu}$) by modeling different types of influence of individual features. In Section \ref{sec:results} we evaluate their performance, in combination with different recommendation algorithms.
\begin{itemize}
    \item
    {\bf Min:} $comp_{iu}$ is the minimum compatibility of $i$'s features with $u$:
    \begin{equation}
    \label{eq:min}
        comp_{iu} = \min\limits_{f\in F} comp_{fui}
    \end{equation}
    This measure is conjunctive: it evaluates an item as incompatible with $u$ if it has at least one totally incompatible feature.

\end{itemize}
\begin{itemize}

    \item
    {\bf Ave:} $comp_{iu}$ is the mean compatibility of $i$'s features:
    \begin{equation}
    \label{eq:mean}
        comp_{iu} = \frac{\sum\limits_{f\in F} comp_{fui}}{|F|}
    \end{equation}
    where $| \cdot |$ denotes set cardinality. This measure is additive (disjunctive) and equally balances the influence of all the features on compatibility.

\end{itemize}
The following aggregation measures estimate overall compatibility in function of the distance between the features of $i$ (stored in the $\vec{{\bf i}}$ vector) and those of an $ideal$ item which best matches $u$'s idiosyncrasies ($\overrightarrow{{\bf{ideal_u}}}$). For each $f \in F$, $\overrightarrow{{\bf{ideal_{uf}}}}$ is the most compatible value of $f$ on the basis of $u$'s estimated aversion presented in Section \ref{sec:featureCompatibility}. This value is $1$ for $f \in F^\uparrow$, while it is the feature value associated to the minimum aversion for $f \in F^V$; e.g., $\overrightarrow{{\bf{ideal_{uf}}}} = 2.7$ in the right graphic of Figure \ref{fig:aversion} ($X$ axis). The overall compatibility of $i$ with $u$ is thus computed as follows:
\begin{itemize}

    \item
    {\bf Cos:} $comp_{iu}$ is the Cosine similarity between $\vec{{\bf i}}$ and $\overrightarrow{{\bf{ideal_u}}}$:
    \begin{equation}
    \label{eq:cosine}
        comp_{iu} = \frac{\vec{{\bf i}} \cdot \overrightarrow{{\bf{ideal_u}}}} {\Vert \vec{{\bf i}}\Vert_F \; \Vert \overrightarrow{{\bf{ideal_u}}}\Vert_F}
    \end{equation}
    where $\cdot$ is the scalar vector product and $\Vert \cdot \Vert_F$ is the Frobenius Norm.
    The smaller is the angle between $\vec{{\bf i}}$ and $\overrightarrow{{\bf{ideal_u}}}$, the more compatible is $i$ with $u$.
    \item
    {\bf RMSD:} $comp_{iu}$ is the complement of the Root Mean Square Deviation between $\vec{{\bf i}}$ and $\overrightarrow{{\bf{ideal_u}}}$:
    \begin{equation}
    \label{eq:rmsd}
        comp_{iu} = v_{max} +1 - \sqrt{\frac{1} {|F|}* \sum\limits_{f\in F} ( \vec{{\bf i}}_f  - \overrightarrow{{\bf ideal}_{uf}} )^2}
    \end{equation}
    The smaller is the distance between $\vec{{\bf i}}$ and $\overrightarrow{{\bf{ideal_u}}}$, the more compatible is $i$ with $u$.
\end{itemize}

\subsection{Preference-based Item Evaluation}
\label{sec:preferences}
While compatibility indicates whether the user can safely experience an item, it does not mean that (s)he likes it. User preferences have to be taken into account for this purpose.

In our domain, the only preference that we consider is the user's interest in the category of the item to be evaluated. Thus, the preference value of a user $u$ for an item of category $c \in C$ corresponds to the value of $u$'s preference for $c$ stored in $u$'s profile. We denote this value as $p_{uc}$.

It is worth mentioning that, if more preferences had to be modeled, a classical Multi-Criteria Decision Analysis approach might be applied to compute an overall preference estimation as a weighted function of preferences for individual attributes \cite{Adomavicius-Kwon:07,Adomavicius-Kwon:11,Jannach-etal:14}. However, this is out of the scope of the present work.

\subsection{Rating Prediction}
\label{sec:overallItemEvaluation}
In order to take personal balance between compatibility and preferences into account, we propose to identify user-dependent evaluation criteria by exploiting the user's idiosyncrasies and preferences in combination with the ratings of items (s)he provides, which we consider as the ground-truth revealing her/his priorities.
For this purpose, we define the overall evaluation of items, which produces the estimated rating for each user, as a weighted mean of items compatibility and user preferences:
\begin{equation}
    \hat{r}_{ui} = \alpha * comp_{iu} + (1-\alpha) * p_{uc_i}
    \label{eq:est_rating}
\end{equation}
where $\alpha$ takes values in the [0, 1] interval and $p_{uc_i} \in L$ is the preference-based evaluation of $i$, given $u$'s profile. This model, henceforth referred as {\bf Ind} (i.e., Individual), identifies a specific $\alpha$ value for each user to optimize item recommendation to her/him.
We identify the value of $\alpha$ for each $u \in U$ as the one that minimizes the distance between estimated ratings and ground-truth ones.

\section{Validation methodology}
\label{sec:validation}
We aim at assessing the usefulness of modeling both compatibility and preference aspects for recommendation, with respect to taking only one of these aspects into account. Moreover, we aim at evaluating the usefulness of a personalized balance between these two types of information, specified by the $\alpha$ parameter of Equation \ref{eq:est_rating}. For these purposes we compare our model to baseline recommenders which (i) uniformly manage compatibility and user preferences, i.e., they ignore the possibly different impact on decision-making) or (ii) focus on a single aspect, either compatibility or preferences.
Specifically, we consider the following baselines:
\begin{itemize}
    \item
    {\bf Multi-Criteria (MC):} this recommender system estimates item ratings by uniformly treating idiosyncratic features and preferences on the basis of the aggregation measures described in Section \ref{sec:itemCompatibility}. It computes $\hat{r}_{ui}$ by fusing $u$'s preference for the category of $i$ ($p_{uc_i}$) with the compatibility of individual features ($comp_{fui}$) by means of a single aggregation function; e.g., the mean of all these values as in Ave (Equation \ref{eq:mean}).

    \item
    {\bf C-only:} this is a setting of our recommendation model (Equation \ref{eq:est_rating}) in which $\alpha=1$ to evaluate items exclusively on the basis of their compatibility with the user.
    \item
    {\bf Pref-only:} in this setting of our model, $\alpha=0$ to evaluate items on the exclusive basis of the user's preferences.
\end{itemize}
We did not select as baselines any collaborative or feature-based recommenders such as those proposed in \cite{Han-etal:05} or \cite{Adomavicius-Kwon:07} because there is not enough data to train the recommenders.

We compare our model to the above baselines on both the autistic users dataset (henceforth denoted as AUT) and on the neurotypical users one (NOR). For the comparison we configure all the algorithms on each aggregation measure of Section \ref{sec:itemCompatibility}. For each algorithm, we denote the specific configuration we apply by appending its name to that of the algorithm; e.g., Ind$_{Cos}$ represents the application of the Cos aggregation measure to model Ind.

To evaluate recommendation performance we focus on ranking capability (MRR and MAP), accuracy (Precision, Recall and F1), error minimization (MAE and RMSE) and user coverage.
We perform a 5-fold cross validation in which, for every fold, we use 80\% as training set and 20\% as test set. As the Ind models have to optimize the $\alpha$ parameter, we train each of them to find the best user-specific setting by optimizing its results with respect to MAP. Moreover, to be sure that the baselines are consistently evaluated, we run the other algorithms (MC, C-only and Pref-only, which do not need any training) on the same test sets used for Ind.

\section{Evaluation results and discussion}
\label{sec:results}
Tables \ref{tab:autistici} and \ref{tab:neurotipici} show the Top-N evaluation results with N=5; i.e., the list of suggested items has length=5. The tables omit the results concerning user coverage because it is 100\% in all the cases.

We consider two categories of algorithms, i.e., the configurations of our model on the various aggregation measures and the corresponding ones of the baselines, and we use the following notation:
the best value across all algorithms is printed in bold; the best value obtained by the other category of algorithms is underlined (when our model obtains the best value we underline the best value achieved by the baselines, and {\em vice versa}). Stars indicate significant differences according to a Student T-Test between the best performing algorithm from each category; **: p<0.01; *: p<0.05.
It can be noticed that:
\begin{itemize}
    \item
    {\it Ind$_{Cos}$ excels in accuracy and ranking capabilities:} on both datasets it outperforms all the other algorithms (baselines and own category) in F1 and MAP, and it has the best Recall of its own category. Moreover, it obtains better MRR values than all algorithms on AUT and most baselines on NOR. As far as Precision is concerned, Ind$_{Cos}$ is the third best algorithm on AUT while it is the fourth best on NOR; in both cases it is outperformed by another member of its own category. {\it The results concerning error minimization are mixed:} on both datasets the MAE of Ind$_{Cos}$ is worse than that of most of the other algorithms, including  those of its own category. However, on NOR, it has the best RMSE of its own category and it achieves better results than most of the baselines.
    \item
    {\it Ind$_{Min}$ excels in error minimization:} on both datasets it obtains the best MAE of all algorithms; moreover, concerning RMSE, it outperforms all algorithms in AUT and the other algorithms of its own category in NOR. {\it Ind$_{Min}$ also has fairly good ranking capability:} it is the second best algorithm for MAP on both datasets and it has the best MRR on NOR, and second best MRR of its own category on AUT. {\it It has good Precision:} it achieves the best one of all algorithms on AUT and the third best on NOR. On both datasets it outperforms most baselines in Recall. Moreover, on AUT it has the third best F1 of all algorithms (lower than Ind$_{Cos}$ and C-only$_{Cos}$), while its accuracy on NOR is in middle position.
    \item

    {\it Ind$_{Ave}$ has the third best MAP  results of its own category, the third and second best MRR on AUT and NOR respectively, } and, concerning these metrics, it is positioned in the higher part of the general classification. As far as Precision, Recall and F1 are concerned, it achieves intermediate results on AUT but it is placed in the top positions of the classification on NOR. It has {\it fairly good error minimization capability (MAE, RMSE),} better than several baselines, and it is placed in second position within its own category on both datasets.

    \begin{table}[t]
\centering
\caption{Results on AUT dataset for N=5. The lines of the table are ordered by MAP. The best values of each measure across all algorithms is printed in bold. The best value obtained by the other category of algorithms is underlined. Stars denote statistical significance: **: p<0.01; *: p<0.05.}
\vspace{-10pt}
\resizebox{\columnwidth}{!}{%
{\def1.4{1.4}

\begin{tabular}{lrrrrrrr}
\toprule
Algorithm       & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Recall} & \multicolumn{1}{c}{F1} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & \multicolumn{1}{c}{MAE} & \multicolumn{1}{c}{RMSE} \\ \midrule
Ind$_{Cos}$     & 0.6290             & \underline{0.6207} & \textbf{0.6046}    & \textbf{**0.5384}  & \textbf{0.8095}    & 0.9927             & 1.4541             \\
Ind$_{Min}$     & \textbf{0.6328}    & 0.5832             & 0.5910             & 0.5125             & 0.7825             & \textbf{0.8691}    & \textbf{*1.3020}   \\
Pref-only       & 0.6220             & 0.5912             & 0.5860             & \underline{0.5114} & 0.7858             & \underline{0.9346} & \underline{1.4276} \\
Ind$_{Ave}$     & 0.6118             & 0.5710             & 0.5736             & 0.4960             & 0.7667             & 0.9168             & 1.3659             \\
C-only$_{Cos}$  & 0.6263             & \textbf{0.6224}    & \underline{0.6001} & 0.4877             & 0.7583             & 1.3675             & 1.6948             \\
Ind$_{RMSD}$    & 0.5978             & 0.5545             & 0.5577             & 0.4799             & 0.7537             & 0.9965             & 1.4533             \\
MC$_{Ave}$      & 0.6255             & 0.5383             & 0.5575             & 0.4489             & 0.7792             & 1.1902             & 1.4861             \\
MC$_{RMSD}$     & 0.6080             & 0.5396             & 0.5463             & 0.4429             & 0.7775             & 1.2172             & 1.5426             \\
MC$_{Min}$      & \underline{0.6305} & 0.5057             & 0.5344             & 0.4352             & \underline{0.7950} & 1.4512             & 1.7943             \\
MC$_{Cos}$      & 0.5917             & 0.5558             & 0.5459             & 0.4336             & 0.7217             & 1.3534             & 1.6236             \\
C-only$_{Min}$  & 0.6065             & 0.4999             & 0.5230             & 0.4166             & 0.7583             & 1.3675             & 1.6816             \\
C-only$_{Ave}$  & 0.5912             & 0.5154             & 0.5270             & 0.4142             & 0.7192             & 1.3045             & 1.6060             \\
C-only$_{RMSD}$ & 0.5825             & 0.5009             & 0.5145             & 0.4036             & 0.7142             & 1.3702             & 1.7168             \\ \bottomrule
\end{tabular}}
}
\label{tab:autistici}
\end{table}

\begin{table}[t]
\centering
\caption{Results on NOR dataset for N=5. We use the same notation of Table \ref{tab:autistici}.}
\vspace{-10pt}
\resizebox{0.96\columnwidth}{!}{%
{\def1.4{1.4}

\begin{tabular}{lccccccc}
\toprule
Algorithm       & \multicolumn{1}{c}{Prec.} & \multicolumn{1}{c}{Recall} & \multicolumn{1}{c}{F1} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & \multicolumn{1}{c}{MAE} & \multicolumn{1}{c}{RMSE} \\ \midrule
Ind$_{Cos}$     & 0.5790             & \underline{0.5406} & \textbf{0.5349}    & \textbf{0.4139}    & 0.7475             & 1.1792             & 1.5232             \\
Ind$_{Min}$     & 0.5791             & 0.5225             & 0.5250             & 0.4120             & \textbf{0.7688}    & \textbf{1.0950}    & \underline{1.4024} \\
Ind$_{Ave}$     & 0.5740             & 0.5261             & 0.5262             & 0.4108             & 0.7555             & 1.1085             & 1.4343             \\
Ind$_{RMSD}$    & \textbf{0.5816}    & 0.5286             & 0.5297             & 0.4108             & 0.7521             & 1.1427             & 1.4758             \\
Pref-only       & \underline{0.5795} & 0.5408             & \underline{0.5347} & \underline{0.4076} & 0.7304             & 1.1416             & 1.5270             \\
C-only$_{Cos}$  & 0.5503             & \textbf{0.5414}    & 0.5255             & 0.4000             & 0.7189             & 1.4374             & 1.7456             \\
MC$_{Ave}$      & 0.5752             & 0.5154             & 0.5213             & 0.3995             & 0.7564             & \underline{1.1238} & {\bf 1.3564}       \\
MC$_{Min}$      & 0.5664             & 0.4956             & 0.5053             & 0.3890             & \underline{0.7583} & 1.1249             & 1.4052             \\
MC$_{RMSD}$     & 0.5568             & 0.4840             & 0.4963             & 0.3767             & 0.7433             & 1.3320             & 1.6255             \\
C-only$_{Ave}$  & 0.5476             & 0.4936             & 0.4979             & 0.3701             & 0.7168             & 1.2122             & 1.4668             \\
C-only$_{Min}$  & 0.5507             & 0.4769             & 0.4899             & 0.3673             & 0.7359             & 1.1704             & 1.4213             \\
C-only$_{RMSD}$ & 0.5460             & 0.4870             & 0.4936             & 0.3651             & 0.7223             & 1.4157             & 1.7281             \\
MC$_{Cos}$      & 0.5274             & 0.5053             & 0.4974             & 0.3535             & 0.6591             & 1.2775             & 1.5795             \\
 \bottomrule
\end{tabular}}}
\label{tab:neurotipici}
\vspace{-3mm}
\end{table}

    \item
    {\it Ind$_{RMSD}$ is the worst performing configuration of our model:} on AUT it obtains the lowest performance results of its own category on all measures, but it still outperforms several baselines in MAP and in other metrics. Differently, in NOR it achieves fairly good F1, the best Precision of all algorithms (including baselines), fairly good MAP and MRR and fairly good error minimization.
    \item
    {\it Pref-only is the best baseline regarding MAP} and it has {\it fairly good MRR and F1} on both datasets. This algorithm achieves the best MAE and RMSE of the baselines on AUT, while on NOR it is outperformed by other baselines. Except for RMSE in AUT and MAE in NOR, it achieves lower results than Ind$_{Cos}$ on all performance metrics.
    \item
    On both datasets {\it C-only$_{Cos}$ has lower ranking capability than the previous algorithms (MAP and MRR) but it has fairly good accuracy,} being the best, or second best baseline on the various measures. Its error minimization capability is definitely lower than that of our model. Notice that the other configurations of C-only (i.e., using Min, Ave and RMSD) have worse performance than this one on both datasets, in all metrics except for error minimization.
    \item
    Similarly, {\it the configurations of MC, except for MC$_{Min}$ in Precision and MRR, have middle to low performance}; they are outperformed by our model or by some other algorithms in both datasets.
\end{itemize}

The evaluation results suggest that Ind$_{Cos}$ is the best recommendation algorithm because it combines good ranking capability with good accuracy. Ind$_{Min}$ achieves better error minimization than this algorithm but, as previously discussed, this is a secondary evaluation criterion for us.

Unfortunately, the low size of the AUT and NOR datasets does not support the statistical significance of results for several metrics. However, the results concerning MAP (and RMSE) on the AUT dataset are significant. This is important because our recommendation model is targeted to autistic people and we can thus rely on the ranking capability results we obtained on them. At the same time, the results are encouraging for neurotypical users. Thus, it is worth investigating performance within a larger experiment that will possibly provide more statistically relevant results on both groups of people.

The evaluation results help us answering our research questions:
\begin{itemize}
    \item
    {\em RQ1:} we can positively answer this question. As far as F1 and ranking capability are concerned, the configurations of Ind that take both preferences and compatibility into account (and, specifically, Ind$_{Cos}$) outperform Pref-only, which only exploits user preferences to recommend items. Moreover, they achieve better results than all the algorithms that only use compatibility information (C-only). These algorithms are outpeformed by Pref-only, too. This means that, not surprisingly, compatibility information alone is not enough to generate relevant recommendations for the user.
    \item
    {\em RQ2:} we can positively answer this question as well. In fact, the Ind configurations outperform the MC ones, regardless of the used aggregation measure, in most evaluation metrics and especially in ranking capability and F1.
\end{itemize}
To summarize, in Top-N PoI recommendation preference information is useful to suggest relevant items. However, better results can be achieved by combining this type of information with a compatibility evaluation aimed at assessing whether the user can serenely experience the recommended items. Interestingly, a uniform management of compatibility and preference information, which does not distinguish the possibly heterogeneous evaluation criteria concerning them, does not bring good results. Conversely, the acquisition of user-specific weights to balance the impact of compatibility and interests in item evaluation improves item suggestion.

\section{Conclusions}
\label{sec:conclusions}
Users with Autism Spectrum Disorder (ASD) are a particularly interesting and challenging target of PoI recommender systems because of their characteristics and needs in relation to places. In order to suggest suitable solutions, which the user can like {\em and} serenely experience, both her/his preferences for PoI categories, traditionally analyzed by researchers, and her/his aversions to sensory features, have to be considered: as a matter of fact, the latter can seriously affect ASD people's experience of the places, causing negative feelings. Notice that it is particularly important to avoid wrong suggestions because there might be critical effects on the person, given her/his “frailty”, causing for example anxiety, irritation and anger, with unpredictable consequences.

In this paper we presented a Top-N recommender of PoIs for ASD people that takes their idiosyncratic aversions to sensory features into account in order to generate suggestions that are expected to be both pleasant and safe for them.
We tested our model on autistic and neurotypical users.
The evaluation results show that, on both user groups, our model outperforms in accuracy and ranking capability baseline recommenders which (i) evaluate items on the sole basis of how closely they meet the user's preferences, or how compatible they are with her/his idiosyncratic aversions to sensory features, and (ii) uniformly manage compatibility and preference information without distinguishing the possibly different contributions of these aspects to item evaluation.
We thus conclude that the integration of possibly heterogeneous evaluation criteria concerning user interests and idiosyncratic aversions is a promising approach to extend the adoption of recommender systems to new user groups with respect to those typically addressed in the state of the art.

Two main limitations of our approach concern the data about PoIs and users available for the experiment.
We plan to address these limitations as follows:
\begin{itemize}
\item
 We will extend information about PoIs in two ways: (i) by starting a VGI campaign with people with ASD and their caregivers, as well as with the general population, to acquire a larger amount of data, and (ii) by extracting sensory information from consumer reviews available in online platforms such as TripAdvisor \cite{TripAdvisor}; e.g., see \cite{Bilici-Saygin:17}.
\item When the PIUMA app will be available, we will be able to learn detailed information about user interests and aversions by coupling an initial bootstrapping of user profiles via form filling with a subsequent refinement based on an analysis of user behavior.
\end{itemize}
Another limitation is related to the questionnaire for gathering user's sensory aversion: it has been derived from a state-of-art SPQ questionnaire, but a correlation between the two has to be computed. We plan to carry out this analysis as well.

\begin{acks}
This work is supported by the COMPAGNIA di SAN PAOLO Foundation. We thank Stefano Cocomazzi, Stefania Brighenti and Claudio Mattutino for their contributions to the work.
\end{acks}

\bibliographystyle{ACM-Reference-Format}

Information Retrieval (cs.IR); Machine Learning (cs.LG)


SlideImages: A Dataset for Educational Image Classification

Abstract: In the past few years, convolutional neural networks (CNNs) have achieved impressive results in computer vision tasks, which however mainly focus on photos with natural scene content. Besides, non-sensor derived images such as illustrations, data visualizations, figures, etc. are typically used to convey complex information or to explore large datasets. However, this kind of images has received little attention in computer vision. CNNs and similar techniques use large volumes of training data. Currently, many document analysis systems are trained in part on scene images due to the lack of large datasets of educational image data. In this paper, we address this issue and present SlideImages, a dataset for the task of classifying educational illustrations. SlideImages contains training data collected from various sources, e.g., Wikimedia Commons and the AI2D dataset, and test data collected from educational slides. We have reserved all the actual educational images as a test dataset in order to ensure that the approaches using this dataset generalize well to new educational images, and potentially other domains. Furthermore, we present a baseline system using a standard deep neural architecture and discuss dealing with the challenge of limited training data.

\section{Introduction}
\label{sec:org8c5fa32}

Convolutional neural networks (CNNs) are making great strides in computer vision, driven by large datasets of annotated photos, such as ImageNet \cite{imagenet_cvpr09}. Many images relevant for information retrieval, such as charts, tables, and diagrams, are created with software rather than through photography or scanning.

There are several applications in information retrieval for a robust classifier of educational illustrations. Search tools might directly expose filters by predicted label, natural language systems could choose images by type based on what information a user is seeking. Further analysis systems could be used to extract more information from an image to be indexed based on its class. In this case, we have classes such as pie charts and x-y graphs that indicate what type of information is in the image (e.g., proportions, or the relationship of two numbers) and how it is symbolized (e.g., angular size, position along axes).

Most educational images are created with software and
are qualitatively different from photos and scans.
Neural networks designed and trained to make sense of the noise and spatial relationships in photos are sometimes suboptimal for born-digital images and educational images in general.

Despite their practical relevance, educational images and illustrations have been comparatively underserved in training datasets and challenges. Competitions such as the
Contest on Robust Reading for Multi-Type Web Images \cite{mtwiContest} and ICDAR
DeTEXT \cite{detextContest} have shown that these tasks are difficult and
unsolved. Research on text extraction such as Morris et al. \cite{self} and Nayef
\& Logier \cite{frenchLocalization} has shown that even noiseless born-digital
images are sometimes better analyzed with neural nets than with handcrafted
features and heuristics. Born-digital and educational images need further benchmarks related to challenging information
retrieval tasks in order to test the generalization of methods for those tasks.

In this paper, we propose SlideImages, a dataset which targets images from educational presentations. Most of these educational illustrations are created with diverse software, so the same symbols are drawn in different ways in different parts of the image. As a result, we expect that effective synthetic datasets will be hard to create, and methods effective on SlideImages will generalize well to other tasks with similar symbols. SlideImages contains eight classes of image types (e.g. bar charts and x-y plots) and a class for photos. The labels we have created were made with information extraction for image summarization in mind.

In the rest of this paper, we discuss related work in §2, details about
our dataset and baseline method in §3, results of our baseline method in §4, and conclude with a discussion of potential future developments in §5.
\section{Related Work}
\label{sec:orgd0b8d5a}

Prior information retrieval publications have used, or  could make use of, document figure classification. Charbonnier et al. \cite{luciaECIR} built
a search engine which allows user to filter images based on type, Aletras \&
Mittal \cite{DBLP:conf/ecir/AletrasM17} automatically
label topics in photos, Kembhavi et al. \cite{diagram} build a system for extracting the relationships between entities in a diagram, which assumes that the input figure is a diagram, Hiippala \& Orekhova extended their dataset by annotating it semantically in terms of Relational Structure Theory, which implies that the same visual features communicate the same semantic relationships, and de Herrera
et al. \cite{DBLP:conf/ecir/HerreraMJSFM15} seek to classify image types to filter
their search for medical professionals.

We intend to use document figure classification as a first step in automatic educational image summarization applications. A similar idea is followed by Morash et
al. \cite{DBLP:journals/taccess/MorashSMHL15}, who built one template for each type of image, then manually classified images and filled out
the templates, and suggested automating the steps of that
process. Moraes et al. \cite{DBLP:conf/assets/MoraesSMC14} mentioned the same idea for their SIGHT (Summarizing Information GrapHics Textually) system.

A number of publications on document image classification
such as Afzal et al. \cite{errorByHalf} and Harley et al. \cite{rvl-cdip} use the RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) dataset, which
covers scanned documents.  While document scans and born-digital educational
illustrations have materially different appearance, these papers show that
the utility of deep neural networks is not limited to scene image tasks.

A classification dataset of scientific illustrations was created for the NOA project \cite{luciaTPDL}.
However, their dataset is not publicly available, and does not draw as
many distinctions between types of educational illustrations.

Jobin et al.'s DocFigure \cite{docfigure} consists of 28 different categories of illustrations
extracted from scientific publications totaling 33,000 images.
\section{Dataset and Baseline System}
\label{sec:orgaa965cd}
Techniques that work well on DocFigure \cite{docfigure} do not generalize to the educational illustrations in our use case scenarios (as we also show in section 4.2).
This suggests a dataset of specifically
educational illustrations is needed.

 CNNs and related techniques are heavily data driven. An approach must consist of both an architecture and optimization technique, but also the data used for that optimization.  In our case, we consider the dataset  our main contribution.

\subsection{SlideImages Dataset}
\begin{table}[!t]
\caption{\label{tab:org783f50b}
Dataset sizes, including reduced datasets for head-to-head comparison}
\centering
\begin{tabular}{lrrrr}
 & Classes & Train & Val & Test\\
\hline
SlideImages & 9 & 2646 & 292 & 691\\
DocFigure & 28 & 19795 & 0 & 13172\\
Head-to-Head SlideImages & 8 & 2331 & 257 & 575\\
Head-to-Head DocFigure & 8 & 11678 & 3886 & 3891\\
\end{tabular}
\end{table}

\begin{figure}[!t]
\centering
\includegraphics[width=.6\linewidth,trim=0 0 0 0, clip]{./dataset_ex.pdf}
\caption{\label{fig:dataset_ex}
Examples of our classes from training data. Clockwise from top left: bar charts, photos, pie charts, slide images, tables, structured diagrams, technical drawings, x-y plots, and maps.}
\end{figure}

When building our taxonomy, we have chosen classes such that one class would have the same types of salient features, and appropriate summaries would also be similar in structure. Our classes are also all common in educational materials. Beyond the requirements of our taxonomy, our datasets needed to be representative of common educational illustrations in order to fit real-world applications. We also aimed to create a dataset which we could legally re-share in order to promote research on the classification of educational images.

Educational illustrations are created by a variety of
communities with varying expertise, techniques, and tools, so choosing a
dataset from one source may eliminate certain variables in educational
illustration.
To identify these variables, we kept our training and test data sources separate.

We have assembled a training dataset from various sources of open access
illustrations. Bar charts, x-y plots, maps, photos, pie charts, slide images,
table images, and technical drawings have been manually selected from Wikimedia
Commons images using the Wikimedia Commons search for related terms, and
choosing a varied group of suitable images. Graph diagrams, which we also call node-edge diagrams or ``structured diagrams,'' have been selected manually from the AllenAI
Diagram Understanding (AI2D) dataset described by Kembhavi et al. \cite{ai2d},
since not all of the diagrams in AI2D contain graph edges. AI2D was an ideal source of images for SlideImages, since Kembhavi et al. chose images which represented topics from grade school science \cite{ai2d}.

The test dataset of SlideImages is derived from a snapshot of SlideWiki open
educational resource platform (https://slidewiki.org/) datastore obtained in 2018.  From that snapshot, we manually selected and labeled 691 images.
We will make our training and test datasets available online and link them in
the camera-ready version of this paper.

\subsection{Baseline Approach}
\label{sec:methodology}
The SlideImages training dataset is small compared to datasets like ImageNet \cite{imagenet_cvpr09}, with over 14 million images, RVL-CDIP \cite{rvl-cdip} with 400,000 images, or even DocFigure \cite{docfigure} with 33,000 images.  Much of our methodology is shaped by needing to confront the challenges of a small dataset. In particular, we aim to avoid overfitting: the tendency of a classifier to identify individual images and patterns specific to the training set rather than the desired semantic concepts.

For our pre-training dataset, a large, diverse dataset is required that contains a large proportion of educational and scholarly images.
We pre-trained on a dataset of almost 60,000 images labeled by Sohmen et al.~\cite{luciaECIR} (NOA dataset), provided by the authors on request. The images are
categorized as composite images, diagrams, medical imaging, photos, or visualizations/models.

We used image augmentation to help combat overfitting.  Distorting an image in ways which do not remove the characteristic qualities helps to disrupt spurious patterns that the network might otherwise pick up.  We used image stretching, brightness scaling, zooming, and color channel shifting, with details shown in our source code.  We also added dropout  with a rate of 0.1 on the extracted features before running our fully connected and output layers.  We used similar image augmentation for pre-training and training.

We use MobileNetV2 \cite{mobilenetv2} as our network architecture. We chose MobileNetV2 since it provides a balance between a small number of
parameters and proven performance on ImageNet. Intuitively, a smaller parameter space implies a model with more bias and lower variance, which is better for smaller datasets.  We initialized our
weights from an ImageNet model and pre-trained for a further 40 epochs
with early stopping on the NOA dataset using the Adam (adaptive moment estimation) \cite{adam} optimizer.  This additional pre-training was intended to cause the lower levels of the network to extract more features specific to born-digital images. We then trained for 40 epochs with Adam and a learning rate schedule.
Our schedule drops the learning rate by a factor of 10 at the 15th and 30th epoch.

\section{Preliminary Results}
\label{sec:org604bce1}

\begin{figure}[!t]
\centering
\includegraphics[width=.5\linewidth,trim=55 168 355 60, clip]{./confusion_matrix.pdf}
\caption{\label{fig:org06abea2}
Confusion matrix of our baseline system on SlideImages. Entries show percent of true members of the class on the left margin labeled as on the bottom margin. Weighted accuracy average is 80\% over all 691 images.}
\end{figure}

We have performed two experiments, in order to show that this dataset represents
a meaningful improvement over existing work, and to establish a baseline. Because our classes are unbalanced, we have reported summary statistics as accuracy averages of each class weighted by number of instances per class.

\subsection{Baseline}
\label{sec:orgc7567cb}
We have used the classifier described in section \ref{sec:methodology} to generate a
baseline for our dataset.
The confusion matrix in Fig. \ref{fig:org06abea2} shows that while
misclassifications do tend towards a few types of errors, none of the classes
have collapsed; while certain classes are likely to be misclassified as another
specific class (such as structured diagrams as slides), those relationships
do not happen in reverse, and a correct classification is more likely. Fig. \ref{fig:org06abea2} shows that our baseline leaves room for improvement, and our test set helps to identify challenges in this task. Viewing individual classification errors of our baseline highlighted a few
problems with our existing training data. Our training data do not include
sufficient structured diagrams with wide, illustrated arrows, or edges which
travel only at 90° increments, such as organigrams charts or many types of UML (Unified Modeling Language) diagrams.
Our photos do not include examples with the background removed, but these are
common in educational images. These problems should be remedied in future
training datasets in this and similar problems.

\begin{table}[htbp]
\caption{\label{tab:org9d01920}
Head-to-head comparison of precision weighted averages.}
\centering
\setlength{\tabcolsep}{6pt}
\begin{tabular}{lccc}
 & SlideImages Train & DocFigure Train & DocFigure Baseline\\
SlideImages Test & 80\% & 78\% & 75\%\\
DocFigure Test & 92\% & 99\% & 99\%\\
\end{tabular}
\end{table}

\subsection{Head-to-head Comparison}
\label{sec:org9b8ff98}
The related DocFigure dataset covers similar images and has much more data than
SlideImages.  To
justify SlideImages, we have created a head-to-head comparison of classifiers
trained in the same way (as described in section \ref{sec:methodology}) on the
SlideImages and DocFigure datasets. All the SlideImages
classes except \emph{slides} have an equivalent in DocFigure. We have shown the reduction in the data used, and the relative sizes of
the datasets, in Table \ref{tab:org783f50b}; the Head-to-Head datasets contain only the matching classes, and in the case of the DocFigure dataset,
the original test set has been split into validation and test sets.

After obtaining the two trained networks, we have tested each network on both
the matching testing set, and the other testing set. Although we were unable to reproduce the VGG-V baseline used by Jobin et al., we used a linear SVM with VGG-16 features and achieved comparable results on the full DocFigure dataset (90\% macro average compared to their 88.96\% with a fully neural feature extractor). We show our results in
Table \ref{tab:org9d01920}. The results show that SlideImages is a more challenging and
potentially more general task; the net trained on SlideImages did even better on
the DocFigure test set than on the SlideImages test set. Despite having a different source and
approximately a fifth of the size of the DocFigure dataset, our training set was
better on our test set.

\section{Conclusions and Future Work}
\label{sec:org9b50a5d}
In this paper, we have presented the task of classifying educational illustrations and images in slides and introduced a novel dataset SlideImages for this task. The classification remains an open problem despite our baseline and represents a useful task for information retrieval. We
have provided a test set derived from actual educational illustrations, and a
training set compiled from open access images. Finally, we have established a baseline system for the classification task.

Other potential avenues for future research include experimenting with the DocFigure
dataset in the pre-training and training phases, and experimenting with text
extraction for multimodal classification.
\section{Acknowledgement}
This work is financially supported by the German Federal Ministry of Education and Research (BMBF) and European Social Fund (ESF) (InclusiveOCW project, no. 01PE17004).
\bibliographystyle{ieeetr}

Computer Vision and Pattern Recognition (cs.CV); Information Retrieval (cs.IR)


Stopped light with storage times greater than one second using EIT in a solid

Abstract: We report on the demonstration of light storage for times greater than a second in praseodymium doped Y$_2$SiO$_5$ using electromagnetically induced transparency. The long storage times were enabled by the long coherence times possible for the hyperfine transitions in this material. The use of a solid state system also enabled operation with the probe and coupling beam counter propagating, allowing easy separation of the two beams. The efficiency of the storage was low because of the low optical thickness of the sample, as is discussed this deficiency should be easy to rectify.

\section{\label{sec:intro}Introduction}

Some of the most significant advances in quantum information
processing have been made using quantum optics-based techniques.  For
example, working practical quantum cryptosystems already exist and
there have been demonstrations of linear optics quantum computing
\cite{obri03}, quantum teleportation, quantum non-demolition
measurements \cite{Pryd04}, quantum feedback and control
\cite{stoc04}. To proceed further it is necessary to have devices such
as single photon sources, quantum memories and quantum repeaters,
where quantum information is exchanged in a controlled fashion between
light fields and material systems. It has been proposed that both the
required control and strong coupling can be readily achieved using an
ensemble approach, where the light field interacts with a large number
of identical atoms. Such a ensemble based approaches now exist for single
photon sources \cite{dlcz}, `cat' state sources \cite{pate03}, quantum
memories \cite{flei02, mois01, eisa04} and quantum repeaters
\cite{dlcz}.  Experiments have demonstrated heralded single photon
sources \cite{kuzm03, jian04} and the mapping of quantum information
on a light field onto spin states of an atomic ensemble \cite{Juls04}.
Experiments using electromagnetic induced transparency have
demonstrated the storage and recall of optical pulses \cite{liu01,
  phil01}.

The quantum systems used for these ensemble based demonstrations have
almost exclusively been atomic vapors. An issue with these
demonstrations is that even for a laser cooled ensembles, the atomic
motion impacts on the devices' performance. Ensembles of solid-state
optical centers provide an alternative to atomic systems where the
relative motion is zero.  In this paper we investigate the use of
solid-state system for ensemble based quantum optics and highlight its
usefulness by stopping a light pulse using electromagnetically induced
transparency (EIT). Unlike an earlier experiment \cite{turu02}, the
current demonstration highlights for the first time two advantages of
using optically active solid state centers: a one thousand fold
increase in storage time and the ability to operate with a less
restrictive beam geometry.

When storing light using EIT characteristics of the field are recorded
as a spin wave in the ensemble. The storage time is determined by the
coherence times of the hyperfine transitions. In principle coherence
times for hyperfine transitions in atomic systems can be very long and
many minutes have been measured in ion traps \cite{fisk95}. However,
these long coherence times in large ensembles suitable for EIT have
not been achieved. Transit time broadening in vapor cells and magnetic
inhomogeneity in trapped systems mean that the longest that light has
been stored atomic systems is a few  milliseconds. In contrast, in
earlier work we have demonstrated techniques to obtain hyperfine
coherence times of tens of seconds in Pr:Y$_2$SiO$_5$\ \cite{frav04a,frav04b}.
Here we show it is possible to utilize these long coherence times to
stop light for similar lengths of time.

EIT is sensitive to atomic movement, with the spin wave being
scrambled once the atoms have moved significantly compared to the
wavevector mismatch between the probe and the coupling beams. To
minimize this wavevector mismatch experiments in atomic systems
typically operate with the beams co-propagating. Because the probe
and coupling beams are close in frequency, in this configuration,
the wavevector mismatch is typically less than 1~cm$^{-1}$. A
consequence of this co-propagating operation is that it is
difficult to separate the probe and the coupling beam. In a
solid-state system, where the optical centers are locked in a
crystal lattice, co-propagating operation is not required, in the
present work the probe and the coupling beams are
counter-propagating. With counter-propagating beams it is easier
to separate the probe and coupling beam whilst maintaining optimum
overlap.

The experimental setup is shown in FIG.~\ref{fig:setup}. Because
of the narrow 2500~Hz optical homogeneous linewidth of the $^3$H$_4$$\rightarrow$$^1$D$_2$
transition in Pr$^3+$:Y$_2$SiO$_5$ a highly frequency stabilized
dye laser was required for the experiment not to be limited by
laser jitter. The laser used was a modified Coherent 699 dye laser
with a linewidth 200~Hz over 1~second time scales. The laser
output was split into two beams, one of which was frequency
shifted and gated by two AOMs and used as the probe beam. The
other beam was frequency shifted and gated using a double pass AOM
setup. This beam was used for the coupling and repumping fields. This coupling/repumup beams was aligned on a beam-splitter to go through the sample counter-propagating with the probe. The
spare port of this right-most beam splitter was used to combine a local
oscillator beam with the transmitted probe beam,
enabling the heterodyne detection of the signal.

The sample used was the same as that used in reference \cite{frav04b}
and consisted of 0.05\% Praseodymium doped in Y$_2$SiO$_5$. It was
4~mm thick along the direction of light propagation.  The sample was
mounted in a bath liquid helium cryostat. Three orthogonal
super-conducting magnets were used to apply a DC magnetic field to the
sample and a six turn rf coil was used to apply a rf field.

\begin{figure}
  \includegraphics[width=0.75\textwidth]{setup}
  \caption{The experimental setup. What is not shown is a beam picked
    of the laser was put in the remaining port of the right most
    beam-splitter. This enabled heterodyne detection of the probe beam
  that was transmitted through the sample.}
  \label{fig:setup}
\end{figure}

The dominant dephasing mechanism for the hyperfine states of the
Pr$^{3+}$ ions is random Zeeman shifting due to fluctuating
magnetic fields from the yttrium nuclei. Dramatic increases in
coherence times can be achieved by operating at a magnetic field
where the transition frequency is insensitive to magnetic field
changes to first order \cite{frav04a}. The magnetic field required
is 78~mT in an  orientation described in Ref.~\cite{frav04a}.
Once the magnetic field is obtained  the
remaining fluctuations have reasonably long correlation times.
This situation enables the effective use of dynamic decoherence control
(DDC) techniques \cite{viol04} and coherence times in excess of
half a minute have been demonstrated \cite{frav04b}.

\begin{figure}
  \includegraphics[width=0.3\textwidth]{energy_levels}
  \caption{Energy level diagram showing transitions driven as part of
    the experiment. The experiment was carried out on the zero phonon
    line of the $^3$H$_4$$\rightarrow$$^1$D$_2$ optical
    transition. The hyperfine levels are shown and these are due to
    the $5/2$ spin of the praseodymium nuclei. In the presence of a
    magnetic field these are linear combinations of the zero field
    states. The probe, coupling and repump beams are labelled
    P, C and R respectively.}
  \label{fig:nrg_levels}
\end{figure}

An energy level diagram showing the transitions driven during the
experiments is shown in Fig.~\ref{fig:nrg_levels}. While the optical
inhomogeneous line widths is a few GHz. The narrow homogeneous
linewidth (of order 1~kHz) and long hyperfine population lifetimes (of
order 1~minute) enabled the experiment to be carried out on an
ensemble with a much smaller range of optical frequencies. At the
beginning of each shot a sequence of the five optical frequencies
(labelled ``R'' in Fig.~\ref{fig:nrg_levels}) was applied
repeatedly. The repump frequencies were applied sequentially rather
that all at once to avoid the possibility of darks states and
nonlinear mixing of the different frequencies in the AOM. The gap in time
between the repumping and each experimental shot was long enough to
ensure that ions had no  remaining optical coherence.
This repumping procedure prepared an ensemble of ions in the desired
hyperfine state and gives a narrow adsorption with an inhomogeneous
width of 100~kHz when measured by sweeping a week probe in frequency
(line given by dots in Fig.~3). When the coupling beam was applied a
narrow transparency was obtained in the absorption of the weak probe
(solid trace in Fig.~3).

\begin{figure}
  \centering
  \includegraphics[width=0.4\textwidth]{eit_exp}
  \caption{Transmission of a weak probe, 10~$\mu$W, as its frequency is swept
    through resonance with the prepared ensemble. The solid line was
    taken with a 1~mW coupling beam on and the dotted line with the
    coupling off.}
  \label{fig:eit_sweep}
\end{figure}

 The repumping
beams were applied after each shot and the 300~kHz span shown was
swept in 4~ms. The transmitted probe beam was detected as a
heterodyne beat signal and the bandwidth of the RF detector was
comparable to 300~kHz, the extra noise at each end of the spectrum
came from dividing out this frequency response. For coupling
intensities above  1~mW the EIT was observed to depend linearly
on the amplitude of the coupling beam. The limiting EIT width at
low intensity was 10~kHz, corresponding to the hyperfine
inhomogeneous linewidth. Below 1~mW the EIT transmission decreased
with decreasing coupling intensity.

It can bee seen from FIG.~\ref{fig:eit_sweep} that the peak
absorption of our ensemble is only about 15\% and, as is discussed
below, this limits the efficiency of the storing process.

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{superfig}
  \caption{ On the left is the time sequence used in the stopped light
    experiments (a) with simple rephasing of the inhomogeneous
    broadening of the spin transitions and (b) with ``bang-bang''
    dynamic decoherence control. In (a) two rephasing pulses are used
    placed 1/4 and 3/4 of the way through the storage time. In (b) N
    rephasing pulses were used (N even). The first rephasing pulse was
    applied 2~ms after the light was stored, the pulses were separated
    by 4~ms and the last pulse was applied 2~ms before the light was
    recalled. The rephasing pulses lasted 22~$\mu$s.  On the right is
    the size of the recalled pulse as a function of time. The
    faster decay was acquired using simple rephasing of the ground
    state spin coherence (a). The slower decay was acquired using
    ``bang-bang''\cite{viol98} decoherence control (b). The inset
    shows the energy of the recalled pulse as a function of the
    energy of the input pulse, the probe pulse length was 20~$\mu$s
    and the delay held constant at 100~ms.  }
  \label{fig:superfig}
\end{figure}

The time sequence for the light storage demonstration is shown on the
left of
FIG.~\ref{fig:superfig}. A 20~$\mu$s long probe pulse was
applied and then the 10~mW coupling beam was turned off to
transfer the optical coherence onto the spin transition. As in the
previous solid state stopped light \cite{turu02} experiment RF
rephasing pulses were used to rephase the inhomogeneous broadening
in the spin transition. Although one RF rephasing pulse is enough
to rephase the spin-wave it also flips the spin-wave's direction.
Therefore when not using co-propagating beams, as is the case
here, it is necessary to use an even number of rephasing pulses.

The size of the pulse of light recalled as a function of delay can be shown with and without dynamic
decoherence control (DDC) and the results are shown in FIG.~\ref{fig:superfig}. The decay constants for the stored
signal output were 0.35 seconds without DDC and 2.3 seconds with
DDC. These decay rates were comparable to measurements of $T_2$
made using the same method as Fraval et al. \cite{frav04a}. The
difference between the present measurements of $T_2$ and those
obtained by Fraval et al. \cite{frav04a} is attributed to not
having tuned the magnetic field as carefully as was achieved by Fraval et al.

Shown in the inset of FIG.~\ref{fig:superfig} is the intensity of the output
pulse as the intensity of the input probe pulse is varied. From
the graph it can be seen that the storage process is linear at low
powers and starts to saturate at higher powers once the input
pulse becomes a significant fraction of a $\pi/2$ pulse. This
demonstration of linearity is important. Previous solid state
experiments \cite{turu02} have been restricted  by laser frequency
jitter to using probe pulses with areas greater than $\pi$. At
such high powers effects such as self induced transparency (SIT)
\cite{mcca69} cannot be ignored.

While the effect was linear and scaled to low powers, the
efficiency was low, of the order of 1\%. This in part can be
improved with better timing and shaping of the probe and coupling
waveforms. However the main reason for the low efficiency is the
low optical absorption at the probe frequency and the accompanying
modest group delay.

The sample used for this experiment was only 4~mm thick,
longer samples as well as multi-pass cells and cavities are simple
means to increase the optical absorption. Preliminary measurements on
a samples with a range of praseodymium concentrations \cite{ala}
suggest that at least two or three fold increases in the optical
thickness can be achieved by increasing the concentration without
significantly increasing the inhomogeneous broadening of the hyperfine
transition.

As it is a goal of this line of research to store and retrieve quantum
mechanical states it worthwhile to consider the effect that rephasing
pulses would have on few photon states stored in the hyperfine
coherences. It has been asserted in a theoretical investigation of
quantum information storage in the solid state \cite{john04} that one
would not be able to apply the RF $\pi$ pulses with sufficient
accuracy. This is not the view of the authors of this paper. In
Ref.~\cite{john04} it was assumed that the $\pi$ pulse would
have to be applied with an accuracy close to 1 part in $N$ (where $N$
is the number of atoms) in order that the few photon pulse not be
swamped by light caused by inaccuracies of the $\pi$ pulse. However
this light will be emitted randomly rather than in the very precise
spatio-temporal mode of the output pulse. This should enable the
output pulse to be easily separated from the background with very high
efficiency.

In conclusion, we have demonstrate stopped light in Pr:Y$_2$SiO$_5$\ for time
scales of several seconds which is three orders of magnitude longer
than any obtained previously. Based on previous measurements of $T_2$
it should be possible to extend this storage time by at least one more
order of magnitude.

For the first time stopped light has been demonstrated in a solid with
the coupling and probe beams counter propagating.  This configuration
is desirable as it allows easy separation of the two beams.  However,
it is only practical if the atoms movement during the storage time is
small compared to the optical wavelength.  Even for ultra-cold systems
this places significant limits storage time.  In a solid where the
atoms are locked into position this isn't a problem.

The efficiency of the storage process required for a quantum memory
should be obtainable by increasing the density of the dopant ions and
by increasing the interaction length.

The authors would like to thank Philip Hemmer for helpful discussions.
We would like to acknowledge the support of the Australian Research Council and the Australian Department of Defense.