Facial Behavior Analysis using 4D Curvature Statistics for Presentation Attack Detection

Abstract: The human face has a high potential for biometric identification due to its many individual traits. At the same time, such identification is vulnerable to biometric copies. These presentation attacks pose a great challenge in unsupervised authentication settings. As a countermeasure, we propose a method that automatically analyzes the plausibility of facial behavior based on a sequence of 3D face scans. A compact feature representation measures facial behavior using the temporal curvature change. Finally, we train our method only on genuine faces in an anomaly detection scenario. Our method can detect presentation attacks using elastic 3D masks, bent photographs with eye holes, and monitor replay-attacks. For evaluation, we recorded a challenging database containing such cases using a high-quality 3D sensor. It features 109 4D face scans including eleven different types of presentation attacks. We achieve error rates of 11% and 6% for APCER and BPCER, respectively.

\section{Introduction}
\label{sec:intro}
The digital transformation of organizational processes to improve efficiency in terms of time and resources is progressing fast.
As a result, the demand for automatic and robust unsupervised authentication methods increases.
However, current authentication methods have limitations that make them prone to advanced spoofing attacks or identity thefts.
These \textit{presentation attacks} reduce trust in procedures for automated border control, financial transfer, and mobile payments using self-service eGates, kiosks, and mobile phones.

Biometric methods take advantage of the individuality of human physiological characteristics.
Typical systems are based on finger print, palm print, or iris \cite{zhang2005palm} identification.
Additionally, the field of behavioral biometrics provides methods to analyse voice \cite{cheyer2016device}, walking gait \cite{cattin2002biometric}, and keystroke dynamics \cite{monrose1997authentication}.
By analyzing temporal dynamics these methods can be more robust against presentation attacks than biometric methods that rely on static appearance.
However, individual characteristics can also be replayed or imitated by another person \cite{arik2018neural}.
Therefore, we focus on \textit{presentation attack detection} (PAD) as a mandatory security check for face authentication systems.
We analyze the plausibility of the individual facial trait of a person based on temporal sequences of three-dimensional scans of their face.
In the following, we will refer to these as 4D face scans.

To account for the huge amount of redundant information in high-resolution 4D face scans we analyze the plausibility of curvature changes at sub-sampled radial stripes.
An illustration of the overall pipeline for our curvature analysis is depicted in \cref{fig:pipeline}.
As a result, our novel framework for anomaly detection can detect elastic masks as well as static and planar presentation attacks based on their abnormal facial behavior.
For static and planar presentation attacks, the curvatures are either very small or are constant over time.
The training based only on genuine faces increases generalization capability as properties of individual attack types cannot be memorized.

\begin{figure*}[tb]
  \centering
  \includegraphics[width=\textwidth]{images/pipeline.pdf}
  \caption{Our suggested representation of 4D face scans reduces the huge amount of biometric characteristics in color and depth image sequences to the graph shown on the right.
    This graph measures the facial expression change at distinct radial stripes over time.
    As such, it is eligible for identifying presentation attacks in an anomaly detection setting using a one-class SVM, for instance.
    This figure is best viewed in color.}
  \label{fig:pipeline}
\end{figure*}

\section{Related Work}
\label{sec:relatedWork}
In the following, we first state the developments in the field of presentation attack detection to put our work into perspective.
After that, we look into related work on curvature analysis of radial stripes.

\subsection{Presentation Attack Detection}
\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{images/LivenessDetectionProgress.pdf}
  \caption{The complexity of presentation attacks increased over time from face photographs to elastic masks.
    To detect them, a variety of suitable countermeasures were developed.}
  \label{fig:progress}
\end{figure}
In recent years, the quality of presentation attacks included in public 3D face databases continuously improved from photographs to sophisticated silicone masks.
In \Cref{fig:progress}, we give an overview on which kind of biometric copies impostors already used in the past and how they are counteracted.

After accurate 2D face recognition systems were implemented in portable devices, methods for detecting static photographs from differences in Fourier spectra like \cite{li2004live,kim2012face} emerged.
In contrast, these methods are vulnerable to large resolution photographs with high-frequency spectra components.
Additionally, PAD methods based on natural face movements like eye blinking \cite{sun2007blinking} or eye movement \cite{jee2006liveness} can be fooled by additional holes at eyes and mouth.
Thus, 2D challenge-response protocols were introduced, where the user is asked to perform facial expressions \cite{tang2018face}, head movements \cite{bao2009liveness} or read words aloud \cite{kollreider2007real}.
More recently, deep fakes \cite{tewari2018self} became a threat.
They can be used to reenact authenticated faces in real-time on a portable monitor screen as the requested response.

At the same time, cheap 3D sensors become available.
They are integrated into mobile phones using time-of-flight, structured-light or stereo 3D scanning technology.
Due to their high potential for PAD several methods for 3D face scans based on curvature \cite{lagorio2013liveness}, depth variance of 3D landmark locations \cite{wang2013face}, 3D face recognition \cite{tang2018face} have been proposed.
The accuracy of the 3D face recognition method in \cite{gilani2018learning} reached the accuracy of 2D methods and is more robust against planar presentation attacks by design.

Similarly, the authors of \cite{raghavendra2015presentation} are able to detect planar presentation attacks based on facial images using a variety of depth-of-field ranges captured by a light field camera.
However, there are sophisticated static 3D masks or detailed wax figures that can fool said methods.
Hence, we propose to combine a challenge-response protocol with the temporal analysis of 3D face to improve robustness against such sophisticated masks.
In fact, the temporal aspect even allows to detect elastic 3D masks.

As a source for training PAD methods, recent databases contain many wax figures \cite{jia2019spoofing} or realistic silicone masks that are deformed by facial expressions \cite{manjani2017detecting}.
The variety of such masks is limited, such that many 2D PAD methods have already seen the same or similar presentation attacks during training.
Thus, properties of individual attack types might get memorized while methods remain vulnerable to different or unseen attacks.
In contrast, our method requires no samples containing presentation attacks during training.
We propose an anomaly detection scenario, where the PAD method is trained only on recordings of genuine faces.

\subsection{Curvature Analysis of Radial Stripes}
According to \cite{katina2016definitions}, 3D anatomical curves provide a much richer characterization of faces than landmarks, which are just individual points along these curves.
Thus, authors of \cite{vittert2017statistical} localize a complete set of anatomical curves along ridges and valleys of the neutral face shape for statistical analysis.
However, many heuristics would be required to obtain robust anatomical curves in case of facial expressions.
Instead of relying on anatomical curves, the representation proposed in \cite{berretti2008face} approximates the facial shape with a set of geodesics.
Each so-called radial curve is obtained by intersecting the facial surface with a set of equiangular planes that pass through an anchor point.
This representation has proven to result in superior performances in 4D facial expression recognition \cite{zhen2017magnifying}, face recognition \cite{drira20133d,berretti2008face}, and even body part analysis in general \cite{bowman2015anatomical,HKUHUB_10722_31166}.

These methods fit 1D polynomial curves in a registration process.
The degree of the polynomials is fixed.
It is assumed that the complexity of the underlying surface is similar for all curves and all expected shapes.
In practice, this results in plausible curvatures values for planar presentation attacks in the same range as for genuine faces due to overfitting.
Therefore, our proposed representation combines the extraction of radial stripes with the calculation of the 3D surface curvature \cite{pauly2002efficient}.
As a larger neighborhood in a 2D surface is taken into account, this approach is more robust with respect to overfitting to noise, outliers, and holes.

\section{Method}
For 4D facial behavior analysis, we have to deal with extremely high spatiotemporal redundancy of 4D face scans.
The differences between both temporally consecutive scans and spatially close point neighbors are very small.
A feature representation should only extract facial expression changes, which constitute the overall facial behavior.
Therefore, we propose a representation of 4D face scans based on equidistant surface curvatures extracted at equiangular radial stripes (\cref{sec:curvature}) and their correlation over time (\cref{sec:temporalRedundancy}).
The latter allows for measuring facial expression changes.
Before modeling, feature extraction and analysis a few pre-processing steps are necessary to ensure a normalized input.

\subsection{Pre-processing Steps}
\label{sec:preprocessing}
Given two synchronized sequences of color and depth images, two pre-processing steps are performed.
First, $68$ anatomical landmarks are localized in the color images using the method described in \cite{kazemi2014one}, transformed to the disparity map, and 3D reconstructed (see \cref{fig:pipeline}, left).
To also remove the 3D reconstructed background, the point cloud is centered at the nose tip landmark.
Then, all points inside a sphere of radius \mbox{$r = \SI{0.1}{m}$} are extracted.
This removes the large proportion of background in the case of 3D face scans in frontal and cooperative scenarios.
For pose normalization, a rigid transformation between consecutive 3D landmarks and the localized 3D landmarks in the first face scan is obtained by a Procrustes analysis.
Afterward, these transformations are applied to all temporally consecutive 3D face scans, which in turn proved to be robust against pose changes.

\subsection{Spatiotemporal Curvature Analysis}
\label{sec:curvature}

After 3D reconstruction, normalization and face extraction based on a sphere centered at the nose tip we have a pose normalized face.
A depiction with landmarks (white) and nose tip (black) can be found in the middle block of \cref{fig:pipeline}.
The black dot also serves as an origin of a reference coordinate system given by the three unit vectors $\vec n_j$ in x-, y-, and z-direction, with $j\in\{1,2,3\}$.
Red, green and blue arrows in the middle image point into the positive direction of these axes.
In the next step, the face is intersected with the yz-plane at regular angle steps.
Then, a radial stripe is extracted by taking all points for which the projection onto the x-axis $\vec n_1$ is lower than a certain threshold $\delta$ and the projection onto the y-axis $\vec n_2$ is positive:

\begin{equation}
\left\{ \vec p_i = \begin{pmatrix} x_i \\ y_i \\ z_i \end{pmatrix} : \left| \vec p_i^T \vec n_1 \right| \leq \delta ; \vec p_i^T \vec n_2 > 0 \right\}
\label{eq:stripe}
\end{equation}

among all points $\vec p_i$ of a single 3D face scan.
Afterward, the x- and y-axis are rotated clockwise around the z-axis by $\alpha = \frac{\SI{360}{\degree}}{N}$ and the process is repeated until $N$ radial stripes are extracted.
For instance, the first five out of $N = 24$ red stripes in \cref{fig:pipeline} denote the subset of points that satisfy \cref{eq:stripe} for $\alpha = \frac{\SI{360}{\degree}}{24} = \SI{15}{\degree}$.

To reduce the feature dimension, we calculate the curvature for each sampled point based on its local neighborhood in the original point cloud.
The use of the curvature is encouraged by its invariance under 3D Euclidean transformations.
The curvature can be calculated from the 1D parametric arc-length representation of each curve as

\begin{equation}
\kappa(s) = \frac{y''}{(1 + y'^2)^\frac{3}{2}}.
\label{eq:curvature}
\end{equation}

Resulting 1D curvatures in case of a 3D scan of a flat surface (e.g. face photograph, monitor) tend to be in the range of genuine face scans.
This is due to overfitting of radial curves to noisy point cloud data.
To alleviate this effect, we follow \cite{pauly2002efficient} in approximating the 3D surface curvature using the \textit{surface variation} for each sampled point

\begin{equation}
\tilde \kappa(i) = \left|\frac{\lambda_0}{\lambda_0 + \lambda_1 + \lambda_2}\right|,
\label{eq:surfaceVariation}
\end{equation}

where $\lambda_0 < \lambda_1 < \lambda_2$ and $\lambda_i$ are the eigenvalues of a principal component analysis applied to a neighborhood with a fixed radius of \SI{6}{\milli \meter} in the original point cloud.

In contrast to previous works \cite{berretti2008face,zhen2017magnifying,drira20133d,bowman2015anatomical}, we take the 2D neighborhood into account.
The surface variation is bounded to $\frac{1}{3}$ for isotropically distributed points and avoids the major impact of the unbounded, unstable second derivative in \cref{eq:curvature}.
The 3D curvature calculation step in \cref{fig:pipeline} shows the resulting values obtained by \cref{eq:surfaceVariation} for an exemplary 3D face scan.

To compare temporally consecutive radial stripes, we sub-sample $M$ points along each radial stripe.
We obtain an equidistant spacing between each point by sampling along their projection to the radial axis of the intersection plane $\vec p_i^T \vec n_i$.
As an important advantage, the degree of equidistant and equiangular sub-sampling can be varied by parameters $M$ and $N$ depending on the desired accuracy and runtime.\\

\subsection{Measuring Facial Expression Change}
\label{sec:temporalRedundancy}
The set of $M$ curvatures for $N$ radial stripes would still be indistinguishable between genuine faces and well-shaped copies like the 3D masks from REAL-f Co.
Therefore, we measure the curvature change between temporally consecutive face scans and perform PAD on the resulting time series representation.
We calculate the point-wise product of the curvature values between temporally consecutive radial stripes.
As the detected position of the nose tip slightly varies, it is necessary to align the radial stripes to each other.
Hence, we compute the maximum cross-correlation as

\begin{equation}
R_t = \max_j \left( \sum_{i=-M}^M\tilde \kappa_t(i) \cdot \tilde \kappa_{t-1}(i + j)\right)
\label{eq:correlation}
\end{equation}

between radial stripes at time steps $t$ and $t-1$, which refer to consecutive frames in a video.
The maximum cross-correlation measures how similar the temporal curvature values are.
The resulting multivariate time series contains values of $R_t$ for each radial stripe.
It measures the individual change of the facial expression.
For instance, the two inner peaks of the graph for $R_t$ in \cref{fig:pipeline} relate to the eye movements and the two outer peaks to the mouth movements.
Estimating $R_t$ for a high-resolution scan takes \SI{90}{\milli \second} on an Intel Core-i7 CPU.

\begin{figure}[tb]
  \centering
  \input{images/curvatureCorrelationTimeSeries.pgf}
  \caption{For most recordings of genuine faces like '22', the facial expression change is larger than in case of the 3D mask recording ('7').
    Even when the facial expression change is similar like in '11' (elastic 3D mask) and '5' (minimal facial expression), there is detectable difference in the standard deviation.}
  \label{fig:curvatureChange}
\end{figure}

\cref{fig:curvatureChange} shows some exemplary temporal trends for $R_t$.
It demonstrates that the overall facial expression change of genuine faces is much larger than of presentation attacks using 3D masks.
In contrast, \cref{fig:curvatureConfidence} depicts the distribution of $\sigma(R_t)$ among all radial stripes and recordings in our database (see \cref{subsec:database}).
The standard deviation of the first and last few radial stripes through the mouth differ between genuine faces and presentation attacks by a large margin.
For the eye regions, the standard deviation is also different, but not well-separated.
Current active 3D sensors struggle to accurately measure the eye region, which is either reflecting or absorbing the projected stripe pattern.

In the last step, we need to distinguish changes induced by presentation attacks and changes attributed to genuine faces based on these findings.
Thus, we calculate $\sigma(R_t)$ as a measurement of the overall facial expression intensity for each radial stripe.
In \cref{sec:experiments}, we will report our automatic PAD results based on this feature representation in an anomaly detection setup using a one-class SVM.

\section{Experiments}
\label{sec:experiments}
There are several public databases for PAD that have been used in the past.
However, they do not match our proposed setting for 4D facial behavior analysis.
In some cases, they contain only 2D attacks using bent photographs (CASIA-SURF \cite{zhang2019dataset}) and monitor replay-attacks (CASIA-FASD \cite{zhang2012face}).
Other databases like 3D-MAD \cite{erdogmus2013spoofing}, CS-MAD \cite{bhattacharjee2018spoofing} and WMCA \cite{george2019biometric} containing static, elastic and partial 3D masks only capture the neutral face appearance without any facial expression.
Thus, we created a new database for this evaluation.

\begin{figure}[tb]
  \centering
  \input{images/curvatureCorrelationConfidence.pgf}
  \caption{Blue and orange curves show the sample distribution of $\sigma(R_t)$ for each radial stripe ($N = 128$).
    In some cases, standard deviations are small for both presentation attacks and genuine faces.
    However, radial stripes corresponding to mouth and eye regions show substantial differences.}
  \label{fig:curvatureConfidence}
\end{figure}

\subsection{Database and Protocol}
\label{subsec:database}
\begin{figure*}[tb]
  \centering
  \includegraphics[width=\linewidth]{database_distr}
  \caption{Our new PAD database contains 4D scans of static presentation attacks, elastic 3D masks and monitors replay-attacks.
    Photographs as static masks were presented without holes (4 samples) and with holes for eyes and mouth (8x).
    Data protection agreements for presenting facial images in compliance with the regulations were signed by all subjects in this paper.}
  \label{fig:database}
\end{figure*}

During the preparations and planning of our recordings, we focused on the cooperative self-service scenario with a frontal pose for face authentication.
Three studio lamps with D65 standard illuminant were used to avoid light flicker and ensure optimal illumination at a constant exposure time of the 3D sensor.
Our application scenarios are the authentication of ID-documents using eGates for cross-border traffic and eKiosks for biometric data enrollment at administrative offices.
Recent mobile phones are also equipped with a 3D sensor.
Hence, the same authentication scenarios can be implemented in mobile devices in the future for mobile payments and contract conclusions.

In addition, we applied a challenge-response protocol.
We asked participants nine questions that reflect common entry regulations, like \textit{Why are you traveling?}, \textit{How long will you stay?} and \textit{Do you carry liquids with you?}
These questions were presented on a video wall behind the 3D sensor.
Participants were instructed to answer orally to induct visemes and facial expressions.

For recording we used a highly accurate structured-light 3D sensor \cite{notni2018irritation}.
Such an active 3D sensor is more exact in homogeneous areas like forehead or cheeks than passive 3D sensors and robust against extraneous light.
Each 3D reconstructed point cloud of a head has $1$ million points.
On average $250k$ of those points constitute the facial region.
Sequences were recorded at \SI{30}{\Hz} for \SI{36}{\second}, during which questions were asked in \SI{4}{\second} intervals.

In total, our database consist of $109$ 4D face scans from $24$ different subjects (nine women, $15$ men).
Eleven different types of presentation attacks were recorded resulting in $45$ scans.
Furthermore, six participants have beards and six wear glasses.
In the latter case, the participants were recorded with and without glasses.
Beards and glasses pose a challenge for active 3D sensors.
A visual overview with all types of included presentation attacks is given in \cref{fig:database}.
Please note, our database was created in a small study and is not balanced with respect to gender, ethnicity or age.

\subsection{Anomaly Detection Results}
\begin{table}[t]
  \caption{Quantitative evaluation for anomaly detection and standard PAD criteria:
	An SVM is only trained on genuine faces (bona fide) to detect presentation attacks during testing.}
  \label{tab:results}
  \centering
  \begin{tabular}{L{5.8cm} L{2.2cm}}
    \toprule
    \textbf{Performance Measure} & \textbf{Result} \\
    \midrule
    Area under ROC Curve (AUROC) & \SI{95.90}{\percent}\\
    Attack Present. Class. Error Rate (APCER) & $\SI{11.46}{\percent} \pm \SI{3.26}{\percent}$\\
    Bona fide Present. Class. Error Rate (BPCER) & $\SI{6.19}{\percent} \pm \SI{1.23}{\percent}$\\
    \bottomrule
  \end{tabular}
\end{table}

For PAD, we combined the standard deviation of $N = 128$ radial curves in a feature vector for each sample.
Then, we trained an one-class SVM only using genuine faces to pose an anomaly detection task.
Evaluation was carried out using a leave-one-out cross-validation scheme over all subjects.
As parameters of the SVM, we chose a polynomial kernel function and $\nu = 0.05$ as the upper bound of training errors.

\Cref{tab:results} contains the resulting quality metrics on our database described before.
The APCER measures the error rate for presentation attacks wrongly classified as genuine.
BPCER is the analogous measure for \textit{bona fide presentations} (genuine faces).
Both need to be low for a system suitable for practical use.
In our case, we can see a twice as high error rate in the classification performance for presentation attacks when compared to the performance on bona fide presentations.

We observed that the partial chin mask presentations (see \cref{fig:database}, left) particularly pose a challenge for our method.
For the most part, the face is still elastic and able to show normal behavior, which can lead to errors.
Analogously, there are misclassifications in cases of especially strong facial expressions when wearing an elastic mask.
On the other hand, the errors in genuine faces can be traced back to few abnormal recordings.
Corresponding participants did not blink over the whole duration of the recording.
Overall, however, our approach performs very robust in all these different attack scenarios.
We are able to reach an AUC of almost 96\% for this classification task.

\subsection{PAD Discussion}
Given our previous findings, we encourage to specifically localize facial regions, which show anomalous behavior in the next step.
It is possible to highlight the eyes or the partial mask shown in \cref{fig:database} (left) for the chin region.
We also like to point out, that the standard PAD evaluation is only for the classification of genuine faces vs. presentation attacks.
In a more complex task the type of attack could also be classified.

We like to stress, that a limited number of available presentation attack types (e.g. few distinctive masks) increases the potential of bias in a database or algorithm.
However, our anomaly detection approach for PAD does not require training samples of such attacks.
It is not susceptible to overfitting by memorizing properties of mask specifics, which can happen for deep learning-based methods when the training set includes presentation attacks.
The resulting small BPCER shows the generalization capability of our method despite the limited number of subjects in the training set and potential biases from emotional expressiveness, interaction context, or extraversion.

Please note, the steps for pose normalization and curvature analysis preserve the metrical properties of the 3D scans.
Thus, we achieve robustness under head position and pose changes.
Furthermore, the infrared pattern projection of an active 3D sensor results in robustness under illumination changes of visible light.

\subsection{Beyond Presentation Attack Detection}
Our described scenario for PAD is unique, but is a result of the fast progress in that area (see \cref{fig:progress}).
The approach of time series analysis of 3D point cloud data is not limited to that application scenario.
In a parallel medical study we apply it in paresis treatment analysis.
Patients with facial palsy are recorded at regular time intervals using a similar 3D sensor, but without the challenge-response protocol.
Each individual recording is a temporal sequence of pre-defined facial exercises that serve as training to regain muscle movement.
In this scenario our method serves as a detector and measuring instrument.
It is meant to support physicians in treating such patients as it indicates subtle improvements.
An example is depicted in \cref{fig:palsyMean}, where an improvement of the cheek movement can be observed over time.
The mean of the cross-correlation $\bar{R}_t$ continuously improved.

Beyond this medical task, it is also possible to apply our method for emotion recognition given suitable 4D data and a multi-class classifier.
At the moment typical emotion recognition benchmarks either lack depth information (2D video datasets) or temporal information (single 3D face scans).

\section{Conclusion}
The process of face identification is of ever-increasing importance in a variety of daily scenarios.
At the same time attacks on such systems increase and get more complex.
In this paper, we proposed a PAD method to robustly detect flexible 3D masks, bent photographs, and monitor replay-attacks using 4D face scans and behavioral analysis.
We sub-sampled the 3D surface curvature at equiangular radial stripes and analyzed temporally consecutive stripes to detect anomalies.
Our proposed representation allows for varying the degree of sub-sampling depending on the desired accuracy and run-time.
Furthermore, we created a new dynamic 4D database for the evaluation of our method as available PAD databases do not fit our setting of 4D anomaly detection.
Our results are very promising as the error rates demonstrate.
However, distinguishing elastic 3D masks from genuine faces with minimal facial expressions remains a challenging task.

\begin{figure}[tb]
  \centering
  \raisebox{-0.5\height}{\includegraphics[width=0.85\linewidth]{palsyMean3.pdf}}%
  \hspace*{-0.7cm}
  \raisebox{-0.3\height}{\includegraphics[width=0.2\linewidth]{legend.pdf}}
  \caption{
    Besides natural fluctuations, $\bar{R}_t$ improved over time for the curves through the left cheek (arrow).
    It indicates, that the treatment method was successful as the nerve reinnervated and allowed for voluntary muscle movements.
    Inner red areas mark radial stripes through the eyes, where recording artifacts can occur.}
  \label{fig:palsyMean}
\end{figure}

\bibliographystyle{abbrv}

Computer Vision and Pattern Recognition (cs.CV)


Environmental Effects on the Faint End of the Luminosity Function

Abstract: Recent studies have demonstrated that many galaxy clusters have luminosity functions (LFs) which are steep at the faint end. However, it is equally clear that not all clusters have identical LFs. In this paper we explore whether the variation in LF shape correlates with other cluster or environmental properties.

\section{Introduction}

Much recent work has been devoted to measuring the galaxy luminosity function
(LF) within rich clusters, particularly with regard to the faint end which has
become accessible to detailed study through various technical and
observational improvements (see the paper by Smith et al. in these
proceedings).
These studies suggest that the LF becomes steep (Schechter (1976) slope
$\alpha \leq - 1.5$) in many clusters,
faintwards of about $M_{B} = -17.5$ or $M_{R} \simeq -19$ (for
$H_{0}$ = 50 km s$^{-1}$ Mpc$^{-1}$), where (generally low surface
brightness) dwarfs begin to dominate (e.g. Smith, Driver \&
Phillipps 1997; Trentham 1997a,b).
Using deep CCD imaging from the Anglo-Australian Telescope,
we have now extended this work (see Driver,
Couch \& Phillipps 1998), in order to examine the
luminosity distribution in and across a variety of Abell and ACO
clusters. In particular, we were interested in any
possible dependence of the dwarf population (specifically
the ratio of the number of dwarfs to the number
of giants) on cluster type or on position
within the cluster.

\section{Dwarfs in Rich Clusters}

A number of
papers (e.g. Driver et al. 1994; Smith et al. 1997; Wilson et al. 1997)
have demonstrated remarkably similar dwarf populations in
a number of morphologically similar, dense rich clusters like (and
including) Coma. This similarity appears not only in the faint end
slope of the LF, around $\alpha = -1.8$, but also in the point at
which the steep slope cuts in, $M_{R} \simeq -19$ (i.e. about $M^{*} + 3.5$).
The latter implies equal ratios
of dwarf to giant galaxy numbers in the different clusters.

However, there clearly do exist differences between some clusters.
For example,
several of the clusters in the Driver et al. (1998) sample do not
show a conspicuous turn up at the faint end (see also Lopez-Cruz
et al. 1997 for further examples). Either these clusters
contain completely different types of dwarf galaxy population or, as we
suggest, the turn up occurs at fainter magnitudes. For a composite
giant plus dwarf LF, this is equivalent to a smaller number of dwarfs
relative to giants.

To simplify the discussion, we will define the dwarf to giant ratio DGR
as the number of galaxies with $-16.5 \geq M_{R} \geq -19.5$
compared to those with $-19.5 \geq M_{R}$),\\
i.e. $\: \: \: \: DGR = \frac{N(-16.5 > M_{R} > -19.5)}{N(-19.5 > M_{R} > -23.5)}$ .

The DGR does not have any obvious dependence on cluster richness
(Driver et al. 1998; see also Turner et al. 1993), but
we can also check for variations with morphological characteristics
of the clusters. For giant galaxies, it is well known that a cluster's
structural and population characteristics are well correlated.
For example, dense regular clusters are of early Bautz-Morgan type
(dominated by cD galaxies) and have the highest fractions of giant
ellipticals (Dressler 1980).
In a similar way, we find that the DGR (i.e. the fraction
of dwarfs) is {\it smallest} in these early Bautz-Morgan type clusters
(Driver et al. 1998).
Next consider the galaxy density. We can characterise the clusters by their
central (giant) galaxy number densities, for instance the number of
galaxies brighter
than $M_{R} = -19.5$ within the central 1 Mpc$^{2}$ area. An alternative
would be to use
Dressler's (1980) measure of the average number of near neighbours.
We then find (solid squares in Figure 1) that the clusters with the least
prominent dwarf populations (low DGRs $\sim 1$)
are just those with the highest projected
galaxy densities (e.g. the Bautz-Morgan Type I-II cluster A3888).
Previously, Turner et al. (1993) had
noted that the rich but low density
cluster A3574, which is very spiral rich (Willmer et al. 1991),
had a very high ratio of low surface brightness (LSB) dwarfs to
giants. This is now backed up by the observations
of clusters
like A204 which are dwarf rich (DGR $\sim 3$), have low central densities
and late B-M types (A204 is B-M III).

\begin{figure}
\plotone{phillippss1.ps}
\caption{Variation of the dwarf-to-giant ratio (DGR), as defined
in the text, with projected density of cluster giants (per square Mpc).
Solid boxes represent the central 1 square Mpc regions of the clusters,
the open boxes the outer regions (data from Driver at al 1998).
The triangles show the variation over
a wider range of radii for Abell 2554 (data from Smith et al 1997).
Note that typical
error bars (due to the combination of Poisson errors and background
subtraction errors) are 10\% in density and 20\% in DGR for the denser regions,
rising to 30\% in density and 50\% in DGR at the lowest densities (and hence
object numbers). The outlier at low density and low DGR (the outskirts
of A22) has a very large error in DGR ($\sim$ 100\%).
\label{fig1}}
\end{figure}

To extend the range of environments studied, we can add in further
LF results from the literature (Figure 2).
A problem here, of course, is the lack of homogeneity
due to different observed wavebands, different object detection
techniques and so forth.
Nevertheless, we can explore the general trends.
Several points are shown for surveys of Coma (hexagons).
These surveys (Thompson \& Gregory 1993, Lobo et al. 1997, Secker \& Harris
1996 and
Trentham 1998) cover different areas and hence different mean projected
densities (see also the next section). All these lie close to the relation
defined by our original data, with the larger area surveys having higher DGRs.
Points (filled triangles)
representing the rich B-M type I X-ray selected clusters studied by
Lopez-Cruz et al. (1997) fall at
somewhat lower DGR than most of our clusters at similar densities. However
we should note that these clusters were selected (from a larger unpublished
sample) {\it only} if they had LFs well fitted by a single Schechter
function. This obviously precludes clusters with steep LF turn-ups
at intermediate magnitudes and hence rules out high DGRs.
The one comparison cluster they do show {\it with} a
turn up (A1569 at DGR $\simeq 4.2$) clearly supports our overall trend.

Ferguson \& Sandage (1991 = FS), on the other hand, deduced a trend in
the opposite direction, from a study of
fairly poor groups and clusters, with the early type dwarf-to-giant
ratio {\it increasing} for
denser clusters. However, this is not necessarily as contradictory to the
present result as it might initially appear. For instance, FS select
their dwarfs morphologically, not by luminosity (morphologically
classified dwarfs and giants significantly overlap in luminosity) and
they also concentrate solely on early type dwarfs. If, as we might expect,
low density regions have significant numbers of
late type dwarf irregulars (e.g. Thuan et al. 1991), then
the FS definition of DGR may give a lower value than ours for these regions.
Furthermore FS calculate their projected densities from {\it all} detected
galaxies, down to very faint dwarfs. Regions with high DGR will therefore
be forced to much higher densities than we would calculate for giants only.
These two effects may go much of the way to reconciling our respective
results. This is illustrated by the open triangles in Figure 2, which are
an attempt to place the FS points on our system; magnitudes have
been adjusted approximately for the different wavebands,
DGRs have been estimated
from the LFs and the cluster central
densities (from Ferguson \& Sandage 1990)
have been scaled down by the fraction of
their overall galaxy counts which are giants (by our luminosity definition).
Given the uncertainties in the translation,
most of the FS points then lie
close to those of our overall distribution.
Finally, a field LF with a steep faint end tail ($\alpha
\simeq -1.5$; e.g. Marzke, Huchra \& Geller 1994, Zucca et al. 1997, Morgan,
Smith \& Phillipps 1998)
would also give a point (filled pentagon) at DGR $\simeq 4$, again
consistent with the trend seen in the clusters.

Nevertheless, there are exceptions. The FS points of lowest density
(the Leo and Dorado groups) also have low DGR (and lie close to {\it our}
main `outlier', the point for the outer region of A22).
The Local Group (shown by the star)
would also be in this regime, at low density and DGR = 2,
as would the `conventional' field with $\alpha \simeq -1.1$ (Efstathiou, Ellis
\& Peterson 1988; Loveday et al. 1992) and hence DGR
$\simeq 1.5$ (open pentagon).
This may suggest that at very low density
the trend is reversed (i.e. is in the direction seen by FS),
or that the cosmic (and/or statistical) scatter
becomes large. More data in the very low density regime is probably
required before we can make a definitive statement on a possible reversal
of the slope of the DGR versus density relation. In particular, the scatter
in the derived faint end of the field LF between different surveys
(see, e.g., the recent discussion
in Metcalfe et al. 1998) precludes using this to tie
down the low density end of the plot.

\begin{figure}
\plotone{phillippss2.ps}
\caption{
As Figure 1, but including data from other observers. Squares are our
data repeated from Figure 1, hexagons are for various Coma surveys detailed
in the text, filled triangles are from Lopez-Cruz's sample and open
triangles are for Ferguson and Sandage's poor clusters and groups.
The open pentagon at low density represents a conventional `flat' field LF,
the filled pentagon a possible
steep ($\alpha \simeq -1.5$) field LF and the Local Group
is represented by the star at DGR = 2.
\label{fig2}}
\end{figure}

\subsection{Population Gradients}

It was suggested by the results on A2554 (Smith et al. 1997), that the dwarf
population was more spatially extended than that of the giants, i.e. the
dwarf to giant ratio increased outwards. This type of population gradient
has now been confirmed by the results in Driver et al. (1998) illustrated
in Figure 1, where we contrast the inner 1 Mpc$^2$ areas (solid symbols)
with the outer regions of the same clusters (open symbols). The triangles show
in slightly more detail the run of DGR with radius (and hence density)
across A2554. A similar effect can be seen for Coma in Figure 2
and can explain the discrepancy between the LFs derived for the core
and as against larger areas.
It is found, too, in Virgo
(Phillipps et al. 1998a; Jones et al., these proceedings),
where the dwarf LSBG population has almost constant
number density across the central areas while the giant density drops
by a factor $\sim 3$.

\section{A Dwarf Population Density Relation}

The obvious synthesis of the above results is a relationship
between the {\it local} galaxy density and the fraction of dwarfs
(i.e. the relative
amplitude of the dwarf LF). The inner, densest parts of rich clusters
have the smallest fraction of dwarfs, while loose clusters and the
outer parts of regular clusters, where the density is low, have high dwarf
fractions. It is particularly interesting
to note the clear overlap region in Figure 1,
where regions of low density on the
outskirts of dense clusters (open squares) have similar DGRs to the
regions of the same density at the centres of looser clusters (solid squares).

The proposed relation
of course mimics the well known morphology - density relation (Dressler
1980), wherein the central parts of rich clusters have the highest early type
galaxy fraction, this fraction then declining with decreasing local galaxy
density.
Putting the two relations together, it would also imply that dwarfs
preferentially occur in the same environments as spirals. This would be in
agreement with the weaker clustering of low luminosity systems in general
(e.g. Loveday et al. 1995), as well as for spirals compared to ellipticals
(Geller \& Davies 1976). Thuan et al. (1991) have previously discussed the
similar spatial distributions of dwarfs (in particular dwarf irregulars)
and larger late type systems.

\section{The Origin of the Relation}

As with the corresponding morphology - density relation for giant galaxies, the
cause of our population - density relation could be either `nature' or
`nurture', i.e. initial conditions or evolution. Some clues may be provided by
the most recent semi-analytic models of galaxy formation, which have been
able to account successfully for the excess of (giant) early type
galaxies in dense environments (e.g. Baugh, Cole \& Frenk 1996), basically
through different merging histories for different types of galaxy.
Does this also work for the dwarfs?

The steep faint end slope of the LF appears to be a generic result of
hierarchical clustering models
(e.g. White \& Frenk 1991; Frenk et al. 1996;
Kauffmann, Nusser \& Steinmetz 1997 = KNS),
so is naturally accounted
for in the current generation of models. The general hierarchical
formation picture envisages (mainly baryonic) galaxies forming at the cores
of dark matter halos. The halos themselves merge according to the general
Press-Schechter (1974) prescription to generate the present day halo mass
function. However the galaxies can retain their individual
identities within the
growing dark halos, because of their much longer merging time scales.
The accretion of small halos by a large one then results in the main
galaxy (or cluster of galaxies, for very large mass halos) acquiring a
number of smaller satellites (or the cluster gaining additional, less
tightly bound, members).

KNS have presented a detailed study of the distribution
of the luminosities of galaxies expected to be associated with a single
halo of given mass.  We can thus easily compare the theoretically
expected numbers of dwarf galaxies
per unit giant galaxy luminosity with our empirical results (Phillipps et al.
1998b).

The KNS models mimic a ``Milky Way system"
(halo mass $5 \times 10^{12} M_{\odot}$), a sizeable group
(halo mass $5 \times 10^{13} M_{\odot}$)
and a cluster mass halo ($10^{15} M_{\odot}$).
Their results imply
that the Milky Way and small group halos have similar numbers
of dwarf galaxies per unit giant galaxy light, whereas the dense cluster
environment has a much smaller number of dwarfs for a given total giant
galaxy luminosity. Thus the predictions of the hierarchical models
(which depend, of course, on the merger history of the galaxies)
are in qualitative agreement with our empirical results if we identify
loose clusters and the outskirts of rich clusters with a population
of (infalling?) groups (cf. Abraham et al. 1996),
whereas the central dense regions of the clusters
originate from already massive dark halos.
If we renormalise from unit galaxy light to an
effective giant galaxy LF amplitude (see Phillipps et al. 1998b)
then the actual expected ratios
($\sim 1$ to a few) are also consistent with our observational results.

By inputting realistic
star formation laws etc., KNS could further identify the
galaxies in the most massive halos with old elliptical galaxies, and
those in low mass halos with galaxies with continued star formation.
This would imply the likelihood that our dwarfs in low density regions
may still be star forming, or at least have had star formation in the
relatively recent past (cf. Phillipps \& Driver 1995 and references
therein). Note, too, that these galaxy formation models would also
indicate that the usual (giant) morphology - density relation and our
(dwarf) population - density relation {\it do} arise in basically the same way.
Finally, we can see that if these semi-analytic
models are reasonably believable,
then we need not necessarily expect the field to be even richer in dwarfs than
loose clusters; the dwarf to giant ratio seems to level off at the
densities reached in fairly large groups.

\section{Summary}

To summarise, then,
we suggest that the current data on the relative numbers of dwarf
galaxies in different clusters and groups can be understood in terms
of a general dwarf population versus local galaxy density relation,
similar to the well known morphology - density relation for giants.
Low density environments are the preferred habitat of low luminosity
galaxies; in dense regions they occur in similar numbers to giants,
but at low densities dwarfs dominate numerically by a large factor.
This fits in with the general idea that low luminosity
galaxies are less clustered than high luminosity ones (particularly
giant ellipticals). Plausible theoretical justifications for the
population - density relation can be found within the context of current
semi-analytic models of hierarchical structure formation.


Heterogeneous Hidden Markov Models for Sleep Activity Recognition from Multi-Source Passively Sensed Data

Abstract: Psychiatric patients' passive activity monitoring is crucial to detect behavioural shifts in real-time, comprising a tool that helps clinicians supervise patients' evolution over time and enhance the associated treatments' outcomes. Frequently, sleep disturbances and mental health deterioration are closely related, as mental health condition worsening regularly entails shifts in the patients' circadian rhythms. Therefore, Sleep Activity Recognition constitutes a behavioural marker to portray patients' activity cycles and to detect behavioural changes among them. Moreover, mobile passively sensed data captured from smartphones, thanks to these devices' ubiquity, constitute an excellent alternative to profile patients' biorhythm.In this work, we aim to identify major sleep episodes based on passively sensed data. To do so, a Heterogeneous Hidden Markov Model is proposed to model a discrete latent variable process associated with the Sleep Activity Recognition task in a self-supervised way. We validate our results against sleep metrics reported by clinically tested wearables, proving the effectiveness of the proposed approach.

\section{Introduction}
\label{sec:intro}

Mental illnesses affect as much as the 2\% of world's population \citep{WHO} and they are estimated to be the cause of about 14\% of global diseases \citep{prince2007no}. Mental disorders lead to quality of life deterioration along with social disabilities that burden patients' professional and social lives \citep{rapaport2005quality}. Therefore, monitoring patients that suffer from mental illnesses is vital to detect trend changes that may indicate behavioural shifts.
Historically, psychiatric patients' mental state monitoring has been assessed via questionnaires, patients' reports, and periodic appointments, which entail long periods before detecting any possible risk. Recent technological advances allow real-time patients monitoring with tools like the smartphone-based active Ecological Momentary Assessment (EMA) \citep{porras2020smartphone}.
Nevertheless, collected data via electronic EMA suffer from patients' subjective insight and presumes patients' ability and interest to fill in the required information \citep{fischer2021possibilities}.
Objective assessment methods can be achieved through passive data collected from patients' smartphones, using information as the visited locations via GPS, smartphone and social media interaction, etc. The ubiquity of smartphone devices allows Human Activity Recognition (HAR) methods \citep{jobanputra2019human, rios2020hidden} to track patients in a non-invasive way, creating a personal profile for each user. To do so, the variety of smartphones' available sensors must be merged into relevant indicators of patients' daily life. These indicators constitute biomarkers that represent patients' state in real-time and can be used to profile their evolution, allowing real-time monitoring and detection of behavioural shifts.
Recently, several studies have identified relations between mental health condition worsening and sleep disruption in psychiatric patients \citep{tseng2019sleep, littlewood2019short, freeman2020sleep}. Considering the available data through patients' smartphones, sleep estimation arises as a natural biomarker to integrate smartphones' data into a single behaviour metric that characterizes each individual's circadian rhythm over time. Furthermore, a wide range of studies have reported that wearable devices show high reliability while detecting asleep-awake states in different populations, consistently reporting good estimators of the sleep activity \citep{de2018validation,chinoy2021performance}, which allows them to be used for validating predictions performed by Sleep Activity Recognition (SAR) models, as they can assess sleep quality during long time windows.

\vspace{-0.2cm}
\section{Sleep Measurement Techniques}
\label{sec:sleep_measurement}

This work aims to identify major sleep episodes based on passively sensed behavioural data, obtaining objective behaviour markers that could be used to recognise major disturbances in psychiatric patients.
Among traditional sleep assessment methods, Polysomnography (PSG) stands as the most reliable technique. Nonetheless, PSG requires medical assistance for its application and sleep studies need to be recorded in an ambulatory setting \citep{smith2020actigraph}. Besides, it is an expensive procedure that does not provide a continuous evaluation of patients' sleep as it is required to assess their behaviour evolution.
Sleep questionnaires and diaries arise as a natural alternative when trying to monitor sleep evolution constantly. Regardless, we discard them as they do not express an objective sleep evaluation \citep{hennig2017predicts}.
In these conditions, wearable devices appear as an excellent alternative: they are not expensive and they collect daily information in a non-intrusive way. Several studies have shown that, even when they are not sufficiently accurate to be used in clinical settings, mainly due to poor specificity and low accuracy, they can be relied on for detecting asleep-awake states \citep{moreno2019validation}. Nonetheless, the percentage of users who possess these devices is minimal, implying that we cannot rely on wearable devices to assess patients' sleep continuously.
Therefore, smartphones constitute the most reliable source of data: most patients have one, they do not require an additional investment, and most people carry them along most of the day; the ubiquity of these devices makes them the perfect passive assessment. However, mobile health data that can be collected this way usually entails several problems: missing data is quite common, signals are typically corrupted by noise, the different number of sensors that smartphones use imply data heterogeneity, and main sleep trends must be detached from false positives while making the predictions. Therefore, the proposed model must be capable of dealing with these problems.
Several frameworks are available in the literature to obtain sleep estimates from smartphone data \citep{gautam2015smartphone, eichi2021dpsleep}, as well as certain HMM implementations that handle mixed discrete and continuous observations \citep{epaillard2015hybrid, missaoui2013multi}. Anyhow, none of these models can simultaneously deal with all the complex requirements previously mentioned, making them not suitable for creating behavioural markers in the considered scenario.
Furthermore, the literature lacks models validated with clinical patients and tested in large samples \citep{aledavood2019smartphone}. For these reasons, we propose our own method for the SAR problem, detailed in Section \ref{sec:sar}.

\vspace{-0.2cm}
\section{Passively Sensing Mobile Data}\label{sec:dataset}

In this section, we aim to specify the details of the dataset used. As previously stated, smartphones' data constitute a tool for patient monitoring due to their omnipresence, reporting objective measures of patients' circadian rhythm that can be used as indicators for detecting behavioural shifts. A data gathering system \citep{eB2} was used to collect passively sensed mobile data from patients' smartphones under clinicians' supervision.
The available population consisted of a set of 57 patients who suffered from different mental health conditions. For 10 of those patients, 386 days of data with associated wearables' sleep metrics were available. Data from this subset of patients was exclusively used for validation purposes and did not take part during the training process. Demographic information for this subset was not available.
 The rest 47 patients, 38 females and 9 males, had an average age of 43 $\pm$ 15 years. For this second subset of patients, 647 days of data were collected. Thus, considering both sets, 1033 days of data were available for this study, with an average of 15 $\pm$ 24 days of data per user. We should clarify that train/test split was done solely depending on the availability of wearable devices' data: those patients who possessed wearable devices were not used during the training phase. Instead, their corresponding data was kept for testing purposes, as wearable devices’ reported sleep metrics allow us to assess the quality of the model’s predictions. This is possible as the proposed model operates in an unsupervised way, which entails that no labelled data is required during training, allowing us to use all wearable devices’ data for evaluation purposes.
Data acquisition was conducted periodically, collecting data from different sensors, i.e. actigraphy, light, smartphone usage (a binarized feature that indicates if the patient is using or not the smartphone), and pre-computed metrics, i.e. the daily amount of steps, which we binarize for each time-slot. Data's granularity  was 10 minutes; hence 144 observations per day were available in absence of missing data. Table \ref{tab:data} summarizes the smartphones' used sensors and Figure \ref{fig:dataset} shows the collected data's structure for one of the patients.
The models  developed to extract relevant sleep trends from this real-world data (incomplete, noisy, and heterogeneous) need to consider its structure, as it encloses different challenges while building optimal methods capable of dealing with these problems.

\comment{
\begin{figure*}[t]
\centering
\includegraphics[width=0.4\textwidth]{example-image-golden}
\caption{Here we will show a couple of figures to illustrate the dataset strucutre}
\label{fig1}
\end{figure*}
}

\begin{figure*}[]
\vspace{-0.4cm}
\centering
    {\includegraphics[width=13cm, height=5.3cm]{figures/dist_user_workplace_5_daily_mindcare_test_12_2021-07-01_repr}}

\caption{Patient's passively sensed data. The left figure shows five months of data, the right figure displays intra-day features' values.}
\label{fig:dataset}
\vspace{-0.6cm}
\end{figure*}

\begin{table}[]
\fontsize{10.0}{12}\selectfont
\centering
\begin{tabular}{ccc}
\specialrule{.1em}{.05em}{.05em}
\textbf{Feature} & \textbf{Domain} & \textbf{\% Missing values} \\ \specialrule{.1em}{.05em}{.05em}
Actigraphy                      & $\mathbb{R}$             & 26.27 \%                   \\
Light                           & $\mathbb{R}$               & 59.22 \%                  \\
Steps                           & $\mathbb{Z}$               & 21.98 \%                      \\
App usage                       & $\mathbb{Z}_{2}=\{0,1\}$     & 5.33 \%                       \\ \specialrule{.1em}{.05em}{.05em}
\end{tabular}
\caption{Passively sensing mobile data features used and missing observations percentage per feature.}
\label{tab:data}
\vspace{-1.0cm}
\end{table}

\vspace{-0.5cm}
\section{Sleep Activity Recognition}
\label{sec:sar}

Passively sensing mobile data's characteristics, enumerated in Sections \ref{sec:sleep_measurement} and \ref{sec:dataset}, demarcates the features that we demand from the proposed model. Firstly, it needs to be able to manage data heterogeneity, handling gaussian and categorical observations at the same time. Secondly, it needs to perform inference on missing data, critical while dealing with healthcare datasets. Finally, it needs to allow semi-supervised training, granting users the possibility to fix certain observation probabilities, therefore manually designing certain aspects of the model’s behaviour. E.g., fixing to zero the probability of observing smartphone usage for those hidden states that we aim to interpret as sleeping activity.
Fullfilling these requirements, we present a model easy to train and with a low number of parameters, which allows it to be implemented in a production setting and to be used by clinicians as a tool to monitor patients’ circadian rhythm, unveiling a powerful mechanism to detect behavioural changes.

To merge the heterogeneous input features that define the collected data into a SAR indicator, capable of detecting different activity levels along the day and identifying sleep episodes among them, we require the proposed model to be capable of projecting the high dimensional input features into a lower dimensional space. To do so, latent variable models arise as a natural alternative \citep{bishop1998latent}. Considering the time dependence of our data, Hidden Markov Models (HMMs) \citep{rabiner1989tutorial} constitute a natural way of modelling a discrete latent process associated with the SAR task.
HMMs are generative models that assume that the phenomena being modelled is a Markov process \citep{ethier2009markov} with associated hidden states. HMMs' objective is to learn about these hidden states sequence, denoted $S=\left\{s_{1}, s_{2}, \ldots, s_{T}: s_{t} \in 1, \ldots, I\right\}$, with $t=\left\{1, 2, \ldots, T\right\} \in  \mathbb{N}$, by means of the generated observations, $Y=\left\{\mathbf{y}_{1}, \mathbf{y}_{2}, \ldots, \mathbf{y}_{T}: \mathbf{y}_{t} \in \mathbb{R}^{M}\right\}$. To do so, they use an observation model $p(\mathbf{y}_t | s_t)$. Therefore, each observation $\mathbf{y}_t$ depends exclusively on its associated state $s_t$.
Here, we aim to use this hidden states sequence $S$ as the low dimensional representation of the input features. Therefore, each possible state of the HMM is associated with different activity levels, being  sleep episodes one of them.

Nevertheless, standard HMM implementations use Multinomial and Gaussian observation models, depending on the probability distribution chosen to model the emission probabilities $p(\mathbf{y}_t | s_t)$.
Due to the heterogeneity of the collected data, neither of these options would fit our requirements. While some of the used features, like the actigraphy and light sensors, can be naturally modelled via Gaussian distributions, others, as the app usage and steps count (binarized and converted into an movement indicator), are better modelled with a Bernoulli distribution.
To manage the heterogeneity of our input sources, we developed a variation of ordinary HMMs which we refer to as Heterogeneous Hidden Markov Model (HHMM) \footnote{\tiny{https://github.com/fmorenopino/HeterogeneousHMM}} \citep{moreno2022pyhhmm}. The proposed HHMM can simultaneously manage categorical and continuous data  thanks to its observation model, visible in Fig. \ref{hhmm}, which is responsible for jointly modelling the continuous and discrete observations, $\mathbf{y}_t$ and $\mathbf{l}_t$. The joint distribution of the proposed model is stated in Eq. \ref{joint_main}, where $N \in  \mathbb{Z}$ represents the number of sequences. More information about the HHMM, its parameters inference procedure, and how it performs inference on missing data can be found in the Appendixes \ref{subsec:inference} and \ref{subsec:missing}.

\begin{equation}
\footnotesize
\begin{aligned}
    p(S, Y, L) =& \prod_{n=1}^{N}\left(p\left(s_{1}^{n}\right) \prod_{t=2}^{T_{n}}  p\left(s_{t}^{n} | s_{t-1}^{n}\right)\right) \\
    &\left(\prod_{t=1}^{T_{n}} p\left(\mathbf{y}_{t}^{n} | s_{t}^{n}\right)\right)\left(\prod_{t=1}^{T_{n}} p\left(l_{t}^{n} | s_{t}^{n}\right)\right)
\end{aligned}
\label{joint_main}
\vspace{-0.3cm}
\end{equation}

\begin{figure}[]
\centering
\begin{tikzpicture}[scale=0.9, transform shape]
\tikzstyle{main}=[circle, minimum size = 10mm, thick, draw =black!80, node distance = 10mm]
\tikzstyle{connect}=[-latex, thick]
\tikzstyle{box}=[rectangle, draw=black!100]
  \node[main] (S1) [] {$s_{t-1}$};
  \node[main] (S2) [right=of S1] {$s_{t }$};
  \node[main] (S3) [right=of S2] {$s_{t+1}$};
  \node[main] (St) [right=of S3] {$s_{T}$};

  \node[main,fill=black!10] (O1) [below=of S1] {$y_{t-1}$};
  \node[main,fill=black!10] (O2) [right=of O1,below=of S2] {$y_{t}$};
  \node[main,fill=black!10] (O3) [right=of O2,below=of S3] {$y_{t+1}$};
  \node[main,fill=black!10] (Ot) [right=of O3,below=of St] {$y_{T}$};

  \node [main,fill=black!10] (L1) at (1,-1.2) {$l_{t-1}$};
  \node [main,fill=black!10] (L2) at (3,-1.2) {$l_{t}$};
  \node [main,fill=black!10] (L3) at (5,-1.2) {$l_{t+1}$};
  \node [main,fill=black!10] (Lt) at (7,-1.2) {$l_{T}$};

  \path (S3) -- node[auto=false]{\ldots} (St);
  \path (S1) edge [connect] (S2)
        (S2) edge [connect] (S3)
        (S3) -- node[auto=false]{\ldots} (St);

  \path (S1) edge [connect] (O1);
  \path (S2) edge [connect] (O2);
  \path (S3) edge [connect] (O3);
  \path (St) edge [connect] (Ot);
  \path (S1) edge [connect] (L1);
  \path (S2) edge [connect] (L2);
  \path (S3) edge [connect] (L3);
  \path (St) edge [connect] (Lt);

  \draw[dashed]  [below=of S1,above=of O1];
\end{tikzpicture}
\caption{Heterogeneous HMM architecture. Gray represents observed data.}
\label{hhmm}
\vspace{-0.4cm}
\end{figure}

\vspace{-0.1cm}
\section{Experiments}
\label{sec:experiments}
We will now train the proposed model and evaluate its performance in comparison with different baseline methods. We should remark that because of the dataset used, where just a relatively small portion of data has the corresponding labels indicating the sleep activity of the patients, as stated in Section \ref{sec:dataset}, it is not feasible to train supervised methods due to this lack of labelled data.

HHMM's parameters are trained via the Baum-Welch algorithm using a subset the 47 patients with 647 sequences of daily data. The remainder 10 patients are retained for model evaluation purposes. The optimal number of hidden states is selected using two different model selection methods, the Bayesian Information Criteria (BIC) and the Akaike Information Criterion (AIC). For further details regarding the selection criteria for the optimal number of hidden states, as well as its projection into a binary output (awake or asleep), we refer the reader to Appendix \ref{sec:appendix_experiments}.
\comment{
*LARGO: no voy a hablar aquí de semi-supervised training
The training process with $N=6$ was performed in a semi-supervised manner. To do so, certain discrete emission probabilities were fixed in order to ban the model from accessing specific states if the smartphone was being used. Hence, we are manually associating certain hidden states to low activity periods. HHMM's predictions were then binarized into awake/asleep categories.
}
Once the optimal model's parameters are obtained, we proceed to evaluate the accuracy of the model in the task of detecting sleep episodes. To do so, we use the subset of 10 patients with 386 days of data for whom associated wearables' sleep metrics were available, reporting the sleep state of the patients as awake, light sleep, deep sleep or REM phase. We binarize this output into two categories: awake or asleep, so we can directly compare them with HHMM's final binary predictions.

\begin{table*}[]
\fontsize{10.0}{15}\selectfont
\centering
\begin{tabular}{ccccc}
\specialrule{.1em}{.05em}{.05em}
\multicolumn{2}{c}{\textbf{Method}}                                                                      & \textbf{Accuracy}                & \textbf{Specificity}             & \textbf{Sensitivity}             \\ \specialrule{.1em}{.05em}{.05em}
\multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}Random Classifier (Most Frequent)\end{tabular}}} & $0.6798 \pm 0.043$                   & $0.9973 \pm 0.0510$                  & $0.0026 \pm 0.0511$                  \\ \hline
\multicolumn{2}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}Random Classifier (Uniform)\end{tabular}}}       & $0.4967 \pm 0.0426$                  & $0.5011 \pm 0.0502$                  & $0.4983 \pm 0.0762$                  \\ \hline
\multirow{2}{*}{\textbf{K-means}}                     & \textbf{Imputing Method 1}                     & $0.7393 \pm 0.1156$                  & $0.6338 \pm 0.1651$                  & $0.9641 \pm 0.0324$                  \\
                                                      & \textbf{Imputing Method 2}                     & $0.7386 \pm 0.1167$                  & $0.6356 \pm 0.1672$                  & $0.9587 \pm 0.0767$                  \\ \hline
\multirow{2}{*}{\textbf{GMM}}                         & \textbf{Imputing Method 1}                     & $0.5126 \pm 0.1327$                  & $0.2968 \pm 0.1926$                  & $0.9717 \pm 0.0416$                  \\
                                                      & \textbf{Imputing Method 2}                     & $0.7386 \pm 0.1167$                  & $0.6356 \pm 0.1672$                  & $0.9587 \pm 0.0767$                  \\ \hline
\multicolumn{2}{c}{\textbf{HHMM}}                                                                        & $0.8780 \pm 0.0804$ & $0.9052 \pm 0.0779$ & $0.8207 \pm 0.2382$ \\ \specialrule{.1em}{.05em}{.05em}
\end{tabular}
\caption{Experiments results.}
\label{tab:results}
\vspace{-0.2cm}
\end{table*}

In order to illustrate HHMM performance, we include two baseline models for comparison purposes. The first of them is a K-means algorithm that cluster each datapoint as asleep/awake. Two variations of this K-means model are considered, each of them uses different missing data imputation techniques in order to cluster each datapoint. The first imputation method (\textit{Imputation Method 1} in Table \ref{tab:results}) uses the mean to infer the missing values on the continuous features of the dataset, and it sets to zero the missing observations on the binary features. The second imputation method (\textit{Imputation Method 2} in Table \ref{tab:results}) still infers continuous missing values using the mean, but it substitutes missing observations on the binary features with the most common values (per sequence).
Besides the K-means algorithm, the second baseline considered is a Gaussian Mixture Model (GMM), which also uses the two previously explained imputation techniques to deal with missing values. Furthermore, besides these two baselines, two dummy classifiers are tested on the labelled dataset. The first of these dummy classifiers follows a “most frequent” approach, always returning the most frequent class label. The second one is “uniform”, as it generates predictions uniformly at random from the list of unique classes observed. The inclusion of these two dummy classifiers provides a quantification of the label distribution of the dataset, which helps to understand class imbalances, as the portion of time the patients are asleep is considerable smaller than the time they are awake.

As done with the HHMM, we train the unsupervised baseline methods in the 47 patients within the training set. The trained methods are evaluated then in the cohort of 10 patients with associated labelled wearables’ data. The results reported by these baselines, together with HHMM's performance, are collected in Table \ref{tab:results}. As these results show, no baseline methods report better performance than the proposed algorithm in terms of accuracy. Regarding specificity and sensitivity, some of the utilised baselines report better results if we consider them separately, due to class imbalances, but the HHMM report the best trade-off between both metrics.
 For illustrative examples of the HHMM's predictions, we refer the reader to Figures \ref{fig:predictions1} and \ref{fig:predictions2} of the Appendix \ref{sec:appendix_experiments}, which show two examples of the HHMM's reported predictions for different patients.

\vspace{-0.3cm}
\section{Conclusion}
\label{sec:conclusion}

We introduce a new approach that performs Sleep Activity Recognition from passively sensed data, providing an unsupervised method that can operate in realistic scenarios and produces sleep activity indicators. Accordingly, the model comprises a tool that can be used by clinicians to monitor patients’ circadian rhythm.

A discrete latent variable model with temporal dependences is used to monitor patients' activity cycles, integrating smartphones’ data  into  a  single  behaviour  metric. The proposed model can deal with passively sensed data associated problems: corrupted signals, heterogeneity, and missing values. We validated our results against wearables' reported sleep metrics, proving the effectiveness of the proposed approach.
Therefore, Heterogeneous Hidden Markov Models comprise a powerful tool to supervise patient's evolution in real-time, providing a behavioural marker to portray patients' activity cycles and to detect behavioural changes.
\comment{

\section{Acknowledgments}

This work has been supported by Spanish government Ministerio de Ciencia, Innovación y Universidades under grants FPU18/00470, TEC2017-92552-EXP and RTI2018-099655-B-100, by Comunidad de Madrid under grants IND2017/TIC-7618, IND2018/TIC-9649, IND2020/TIC-17372,  and Y2018/TCS-4705, by BBVA Foundation under the Deep-DARWiN project, and by the European Union (FEDER) and the European Research Council (ERC) through the European Union’s Horizon 2020 research and innovation program under Grant 714161.
}

Signal Processing (eess.SP); Machine Learning (cs.LG)