Ensemble Machine Learning Model Trained on a New Synthesized Dataset Generalizes Well for Stress Prediction Using Wearable Devices

Abstract: Introduction. We investigate the generalization ability of models built on datasets containing a small number of subjects, recorded in single study protocols. Next, we propose and evaluate methods combining these datasets into a single, large dataset. Finally, we propose and evaluate the use of ensemble techniques by combining gradient boosting with an artificial neural network to measure predictive power on new, unseen data.Methods. Sensor biomarker data from six public datasets were utilized in this study. To test model generalization, we developed a gradient boosting model trained on one dataset (SWELL), and tested its predictive power on two datasets previously used in other studies (WESAD, NEURO). Next, we merged four small datasets, i.e. (SWELL, NEURO, WESAD, UBFC-Phys), to provide a combined total of 99 subjects,. In addition, we utilized random sampling combined with another dataset (EXAM) to build a larger training dataset consisting of 200 synthesized subjects,. Finally, we developed an ensemble model that combines our gradient boosting model with an artificial neural network, and tested it on two additional, unseen publicly available stress datasets (WESAD and Toadstool).Results. Our method delivers a robust stress measurement system capable of achieving 85% predictive accuracy on new, unseen validation data, achieving a 25% performance improvement over single models trained on small datasets.Conclusion. Models trained on small, single study protocol datasets do not generalize well for use on new, unseen data and lack statistical power. Ma-chine learning models trained on a dataset containing a larger number of varied study subjects capture physiological variance better, resulting in more robust stress detection.

\section{Introduction}

\noindent Stress can be defined as any type of change that causes physical, emotional or psychological strain. Such change in the environment elicits the activation of a cascade of biological responses (stress response) in the brain and in the body \cite{McEwen1998}. The stress response serves an important evolutionary role of helping the adaptation of the organism to the dynamically changing external and internal environment. This is achieved through mobilization of energy and its appropriate redistribution to organs that most immediately serve the adaptational response. In this sense the biological stress response is adaptive and beneficial in the short term. The risk lies with the fact that the biological systems and molecules sub-serving the stress response exert considerable effects and bio-energetic demands on the organism. If not properly shut down or controlled by the body’s feedback mechanisms, the long-term exposure to stress will have detrimental effects. In humans this may include an increased risk to develop metabolic, cardiovascular and mental disorders, resulting in significantly compromised quality of life and shortened life expectancy \cite{McEwen1993, McEwen2006}.\\

\noindent Wearable devices for personal health monitoring have increased significantly in technical sophistication and are capable of measuring a wide variety of physiological signals. Continuous measurement of these signals using wearables enable researchers to record and extract useful information to detect and monitor a variety of potential health-related events, including stress.\\

\noindent Despite limitations with battery time and number of available sensors in certain models, compared to controlled laboratory measurement devices, wearables are non-intrusive and easier to use. This ease has facilitated many experiments using wearables \cite{Sriramprakash2017, Limas2018, Schmidt2018, Nkurikiyeyezu2019,Eskandar2020,Siirtola2020, Indikawati2020g, Li2020, Iqbal2021g, Liapis2021, Ninh2021,Khan2022}, and predominantly utilizing Empatica's latest E4 device \cite{Empatica2022}, which have yielded a number of well-studied public datasets \cite{Kraaij2015,Birjandtalab2016,Schmidt2018,Haouij2018,Svoren2020,Sabour2021, Amin2022, Hosseini2022}. Table \ref{tab:datareviewed} provides a summary of the datasets which were considered in this study. The table also shows details about these datasets including their number of subjects, available biomarkers, their wearable devices, and labeling strategies, which will be discussed in more details.\\

\begin{sidewaystable}
\setlength\tabcolsep{1pt}
\centering
\caption{\label{tab:datareviewed}Summary of public wearable device stress-related datasets used in this study.}
\resizebox{\textwidth}{!}{
\begin{tabular}{p{8cm}cccccLLL}
\hline\hline
\textbf{Dataset}                                & \multicolumn{1}{l}{\textbf{Year |}} & \multicolumn{1}{l}{\textbf{Subjects |}} & \multicolumn{1}{l}{\textbf{Female |}} & \multicolumn{1}{l}{\textbf{Male |}} & \textbf{Duration |} & \textbf{Biomarkers |}                          & \textbf{Devices |}                                                             & \textbf{Labeling/Scoring}                                       \\
\hline
SWELL                                           & 2014                              & 25                                    & 8                                   & 17                                & 138 min           & EDA, HRV, ECG                                & Facial expression, body postures, Mobi                                   & Periodic: Neutral, Time Pressure, Interruptions                           \\
\rowcolor[rgb]{0.753,0.753,0.753} Neurological Status (NEURO)                             & 2017                              & 20                                    &                &              & 31 min            & ACC,EDA, TEMP, HR, SPO2                      & Empatica E4                                                                  & Periodic: Relax, Physical Stress, Emotional Stress, Relax, Emotional Stress, Relax \\

WESAD         & 2018                              & 15                                    & 3                                   & 12                                & 120 min           & ACC, EDA, BVP, IBI, HR, TEMP, ECG, EMG, RESP & RespiBAN, Empatica E4                                                        & Periodic: Preparation, Baseline, Amusement, Stress, Meditation, Recovery  \\
\rowcolor[rgb]{0.753,0.753,0.753} AffectiveROAD & 2018                              & 10                                    & 5                                   & 5                                 & 118 min           & EDA, HR, TEMP                                & Empatica and Zephyr BioHarness 3.0 chest belt                                & Scored by observer                  \\
Toadstool                                       & 2020                              & 10                                    & 5                                   & 5                                 & 50 min            & ACC, EDA, BVP, IBI, HR, TEMP                 & Empatica E4                                                                  & Periodic: Game play under time pressure                                                         \\ \rowcolor[rgb]{0.753,0.753,0.753} UBFC-Phys                                        & 2021                              & 56                                    & 46                                  & 10                                & 20 min           & EDA, BVP            & Empatica E4 & Self-report                        \\
A Wearable Exam Stress Dataset for Predicting Cognitive Performance in Real-World Settings                                       & 2022                              & 10                                    & 2                                   & 8                                 & 180 min            & EDA, HR, BVP, TEMP, IBI, ACC                 & Empatica E4                                                                  & Periodic                                                         \\
\rowcolor[rgb]{0.753,0.753,0.753} A multimodal sensor dataset for continuous stress detection of nurses in a hospital                                        & 2022                              & 15                                    & 15                                  & 0                                & Varying           & EDA, HR, ST, BVP, ACC, IBI            & Empatica E4 & Self-report                        \\
\hline\hline
\end{tabular}
}
\end{sidewaystable}

\noindent Currently, stress-related and personalized questionnaires are mainly used to measure or score (label) stress in real-life and outside of a laboratory context. However, this technique does not allow for continuous monitoring, and often suffers from bias such as demand effects, response and memory biases. Technology offers a solution, by combining the large quantities of sensor-based physiological data recorded using wearable devices with the use of machine learning techniques, and specifically for the purpose of measuring stress, supervised machine learning techniques. In supervised learning, models are trained using data that is accurately labeled for the response you are predicting for; in the context of this paper, the labeling would be for elevated levels of stress with labels as binary yes/no indicators or a numeric scale to indicate stress level, generally a range between 0 (no perceived stress) and 1 (maximum perceived stress).\\

\noindent As shown in  Table \ref{tab:datareviewed}, the datasets included in this study were labeled using one of two methods: (i) periodic \cite{Kraaij2015, Birjandtalab2016, Schmidt2018, Svoren2020, Amin2022}, where specific time frames during the experiment were either labeled as stressed or non-stressed, while the test subject was placed under that perceived condition (a stressful test or action, or non-stressed, restful period), or (ii) scored as experiencing stress or no stress during a particular period, either by completing a self-scoring evaluation \cite{Sabour2021, Hosseini2022}, or by an observer \cite{Haouij2018} who perceived a level of stress by observing the emotional reaction of the subject during that period.\\

\noindent When applying machine learning to the task of measuring stress on a biomarker dataset, a robust model should to be able to generalize well from the input (training) data for any new data from within the problem domain (validation data). Generalization in this context, therefore, refers to how well a trained machine learning model can perform on unseen data, i.e., data not included when initially training the model. In order to generate such a model, the training dataset should be sufficiently large and diverse. Within the context of this paper, this implies recording data samples under varying experimental conditions, and across demographics. Variance in the context of machine learning relates to the variety of predictions made by the model, while Bias refers to the distance of the predictions from the actual (true) values. A highly-biased model implies its predicted values are far from the actual, true values. A generalized model offers the best trade-off between bias and variance, thereby delivering the best predictive performance.\\

\noindent Power analysis assists researchers in determining the smallest sample size suitable to detect the effect of a given experiment at a desired level of significance, as collecting larger samples are likely costlier and much harder. One of the recurrent questions psychology researchers ask is: \emph{"What is the minimum number of participants I must test?"} \cite{Brysbaert2019}. The high number of participants required for an 80\% powered study often surprises cognitive psychologists, because in their experience, replicable research can be done with a smaller number. Given that an effect size of \emph{d = 0.4} is a good first estimate of the smallest effect size of interest in psychological research \cite{Brysbaert2019}, over 50 subjects will be required for a simple comparison of two within-participants conditions to run a study with 80\% power. Of the datasets included in this study, only UBFC-Phys \cite{Sabour2021} contains biomarker data for more than 50 subjects.\\

\noindent In this work, we investigate whether models built on datasets with a small number of study participants (\textless 50) are capable of generalizing well for use on new, unseen data, thereby showing transferability and generalizability. We further propose to improve the statistical power of training datasets by merging a number of small datasets to create a larger dataset named StressData (with \textgreater 90 subjects) to improve machine learning model generalizability to new unseen datasets. We also propose a new method of synthesizing data from a number of available small datasets to make a larger  balanced dataset. We call this new dataset SynthesizedStressData and show that it can significantly improve generalizability of various models to new unseen data. Finally, we propose to combine the predictive power of two unique machine learning algorithms using ensemble methods, and investigate whether our proposed approach offers improved predictive power over singular models in the literature.

\section{Related Work}

\noindent A growing number of studies have been conducted in recent years with the aim of building robust stress detection and measurement machine learning models. Few studies have further examined the reproducibility and generalizability of results previously reported. Mishra \emph{et al.} \cite{mishra2020} took the first steps towards evaluating the performance of models built using data from one study, and testing their performance on data from other studies. Reproducing the results achieved in previous studies rely on the availability of the stress biomarker data, and most often this was not the case.\\

\noindent Where experimental data was made available and the studies utilized Electrodermal Activity (EDA) and Heart Rate (HR) biomarkers, the reported results and associated models and datasets were reviewed as listed in Table \ref{tab:modelsreviewed}. The publicly available WESAD \cite{Schmidt2018} dataset containing sensor biomarker data for 15 subjects was the most commonly used \cite{Schmidt2018, Nkurikiyeyezu2019, Indikawati2020g, Li2020, Iqbal2021g, Liapis2021, Ninh2021}. The NEURO \cite{Birjandtalab2016} dataset was utilized in experiments by Jiménez-Limas \emph{et al.} \cite{Limas2018} and Eskandar \emph{et al.} \cite{Eskandar2020}, while Siirtola \emph{et al.} \cite{Siirtola2020} and Ninh \emph{et al.} \cite{Ninh2021} utilized the AffectiveROAD dataset containing sensor biomarker data for 9 subjects. Sriramprakash \emph{et al} \cite{Sriramprakash2017} and Khan \cite{Khan2022} utilized the SWELL \cite{Kraaij2015} dataset containing sensor biomarker data for 25 subjects, while Nkurikiyeyezu \emph{et al.} \cite{Nkurikiyeyezu2019} utilized both the WESAD and SWELL datasets.\\

\noindent Accuracy metrics reported ranged between 81.13\% \cite{Ninh2021} and 99.80\% \cite{Li2020}. LOSO cross-validation were utilized in three experiments \cite{Schmidt2018, Siirtola2020, Khan2022}. Indikawati \emph{et al.} \cite{Indikawati2020g} used a 60/40 train/validation split for validating their model performance, while Li \emph{et al.} \cite{Li2020} used a 70/30 train/validation split. Jiménez-Limas \emph{et al.} \cite{Limas2018} and Eskandar \emph{et al.} \cite{Eskandar2020} both used an 80/20 split for their experiments utilizing the NEURO dataset. The remaining studies utilized n-Fold cross-validation.\\

\begin{sidewaystable}
\centering
\caption{\label{tab:modelsreviewed}Summary of related works reviewed.}
\renewcommand{\arraystretch}{1.5}
\resizebox{\textwidth}{!}{
\begin{tabular}{Lcccccccc}
\hline\hline
\textbf{Paper}                                                                                & \multicolumn{1}{l}{\textbf{Year}} & \textbf{Model}                & \textbf{Dataset} & \multicolumn{1}{l}{\textbf{Accuracy}} & \multicolumn{1}{l}{\textbf{Subjects}} & \multicolumn{1}{l}{\textbf{Features}} & \textbf{Cross Validation}     & \textbf{Window}    \\
\hline
\rowcolor[rgb]{0.753,0.753,0.753} Stress Detection in Working People \cite{Sriramprakash2017}                                                                    & 2017                     & SVM                          & SWELL                & 92.75\%                             & 25                           & 17                           & 10-Fold          & 60s                \\
Feature selection for stress level classification into a physiologycal signals set \cite{Limas2018}
& 2018                     & Linear Regression & NEURO                & 81.38\%                             & 20                           & 7                           & 80/20 Split             & 5m     \\
\rowcolor[rgb]{0.753,0.753,0.753} Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection \cite{Schmidt2018}                            & 2018                     & Random Forest, LDA, AdaBoost & WESAD                & 93.00\%                             & 15                           & 82                           & LOSO             & 0.25s, 5s, 60s     \\
he Effect of Person-Specific Biometrics in Improving Generic Stress Predictive Models \cite{Nkurikiyeyezu2019}                & 2019                     & Random Forest, ExtraTrees    & WESAD, SWELL         & 93.90\%                             & \multicolumn{1}{l}{15, 25}   & 94                           & 10-Fold          & 5min, 10min        \\
\rowcolor[rgb]{0.753,0.753,0.753} Using Deep Learning for Assessment of Workers' Stress and Overload \cite{Eskandar2020}  & 2020                     & Neural Network         & NEURO        & 85\%                             & 20                            & 4                          & 80/20 Split              & 20s  \\
Comparison of Regression and Classification Models for User-Independent and Personal Stress Detection \cite{Siirtola2020}      & 2020                     & Bagged tree ensemble         & AffectiveROAD        & 82.30\%                             & 9                            & 119                          & LOSO             & 60s, 0.5s overlap  \\
\rowcolor[rgb]{0.753,0.753,0.753} Stress Detection from Multimodal Wearable Sensor Data \cite{Indikawati2020g}                                                   & 2020                     & Random Forest                & WESAD                & 92.00\%                             & 15                           & 4                            & 60/40 Split      & 0.25s              \\
Stress detection using deep neural networks \cite{Li2020}                                                                      & 2020                     & Neural Network               & WESAD                & 99.80\%                             & 15                           & 8                            & 70/30 Split      & 5s                 \\
\rowcolor[rgb]{0.753,0.753,0.753} A Sensitivity Analysis of Biophysiological Responses of Stress for Wearable Sensors in Connected Health \cite{Iqbal2021g}      & 2021                     & Logistic Regression          & WESAD                & 85.71\%                             & 14                           & 5                            & 14-Fold          & 60s                \\
Advancing Stress Detection Methodology with Deep Learning Techniques \cite{Liapis2021}                                         & 2021                     & L-SVM                        & WESAD                & 93.20\%                             & 15                           & 36                           & 5-Fold           &                    \\
\rowcolor[rgb]{0.753,0.753,0.753} Analysing the Performance of Stress Detection Models on Consumer-Grade Wearable Devices \cite{Ninh2021}                        & 2021                     & SVM                          & WESAD, AffectiveROAD & \multicolumn{1}{l}{87.5\%, 81.13\%} & \multicolumn{1}{l}{15, 9}    & 1                            &                  & 60s, 30s           \\
Semi-Supervised Generative Adversarial Network for Stress Detection Using Partially Labeled Physiological Data \cite{Khan2022} & 2022                     & Neural Network               & SWELL                & \multicolumn{1}{l}{ 90.00\%}        & 25                           & 30                           & LOSO             & 60s
    \\
\hline\hline
\end{tabular}
}
\end{sidewaystable}

\noindent Feature-engineering techniques were employed in all the models reviewed apart from Indikawati \emph{et al.} \cite{Indikawati2020g}, where Accelerometer (ACC), Electrodermal Activity (EDA), Blood Volume Pulse (BVP) and temperature (TEMP) signals were directly used, and Eskandar \emph{et al.} \cite{Eskandar2020} who utilized HR, EDA, TEMP and arterial
oxygen levels (SpO2). Iqbal \emph{et al.} \cite{Iqbal2021g} used a chest-worn RespiBAN device with the Empatica E4 and included biomarker data for RR-interval (RRI) and respiratory rate (RspR). Four of the models \cite{Schmidt2018, Nkurikiyeyezu2019, Siirtola2020, Indikawati2020g} employed tree-based machine learning models while Iqbal \emph{et al.} \cite{Iqbal2021g} and Jiménez-Limas \emph{et al.} \cite{Limas2018} utilized Logistic Regression and Linear Regression, respectively. Support Vector Machines (SVM) were utilized in experiments by Sriramprakash \emph{et al}  \cite{Sriramprakash2017}, Liapis \emph{et al.} \cite{Liapis2021} and Ninh \emph{et al.} \cite{Ninh2021}. Artificial neural network models were utilized in the remaining three studies \cite{Eskandar2020, Li2020, Khan2022} and reported the highest predictive accuracy across all results, followed by tree-based models. The most common feature-engineering technique employed consisted of generating new features from statistical summaries of biomarker data, based on grouped sliding windows ranging between 0.25 seconds to 60 seconds. Nkurikiyeyezu \emph{et al.} \cite{Nkurikiyeyezu2019} and Jiménez-Limas \emph{et al.} \cite{Limas2018} employed larger sliding windows of 5 minutes and 10 minutes, respectively.\\

\noindent In the related works reviewed, experimentation was performed on a single dataset, most notably WESAD, with a limited number of subject sensor biomarker data (15 subjects). Only Nkurikiyeyezu \emph{et al.} \cite{Nkurikiyeyezu2019} validated their experimentation results by using two unique datasets, WESAD and SWELL (15 and 25 subjects, respectively). As noted previously, Brysbaert \emph{et al.} \cite{Brysbaert2019} recommended over 50 subjects to achieve statistical power of over 80\%, which brings into question whether models built on datasets containing less than 50 subjects produce results that can be considered statistically significant, and whether these models generalize well to predict accurately on new, unseen data.\\

\noindent To address this question, in this paper we will test the generalizability of machine learning models built on one, small dataset by validating these models on different, unseen datasets. The ability of machine learning models to generalize well is crucial to providing a technology solution that can be utilized and applied outside of experimental conditions, thereby providing a step forward towards enabling health practitioners to better use sensor biomarker data recorded using wearable devices.\\

\noindent In addition, in order to fill the gap of lack of substantially large public datasets available for stress research, we propose to merge four small public stress biomarker datasets into a single larger dataset (StressData), representing a total of 99 test subjects, thereby providing statistical power of over 85\%. Although we show that the improved statistical power of a larger number of subjects within a training dataset leads to improved predictive performance, a challenge with the merging approach is the resulting class imbalance created due to the varying experimental protocols utilized during biomarker recording. To address this challenge, we propose using random sampling of StressData segments to generate a 200-subject synthetic training dataset, which we name SynthetizedStressData. Finally, we build two unique machine learning models utilizing the well-known XGBoost (XGB) gradient boosting algorithm and an artificial neural network (ANN), then combine their predictive power using ensemble methods to deliver a robust stress detection system. We show that while our ensemble model trained on StressData leads to small improvements, utilizing our SynthesizedStressData dataset results in a predictive performance improvement of over 25\%, compared to another model trained on a small dataset with low statistical power.

\section{Methods}
\subsection{Datasets}

\noindent Based on the prior experiments and reviewed literature (Table \ref{tab:modelsreviewed}), publicly available datasets including WESAD \cite{Schmidt2018} and SWELL \cite{Kraaij2015} were utilized in this study. Additionally, since these datasets all included  Empatica E4 sensor biomarker data, the Toadstool \cite{Svoren2020}, UBFC-Phys \cite{Sabour2021}, Non-EEG Dataset for Assessment of Neurological Status (NEURO) \cite{Birjandtalab2016}, Wearable Exam Stress Dataset (EXAM) \cite{Amin2022}, AffectiveROAD \cite{Haouij2018} and Multimodal Sensor Dataset for Continuous Stress Detection of Nurses in a Hospital \cite{Hosseini2022} public datasets, which also are collected using Empatica E4, were considered. Table \ref{tab:datareviewed} provides a summary of all  datasets considered. The AffectiveROAD dataset was excluded from our experimentation due to its labeling protocol. In this dataset, subjects were self- and observer-scored on a scale that limited conversion to a binary stressed/non-stressed indicator for model training and evaluation purposes. Subjects were scored while driving in inner city and highway scenarios, and it is not clear which scenario under normal circumstances would be considered more or less stressful.\\

\noindent Additionally, the Multimodal Sensor Dataset for Continuous Stress Detection of Nurses in a Hospital \cite{Hosseini2022} was not included in our experimentation for similar reasons. In this dataset, thirteen potential scenarios could be marked by the subject as stressful by using the Empatica E4 event marker button. We found a substantial number of marked sections that exceeded the expected duration of the event, with a further number of short events with no potential for a cooling down period to separate the perceived stressful event from a subject baseline (non-stressed).\\

\noindent The SWELL, WESAD, NEURO and UBFC-Phys datasets were specifically labeled for stress, either using self-report scoring (UBFC-Phys) or periodic (SWELL, NEURO, WESAD, EXAM), where stressful situations were applied during specific time periods of the experiment. The Toadstool dataset contained sensor biomarker data recorded during periods of game-play where stress may have been perceived as high, but was not labeled. However, it was included to validate the XGB, ANN and ensemble models developed as part of this study. The EXAM dataset was also included for some of our experiments. This dataset was recorded during single sessions of 10 students writing mid-term and final exams, with exam scores provided, but no specific stress-based scoring included.\\

\noindent Of the biophysical and biochemical markers normally used to measure stress \cite{Iqbal2021g}, in this study we opted to focus on Electrodermal Activity (EDA), and Heart Rate (HR). Previous works have included Accelerometer (ACC) bio-marker data as a feature, however considering that at least one of the datasets (Toadstool) involved substantial subject physical movement as part of their experimental setup, this signal was excluded. Temperature (TEMP) and Skin Temperature (ST) was further excluded due to being missing in two additional datasets (SWELL, UBFC-Phys).\\

\noindent While not all datasets included an HR biomarker, this can be algorithmically derived using Heart Rate Variation (HRV) or Blood Volume Pulse (BVP), and where required was generated using a Python script utilizing the BioSPPy library \cite{BioSPPy}.\\

\subsection{Dataset analyses}
\noindent The statistical programming language R \cite{RSoftware} version 4.0.4 was used for analysis in this study. The Empatica-R library \cite{EmpaticaR} was additionally utilized to read the raw Empatica E4 wearable device data into an R data table for processing. Next, sensor signals were converted to the same sampling frequency, as the Empatica E4 samples the BVP sensor data at 64 Hz, the EDA sensor data at 4 Hz, and HR sensor data in spans of 10 seconds. Stress labels within the datasets were then standardized to binary indicators with non-stressed set to zero (0) and stressed set to one (1).\\

\noindent Initial data exploration was performed on only the NEURO, WESAD, SWELL and UBFC-Phys datasets, considering the WESAD and SWELL datasets were utilized in prior work as listed in Table \ref{tab:modelsreviewed}, and all four contained at least the EDA and HR biomarker data for a substantial percentage of the observations available. Not all biomarker sensor data were available for all subjects within the SWELL and WESAD datasets, and in these cases the subjects were excluded. This resulted in a total of 9 subjects from the SWELL dataset being included (of 25) containing a total of 157,739 observations, and 14 of the 15 subjects from the WESAD dataset, containing a total of 26,385 observations.\\

\noindent Statistical summaries were produced for each dataset, as detailed in Table \ref{tab:datastats}, indicating substantial class imbalance across the datasets, with the NEURO and UBFC-Phys datasets containing 70.1\% and 65.8\% labels marked as stressed, respectively. In contrast, WESAD and SWELL have labels marked as non-stressed in 63.6\% and 71.4\% of total observations. The Table also shows the mean and Standard Deviation (SD) of the biomarkers across the four datasets. Substantial variation is further noted across each dataset for both the EDA and HR biomarkers.

\begin{table}
\setlength\tabcolsep{4pt}
\centering
\caption{\label{tab:datastats}Statistical summary of included public wearable device stress-related datasets.}
\resizebox{\textwidth}{!}{
\begin{tabular}{ccccccc}
\hline\hline
\textbf{Dataset}                        & \multicolumn{1}{c}{\textbf{EDA}}    & \multicolumn{1}{c}{\textbf{EDA}}  & \multicolumn{1}{c}{\textbf{HR}}     & \multicolumn{1}{c}{\textbf{HR}}   & \multicolumn{1}{l}{\textbf{Non-Stressed}} & \multicolumn{1}{l}{\textbf{Stressed}}  \\
                                        & \multicolumn{1}{c}{\textbf{(Mean)}} & \multicolumn{1}{c}{\textbf{(SD)}} & \multicolumn{1}{c}{\textbf{(Mean)}} & \multicolumn{1}{c}{\textbf{(SD)}} & \multicolumn{1}{l}{}                      & \multicolumn{1}{l}{}                   \\
\rowcolor[rgb]{0.753,0.753,0.753} NEURO & 1.7                                 & 2.1                               & 75.4                                & 13.4                              & 29.90\%                                   & 70.10\%                                \\
WESAD                                   & 2.2                                 & 2.7                               & 76.5                                & 10.5                              & 63.60\%                                   & 36.40\%                                \\
\rowcolor[rgb]{0.753,0.753,0.753} SWELL & 1                                   & 0.2                               & 75.8                                & 10.1                              & 71.40\%                                   & 28.60\%                                \\
UBFC                                    & 1.4                                 & 2                                 & 92.2                                & 16                                & 34.20\%                                   & 65.80\%

\\
\hline\hline
\end{tabular}
}
\end{table}

\noindent Histograms (Figure \ref{fig:figure1}) of each biomarker across the datasets show the EDA biomarker being significantly right-skewed. It should be noted that, the SWELL dataset contains on average 4\% or more observations than the NEURO, WESAD or UBFC-Phys datasets, while the UBFC-Phys dataset contains the largest number of individual subjects, but each recording segment being significantly shorter compared to the SWELL, NEURO and WESAD datasets.\\

\begin{figure}[h]
\centering
\fbox{\includegraphics[width=\textwidth]{figure1_1col.jpg}}
\caption{\label{fig:figure1}Biomarker histograms for NEURO, SWELL, UBFC-Phys and WESAD datasets.}%
\end{figure}

\noindent Box plots highlight this variation further (Figure \ref{fig:figure2} and Figure \ref{fig:figure3}), with significant outliers occurring for the HR biomarker (Figure \ref{fig:figure3}) of the UBFC-Phys dataset.\\

\begin{figure}
\centering
\fbox{\includegraphics[width=\textwidth]{figure2_1col.jpg}}
\caption{\label{fig:figure2}EDA biomarker variance across NEURO, SWELL, UBFC-Phys and WESAD dataset subjects.}%
\end{figure}

\begin{figure}
\centering
\fbox{\includegraphics[width=\textwidth]{figure3_1col.jpg}}
\caption{\label{fig:figure3}HR biomarker variance across NEURO, SWELL, UBFC-Phys and WESAD dataset subjects.}%
\end{figure}

\noindent In addition to box plots and statistical summaries, plots were generated for each of the four datasets to investigate correlation between the HR and EDA biomarkers and the labeled, binary stress metric (Figure \ref{fig:figure4}). Correlation was consistently weak across all the datasets, with the lowest correlation observed in the UBFC-Phys dataset. Within the NEURO dataset, we noted higher correlation between the binary stress label and the HR biomarker, while correlation was low between the binary stress label and the EDA biomarker. This correlation was slightly higher for the SWELL and WESAD biomarkers. This difference in correlation across stress biomarker datasets can affect transferability of experimental results, and further limit the ability of machine learning models trained on any of these individual datasets to generalize well for new, unseen data.

\begin{figure}[!h]
\centering
\fbox{\includegraphics[width=\textwidth]{figure4_1col.jpg}}
\caption{\label{fig:figure4}Correlation between biomarkers and stress metric of NEURO, SWELL, UBFC-Phys and WESAD datasets.}%
\end{figure}

\subsection{Machine Learning Models} \label{sec:ML}

\noindent One of the most crucial steps in building machine learning models is the choice of algorithm. A common approach for optimal algorithm selection involves conducting exhaustive grid-searches using a wide variety of algorithms and approaches, and determining the best potential algorithm based on the results achieved. By examining the reported results of previous experiments and their associated algorithms (summarized in Table \ref{tab:modelsreviewed}), we find tree-based algorithms including Random Forest performing well compared to other algorithms, in line with findings reported by Misha \emph{et al.} \cite{mishra2018}, who noted that Random Forest is often the optimal model when using EDA, HR and RRI biomarker data as input features for stress-related machine learning models.\\

\noindent Gradient Boosting algorithms, compared to Random Forest, offer improved efficiency and predictive performance, at the cost of being prone to over-fitting. Random Forest builds a number of decision trees independently, where each decision tree is a simple predictor with all results aggregated into a single result. The Gradient Boosting algorithm, however, is an ensemble of weak predictors, usually decision trees. Additionally, in Random Forest, the results of the decision trees are aggregated at the end of the process, while Gradient Boosting instead aggregates the results of each decision tree along the way to calculate the final result. Popular Gradient Boosting algorithms include LightGBM \cite{LightGBM}, CatBoost \cite{CatBoost} and XGBoost (XGB) \cite{XGBoost}.\\

\noindent For this study, XGB was selected as the primary machine learning algorithm due to its popularity, availability as an R package, and having been extensively utilized since its original publication in 2016 \cite{XGBoost}. XGB provides a highly efficient implementation of the stochastic gradient boosting algorithm and access to a suite of model hyperparameters, designed to provide control over the model training process.

\subsection{Model Generalization} \label{sec:modelgen}
\noindent To test the generalizability of machine learning models built on datasets containing sensor biomarker data of a small number of subjects, two initial experiments were performed, as follows:
\begin{enumerate}
  \item Train XGB on SWELL, test on NEURO and WESAD with no additional feature-engineering, using EDA and HR biomarkers only.
  \item Train XGB on SWELL, test on NEURO and WESAD and generate additional features using statistical summaries.\\
\end{enumerate}

\noindent The SWELL dataset was selected for training as it contained the largest number of observations, nearly 6 times more than that of either WESAD or NEURO. The UBFC-Phys dataset was excluded as it contained 56 subjects, substantially larger than WESAD, NEURO or SWELL, which were used in the previous works that were reviewed. As the stress metric within the datasets was binary, logistic regression was selected as the XGB learning objective. Due to the metric imbalance within each dataset, class balancing was performed by using the \emph{scale\textunderscore pos\textunderscore weight} parameter provided by XGB, which has the effect of weighing the balance of positive examples, relative to negative examples, when boosting decision trees.\\

\noindent Optimal hyperparameters for the XGB algorithm were identified using a grid search with 10-Fold cross-validation, and this was repeated for each experiment, to ensure the most suitable parameters were utilized prior to model training. The XGB algorithm was run for 5,000 rounds with early-stopping set to 3 rounds. This ensures that training stops when the evaluation metric fails to improve after 3 rounds of training in order to prevent over-fitting. For the initial two experiments, training was automatically stopped after 93, and 87 rounds, respectively.\\

\noindent Prior to performing experiments where feature-engineering was applied, we investigated the dominance of biomarkers and found HR feature as a more important biomarker, relative to EDA, as shown in Figure \ref{fig:figure5}.\\

\begin{figure}[h]
\centering
\fbox{\includegraphics[width=\textwidth]{figure5_1col.jpg}}
\caption{\label{fig:figure5}Feature importance for experiment 1, i.e. before biomarker feature engineering.}%
\end{figure}

\noindent Feature importance was determined using the built-in \emph{xgb.plot.importance} function provided by the XGBoost R package. Here, importance is calculated based on overall gain, where gain implies the relative contribution of the corresponding feature to the model, calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.\\

\noindent Next, an additional 22 features were generated by calculating statistical summaries using a tumbling window approach at 25-second intervals. Intervals between 0.25 seconds and 60 seconds are common in the literature (see Table \ref{tab:modelsreviewed}), and 25-second intervals were selected based on the highest level of correlation observed between the biomarkers and stress label, after a substantial amount of experimentation. Statistical summaries including the mean, median, max, min, standard deviation, variance, skewness and kurtosis of each biomarker were calculated as new features, with an additional feature for the covariance between the EDA and HR biomarker. The resulting variable importance after model training and evaluation is detailed in Figure \ref{fig:figure6} (top 10, in order of importance).\\

\begin{figure}[h]
\centering
\fbox{\includegraphics[width=\textwidth]{figure6_1col.jpg}}
\caption{\label{fig:figure6}Feature importance for experiment 2, i.e. after biomarker feature engineering.}
\end{figure}

\noindent The HR biomarker and its derived features are again observed to be the dominant features, with EDA-based features serving a secondary role. These 10 features were utilized in all our experiments requiring feature engineering. The remaining 12 features were not utilized to prevent potential over-fitting to specific datasets, notably UBFC-Phys, with 56 subjects, and substantially shorter recording segments compared to the NEURO, WESAD and SWELL datasets.\\

\subsection{Merging Unique Stress Biomarker Datasets to Form StressData}

\noindent Merging was performed by combining the four smaller datasets, i.e.  NEURO, WESAD, SWELL and UBFC-Phys row-wise without randomization, to form the single larger dataset, StressData.  This includes a total of 99 unique subjects, constituting a total of 244,399 observations, as follows:

\begin{itemize}
  \item All 20 subjects from the NEURO dataset.
  \item All 56 subjects from the UBFC-Phys dataset.
  \item 14 of the 15 WESAD dataset subjects contained sufficient biomarker data for inclusion.
  \item 9 of the 25 SWELL dataset subjects contained sufficient biomarker data for inclusion.
\end{itemize}

\noindent Two new experiments, numbered 3-4, were then performed on the StressData dataset, as follows:

\begin{enumerate}
\setcounter{enumi}{2}
  \item Train XGB on StressData using LOSO validation, with no feature-engineering, using EDA and HR biomarkers only.
  \item Train XGB on  StressData using LOSO validation, apply  feature-engineering (10 total features), as explained in subsection \ref{sec:modelgen}.
\end{enumerate}

\subsection{An Artificial Neural Network Model for Stress Detection}

\noindent Of the prior works reviewed, we found three instances of using Artificial Neural Networks (ANN). Li \emph{et al.} \cite{Li2020} utilized a Deep Convolutional Network (DCNN) consisting of three hidden layers containing 32 and 16 units, respectively. This was followed by a Sigmoid activation layer, with max pooling applied at the end of each convolution. Predictive accuracy using the WESAD dataset was reported as 99.80\%. Khan \cite{Khan2022} tested Long-Short Term Memory Network (LTSM),  Convolutional Neural Networks (CNN) and Bi-Directional Long-Short Term Memory networks, using both supervised and semi-supervised methods and found supervised models to perform better than semi-supervised models, with accuracy across predictions averaging higher than 90\% on the SWELL dataset. Eskandar \emph{et al.} \cite{Eskandar2020} utilized a LTSM model fed with features extracted by using a CNN with a reported accuracy of 85\% (F1 Score), noting difficulty on predicting the baseline relaxed state when using the NEURO dataset.\\

\noindent Based on recommendations from available literature \cite{Stathakis2009}, we opted for an architecture utilizing a dense network, because of the time-series nature of the sensor data. Our proposed network consists of three layers. The input layer contained 10 neurons to receive the 10 input features. The hidden layer included 5 neurons (half the input features), connected to a final linear layer.\\

\noindent Mean Squared Error (MSE) was selected as the loss function, a frequently used measure of the differences between values predicted by a model and the expected values:

$$MSE = \sum_{i=1}^{D}(\hat{y_i}-y_i)^2$$

\noindent where N is the number of observations, $\hat{y}$ is the predicted value and \emph{y} is the expected value. Typically, binary and multi-class problems would utilize cross-entropy as a loss function instead:
$$-{(y\log(p) + (1 - y)\log(1 - p))}$$
\noindent where log is the natural log, \emph{y} is the binary indicator (0 or 1), and \emph{p} is the predicted probability observation. However, we found MSE to outperform cross-entropy by a factor of 10\% for predictive accuracy. The precision, recall and F-score metrics were also calculated given the classification nature of the labeling of the SWELL, WESAD, NEURO and UBFC-Phys datasets.\\

\noindent The Rectified Linear Unit (ReLU) \cite{RELU} activation function was applied after both the input and hidden layers. Training was performed on a dual-GPU system using the Keras and Tensorflow libraries for a maximum of 200 epochs using a batch size of 512. Early stopping was employed to prevent over-fitting if the validation loss stops improving for a duration of 5 or more epochs.\\

\subsection{The Proposed Ensemble Model}

\noindent Ensembling \cite{Zhang2012} is a widely-used technique known to improve a decision system's robustness and accuracy. Algorithms such as Gradient Boosting and Random Forest utilize ensemble methods to combine the individual results of a large number of independent decision trees into a single prediction. This approach has been further extended to combine the predictions of unique, individual machine learning models via majority voting, weighted-scoring and other blending techniques to deliver more powerful and robust models.\\

\noindent To apply ensembling techniques to our own work, we blended the predictions from our XGB model with those from our ANN model using simple averaging. This resulted in four new experiments as follow:

\begin{enumerate}
\setcounter{enumi}{4}
  \item Train the proposed ensemble model (XGB + ANN) on SWELL and validate on NEURO and WESAD with no feature-engineering, using EDA and HR biomarkers only.
  \item Train the proposed ensemble model (XGB + ANN) on SWELL and validate on NEURO and WESAD with feature-engineering (10 total features), as explained in subsection \ref{sec:modelgen}.
  \item Train the proposed ensemble model (XGB + ANN) on StressData using LOSO validation and feature-engineering (10 total features), as explained in subsection \ref{sec:modelgen}.
  \item Train the proposed ensemble model (XGB + ANN) on StressData (excluding WESAD) and validate on WESAD with feature-engineering (10 total features), as explained in subsection \ref{sec:modelgen}.
\end{enumerate}

\subsection{Class Balancing} \label{sec:classbalancing}
\noindent Reviewing the initial results from experiment 7 when using XGB, ANN and their ensemble against the labeled metric for all 99 StressData subjects, we noted predictions for UBFC-Phys vastly outperformed those for WESAD, SWELL and NEURO across both XGB and ANN models. This was attributed to two potential factors; i) the substantially larger subject size of the UBFC-Phys dataset and ii) the substantially shorter recording segment size of UBFC-Phys samples.\\

\noindent We further noted a reduction in predictive accuracy in Experiment 8, when the WESAD dataset was excluded from StressData and used as a new, unseen validation dataset. This implies that LOSO validation alone may not be a good predictor of overall model generalization ability, likely due to an imbalance across both labeled metric and the significant variance in recording segment sizes across the datasets when merged into StressData.\\

\noindent To address this imbalance, we combined the EXAM dataset with StressData, resulting in a total of 129 subjects. Next, we split the stressed and non-stressed periodic segments of all 129 subjects into two groups. We noted the shortest segment across all subjects were from the UBFC-Phys dataset, at 201 seconds. We therefore further split each stressed and non-stressed subset across all subjects into smaller sets of 180 seconds each (3 minute intervals).\\

\noindent This resulted in a total of 3,758 samples of 3-minute segments, labeled as either stressed (2,962 samples) or non-stressed (796 samples). From these segments, random sampling can be performed to build a substantially larger balanced training dataset of any number and combination of segments. We named this new dataset SynthesizedStressData.\\

\noindent In addition to the 8 aforementioned experiments, we performed two final experiments:

\begin{enumerate}
\setcounter{enumi}{8}
  \item Train the proposed ensemble model (XGB + ANN) on SynthesizedStressData and validate using LOSO. For this experiment, 200 training subjects were generated by using random sampling from the 3,758 segments to simulate two physiological states: a 6-minute long non-stressed baseline period, followed by a 6-minute long stressed period. Feature engineering was applied as explained in subsection \ref{sec:modelgen}.
  \item Train the proposed ensemble model (XGB + ANN) on SynthesizedStressData (excluding the WESAD data) and test on the WESAD dataset for validation. For this experiment, all WESAD samples were first excluded from SynthesizedStressData, resulting in a smaller total of 115 subjects. To emulate the WESAD experimental protocol \cite{Schmidt2018}, 200 training subjects were generated using random sampling to emulate two physiological states: a 12-minute long non-stressed period, followed by a 12-minute long stressed period. Feature engineering was applied as explained in subsection \ref{sec:modelgen}.\\
\end{enumerate}

\section{Results and Discussion}

\noindent Experiments 1 to 2 tested two approaches, the first excluding feature engineering, with the second including feature engineering. These were performed for measuring generalization of models built on a single dataset (SWELL) containing a small number of test subjects, but with large amount of observations. These models were then validated against two new, unseen datasets (WESAD, NEURO) containing a substantially smaller amount of observations.\\

\noindent The results indicate low predictive power with the best model achieving 68\% accuracy using a 70/30 Train/Test split during training, with feature-engineering. More importantly, the results show that a substantially larger number of observations (from a small number of test subjects) within the training dataset did not assist in generalization. Once the datasets are merged to form the larger StressData dataset for training a machine learning model, predictive accuracy of 63\% is achieved with no feature engineering (Experiment 3), increasing to 72.36\% (Experiment 4) when using feature engineering, both using LOSO validation.\\

\noindent Repeating Experiments 1 and 2 but using a weighted ensemble approach (XGB + ANN), predictive accuracy slightly improves to 51\% (NEURO) and 70\% (WESAD) for Experiment 5, with no feature engineering, then 58\% (NEURO) and 68\% (WESAD) for Experiment 6, with feature engineering applied, when training on only the SWELL dataset. Experiment 7 combines the benefits of a larger dataset (StressData) with feature engineering and a weighted ensemble approach, reporting predictive accuracy of 80.33\% when using LOSO validation. The results from the ten aforementioned experiments are detailed in Table \ref{tab:results}.\\

\begin{sidewaystable}
\centering
\caption{\label{tab:results}Summary of experimental results.}
\renewcommand{\arraystretch}{1.5}
\resizebox{\textwidth}{!}{
\begin{tabular}{clllccccc}
\hline\hline
\textbf{Experiment}                  & \textbf{Model} & \textbf{Train Data} & \textbf{Validation Data} &  \textbf{F/E} & \textbf{Accuracy} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}  \\
\hline
\rowcolor[rgb]{0.753,0.753,0.753} 1                              & XGB      & SWELL                 & NEURO, WESAD    &                            & 50\%, 66\%                   & 0.59, 0.27                    & 0.36, 0.29                 & 0.45, 0.28              \\
2                              & XGB      & SWELL                 & NEURO, WESAD    & $\bullet$ & 50\%, 68\%                   & 0.53, 0.36                    & 0.24, 0.59                 & 0.33, 0.45              \\
\rowcolor[rgb]{0.753,0.753,0.753} 3                              & XGB      & StressData            & LOSO            &                            & 63.00\%                      & 0.44                          & 0.68                       & 0.50                    \\
4                              & XGB      & StressData            & LOSO            & $\bullet$ & 72.36\%                      & 0.47                          & 0.66                       & 0.47                    \\
\rowcolor[rgb]{0.753,0.753,0.753} 5                              & Ensemble & SWELL            & NEURO, WESAD            & & 51\%, 70\%                      & 0.67, 0.55                          & 0.30, 0.44                       & 0.38, 0.56                    \\
6                              & Ensemble & SWELL            & NEURO, WESAD            & $\bullet$ & 58\%, 68\%                      & 0.66, 0.45                          & 0.54, 0.16                       & 0.58, 0.27                    \\
\rowcolor[rgb]{0.753,0.753,0.753} 7                              & Ensemble & StressData            & LOSO            & $\bullet$ & 80.33\%                      & 0.56                          & 0.75                       & 0.47                    \\
8                              & Ensemble & StressData (excl. WESAD)  & WESAD            & $\bullet$ & 59.00\%                      & 0.33                          & 0.22                       & 0.24                    \\

\rowcolor[rgb]{0.753,0.753,0.753} 9                              & Ensemble & SynthesizedStressData  & LOSO            & $\bullet$ & 89.00\%                      & 0.90                          & 0.88                       & 0.89                    \\
10                              & Ensemble & SynthesizedStressData (excl. WESAD) & WESAD           &  $\bullet$ & 85.00\%                         & 0.90                          & 0.74                       & 0.78                          \\

\hline\hline
\end{tabular}
}
\end{sidewaystable}

\noindent Figure \ref{fig:figure7} shows the numerical prediction plot for a randomly selected subject (S9) from the SWELL dataset (Experiment 7), with red detailing the stress period and blue indicating the ensemble predictions. The result shows good correlation with the period labeled as stressed. We note from 15,600 seconds onward, the subject is still showing an elevated level of stress, while the labeled recording period has been marked as non-stressed, likely not allowing for a cool down stage between stress and non-stress periods during biomarker recording.

\begin{figure}[t]
\centering
\fbox{\includegraphics[width=\textwidth]{figure7_1col.jpg}}
\caption{\label{fig:figure7} Ensemble model predictions (blue) on subject S9 from the SWELL dataset (Experiment 7).}%
\end{figure}

\noindent Figure \ref{fig:figure8} shows the numerical prediction plot for a randomly selected subject (here W14) from the WESAD dataset (Experiment 7), with red detailing the stress period and blue indicating the ensemble predictions. This result shows the ensemble model reacting well when subjects are subjected to a period of stress during the experiment.\\

\begin{figure}[t]
\centering
\fbox{\includegraphics[width=\textwidth]{figure8_1col.jpg}}
\caption{\label{fig:figure8}Ensemble model predictions (blue) on subject W14 from the WESAD dataset (Experiment 7).}%
\end{figure}

\noindent Comparing the results to those from Experiments 1 to 4, we note slightly better results for the ensemble (using LOSO validation) when both a larger, more varied training dataset is utilized in combination with feature engineering. Various combinations of weighting were tested and utilized in order to achieve the highest accuracy rates, with the XGB model generally outperforming the ANN in most experiments:

\begin{itemize}
  \item Experiment 5 - 60\% (XGB) and 40\% (ANN).
  \item Experiment 6 - 80\% (XGB) and 20\% (ANN).
  \item Experiment 7 - 40\% (XGB) and 60\% (ANN) when predicting on biomarker segments exceeding 16 minutes in duration, and 70\% (XGB) and 30\% (ANN) for biomarker segments shorter than 16 minutes in duration.
  \item Experiment 8 - 30\% (XGB) and 70\% (ANN).
\end{itemize}

\noindent However, when training StressData with the WESAD dataset excluded (Experiment 8), predictive accuracy reduced to 59\% when testing on WESAD as an unseen validation set. Various class balancing methods were applied to StressData including over, under and both-sampling across the labeled stress metric, with both-sampling producing the highest score of 59\%, compared to over and under-sampling. This lower accuracy result implies that LOSO validation alone (Experiment 7) is not a good measure of model generalization, as the training process includes the full StressData dataset, apart from the single subject being left out per training and scoring iteration. This result further implies that training on disparate datasets merged without extensive class and feature balancing across both labeled metric as well as the recording segment size of biomarkers, may not result in a well-generalized model.\\

\noindent Next, in Experiment 9, the EXAM dataset was included into StressData, constituting a total of 129 unique subjects, forming SynthesizedStressData. Random sampling was then utilized for class balancing as described in section \ref{sec:classbalancing} to construct 200 training subjects consisting of two physiological states: 6 minutes of a non-stressed baseline, and 6 minutes of a stressed period, sampled from the pool of 3,758 3-minute StressData segments (2962 stressed and 796 non-stressed). Validation was performed using LOSO, achieving a mean accuracy of 89\% on this instance of  SynthesizedStressData.\\

\noindent Precision, Recall and F1 score metrics are higher compared to experiments 1 through 8. Recall represents the model's ability to correctly predict the positives (stressed) out of actual positives (stressed), while Precision measures the number of predictions made that are actually positive (stressed), out of all positive predictions (stressed). In both cases, a higher score is desired. The F1 score is a good measure to use when seeking a balance between Precision and Recall. The prediction plot of subject X188 (from the SynthesizedStressData) is shown in Figure \ref{fig:figure9}, with a near-perfect match when predicting for both baseline and stressed periods.\\

\noindent Adjusting ensemble weighting based on varying biomarker segment sizes (Experiment 7) during the prediction phase, is not an ideal approach. Therefore, this method was not utilized in Experiment 9, where a fixed weighting of 60\% (XGB) and 40\% (ANN) (similar to Experiment 5) was  applied, given the strength of the XGB model over the ANN model in prior experiments.\\

\begin{figure}[t]
\centering
\fbox{\includegraphics[width=\textwidth]{figure9_1col.jpg}}
\caption{\label{fig:figure9}Predictions on subject X188 from Experiment 9.}%
\end{figure}

\noindent Finally, for Experiment 10, the ensemble model trained on SynthesizedStressData was applied to the WESAD dataset to compare predictive power to those from Experiments 1, 2, 5, 6, and specifically Experiment 8. For this experiment, all WESAD data in StressData was removed and used as a validation set (similar to Experiment 8), resulting in a smaller total of 115 subjects used for training. Random sampling was again utilized to build a new instance of the SynthesizedStressData dataset as described in section \ref{sec:classbalancing}, to construct 200 training subjects consisting of two physiological states: 12 minutes of a non-stressed baseline, and 12 minutes of a stressed period, sampled from the pool of 3,600 3-minute segments (2917 stressed and 683 non-stressed).\\

\noindent A 12-minute interval was used to closely emulate the experimental protocol used when building the WESAD dataset, specifically the stressed period (approx. 10 minutes of stress \cite{Schmidt2018}). The random sampling, training and validation process was repeated for 10 iterations, with accuracy scores averaged across the 10 iterations. The results of these iterations are detailed in Table \ref{tab:resultsten}.\\

\noindent The ensemble method outperformed both the XGB and ANN models, with a mean accuracy rate of 85\%, and a very low standard deviation (0.029) across all 10 trials. A fixed weighting of 45\% (XGB) and 55\% (ANN) was used, showing more stability, and closer to an ideal balanced 50\%/50\% weighting. This result, compared to those from Experiment 8, shows a well generalized machine learning model capable of predicting accurately on new, unseen data (WESAD).\\

\begin{table}[t]
\centering
\caption{\label{tab:resultsten}Summary of accuracy scores for 10 trials of Experiment 10.}
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{lcccccc}
\hline\hline
\textbf{Iteration} & \textbf{XGB}   & \textbf{ANN}   & \textbf{Ensemble} & \textbf{Precision} & \textbf{Recall} & \textbf{F1}     \\
1                  & 0.83           & 0.77           & 0.81              & 0.84               & 0.72            & 0.81            \\
2                  & 0.83           & 0.84           & 0.83              & 0.89               & 0.70            & 0.74            \\
3                  & 0.81           & 0.83           & 0.85              & 0.94               & 0.72            & 0.77            \\
4                  & 0.82           & 0.88           & 0.82              & 0.90               & 0.69            & 0.73            \\
5                  & 0.84           & 0.84           & 0.85              & 0.92               & 0.75            & 0.78            \\
6                  & 0.84           & 0.89           & 0.86              & 0.95               & 0.77            & 0.82            \\
7                  & 0.85           & 0.93           & 0.89              & 0.93               & 0.86            & 0.88            \\
8                  & 0.81           & 0.83           & 0.84              & 0.89               & 0.74            & 0.78            \\
9                  & 0.83           & 0.87           & 0.87              & 0.92               & 0.80            & 0.85            \\
10                 & 0.80           & 0.82           & 0.80              & 0.83               & 0.61            & 0.66            \\
\hline
\rowcolor[rgb]{0.753,0.753,0.753} \textbf{MEAN}      & \textbf{0.82}  & \textbf{0.82}  & \textbf{0.85}     & \textbf{0.90}      & \textbf{0.74}   & \textbf{0.78}   \\
\rowcolor[rgb]{0.753,0.753,0.753} \textbf{SD}       & \textbf{0.015} & \textbf{0.045} & \textbf{0.029}    & \textbf{0.041}     & \textbf{0.065}  & \textbf{0.062} \\
\hline\hline
\end{tabular}
\end{table}

\noindent Figure \ref{fig:figure10} shows a prediction plot of subject W15 from the WESAD dataset. We note a slight stress over-prediction during the baseline condition, with high correlation between prediction and label across the 24 minute experimental period.\\

 \noindent To emulate a real-world scenario where a new, unseen dataset is provided with a potentially stressful experimental scenario, we trained and applied the full SynthesizedStressData (3758 samples, 129 subjects) ensemble model to the Toadstool dataset, using a balanced 50\% (XGB) and 50\% (ANN) weighting.\\

 \noindent This dataset contains sensor biomarker data recorded during periods of game-play where levels of subject stress may have been perceived as high. No labeling is provided for this dataset to compare predictive accuracy. Figure \ref{fig:figure11} shows a plot of the ensemble predictions for subject T2. We note elevated levels of stress when game-play starts, with a period of stabilization, prior to an increase as game-play reaches the end stage. While no definitive conclusions can be made from this test, it does show correlation between predictions and perceived levels of stress in this particular experimental scenario.

\begin{figure}
\centering
\fbox{\includegraphics[width=\textwidth]{figure10_1col.jpg}}
\caption{\label{fig:figure10}Predictions on subject W15 from the WESAD dataset (Experiment 10).}%
\end{figure}

\begin{figure}[!h]
\centering
\fbox{\includegraphics[width=\textwidth]{figure11_1col.jpg}}
\caption{\label{fig:figure11}Predictions on subject T2 from the Toadstool dataset.}%
\end{figure}

\section{Conclusions and Discussion}

\noindent In this work, we evaluated the generalization ability of machine learning models trained on datasets containing stress biomarker data for a small number of study subjects. We found these models performed poorly, and proposed a methodology to engineer merging several small datasets into a single larger dataset which, when combined with feature-engineering, delivered a more robust stress detection model with significantly improved performance. This study was limited to utilizing the EDA and HR biomarkers generally available from medical-grade wearable devices such as the Empatica E4, for which a number of datasets are publicly available and specifically labeled for stress. While several commercially available devices are capable of measuring and recording additional sensor data including skin temperature, blood-volume-pulse and data from an accelerometer, these signals were excluded for this study as both accelerometer and temperature data are likely to be influenced by the study setting and experimental environment.\\

\noindent We found the recommendations by Brysbaert \emph{et al.} \cite{Brysbaert2019}, with regards to number of study subjects required to achieve sufficient statistical power (\textgreater 85\%) to correlate with our own findings, and showed model generalization to occur with more test subjects, compared to low generalization for a smaller number of subjects, as typically found in publicly available stress datasets. This finding has important implications for machine learning researchers who would normally expect generalization to occur given a sufficient number of observations, regardless of the number of individual test subjects that contributed to those observations. However, when the problem domain relates to behavioral science or a dependency on physiological biomarker data, a varied dataset consisting of biomarker data recorded from a sufficiently varied number of test subjects is equally important as, or more important than, dataset size and observation count alone.\\

\noindent While prior studies \cite{Nkurikiyeyezu2019} suggested that generalization is only possible with personalized models built for individual subjects, we found that we could reproduce our results on new, unseen data. Stress is not a binary condition, and the approach developed in this study delivers a methodology for predicting both a binary condition, as well as a level of stress experienced via regression, with accuracy levels \textgreater 85\%. A substantial challenge in using public datasets for training machine learning models to predict physiological responses, such as acute stress, are the varied experimental protocols used during recording, combined with large variance in recorded biomarker segment size. This can lead to class imbalance which is hard to overcome using traditional methods such as over and/or under-sampling of observations used for training.\\

\noindent Finally, we demonstrated the use of ensemble methods to combine the individual predictive power of unique machine learning algorithms, into a more powerful and robust stress detection model. By utilizing non-complex feature-engineering techniques with a tumbling window of 25 seconds, we presented a methodology that can deliver near real-time stress prediction from wearable sensor biomarker data. We further utilized random sampling of very small segments of either baseline or stressed periods, to build a balanced training dataset that performed well when using both LOSO validation, or when predicting on a new, unseen dataset (WESAD).\\

\noindent While the methods presented is not a substitute for large, validated and well-labeled datasets, the current lack of public data available for stress-related machine learning research limits the ability for researchers to reproduce prior findings \cite{Hullman2022}, and towards that end the full source code and feature-engineered StressData and SynthesizedStressData datasets used in this study are made available for future research use via the code-sharing platform GitHub at https://github.com/xalentis/Stress. We believe that the results and data presented herein can help advance the emerging field of machine learning for stress measurement from wearable devices.

 \bibliographystyle{elsarticle-num}

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Signal Processing (eess.SP)


Wasserstein Distance Based Domain Adaptation for Object Detection

Abstract: In this paper, we present an adversarial unsupervised domain adaptation framework for object detection. Prior approaches utilize adversarial training based on cross entropy between the source and target domain distributions to learn a shared feature mapping that minimizes the domain gap. Here, we minimize the Wasserstein distance between the two distributions instead of cross entropy or Jensen-Shannon divergence to improve the stability of domain adaptation in high-dimensional feature spaces that are inherent to object detection task. Additionally, we remove the exact consistency constraint of the shared feature mapping between the source and target domains, so that the target feature mapping can be optimized independently, which is necessary in the case of significant domain gap. We empirically show that the proposed framework can mitigate domain shift in different scenarios, and provide improved target domain object detection performance.

\section{Introduction}
\label{sec:intro}
Object detection is a fundamental computer vision task that aids in scene understanding using visual sensors. It involves locating and classifying all instances of different objects such as people, vehicles, etc. in images. With the advent of large datasets, e.g. ImageNet \cite{imagenet}, there is a need for algorithms that can learn the basic concepts that make up a particular object by learning diverse feature representations of different objects. This led to the rise of deep neural networks (DNN) based object detection algorithms \cite{girshick2014,girshick2015,ren2015,he2015,he2017,redmon2016,liu2016} that give a significantly improved detection performance over the previously developed algorithms that use hand-crafted features \cite{dalal2005histograms}. The DNN-based algorithms are able to learn the basic concepts or attributes of different objects by encoding the hidden semantic information among the pixels in the millions of parameters in a single DNN architecture \cite{zeiler2014visualizing}. There are two categories of DNN-based object detection algorithms: one-step and two-step. In two-step algorithms such as Faster-RCNN, Mask-RCNN, and FPN \cite{girshick2014,girshick2015,ren2015,he2015,he2017}, the detector extracts feature maps of images in the first step, and generates coarse region proposals based on the feature maps, and then classifies each proposal into different objects or background in the second step. In one-step algorithms such as YOLO \cite{redmon2016} and SSD \cite{liu2016}, the detector performs feature extraction, bounding box regression and object classification together in a single step, thereby significantly reducing computational time cost. So, one-step algorithms are more suitable for real-time object detection.

During the training process, the back-bone neural network of a detector (feature generator) is first pre-trained on an image classification dataset such as ImageNet \cite{imagenet}, and then, the entire network is trained using object detection datasets such as Pascal-VOC \cite{pascal-voc-2012} or MS-COCO \cite{mscoco} that have images with annotated bounding boxes and labels of objects. For good generalization performance, the DNN-based object detection algorithms need a large number of such annotated images to train the millions of parameters. Even in this case, the detectors will give good performance on such data that exhibits similar feature representations and distributions as the training data. The performance of the algorithms will degrade when they are applied on data that is significantly different from the training data. Henceforth, we will call the large datasets, on which object detectors are trained as the source domain, and the data collected in test conditions as the target domain. Some examples of the different train-test conditions are different weather conditions, different sensors, real and synthetic imagery, etc.

In some cases, the DNN-based algorithms can be "fine-tuned" by using several hundred or thousands of labeled samples from the target domain. However, manually locating and labeling objects in images requires large amount of time and labor, which may not be realistic in practical applications. There are also weakly-supervised and semi-supervised algorithms \cite{tang2016,bilen2016,inoue2018}, in which a small number of annotations or label information from the target domain are available. However, more often than not, it is not possible to obtain labeled data from the target domain. Hence, in this paper, we propose to perform unsupervised domain adaptation that transfers the knowledge learned by the DNN-based object detector from the source domain to the target domain so that the performance of the detector in target domain is improved without the need for labeled training data from the target domain.

\section{Unsupervised Domain Adaptation}
\label{sec:motivation}

Unsupervised domain adaptation (UDA) bridges the domain shift between the source and target domains, where only the source label information is available. Until recently, research in UDA has been focused on the applications of image classification and semantic segmentation. Our work focuses on UDA for object detection. Several algorithms \cite{ganin2014,tzeng2017,tzeng2014,long2015,saito2018cvpr,balaji2019} have been proposed to mitigate the domain shift by minimizing different kinds of distance measures between the source and target feature distributions. Inspired by the generative adversarial networks (GAN) \cite{goodfellow2014}, recently adversarial domain adaptation algorithms have gained popularity. In such algorithms, the feature extractor is considered as the generator, and the source/target domain classifier is considered as the discriminator. The feature extractors are trained to generate similar distributions for both source and target domains to fool the domain classifier. The domain classifier is trained to distinguish the feature distributions from the source and target domains. After the training is completed, the feature extractor should be able to generate feature distributions for source and target domains in such a way that the domain classifier cannot distinguish them, thus achieving domain adaptation. They are considered adversarial because the feature extractor and the domain classifier are treated as adversaries to each other during the training process.

In one such technique, Ganin and Lempitsky \cite{ganin2014} inserted a gradient reversal layer (GRL) between the feature extractor and the domain classifier to flip the sign of the gradient from the domain classifier before back propagating to the feature extractor and performing adversarial training by pushing the feature extractor and the domain classifier in opposite directions. This algorithm was used to implement domain adaptation for image classification task in \cite{ganin2014}. Following the same idea, two recent approaches applied GRL-based domain adaptation for object detection task \cite{chen2018,saito2018cvpr} using Faster-RCNN framework. In both of these, two domain classifiers were used to align feature distributions at two different image levels. However, they hold different opinions on the validity of using the features generated by the region proposal network (RPN) for domain alignment. In \cite{chen2018}, the authors applied an instance-level alignment to the region-of-interest (ROI) features aiming to change the RPN for domain adaptation. In \cite{saito2018cvpr}, the authors argue that the proposals generated by the source network may not localize objects in the target domain accurately, which may degrade the adaptation performance. However, GRL requires the source and target domain feature extractors to share the same neural network which is not necessarily an amiable constraint for domain adaptation. This constraint reduces the number of parameters of the framework but enforces the exact consistency between the source and target mappings, which makes the optimization poorly conditioned \cite{tzeng2017}. Instead of GRL, we follow the adversarial training scheme of GAN, but do not enforce any consistency constraint between the source and target feature extractors.

In adversarial domain adaptation, the metric that is used to measure the distance between the source and target domain feature distributions, is crucial for the performance of the domain discriminator and the final adapted model. The adversarial discriminative domain adaptation \cite{tzeng2017} uses a domain classifier with cross entropy (CE) loss to distinguish the feature vectors from the source domain and the target domain. The minimization of CE between two distributions for domain adaptation is equivalent to the minimization of the Kullback-Leibler (KL) divergence between the two distributions \cite{arjovsky2017}. When the feature distributions are low-dimensional manifolds in a high-dimensional space, it is likely that the distributions might have a negligible intersection, and Kullback-Leibler (KL) divergence is not defined. This is of great concern in object detection models, where the ambient feature space of the backbone network usually has very high dimensionality. To avoid this, one can use Jensen-Shannon (JS) divergence, which is a symmetrized and smoothed version of KL divergence, that measures the sum of KL divergences between the respective distributions and the mean of the two distributions. However, the discriminator may learn very quickly to distinguish between the two domain distributions, and the JS divergence may be locally saturated. This may lead to the vanishing gradient problem \cite{arjovsky2017}. CE, KL and JS divergence are not very stable metrics for domain adaptation in high-dimensional spaces. So, we use Wasserstein distance \cite{arjovsky2017,balaji2019} as the distance metric between the source and target distributions in our discriminator. The Wasserstein distance or Earth-Mover distance between two distributions, $P_s$ and $P_t$ is given by
\begin{equation}
W\left(P_s, P_t\right) = \inf_{\gamma\in\Pi\left(P_s, P_t\right)} E_{(x,y) \thicksim \gamma} \left[\left\|x - y\right\|\right].
\end{equation}
However, as stated in \cite{arjovsky2017}, this infimum (greatest lower bound) is intractable and so, one can use the Kantorovich-Rubinstein duality of the Wasserstein distance, which is given by
\begin{equation}
\label{eq:wd}
W\left(P_s, P_t\right) = \sup_{\|f\|_L \leq 1} E_{x \thicksim P_s} \left[f\left(x\right)\right] - E_{x \thicksim P_t} \left[f\left(x\right)\right],
\end{equation}
where the supremum (least upper bound) is over all 1-Lipschitz functions $f:X \rightarrow \mathbb{R}$. If $f$ is a K-Lipschitz function, then the above equation leads to a scaled version of Wasserstein distance $ KW\left(P_s, P_t\right)$, which is also a valid metric to optimize for domain adaptation. We detail how we use this metric for domain adaptation in Section~\ref{sec:ga}.

In summary, we propose an unsupervised adversarial domain adaptation framework for Faster-RCNN. We chose Faster-RCNN as the base object detection framework because of its superior performance, and the ease of extensibility. We align the source and target domains on both the global feature level and the region proposal feature (local) level using the Wasserstein distance metric. For the global feature level alignment, the source and target networks do not share any layers for the backbone feature generator. For the local feature level alignment, we incorporate the object detection loss to stabilize the adversarial training similar to \cite{odena2017,sankaranarayanan2018learning}. Hence, the source and target domain networks share the same RPN and the classifier.

\section{Proposed Method}
\label{sec:propmeth}
The overview of our proposed domain adaptive Faster-RCNN framework is shown in Fig.1. In Phase 1, we extract the global feature maps of imagery from source and target domains using their individual backbone feature generator networks, and perform the alignment of the feature spaces using the Wasserstein distance measure. Then, in Phase 2, the aligned global feature maps are fed into the RPN to generate region proposals and extract ROI feature maps. We perform the local alignment of ROI proposal features between the source domain and target domain, again using the Wasserstein distance measure. At the same time, the object detection loss from the labeled source domain is used to stabilize and introduce categorical information into the adversarial domain adaptation training. Since the Faster-RCNN is a two-step algorithm, we denote the backbone feature generator networks of the source and target domain as global mappings $M_s^g$ and $M_t^g$, the shared RPN as local mapping $M^l$, and the shared classifier as $C$. Here, we assume that we have the source images $X_s$ and the corresponding labels or annotations $Y_s$ drawn from a distribution $p_s(x,y)$. The target domain images $X_t$ and labels $Y_t$ are drawn from a distribution $p_t(x,y)$. However, for the target domain, we only have the images $X_t$, but not the labels $Y_t$. The objective is to learn a target domain mapping $M^g_t$ that can map the target domain images $X_t$ to global feature space, and adapt the RPN $M^l$ and the classifier $C$, to generate region proposals and assign right labels $Y_t$, thus generating object annotations for the target images $X_t$, using just the knowledge of source domain mappings $M^g_s$, $M^l$ and classifier $C$.

\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{Picture1-3.pdf}
\end{center}
\caption{Proposed framework.}
\label{fig:framework}
\end{figure*}

\subsection{Phase 1 - Global Alignment}
\label{sec:ga}
In Phase 1, first we pass the images from the source domain $X_s$ and from the target domain $X_t$ through the backbone feature extractor networks represented by $M_s^g$ and $M_t^g$ respectively to obtain their global feature representations $M_s^g(X_s)$ and $M_t^g(X_t)$. The goal of global alignment is to minimize the distance between the distributions of the source and target global features. The source domain object detector is pre-trained on the source domain imagery and the source global mapping $M_s^g$ is fixed during global alignment. As mentioned in Section~\ref{sec:motivation}, unlike \cite{chen2018} and \cite{saito2018cvpr}, the feature generators of the source and target networks, $M_s^g$ and $M_t^g$ do not share the parameters in any layer, which intuitively gives more freedom for the target global mapping to explore the parameter landscape and find the optimal parameters that would push $M_t^g(X_t)$ as close as possible to $M_s^g(X_s)$. We initialize the target global mapping $M_t^g$ with that of the fixed source mapping $M_s^g$.

Next, we estimate the Wasserstein distance between the source domain and target domain distributions $M_s^g(X_s)$ and $M_t^g(X_t)$ using Eq.~\ref{eq:wd}. We use another neural network $D^g$, similar to the discriminator in GAN, to encode the 1-Lipschitz continuous functions $f$. So the Wasserstein distance between the two distributions can be written as
\begin{equation}
W(M_s^g,M_t^g) = \sup_{\|D^g\|_L \leq 1} \mathbb{E}_{X_s \thicksim p_s}(D^g(M_s^g(X_s))) - \mathbb{E}_{X_t \thicksim p_t}(D^g(M_t^g(X_t))).
\end{equation}
We can optimize for the parameters of the network $D^g$, and thus the optimal 1-Lipschitz function, by back-propagating the gradient of the right-hand side term in the above equation. The encoding $D^g$ is called as \textit{critic} in the context of Wasserstein distance. In \cite{arjovsky2017}, the parameters of $D^g$ are clipped after every update to satisfy the Lipschitz continuity requirement. Instead, here we use spectral normalization \cite{miyato2018} for each layer in the global critic network to enforce this requirement. Also, to mitigate the Wasserstein distance estimation problem that arises due to small training batch size used in object detection, we use a patch-based critic network similar to those used in \cite{chen2018} and \cite{johnson2016} to estimate the Wasserstein distance. In particular, we treat each activation of the last convolution layer of the critic network as a feature map of one patch of an image.

Finally, after the Wasserstein distance is estimated, we can minimize this distance measure by optimizing for the parameters of the target global mapping or feature extractor $M_t^g$. The two-step iterative optimization for global alignment can be formulated as following:
\begin{equation}
\max_{D^g} W(M_s^g,M_t^g) = \frac{1}{n} \sum_{i} D^g(M_s^g(X_s^i)) - \frac{1}{n} \sum_{i} D^g(M_t^g(X_t^i)),
\end{equation}
\begin{equation}
\min_{M_t^g} W(M_s^g,M_t^g) = - \frac{1}{n} \sum_{i} D^g(M_t^g(X_t^i)),
\end{equation}
where $n$ is the batch size of one mini-batch. Since the source global mapping $M_s^g$ is fixed, we update only the target global mapping $M_t^g$ in the second step.

\subsection{Phase 2 - Local Alignment}
\label{sec:la}
In global alignment, we optimize the backbone feature extractor of target network to align the global feature distributions of the source and target domains. In Phase 2, we further align the distributions of local ROI features in the two domains. We also would like to use the source domain object detection loss to stabilize the adversarial training process. So, for the local alignment, we constrain the source and target networks to share the same RPN and classifier as shown in Fig.~\ref{fig:framework}. We use Wasserstein distance to measure the domain gap at the local level as well. So, we use a neural network $D^l$, similar to $D^g$ in global alignment step, to encode the \textit{critic} that estimates the distance between the source and target domain distributions at the local level. The traditional adversarial technique that we used to update the feature generator at the global level cannot be used here as the source and target networks share the RPN. So, we use GRL \cite{ganin2014} to update the local critic network and the RPN (feature generator) at the same time. Here, we denote global features of an image $i$ from the backbone networks of source and target networks as $f_s^i$ and $f_t^i$. Then loss functions for local alignment can be formulated as following:
\begin{equation}
\max_{D^l} W(M^l(f_s),M^l(f_t)) = \frac{1}{nm} \sum_{i,j} D^l(M_{j}^{l}(f_s^i)) - \frac{1}{nm} \sum_{i,j} D^l(M_{j}^l(f_t^i)),
\end{equation}
\begin{equation}
\min_{M^l} W((M^l(f_s),M^l(f_t)) = \frac{1}{nm} \sum_{i,j} D^l(M_{j}^{l}(f_s^i)) - \frac{1}{nm} \sum_{i,j} D^l(M_{j}^l(f_t^i)),
\end{equation}
where $n$ is the batch size (number of images) of one mini-batch and $m$ is the number of proposals generated by the RPN for each image. One can observe that the loss functions of the local critic network $D^l$ and the RPN (feature generator) are exact opposite of each other. In GRL, the gradient that is used to update the local discriminator network is reversed in sign, and then used to update the RPN. We also iteratively update the classifier along with the RPN to minimize the source domain object detection loss $L_{det}$ while performing the local alignment.

\section{Experimental Results}
\label{sec:results}
In this section, we evaluate the adaptation performance of the proposed framework on two domain-shift scenarios: different weather conditions (Cityscapes \cite{cordts2016} to Foggy Cityscapes \cite{sakaridis2018}), and between synthetic and real imagery (SIM10k \cite{johnson2016} to Cityscapes). We compare the performance of our proposed framework with that of four algorithms: Faster-RCNN, domain adaptive Faster-RCNN (DA-Faster) \cite{chen2018}, BDC-Faster and the strong and weak alignment Faster-RCNN (SW-Faster) \cite{saito2018cvpr}. The Faster-RCNN is the model trained on the source data and evaluated on the target data without any domain adaptation. BDC-Faster is the Faster-RCNN model with only a global domain classifier that is optimized using CE loss, which was originally reported in \cite{saito2018cvpr}. We included this particular model in our experiments because the comparison between our model and BDC-Faster shows the advantages of using Wasserstein distance over the CE loss for domain adaptation. Since our task focuses on unsupervised domain adaptation in feature space, for a fair comparison, we do not compare our model to the one of the models proposed in \cite{saito2018cvpr}, where additional images are generated by using cycle GAN \cite{CycleGAN2017}, and the additional images are then used for domain adaptation. For all the experiments, we use the Faster-RCNN as the base detector and the VGG16 \cite{simonyan2014} as the backbone feature generator network. The algorithm, implementation details including the global and local critic network architectures are presented in the supplementary material.

\subsection{Adaptation across different weather conditions}
\label{sec:city2foggy}
In this experiment, we evaluate the adaptation ability of our model under different weather conditions. We use the Cityscapes as the source domain and the Foggy Cityscapes as the target domain. The Cityscapes dataset includes the images of street scenes of European cities captured in clear weather. The original size of each image is 2048 $\times$ 1024. The whole dataset includes 2,975 training images and 500 test images which cover 8 object categories: \textit{person, rider, car, truck, bus, train, motorcycle,} and \textit{bicycle}. The Foggy Cityscapes dataset is rendered from Cityscapes by adding the fog noise. The images in the target domain are generated from the source domain, which implies that the source and target domains share the same semantic information. Since the original Cityscapes dataset was created for semantic segmentation, and does not have any bounding-box annotation, similar to \cite{chen2018}, we manually create the ground truth bounding-box annotations for each object by taking the tightest rectangle of its instance mask.

\textbf{Results:} As shown in Table~\ref{tab:city2foggy}, the proposed model with just global alignment can achieve an mAP of 32.0\%, which is much better than the performance of the baseline model without any adaptation. It is also better than the BDC-Faster which uses a global domain classifier with the CE loss. This reinforces our argument that the Wasserstein distance is not only more stable but also more effective compared to the CE loss for domain adaptation. Our model is also better than the DA-Faster and the SW-Faster with only global alignment and regularization term. This confirms our argument that the source and target feature extractor networks sharing the same parameters may limit the adaptation performance. From the results, we can also see that local alignment does not contribute as much as the global alignment to the final performance of the model. This is expected in this experiment because Cityscapes and Foggy Cityscapes share exactly the same bounding-box annotations and categorical information. So, conditioned on the aligned global feature distributions, the distribution of bounding-box annotations and labels for these two datasets would be very similar. The local alignment, which is meant to align the distributions of bounding box locations and labels between the two domains would not be very helpful.

\begingroup
\begin{table}
\begin{center}
\setlength{\tabcolsep}{2.5pt}
\begin{tabular}{|l|cccc|ccccccccc|}
\hline
Method & G &I &L &C &bus &bicycle &car &mcycle &person &rider &train &truck &mAP\\
\hline\hline
Faster-RCNN & & & & &24.2 &26.4 &31.6 &14.0 &24.0 &32.0 &9.1 &11.7 &21.6\\
BDC-Faster & \checkmark & & & &29.2 &28.9 &42.4 &22.6 &26.4 &37.2 &12.3 &21.2 &27.5\\
DA-Faster & \checkmark &\checkmark & & &25.0 &31.0 &40.5 &22.1 &35.3 &20.2 &20.0 &27.1 &27.6\\
SW-Faster & \checkmark & & &\checkmark &38.0 &31.2 &41.8 &20.7 &26.6 &37.6 &19.7 &20.5 &29.5\\
SW-Faster & \checkmark & &\checkmark &\checkmark &36.2 &35.3 &43.5 &30.0 &29.9 &42.3 &32.6 &24.5 &34.3\\
\hline
Ours & \checkmark & & & &41.6 &32.3 &44.0 &24.4 &30.6 &40.5 &21.5 &21.4 &32.0\\
Ours & \checkmark &\checkmark & & &39.9 &34.4 &44.2 &25.4 &30.2 &42.0 &26.5 &22.2 &33.1\\
\hline
\end{tabular}
\end{center}
\caption{Results on adaptation from Cityscapes to Foggy Cityscapes(\%). G, I, L, C represent the global alignment, local alignment on ROI, local alignment on the intermediate layer of VGG, and context-vector based regularization respectively.}
\label{tab:city2foggy}
\end{table}
\endgroup

\subsection{Adaptation from synthetic to real imagery}
\label{sec:sim2city}
For computer vision tasks such as object detection, manually labeling or annotating real data costs a large amount of time and labor, which is not practical. However, for the synthetic data, it is easy and convenient to generate the ground truth labels and annotations for a large number of objects while generating the imagery. The natural goal here is to transfer the object detection model trained on the synthetic data to the real data. So we use the SIM10k dataset as the source domain and the Cityscapes dataset as the target domain. Following \cite{chen2018}, we evaluate the model only on the AP of \textit{car}. SIM10k is a dataset of the synthetic driving scenes rendered from the game \textit{Grand Theft Auto (GTAV)} which includes 10,000 training images. The model is evaluated on the test set of the Cityscapes (500 images).

\textbf{Results:} As shown in Table~\ref{tab:sim2city}, our model with the global alignment can achieve an AP of 39.8\% which is very close to the state of art result. Moreover, we achieve this result with only a few epochs of global domain alignment training, which is very efficient time-wise. These results confirm that the proposed model can reduce the domain shift between the synthetic data and the real data. In Phase 2, we fix the VGG16 backbone of the target network and perform the local alignment on the RPN. According to the experimental results, the local alignment can further improve the performance since the bounding box distributions of SIM10k and Cityscapes are different.

\begin{table}
\begin{center}
\begin{tabular}{|l|cccc|c|}
\hline
Method & G &I &L &C &AP on car\\
\hline\hline
Faster-RCNN & & & & &34.5 \\
BDC-Faster &\checkmark & & & &31.8 \\
DA-Faster &\checkmark &\checkmark & & &38.9 \\
SW-Faster & \checkmark & & &\checkmark &36.4\\
SW-Faster & \checkmark & &\checkmark &\checkmark &40.1\\
\hline
Ours &\checkmark & & & &39.8\\
Ours &\checkmark &\checkmark & & &40.6\\
\hline
\end{tabular}
\end{center}
\caption{Results on adaptation from SIM10k to Cityscapes(\%)}
\label{tab:sim2city}
\end{table}

\section{Discussion}
\label{sec:disc}
Figs.~\ref{fig:sampleimg1} and \ref{fig:sampleimg2} illustrate the improvement of object detection results after domain adaptation between Cityscapes-Foggy Cityscapes and between SIM10k-Cityscapes respectively. The images in the top row show the bounding boxes of detected objects before domain adaptation and the corresponding images in the bottom row show the bounding boxes of detected objects after domain adaptation. One can see that the objects that were missed before domain adaptation due to the domain gap, are detected after domain alignment in both cases. The locations of the detected objects are also more accurate after adaptation. Moreover, in the case of synthetic to real imagery (SIM10k-Cityscapes), one can also observe that the number of false detections before domain alignment has been significantly reduced after the adaptation.

From the results presented in Tables~\ref{tab:city2foggy} and \ref{tab:sim2city}, one can see that most of the improvement in object detection performance is obtained after the global alignment step (Phase 1) itself. The reasons for this are the two main contributions of this paper - using Wasserstein distance to measure the domain gap and optimizing for the target feature generator independent of the source network - as explained in Section~\ref{sec:results}. However, the local alignment does not provide significant improvement in the detection performance. One of the reasons is the small batch size used for Wasserstein distance estimation (learning the critic network) due to constraints on the available computational resources (GPU memory). Another reason is that currently, there is no regularization between global and local alignment steps. The local alignment results may be further improved by using a larger batch size for Wasserstein distance estimation, or by simply improving the sample efficiency of Wasserstein distance estimation. Adding a regularization term to unify the training of the global and local domain critic networks may also improve the domain adaptation results.

\begin{figure}[h]
\begin{center}
\includegraphics[width=\textwidth]{r1.pdf}
\end{center}
\caption{Results from Cityscapes to Foggy cityscapes before and after domain adaptation}
\label{fig:sampleimg1}
\end{figure}

\begin{figure}[h]
\begin{center}
\includegraphics[width=\textwidth]{r2.pdf}
\end{center}
\caption{Results from Sim10k to Cityscapes before and after domain adaptation}
\label{fig:sampleimg2}
\end{figure}

\section{Conclusion}
\label{sec:conc}
We have proposed an unsupervised domain adaptive framework based on Faster-RCNN for object detection. We perform the alignment of the source and target domain distributions at two levels - global and local. We use Wasserstein distance metric to estimate the distance between the distributions of the two domains, which mitigates the gradient vanishing problem in high-dimensional spaces of object detection task. Additionally at the global level, different from previous approaches, the source and target networks in our model do not share any feature extraction layers, which provides more flexibility for the target network to learn the required feature mapping that minimizes domain shift even in the case of significant domain gap. During the local alignment, we also iteratively minimize the object detection loss along with domain adaptation loss to stabilize the adversarial training. We have validated our framework by testing it on datasets exhibiting different domain shift such as different weather conditions, synthetic to real imagery, and comparing it to other unsupervised domain adaptation techniques in terms of object detection performance.

\bibliographystyle{unsrt}

Computer Vision and Pattern Recognition (cs.CV)