Nonverbal Robot Feedback for Human Teachers

Abstract: Robots can learn preferences from human demonstrations, but their success depends on how informative these demonstrations are. Being informative is unfortunately very challenging, because during teaching, people typically get no transparency into what the robot already knows or has learned so far. In contrast, human students naturally provide a wealth of nonverbal feedback that reveals their level of understanding and engagement. In this work, we study how a robot can similarly provide feedback that is minimally disruptive, yet gives human teachers a better mental model of the robot learner, and thus enables them to teach more effectively. Our idea is that at any point, the robot can indicate what it thinks the correct next action is, shedding light on its current estimate of the human's preferences. We analyze how useful this feedback is, both in theory and with two user studies---one with a virtual character that tests the feedback itself, and one with a PR2 robot that uses gaze as the feedback mechanism. We find that feedback can be useful for improving both the quality of teaching and teachers' understanding of the robot's capability.

\section{Nonverbal Feedback}

\subsection{Assumptions on robot learning algorithm}
We focus on robot learners that infer \emph{reward functions} from demonstrations via IRL~\cite{Ng_2000}. The benefit of IRL is that this underlying reward function typically generalizes better across tasks, compared to directly learning a mapping from states to actions, as in behavior cloning~\cite{Pomerleau_1991,Ross_2011}. In addition, humans are naturally inclined to infer the objectives of other agents~\cite{Baker_2009,Jara_2016}, and thus might expect robot learners to do the same. We parameterize the reward function $\mathcal{R}_\theta$ as a linear combination of reward features $\phi(\cdot)$ with weights $\theta$,
   $ \mathcal{R}_\theta(s, a) = \theta^\top \phi(s, a) $,
where $s$ is the state and $a$ is the action. There is no limitation on what these reward features can be, so this assumption is not restrictive~\cite{Abbeel_2004}.

The robot maintains a belief $b(\theta)$ over reward function parameters. We model humans as providing demonstrations $(s,a)$ that are approximately optimal, according to a Boltzmann distribution~\cite{Ortega_2016}. This induces the following observation model that links human actions to the reward parameters,
\begin{equation}
    p(a|s,\theta)\propto e^{\beta Q^*_\theta(s, a)}\, ,
    \label{eqn:softmax}
\end{equation}
where $\beta$ specifies the level of suboptimality and $Q^*_\theta(s,a)$ is the action-value function: the discounted sum of future rewards, after taking action $a$ in state $s$ and acting optimally thereafter with respect to $\mathcal{R}_\theta$.
As in Bayesian IRL~\cite{Ramachandran_2007}, the robot uses this observation model to update its belief over $\theta$:
\begin{equation}
    b'(\theta) \propto p(a | s, \theta) \, b(\theta) \, .
    \label{eqn:bayes_irl}
\end{equation}

\subsection{Generating feedback}
We propose and investigate a form of feedback where at every step, the robot communicates what it believes the optimal action is. Intuitively, this should help human teachers understand what the robot knows and does not know as they proceed through teaching the task, enabling them to adapt what they teach and recognize when they can stop teaching. In addition, this form of feedback is minimally disruptive and allows teachers to maintain control of teaching, in contrast to tests of comprehension and active learning, respectively. We leave open the question of how this feedback should be communicated; we experiment with movement of a virtual avatar and gaze on a real robot.

\noindent\textbf{Feedback target.}
The robot first uses its current belief over reward parameters to predict the human's next action, using the observation model from \eref{eqn:softmax}:
\begin{equation}
    \hat{a}=\underset{a}{\arg\max} \int_{\theta}p(a|s,\theta) \, b(\theta) \, d\theta \, .
\end{equation}
Based on this prediction, it determines the most likely next state, i.e., where the human will go next:
\begin{equation}
    \hat{s}=\underset{s'}{\arg\max} \, p(s'|s,\hat{a}) \, .
\end{equation}
In a deterministic environment, this is $\hat{s}=f(s,\hat{a})$, with $f$ being the dynamics.
The robot then communicates this prediction to the human teacher, for instance by gazing in the direction of $\hat{s}$.

\noindent\textbf{Feedback speed.}
If the robot can control the \emph{speed} of feedback (e.g., the speed at which it moves its end effector or gazes), it can use this to convey how \emph{confident} it is in its prediction; slower speeds indicate lower confidence~\cite{Hough_2017,Zhou_2017}. Given a maximum speed of $v_{max}$, we thus set feedback speed as
\begin{equation}
    v = v_{max} \; p(\hat{s}|s,\hat{a})\int_{\theta}p(\hat{a}|s,\theta) \, b(\theta) \, d\theta \, .
\end{equation}

\noindent\textbf{Grounding in our experimental domain. }
To make this more concrete, consider the domain of decluttering: objects need to be sorted appropriately into bins, and only the human knows the correct sorting mechanism. States are locations of objects and bins,
and actions place objects into bins. Every action $a$ corresponds to a particular bin $B_a$, and we assume the environment is deterministic; taking action $a$ in state $s$ means putting the object from location $s$ into $B_a$. $\phi(s,a)$ is a feature vector that consists of descriptors of object-bin match, e.g., distance, color match, shape match, etc.

Each human action teaches the robot about the relationship between the features and the reward, i.e., about the correct weights $\theta^*$. Once the human has selected an object, the robot predicts, according to its current belief, which action $\hat{a}$ the human will take next, and indicates $B_{\hat{a}}$, for instance via gaze. Each $\theta$ assigns different probabilities to different bins, and the robot combines all this information to compute a confidence in its estimate of bin $B_{\hat{a}}$, which it uses to adjust its feedback speed.

\section{Analysis with Ideal Users: Why Do We Expect Feedback to Help?}
\label{sec:analysis_theory}
\subsection{Model of human teachers that incorporates feedback}
To provide a theoretical justification for feedback, we construct potential models of human teaching that incorporate feedback, and analyze how feedback improves teachers' decisions. These models are based on algorithmic teaching~\cite{Zhu_2015,Goldman_1995,Balbach_2009}, and thus have two components: tracking the learner's state (e.g., what does the learner know), and selecting informative demonstrations based on this.

\noindent\textbf{Tracking learner state.}
We assume our teacher knows how the learner performs an update after every example via \eref{eqn:bayes_irl}. However, the teacher is still missing information about the learner, in particular the learner's \emph{prior} belief $b_0$, and the feature space $\Theta$ that the learner assumes---which may differ from the teacher's. A sophisticated teacher is aware of this \emph{uncertainty}. At every step, she has a belief over what the robot's belief might be, that takes into account the robot's learning updates, its feedback, and the uncertainty over which prior and feature space the robot is using.

After the first time step, this teacher computes the probability of the robot's new belief given the robot's feedback target $x_0$ and speed $v_0$ for the object she picked up, $s_0$, and the action she took, $a_0$:
\begin{equation}
    p(b_1|s_0,a_0,x_0,v_0)
    =\int_{b_0,\Theta} \underbrace{p(\Theta)p(b_0|\Theta)}_{\text{teacher priors}}\underbrace{p(x_0,v_0|b_0,s_0)}_{\text{feedback generation \eref{eqn:gaze_g_prob}\eref{eqn:gaze_v_prob}}}
    \underbrace{p(b_1|b_0,s_0,a_0,\Theta)}_{\text{robot learning \eref{eqn:bayes_irl}}} d\Theta \, d b_0 \, .
\end{equation}
Since we assume the teacher knows how the learner performs belief updates, for the robot learning distribution, $p(b_1|b_0,s_0,a_0,\Theta)$, the teacher places all probability mass on the correct new belief according to \eref{eqn:bayes_irl}. For the feedback generation distribution, $p(x_0,v_0|b_0,s_0)$, we use how the robot actually generates feedback. From $x_0$, the teacher recovers $\hat{a}_0=f^{-1}(s_0,x_0)$, the action that the robot predicted, based on inverse dynamics $f^{-1}$.
From there, the teacher computes the probability of that action under the belief $b_0$:
\begin{equation}
    p(x_0|b_0,s_0)=p(\hat{a}_0|b_0,s_0)=\int_{\theta}p(\hat{a}_0|s_0,\theta) \, b_0(\theta) \, d\theta \, .
    \label{eqn:gaze_g_prob}
\end{equation}
For $v_0$, the teacher assigns probability based on a Gaussian around her predicted speed for $\hat{a}_0$:
\begin{equation}
      p(v_0|b_0,s_0,x_0) =p(v_0|b_0,s_0,\hat{a}_0) =\mathcal{N}(v_0 | v_{max} \; p(\hat{a}_0|b_0,s_0), v_{var}) \, .
    \label{eqn:gaze_v_prob}
\end{equation}
The teacher keeps incorporating such updates based on robot feedback for each new demonstration. In practice, we approximate this update by initializing the teacher's priors $p(\Theta)$ and $p(b_0|\Theta)$ with sampled learner prior beliefs $b_0$ and possible feature spaces $\Theta$, and iteratively updating this set of samples with the evidence, analogous to running a particle filter~\cite{Gordon_1993} (see \sref{sec:feedback_impact} for details).

We also consider a less sophisticated teacher model, who does not account for the uncertainty over the robot's prior and feature space. Rather than maintaining a belief over beliefs, this \emph{iterative} teacher starts with a uniform belief over what the robot's reward estimate might be, and updates it at every step. First, the teacher updates based on the feedback:
\begin{equation}
    b_1'(\theta|s_0,x_0,v_0)\propto p(x_0,v_0|\theta,s_0) \, b_0(\theta) \, ,
\end{equation}
with the feedback probabilities computed as in \eref{eqn:gaze_g_prob} and \eref{eqn:gaze_v_prob}, but for a single $\theta$.
Next, the teacher shows $a_0$ and accounts for the fact that the robot will learn:
\begin{equation}
    b_1(\theta|s_0,a_0,x_0,v_0)\propto p(a_0|\theta,s_0) \, b_1'(\theta) \, .
\end{equation}
Motivated by the ``win stay, lose shift'' strategy in cognitive psychology~\cite{Nowak_1993}, iterative teachers only update if the feedback disagrees with their current belief; see Appendix \ref{sec:iterative_teacher} for details.

\noindent\textbf{Selecting examples based on current learner state.}
At every step, our teacher uses the tracked learner state to select the most informative example to give next. This is the example that leads to the largest increase in learner performance. The iterative teacher teaches:
\begin{equation}
(s_t^*,a_t^*) = \underset{s_t,a_t}{\arg\max} \, g(b_{t+1}(\theta|s_t,a_t)),
\end{equation}
where $g(\cdot)$ computes the learner's expected performance on the task (if its belief were $b_{t+1}$) and $b_{t+1}(\theta|s_t,a_t) \propto p(a_t|\theta,s_t) \, b_t(\theta)$, according to the learner's learning update via \eref{eqn:bayes_irl}.

The uncertainty-aware teacher does the same, but in expectation over the robot's current belief and the feature space the robot uses:
\begin{equation}
    (s_t^*,a_t^*)= \underset{s_t,a_t}{\arg\max} \, \mathbb{E}_{b_{t},\Theta} \, [ g(b_{t+1}(\theta|s_t,a_t)) ]
\end{equation}

\subsection{Impact of feedback}
\label{sec:feedback_impact}
We now investigate how much feedback helps for our models of human teaching. If the teacher has a perfect model of the learner, then feedback is not necessary~\cite{Balbach_2009}. So, we focus on the kinds of mismatches that might occur---the learner might start off with a bad prior, or not know the correct feature space. For these simulated experiments, we randomly sampled $N$ bins and $O$ objects; each bin and object is represented by a $d$-dimensional feature vector. We set $N = 3$, $O = 50$, and $d = 3$.

\noindent\textbf{Manipulated variables.}
We manipulate the teacher type: \emph{iterative}, \emph{uncertainty-aware}, and \emph{random}. The \emph{random} teacher is a baseline, that chooses random teaching examples rather than maximally informative ones. The learners vary along two factors: whether they provide feedback or not, and whether the teacher-learner mismatch is the prior (\emph{prior mismatch}), the learner missing a feature (\emph{feature mismatch}), or the learner learning a separate $\theta$ for each bin (\emph{reward generalization mismatch}). For details, please refer to Appendix \ref{sec:supp_theoryimpl}. Note that in the latter two mismatch conditions, the learner cannot learn the task, because it is reasoning over the incorrect space of $\theta$s.

\noindent\textbf{Learner performance.}
We measure learning performance with expected soft classification error:
\begin{equation}
    g(b_t(\theta)) = \mathbb{E}_{\theta \sim b_t(\theta)} \left[ \frac{1}{|S|} \sum_{s \in S} p(a^*(s) | s, \theta) \right] \, ,
    \label{eqn:learner_perf}
\end{equation}
where $a^*(s) = \text{argmax}_a \mathcal{R}_{\theta^*}(s, a)$ denotes the correct bin to place the object in.

\begin{figure*}
\centering
\begin{minipage}{.26\linewidth}
  \centering
  \includegraphics[width=1.0\linewidth]{figs/sim_results.pdf}
  \captionof{figure}{For mismatched priors, algorithmic teaching with feedback achieves higher learner performance \eqref{eqn:learner_perf}. Standard error bars are computed from $100$ trials.}
  \label{fig:sim_teaching_effectiveness}
\end{minipage}\enskip%
\begin{minipage}{.72\linewidth}
  \centering
  \includegraphics[width=1.0\linewidth]{figs/sim_results_tracking.pdf}
  \captionof{figure}{With feedback, iterative and uncertainty-aware teachers (orange) can track the learner's state accurately for most mismatch conditions. In contrast, teachers significantly overestimate learner performance when there is no feedback. Standard error bars are computed from 100 trials with ten random teaching sequences each. The y-axis is the teacher's estimate of learner performance minus true performance \eqref{eqn:learner_perf}; the x-axis is the number of teaching examples.}
  \label{fig:sim_teaching_mentalmodel}
\end{minipage}
\vspace{-1.5em}
\end{figure*}

\noindent\textbf{Analysis.}
As one might expect, we found that the effect of feedback depends on the teacher-learner mismatch and teacher type. We found two main positive effects: feedback allows the teacher to 1) select more effective teaching examples, and 2) track the capabilities of the learner more accurately.

Feedback leads to more effective teaching for only the \emph{prior mismatch} condition (\figref{fig:sim_teaching_effectiveness}).
Across all mismatch conditions, uncertainty-aware teachers very quickly narrow down which learner model is correct, and can nearly-perfectly estimate the learner's performance (\figref{fig:sim_teaching_mentalmodel}).\footnote{This makes the potentially strong assumption that the uncertainty-aware teacher's possible learner models contain the true one. This is reasonable though, in the case of \emph{feature mismatch} for interpretable reward features, because then the teacher could just be reasoning over the power set of features.} Although the learner cannot learn the task in the \emph{features mismatch} and \emph{reward generalization mismatch} conditions, there is still a benefit to the teacher estimating the learner's performance correctly: they have a better idea of when to stop teaching (i.e., when their estimated learner performance stops increasing), and they have a more accurate estimate of the learner's performance at the end of training. Uncertainty-aware teachers, because they maintain a belief over what the learner's feature space might be, can also narrow down which feature(s) the learner is missing, which gives them a better idea of the learner's capabilities for future tasks.

\noindent\textbf{Summary.}
Our analysis on algorithmic-teaching-based human models suggests that feedback helps teachers. When the learner can learn the task, feedback makes it easier for teachers to select effective teaching examples. Feedback also improves the teacher's estimate of the learner's capabilities, so when the learner cannot learn the task, feedback enables users to realize this.

\section{Analysis in Practice: Feedback Helps}
\begin{wrapfigure}[17]{R}{0.35\textwidth}
\vspace{-3.5em}
\centering
\frame{\includegraphics[width=.28\columnwidth]{figs/digital1_cropped.pdf}}\\
\vspace{0.4em}
\frame{\includegraphics[width=.28\columnwidth]{figs/digital2_cropped.pdf}}
\caption{AMT interface. Correct object-bin pairings are shown (top, bin 3), and learners' feedback explicitly signals their prediction (bottom).}
\label{fig:amt_task}
\end{wrapfigure}
We next investigate whether real human teachers also benefit from feedback. In this section, we test the benefits of the feedback itself with virtual learners that explicitly predict the teacher's action with varying confidence. In the next section, we test whether these benefits still exist when the feedback is realized through \emph{gaze} on a real robot.

\subsection{Design}
\label{sec:study_design}
We recruited 87 participants (ages 22-71, 41\% female) on Amazon Mechanical Turk (AMT) to teach a decluttering task to an agent. Participants demonstrated correct object-bin pairings to the agent; we instructed them to give as few teaching examples as possible while still ensuring that the agent learns the correct sorting rules. The decluttering setup (\figref{fig:amt_task}) consisted of three numbered bins surrounded by two rings of objects, with the following sorting rules:
1) an object lying on an inner ring belongs to the closest bin, and 2) an object lying on an outer ring belongs to the bin with the same shape as it.

Users clicked on the object they wanted to demonstrate, and it would be moved into the correct bin. When feedback was activated, the learner avatar would move toward its best guess for the corresponding bin and declare that it thought the object belonged there (\figref{fig:amt_task}, bottom). Then the object would be moved to its bin, and the learner would acknowledge whether they were mistaken.

\noindent\textbf{Manipulated variables.} We manipulated the \emph{feedback}: no feedback, full feedback that indicates the learner's best-guess bin with variable speed corresponding to its confidence level, or partial feedback that indicates the best-guess bin with a fixed speed. We also manipulated \emph{learner prior belief}: a uniform prior the teacher's feature space $\Theta$, a biased prior over $\Theta$ that heavily prefers sorting objects into their closest bin, or a uniform prior over weights in a mismatched feature space different from the teacher's (please refer to Appendix \ref{sec:supp_practiceimpl} for details). We did not conduct the 3 by 3 factorial---instead, we analyze the impact of prior condition separately from the impact of partial feedback, since we did not hypothesize any interactions there. We thus did a 2 by 3 study with full feedback and no feedback, and only tested the partial feedback on the prior belief condition.

\noindent\textbf{Dependent measures.}
\noindent\textbf{\emph{Objective---}} We measure the \emph{learner performance} \eqref{eqn:learner_perf} throughout teaching. We record the \emph{teaching sequence length} and measure proxy metrics for teaching strategy, e.g., the \emph{proportion of objects demonstrated from each ring}.
\textbf{\emph{Subjective---}} We also ask open-ended and Likert scale questions about the confidence of the teacher in the learner's understanding, their ability to track the learner's progress, the helpfulness of feedback, and the effects of feedback on teaching.
\textbf{\emph{Combined---}} In addition, we compute a \emph{mental model discrepancy} metric~\cite{Cakmak_2010b}: the human's Likert estimate of the robot's understanding of a sorting rule (scaled to the 0-1 range), minus the learner performance for objects classified by that rule.

\noindent\textbf{Hypotheses.} We hypothesize that for all learner prior conditions, \textbf{H1}: Feedback will allow the teacher to better track the learner's progress and understanding, \textbf{H2}: The teacher will adapt their strategy according to their improved estimate of learner capabilities, and \textbf{H3}: This adjustment, enabled by feedback, will ultimately result in increased learning performance.

\noindent\textbf{Subject allocation.}
Participants were randomly allocated between-subjects across the learner prior conditions. Full versus no feedback was within-subjects, because people may vary significantly in terms of their teaching strategies and capabilities, and we wanted to be able to make direct comparisons. We fixed the ordering of conditions: no feedback, followed by feedback. This is because the other ordering would have been too biasing---seeing the feedback first enables participants to figure out how the robot learns, and they can use that information in the no feedback condition if that happens after. Whereas with our ordering, experiencing the no feedback condition first does not inform participants' teaching strategy, because they do not get any signal on what the robot has learned. To mitigate the risk of learning effects, we had participants first complete a practice teaching task and pass a quiz on the sorting rules.

\subsection{Results}
\noindent\textbf{H1: Feedback improves tracking of robot understanding.} In learner prior conditions with high mental model discrepancy at the end of teaching without feedback, adding feedback reduces this discrepancy (\figref{fig:mental_inner}).
A 2 by 3 factorial repeated-measures ANOVA with feedback and prior belief as factors showed a significant interaction effect on both outer ($F(2,84)=19.78, p<.0001$) and inner ($F(2,84)=4.51, p=0.013)$) ring mental model discrepancy. The post-hoc Tukey HSD found that feedback significantly decreases discrepancy in the uniform prior condition for both the outer ($p=0.001$) and inner ($p=0.002$) ring rules, and in the missing feature condition for outer ($p<0.0001$). Subjective Likert responses also support this hypothesis; see Appendix \ref{sec:supp_likert} for details.

\begin{wrapfigure}[18]{R}{0.5\textwidth}
\vspace{-1.5em}
\centering
\includegraphics[width=0.49\textwidth]{figs/mental_model.pdf}
\caption{Mental model discrepancy (defined in \sref{sec:study_design}) for the inner and outer ring sorting rules. \textbf{\emph{Lower magnitude is better; positive values mean that the teacher overestimates the learner capability, and negative values mean they underestimate.}} (**) indicates $p<0.005$ and (***) $p<0.0005$.}
\label{fig:mental_inner}
\end{wrapfigure}
\noindent\textbf{H2: Feedback changes teaching strategy.}
In the biased prior and the missing feature conditions, feedback leads teachers to demonstrate more outer-ring objects (\figref{fig:prop_objects}); an ANOVA found a significant main effect for feedback on this measure ($F(1,85)=31.49, p<.0001$). This is reasonable, since in both conditions, the learner has trouble learning the outer ring rule. An ANOVA on the teaching sequence length found an interaction effect between the feedback and learner mismatch ($F(2, 63)=6.613, p=.0025$). A post-hoc Tukey HSD found that the number of teaching examples is significantly higher for the missing feature case when people receive feedback---they persist in teaching the learner the outer ring rule, until they finally realize it is impossible.

\begin{figure*}[!b]
\vspace{-2em}
\centering
\includegraphics[width=1.0\textwidth]{figs/learner_accuracy.pdf}
\caption{Learner performance \eqref{eqn:learner_perf} versus teaching iteration in the online study (left three) and in the in-person study with a biased prior (right-most). The x-axis is the number of teaching examples.}
\label{fig:learner_accuracy}
\end{figure*}

\noindent\textbf{H3: Feedback improves learning performance.}
Feedback during teaching leads to improved learner performance in the biased prior case (\figref{fig:learner_accuracy}), in line with our results on ideal teacher models (\sref{sec:analysis_theory}).
We ran a 2 by 3 factorial repeated-measures ANOVA on final learner performance with feedback and prior belief as factors, and number of examples as a covariate. We found a significant interaction effect between feedback and prior belief ($F(2,1925)=29.74$, $p<.0001$). A post-hoc Tukey HSD found that feedback significantly improves performance for the biased prior case, significantly decreases it for missing feature, and makes no difference for uniform prior. Since the learner cannot learn the task at all in the missing features condition, it is more important that the teacher recognizes the robot's learning limitations---which feedback does help with.

We found no significant differences between full versus partial feedback in terms how effective teachers were (i.e., final learning performance), the proportion of demonstrated objects from the outer ring, and the number of teaching examples shown. However, learners with partial feedback do tend to overestimate learner performance for the outer ring rule (\figref{fig:mental_inner}), possibly because they assume the learner is consistently confident in its prediction.

\noindent\textbf{Summary.} Overall, our results with human teachers were remarkably similar to what we saw with ideal teacher models: 1) feedback improved learning performance in the biased prior condition; and 2) although feedback did not improve performance in the uniform prior or missing feature conditions, it reduced discrepancy between the teacher's model of the learner and the actual learner.

\section{Analysis in Practice: Robot Gaze}
\begin{wrapfigure}[9]{R}{0.48\textwidth}
\vspace{-3.5em}
\centering
\includegraphics[width=0.47\textwidth]{figs/proportions.pdf}
\caption{Proportion of demonstrated objects from the outer ring on AMT (left) and in-person (right). (*) indicates $p < 0.05$ and (***) $p <0.0005$.}
\label{fig:prop_objects}
\end{wrapfigure}
We next conducted an in-person study with a PR2 robot. In the real world, gaze cues are more natural and less disruptive than explicit communication. We did not explicitly inform users about the intent of the robot's gaze because we hoped to evaluate whether this choice of nonverbal robot feedback is naturally interpretable---in other words, do humans naturally pick up on gaze as a feedback mechanism for action prediction?

\prg{Design}
We replicated the virtual interface in a real-world decluttering task, using the same layout. Instead of a virtual learner that explicitly moves to its best-guess bin, in this study the PR2 relies on only gaze for feedback, changing its head orientation at varying speeds (Appendix \ref{sec:supp_practice}, \figref{fig:real_task}).

We chose to only test the biased prior condition, because this is where feedback has the largest positive impact on learner performance---for both ideal teacher models and human teachers of virtual learners that provide feedback explicitly. We maintain the same hypotheses from the AMT study.

We recruited 17 users (ages 18-26, 47\% female) from the general student population of our university---a limitation we discuss in \sref{sec:discussion}. Each taught the PR2 the same task twice: first with no feedback, and then with gaze feedback. Participants were told that the robot was reset between the two tasks, so they would not assume the agent retained knowledge from previous demonstrations.

\prg{Results}
Nine participants realized the gaze was related to the robot's belief of the sorting rule. The others misunderstood the gaze as the robot either observing where the demonstrated object was selected from or acknowledging its understanding (since gazing at the bin in front resembles a nod).

We thus report findings separately for those who understood the true intent of the gaze (G1) and those who did not (G2). We ran a repeated-measures ANOVA for each dependent measure, with feedback as a factor. For G1, feedback decreases mental model discrepancy for only the inner ring rule ($F(1, 8) = 16.681, p = 0.0035$), partially supporting \textbf{H1} (\figref{fig:discrepancy_pr2} in Appendix). Subjective Likert responses from G1 also support \textbf{H1}; see Appendix \ref{sec:supp_likert_realworld} for details. Feedback also increases the proportion of examples from the outer ring ($F(1, 8)=9.25, p<.02$) (\figref{fig:prop_objects}), as in the AMT study, and improves learner performance ($F(1,8)=9.53$, $p=.01$) (\figref{fig:learner_accuracy}); this supports hypotheses \textbf{H2} and \textbf{H3}. None of this is true for G2. For both G1 and G2, we found no significant effect of feedback on the teaching sequence length. In addition, virtually no participants were aware of the variations in speed of the robot's head motion, which suggests that more natural and noticeable gaze patterns (e.g., modulating acceleration) should be explored.

\section{Discussion}
\label{sec:discussion}
\noindent\textbf{Summary.}
We find that our proposed form of nonverbal robot feedback, predicting the teacher's next action, helps improve the teacher's effectiveness and mental model of the learner. Findings in practice echo those from our algorithmic teaching model, although we find that communicating confidence does not help significantly in practice. However, it is encouraging that despite our simple implementation of gaze, half of participants naturally interpret it as a feedback mechanism for action prediction.

\noindent\textbf{Limitations and future work.}
Our work is limited in several ways. The main limitation is that our experiments are on a relatively simple, non-sequential task. In the real world, for more complex tasks with larger action spaces, gaze may have less communicative power; investigating effective combinations of gaze and other feedback channels is a promising future direction. In addition, due to the small and biased sample, the results of our study with gaze on the PR2 should be interpreted as trends. Further work is necessary to explore the viability of gaze as a feedback mechanism, e.g., explicitly explaining to users the purpose of gaze, or making the gaze itself more human-like.

Finally, the form of feedback we study only improves learning when the learner can learn the task, but the teacher's model of the learner is not perfect (e.g., a mismatched prior). However, even when the task is not learnable, this form of feedback is still important in helping teachers recognize the robot's limitations and correctly estimate its (lack of) learning.
\section{Introduction}
If robots are to be useful to humans, they need to do more than optimize reward functions---they need to be able to figure out what reward functions should be optimized in the first place.
Inverse Reinforcement Learning (IRL)~\cite{Ng_2000} enables robots to infer preferences from human demonstrations. For instance, by collecting data of human drivers, we can infer a human-like driving style~\cite{Abbeel_2004,Levine_2012}. Traditionally, IRL is applied in settings where data is collected offline from people who have no idea that a robot is supposed to learn from this data. When talking about learning preferences for the purpose of assisting people---which is arguably the goal for most robots and AI agents---there is an opportunity to explicitly involve people in teaching the robot about what they want. It can be much more effective for me to actively teach my robot how to organize my kitchen, for instance, instead of having the robot collect data of me putting things away over and over again until it eventually figures out my preferences. When I teach, I get the chance to select examples that might be especially informative to the robot---ones that effectively illustrate the core of my approach.

Unfortunately, effective teaching is tricky even when we teach other people. We have to figure out what the person knows and does not know, what teaching strategy works best for them as an individual, etc. Teaching robots is monumentally harder. We have much poorer mental models of how robots learn compared to our mental models of how humans do. What is more, when we teach humans we receive a great deal of feedback from them.
One traditional way we get feedback is through tests, by asking questions to probe the learner's understanding. More interestingly though, human students continually provide (nonverbal) feedback to the teacher \emph{during the teaching process itself}. They look confused or bored, nod along, fidget, or gaze at various things~\cite{Angelo_1993,Webb_1997}.
\begin{wrapfigure}{R}{0.43\textwidth}
    \centering
    \includegraphics[width=0.42\textwidth]{figs/newthoughtbubble.png}
    \caption{The robot uses its learned preferences to anticipate the teacher's next action (i.e., placing the object in the yellow bin), and then uses  gaze to communicate its belief to the teacher.}
    \label{fig:frontfig}
\end{wrapfigure}

Our goal is for robots to provide similar feedback, at the same time as the teacher is providing examples. We take a step towards that in this paper, based on a simple idea: to use the states that the teacher is in as opportunities to inform the teacher about what the robot expects them to do, according to the robot's current understanding of the task.
We resort to \emph{nonverbal cues} (e.g., gaze) as an intuitive and minimally-disruptive way for the robot to signal how it expects the teacher to act next.
For example, imagine teaching the robot how to organize objects in a decluttering task (\figref{fig:frontfig}). Every time you pick up an object, the robot gazes at where it thinks the object should go. This helps you gain a better understanding of the robot's current hypothesis about your preferences, which helps you figure out a better next example to provide. It also helps you realize when the robot has finished learning, and you can stop teaching.

Our main contributions are as follows: 1) we propose a form of feedback that robots can provide while teaching is ongoing, that consists of a prediction of the teacher's next action along with the confidence; 2) we provide a theoretical motivation for why this form of feedback should improve teaching effectiveness, by introducing an algorithmic teaching model that takes feedback into account; and 3) we test out this feedback with real people, both in an online user study with virtual learners and in an in-person study with a real robot, where gaze is the feedback mechanism. We find that feedback can improve the teacher's estimate of the learner's understanding, quantifiably change the teacher's strategy, and result in higher learner accuracy. Results with real users support our theoretical analysis. With gaze, we find that these effects are stronger when participants know explicitly about the feedback they should expect---otherwise, some participants interpret gaze differently, e.g., as acknowledgement or as purely functional.

\section{Related Work}
\noindent\textbf{Improving quality of human input.}
Teaching robots via demonstration~\cite{Argall_2009} is challenging; humans may have trouble providing useful demonstrations~\cite{Cakmak_2010, Khan_2011} and knowing when to stop teaching~\cite{Sena_2018}.
Robots can take a more active role in learning, by asking for additional demonstrations where they are uncertain~\cite{Inamura_1999,Shon_2007,Chernova_2009,Silver_2012} or know information is missing~\cite{Miura_2005}, or by asking clarification questions~\cite{Whitney_2017}. Robots can ask for different kinds of teacher input as well---e.g., labels, feature queries, or demonstrations---to maximize usefulness~\cite{Cakmak_2012}. However, when robots are active learners, humans lose control over the teaching process, which can make them feel frustrated and disengaged~\cite{Chao_2010,Cakmak_2010b}.

We take an orthogonal approach, in which humans maintain control of teaching, while the robot tries to be \emph{transparent} about what it has learned. Transparency enables \emph{algorithmic teaching}, in which the teacher's understanding of how the learner learns enables her to select teaching examples that optimally teach this learner.\footnote{Since the teacher knows the task, algorithmic teaching is typically more efficient than active learning~\cite{Zhu_2015}.} Robot learners can be more transparent by demonstrating their current learned policy~\cite{Calinon_2007}, allowing teachers to ask questions that probe their understanding~\cite{Steels_2001,Lim_2014},
or showing where they succeed and fail~\cite{Sena_2018}. However, these approaches require people to stop teaching and separately test the robot. In contrast, we focus on how robots can provide feedback \emph{at the same time} as humans are teaching---similar to the nonverbal feedback cues that human students give. This form of feedback requires minimal context switching from the teacher and does not add to the total interaction time. On the real robot, we implement this feedback as gaze.

\noindent\textbf{Robot gaze.}
Robots can use gaze to be transparent in social interactions with humans (see~\citet{Admoni_2017} for a detailed survey). Robot gaze can also be useful in human-robot collaboration tasks, by disambiguating referenced objects~\cite{Admoni_2016,Boucher_2012}, communicating which action the robot is about to take~\cite{Moon_2014}, and influencing the human's actions~\cite{Admoni_2014}.

In this work, we also use robot gaze to communicate to humans, but specifically while the robot is \emph{learning}. \citet{Thomaz_2008} explored this for reinforcement learning (RL) agents: when agents use gaze to communicate what action they are about to take, humans are able to provide more informative rewards, thus speeding up learning. However, in RL the robot acts while the person observes.
When robots instead learn from human demonstrations, the opposite is true, which makes the robot's learning opaque. Taking inspiration from the RL setting, we aim at having the robot always convey what it thinks is the optimal action to take, so that the person can adjust their demonstrations appropriately.
\section{Analysis with Ideal Users}
\subsection{Model of iterative teachers}
\label{sec:iterative_teacher}
Motivated by the ``win stay, lose shift'' strategy in cognitive psychology~\cite{Nowak_1993}, we model iterative teachers as only updating their belief if the feedback disagrees with their current belief in the learner's state. In other words, if the learner predicts the bin that the teacher's maximum a posteriori (MAP) estimate of the learner's $\theta$ deems most likely, and the corresponding velocity for this $\theta_\text{MAP}$ is within $0.05 * v_\text{max}$ of the learner's feedback speed, then the teacher does not update based on this feedback. Thus, if the teacher has a perfect model of how the learner learns, then whether the learner provides feedback or not, the teacher would maintain the same belief over the learner's hypothesis, and thus teach in the exact same way. If we did not make integrating feedback conditional, then this teacher would potentially teach \emph{worse} in this situation, since the additional feedback-based updates cause the teacher's estimate to diverge from the true one.

\subsection{Implementation details}
\label{sec:supp_theoryimpl}
\noindent\textbf{Learner models.}
In the \emph{prior mismatch} condition, we bias the learner toward a randomly-selected $\theta'$ by setting the prior as the softmax of the distance between $\theta$s:
\begin{equation}
    p(\theta) \propto e^{\beta' \, \theta^\top \theta'},
\end{equation}
with $\beta' = 50$ for a fairly strong bias. In the \emph{feature mismatch} condition, the learner ignores the last feature dimension. In other words, the set of $\theta$s that the learner considers all have $\theta_d = 0$. In the \emph{reward generalization mismatch} condition, the learner learns a separate $\theta$ for each bin, i.e., reward features are computed in terms of only the object $s$, as in $\phi(s)$. So the space of $\theta$s the learner is considering now has dimensions $B*d$.

For learning updates, we set $\beta = 20$ for a learner that assumes demonstrations are close to optimal. For feedback, we set $v_\text{max} = 1.0$ and enforce a minimum speed of $0.05$. We sampled $1000$ $\theta$s to approximate the learner's belief.

\noindent\textbf{Teacher models.}
In the \emph{prior mismatch} condition, the uncertainty-aware teacher considers $P+1$ possible learner models, one of which is the default (i.e., uniform prior $b_0$); the other $P$ are biased towards different $\theta$s, including the one that the learner is actually biased towards. In the \emph{feature mismatch} condition, the uncertainty-aware teacher considers $d+1$ possible learner models: one with all features, and $d$ that are each missing a different one of the $d$ features. Finally, in the \emph{reward generalization mismatch} condition, the uncertainty-aware teacher considers two possible learner models, one of which has reward features $\phi(s,a)$ that depend on both the object and bin, and the other of which has reward features $\phi(s)$ that depend only on the object. We assume uncertainty-aware teachers start off believing that the learner likely has a uniform prior $b_0$, considers all $d$ features, and uses reward features $\phi(s,a)$---so we set the teachers' priors (i.e., $p(b_0|\Theta)$ and $p(\Theta)$) to place equal probability on all of the models, except ten times more probability on this most-likely learner model. For teacher updates, we assume $v_\text{var} = 0.0025$.

\section{Analysis in Practice}
\label{sec:supp_practice}
\subsection{Participant consent and compensation}
Both the AMT and in-person user studies, along with their corresponding consent forms, received IRB approval. Participants were paid \$3.75 for the AMT study, which lasted around 15 minutes, and participants were given a \$10 Amazon gift card for the in-person study, which lasted just under 20 minutes.

\subsection{Implementation details}
\label{sec:supp_practiceimpl}
We generated 1024 $\theta$s for the learner to reason over, including those that corresponded to conceptually intuitive sorting rules (e.g., objects belong to bins with their same shape). In the uniform prior case, the belief was uniform over these $\theta$s, and in the biased prior case, the learner heavily preferred sorting all objects into their closest bin. In the mismatched features condition, we removed the shape dimension feature in the learner's belief space.

\subsection{Analysis of Likert responses from AMT study}
\label{sec:supp_likert}
\begin{figure*}[t]
    \centering
    \includegraphics[width=0.9\textwidth]{figs/likert_barsv2.pdf}
    \caption{Likert scale responses from the AMT study, in which users taught learners with and without feedback. (*) indicates $p<0.05$, (**) indicates $p<0.005$ and (***) indicates $p<0.0005$. Labels correspond to the Likert questions in Table \ref{tab:turk_likert_anova}.}
    \label{fig:turk_likert}
\end{figure*}

For the AMT study, we ran a two-way repeated measures ANOVA on each of the subjective measures (i.e., Likert questions) with the feedback and prior belief as factors. The full details are reported in Table \ref{tab:turk_likert_anova}. For clarity, in the following analysis we will refer to the questions by their corresponding labels; full questions can be found in the first column of Table \ref{tab:turk_likert_anova}.

\noindent\textbf{Tracking robot understanding.}
In the uniform prior condition, when the learner is able to learn both the inner and outer ring sorting rules, participants' subjective Likert responses indicate that when they were given feedback, they were significantly more confident that the learner learned correctly (\figref{fig:turk_likert}, \emph{inner} and \emph{outer}). A post-hoc Tukey HSD found a significant difference between feedback and no feedback for both \emph{inner} ($p=0.004$) and \emph{outer} ($p=0.004$).

Similarly, in the missing feature condition, when the learner is not able to learn the outer ring rule, participants who received feedback were more aware of this; a post-hoc Tukey HSD found a significant difference between feedback and no feedback for \emph{outer} ($p=0.0003$).

Across all prior belief conditions, participants found it overwhelmingly easier to recognize what the learner \emph{does not} understand; the ANOVA found a significant main effect for feedback for \emph{doesn't}. Participants also generally found it easier to recognize what a learner \emph{knows}; a post-hoc Tukey HSD found a significant difference between feedback and no feedback for the uniform prior ($p<0.0001$) and partial feedback ($p<0.0001$) conditions.

\noindent\textbf{User satisfaction.}
For \emph{happy}, a post-hoc Tukey HSD found a significant difference between feedback and no feedback for the missing feature condition ($p=0.0011$). This makes sense, because when feedback was given in the missing feature condition, participants were clearly aware that the robot was unable to learn the task, despite their efforts to teach it.

\noindent\textbf{Full versus partial feedback.}
An ANOVA on each of the subjective measures with the feedback level as a factor (full feedback versus partial feedback) found significant differences between the partial and full feedback condition for only the \emph{stop} ($F(1, 47)= 8.43, p=0.0059$) and \emph{happy} ($F(1, 47) = 5.57, p=0.02$) questions. The average for the partial feedback condition was higher in both cases. We hypothesize that this is because participants found the partial feedback (with constant speed) to be easier to reason about. This suggests that there is a need for investigating other mechanisms for expressing feedback confidence.

{\begin{table}[t]
    \small
    \centering
    \begin{tabular}{lccc}
        \toprule \hline \Tstrut
        \textbf{Statement} & \textbf{Effect} & \textbf{F-score} & \textbf{p-value}  \\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Inner:} ``[Learner] correctly learned the rule that if an\\ object lies in an inner ring, it belongs to the closest bin''\end{tabular} & interaction & $F(3, 83)=7.99$ & $p<0.0001$\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Outer:} ``[Learner] correctly learned the rule that if an\\ object lies in an outer ring, it belongs to the bin with the\\ same shape''\end{tabular} & interaction & $F(3, 83)=14.12$ & $p<0.0001$\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Informative:} ``I found it easy to choose informative\\ teaching examples for [Learner]''\end{tabular} & none & --- & ---\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Stop:} ``It was easy to know when to stop teaching\\ $$[Learner]''\end{tabular} & none & --- & ---\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Knows:} ``It was easy to tell what [Learner] knows about\\ the task rules''\end{tabular} & interaction & $ F(3, 83)=3.89$ & $p=0.01$\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Doesn't:} ``It was easy to tell what [Learner] \textit{doesn't}\\ know about the task rules''\end{tabular} & interaction & $F(1, 83)=87.7$ & $p<0.0001$\\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Happy:} ``I would be happy to teach [Learner] again''\end{tabular} & interaction & $F(3, 83)=6.14$ & $p=0.0008$\\
        \midrule
        \bottomrule
    \end{tabular}
    \vspace{2mm}
    \caption{ANOVA Results for AMT Study Likert Questions.}
    \label{tab:turk_likert_anova}
\end{table}
}

\begin{figure}
\centering
\subfigure[]{\label{fig:real_a}\frame{\includegraphics[width=.24\columnwidth]{figs/carolyn_a_blurred-compressed.pdf}}}
\subfigure[]{\label{fig:real_b}\frame{\includegraphics[width=.24\columnwidth]{figs/carolyn_b_blurred-compressed.pdf}}}
\subfigure[]{\label{fig:real_c}\frame{\includegraphics[width=.24\columnwidth]{figs/carolyn_c_blurred-compressed.pdf}}}
\subfigure[]{\label{fig:real_d}\frame{\includegraphics[width=.24\columnwidth]{figs/carolyn_d_blurred-compressed.pdf}}}
\caption{Real-world experiment flow: the participant a) selects the object to demonstrate, b) picks it up and shows it to the PR2, c) observes the PR2's gaze feedback, and d) places the object in the correct bin.}
\label{fig:real_task}
\end{figure}

\subsection{Analysis of Likert responses from in-person study}
\label{sec:supp_likert_realworld}
\begin{wrapfigure}[17]{R}{0.5\textwidth}
\vspace{-1.5em}
\centering
\includegraphics[width=0.49\textwidth]{figs/mental_model_pr2.pdf}
\caption{Mental model discrepancy (defined in \sref{sec:study_design}) for the inner and outer ring sorting rules. \textbf{\emph{Lower magnitude is better; positive values mean that the teacher overestimates the learner capability, and negative values mean they underestimate.}} (**) indicates $p<0.005$.}
\label{fig:discrepancy_pr2}
\end{wrapfigure}

For the in-person study with the PR2 robot, we ran a repeated-measures ANOVA on each of the subjective measures (i.e., Likert questions) with feedback as a factor. These questions are the same as for the AMT study. The full details are reported in Table \ref{tab:realworld_likert_anova}.

\noindent\textbf{Tracking robot understanding.}
Recall that in this study, the PR2 robot started with a biased prior over weights in $\Theta$, that heavily favors sorting all objects into their closest bin. This biased prior agrees with the inner rule, but contradicts the outer rule. When participants received gaze feedback from the robot, those who understood the true intent of the robot's gaze (G1) better recognized that it correctly learned the \emph{inner} rule ($F(1,8) = 16.66$, $p = 0.0035$), compared to when there was no feedback. Participants in G1 also thought it was easier to give \emph{informative} teaching examples ($F(1,8) = 12.57$, $p = 0.0076$), recognize when to \emph{stop} teaching ($F(1,8) = 14.59$, $p=0.0051$), and recognize what the robot \emph{knows} ($F(1,8) = 139.13$, $p<0.0001$) and \emph{doesn't} know ($F(1,8) = 144.00$, $p < 0.0001$) when the robot provided gaze feedback.

Even participants who misunderstood the intent of the robot's gaze (G2) thought it was easier to recognize when to \emph{stop} teaching ($F(1,7) = 6.66$, $p=0.036$) and recognize what the robot \emph{knows} ($F(1,7) = 5.64$, $p = 0.049$) when the robot provided gaze feedback.

\noindent\textbf{User satisfaction.}
Participants in G1 were more \emph{happy} to teach the robot in the future, when it provided gaze feedback ($F(1,8) = 10.73$, $p = 0.011$).

\begin{figure*}
\vspace{-1em}
\centering
\includegraphics[width=\textwidth]{figs/in_person_likert_bars.pdf}
\caption{Likert scale responses from the in-person study, in which users taught learners with and without feedback. (*) indicates $p<0.05$, (**) indicates $p<0.005$ and (***) indicates $p<0.0005$. Labels correspond to the Likert questions in Table \ref{tab:realworld_likert_anova}.}
\label{fig:realworld_likert}
\end{figure*}

\begin{table}[t]
    \small
    \centering
    \begin{tabular}{lccc}
        \toprule \hline
         \textbf{Statement} & \textbf{Group} & \textbf{F-score} & \textbf{p-value} \Tstrut \\ \midrule

        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Inner:} ``[Learner] correctly learned the rule that if an\\ object lies in an inner ring, it belongs to the closest bin''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 16.66$\vspace{0.35em} \\--- \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p=0.0035$\vspace{0.35em}\\--- \end{tabular} \vspace{0.3em} \\
        \midrule
        \begin{tabular}[c]{@{}l@{}}\textbf{Outer:} ``[Learner] correctly learned the rule that if an\\ object lies in an outer ring, it belongs to the bin with the\\ same shape''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} ---\vspace{0.35em} \\--- \end{tabular} & \begin{tabular}[c]{@{}c@{}} ---\vspace{0.35em}\\--- \end{tabular} \\
        \midrule
        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Informative:} ``I found it easy to choose informative\\ teaching examples for [Learner]''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 12.57$\vspace{0.35em}\\--- \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p=0.0076$\vspace{0.35em}\\--- \end{tabular} \vspace{0.3em} \\
        \midrule
        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Stop:} ``It was easy to know when to stop teaching\\ $$[Learner]''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 14.59$\vspace{0.35em}\\$F(1, 7) = 6.66$ \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p=0.0051$\vspace{0.35em}\\$p=0.036$ \end{tabular} \vspace{0.3em} \\
        \midrule
        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Knows:} ``It was easy to tell what [Learner] knows about\\ the task rules''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 139.13$\vspace{0.35em}\\$F(1, 7) = 5.64$ \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p < 0.0001$\vspace{0.35em}\\$p=0.049$ \end{tabular} \vspace{0.3em} \\
        \midrule
        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Doesn't:} ``It was easy to tell what [Learner] \textit{doesn't}\\ know about the task rules''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 144.00$\vspace{0.35em}\\--- \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p < 0.0001$\vspace{0.35em}\\--- \end{tabular} \vspace{0.3em} \\
        \midrule
        \rule{0pt}{4ex}\begin{tabular}[c]{@{}l@{}}\textbf{Happy:} ``I would be happy to teach [Learner] again''\end{tabular} & \begin{tabular}[c]{@{}c@{}} G1\vspace{0.35em}\\G2\end{tabular} & \begin{tabular}[c]{@{}c@{}} $F(1, 8) = 10.73$\vspace{0.35em}\\--- \end{tabular} & \begin{tabular}[c]{@{}c@{}} $p=0.011$\vspace{0.35em}\\--- \end{tabular} \vspace{0.3em} \\
        \midrule
        \bottomrule
    \end{tabular}
    \vspace{2mm}
    \caption{ANOVA Results for In-Person Study Likert Questions.}
    \label{tab:realworld_likert_anova}
\end{table}

Robotics (cs.RO); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)


Subjective Metrics-based Cloud Market Performance Prediction

Abstract: This paper explores an effective machine learning approach to predict cloud market performance for cloud consumers, providers and investors based on social media. We identified a set of comprehensive subjective metrics that may affect cloud market performance via literature survey. We used a popular sentiment analysis technique to process customer reviews collected from social media. Cloud market revenue growth was selected as an indicator of cloud market performance. We considered the revenue growth of Amazon Web Services as the stakeholder of our experiments. Three machine learning models were selected: linear regression, artificial neural network, and support vector machine. These models were compared with a time series prediction model. We found that the set of subjective metrics is able to improve the prediction performance for all the models. The support vector machine showed the best prediction results compared to the other models.

\section{Introduction}

Cloud computing is increasingly becoming the technology of choice as the next-generation platform for most organization \cite{sajib2018}\cite{SUN2019749}. Cloud computing services are provided by vendors under the following categories: Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS) are available in the cloud market. Different organizations such as banks, universities, etc., are the main consumers of cloud services.

The cloud market is divided into two types of participants: those who use cloud services - \textit{cloud customer}, and those who need to invest in the cloud market - \textit{cloud investors/providers}.  \textit{Both} the parties are interested in cloud service performance in the market. Sufficient knowledge of the cloud market allows the market participants to be aware of opportunities, advantages, and risks associated with it. This can include knowledge of how the market is performing in terms of market competition, market growth, customer satisfaction, etc. \cite{mauboussin2012true}, which can all help in making informed decisions regarding the market. In fact, a comprehensive analysis of the market is of value to the cloud customer due to the long-term commitment involved. Such customers could be those who make IT decisions after determining the influence of emerging technologies.  They must understand how the market performs in terms of customer satisfaction, service quality, and many other factors before deciding to participate in the market. Investors also require information on market performance before making any decision to invest in stocks. In fact, many industry reports have been prepared by different organisations in relation to all the participants \cite{USDEP}\cite{451Research}. Most of these reports are concerned with the evaluation of cloud market size and growth as well as numerous other concerns. These reports tend to be produced after thorough monitoring of the market or after accessing to cloud providers’ performance/sales.

The performance information provided by cloud market reports are based on two types of metrics: \textit{objective metrics} and \textit{subjective metrics}. The objective metrics are based on actual facts without any individual bias. Examples of such metrics include market size and growth. According to 451Research \cite{451Research}, the collection of such data is frequently implemented through extensive market monitoring. Meanwhile, the subjective metrics are based on personal perceptions, opinions or feelings, an example being the information regarding the barriers to cloud adoption \cite{STATE}. Subjective measures are frequently obtained using industry surveys \cite{RightScale}\cite{USDEP}. Industries utilise either objective and subjective measures or a combination of both to produce industry reports. However, it is a challenge to apply objective metrics \cite{vij2016subjective} due to the information sensitivity, while it can be \textit{expensive}, \textit{time consuming} and \textit{less timely} to obtain the information required for objective performance. On the other hand, the gathering of subjective market information can be more practical, since the sharing of subjective evaluation information is preferred over objective evaluation information sharing \cite{vij2016subjective}. Market participants often tend to share similar information on \textit{social media}.

Numerous individuals are becoming increasingly involved in sharing their opinions on a variety of subjects in social media platforms \cite{9001217}\cite{8029820}. It has become increasingly popular in recent years \cite{nepal2015trusting}. Consumers express their complaints or their satisfaction with certain brands and services. It has been reported that one in three social media users prefer to use this outlet rather than contact the company by phone. Moreover, 50\% of social media users share their complaints/concerns about a service/brand at least once a month \cite{STATE}. Hence, social media can be viewed as a \textit{free and up-to-date} source of personal wisdom \cite{yu2012survey}.

Purchase decisions can be affected by consumer's perceptions on services/goods. Thus, subjective measures can be used to predict the future behaviour of the participants. With this subjective measure evaluation of market performance, a better comprehension of objectively measured results can be gained. Any given customer review can include perceptions related to various areas of a product or service experience \cite{goh2013social}. In turn, these perceptions will influence the readers’ perceptions and, subsequently, their purchase decisions \cite{mudambi2010research}. Here, perceptions related to risks, barriers and benefits can be particularly influential on a consumer’s purchasing intention \cite{ruiz2009drivers}. Future behaviour can be predicted based on this intention to purchase \cite{shahabuddin2009forecasting}.

According to Jayaratna et al. \cite{Jayaratna}, subjective measures are applicable in evaluating cloud market performance.  They identified unique subjective metrics that can assess cloud market performance. However, the subjective information is limited to \textit{product/service quality}. In addition, attention has not been given to \textit{use machine learning techniques} for predicting economical/financial performance of cloud services in the cloud market.

 We aim to \textit{explore a set of more comprehensive subjective metrics} that may affect cloud market performance and \textit{analyse the effectiveness of machine learning techniques} in predicting cloud market performance. A comprehensive set of subjective cloud service evaluation metrics were collected via literature survey. The values of subjective metrics were collected from social media and quantified using sentiment analysis. We applied three mainstream machine learning prediction techniques: \textit{linear regression}, \textit{artificial neural network}, and \textit{support vector machine} to ascertain which model predicts cloud market performance most accurately. We considered \textit{cloud market revenue growth} as the indicator of cloud market performance. This paper makes the following contributions:
\begin{enumerate}
\item We identified a set of more comprehensive  subjective metrics that may influence the cloud market performance.
\item We explored an effective machine learning model to predict the cloud market performance based on the identified subjective metrics.

\end{enumerate}

The paper is structured as follows: Section 2 describes our solution. Section 3 depicts the results of our experiment. Section 4 concludes the paper.

\section{Methodology}
We used two sets of metrics (subjective and objective) to evaluate cloud market performance. Cloud market revenue was considered as the objective metrics because it is an important factor in market performance evaluation \cite{mauboussin2012true}. Cloud user's perception on different cloud aspects was used as the subjective metrics. We explored an effective machine learning approach that  predicts cloud market revenue growth based on the cloud user's perceptions on different cloud aspects. The main objective of the proposed approach is to predict the cloud market performance based on a comprehensive set of subjective metrics.
\subsection{Cloud Service Evaluation Subjective Metrics  Identification}
Subjective metrics of customer perception have shown effectiveness in predicting cloud market performance over the objective metrics \cite{Jayaratna}. Therefore, in this research, we aim to explore a comprehensive set of subjective metrics that may influence cloud market performance.

Certain drivers and boundaries drive consumers to/from cloud services according to the surveys conducted by RightScale \cite{RightScale} and the Australian Bureau of Statistics \cite{ABS}. We can thus assume that drivers and boundaries are important for cloud consumers. The knowledge of the benefits and challenges are likely to influence a customer’s decision regarding the purchase of cloud services. Therefore, the benefits and challenges of cloud services will act as drivers of cloud market revenue. Jayaratna et al. \cite{Jayaratna} introduced 13 drivers and boundaries related to product/service quality.

Based on the above, we combined Jayaratna et al. \cite{Jayaratna}'s 13 drivers and boundaries with three more drivers: after-sales experience, market responsiveness and marketing execution  from Gartner \cite{gartner}, as a result of our literature survey. Table \ref{aspects} presents the combined drivers and boundaries used in this paper. We focused on consumer’s perception on same cloud aspects. In this paper, we aim to predict cloud market performance based on the 16 cloud aspects.
\begin{table}[]
\centering
\caption{Subjective aspects affecting cloud purchase decisions}
\label{aspects}
\resizebox{\textwidth}{!}{%
\begin{tabular}{|l|L{13cm}|}
\hline
\textbf{Cloud Aspects} & \textbf{Description}  \\ \hline
Greater scalability  & Flexible to either up-scale or down-scale   \\ \hline
 Faster access to infrastructure  & Easy access to infrastructure without having to purchase them    \\ \hline
  Managing multiple services  & Overheads and difficulty in managing multiple services  \\ \hline
   Security concerns   & Concerns over data breaches, privacy, access
control, etc.  \\ \hline
 Cost savings  & Cost saved by transfer to cloud infrastructure  \\ \hline

 Higher availability  & High availability of cloud services  \\  \hline
 Lack of control & Uncertainty of data location. Uncertainty
regarding legal issues, and dispute resolution   \\ \hline
 Higher performance  & Higher performance of cloud compared to
on-premise infrastructure  \\ \hline
 Lack of expertise/resources   & Lack of specialised people or sufficient resources for managing cloud services  \\ \hline

 IT staff efficiency   & Increase of productivity  \\ \hline
 Provider lock-in   & Difficulties with changing cloud computing service provider  \\ \hline
 Business continuity  & Ability to continually operate even through disasters  \\ \hline
 Move from CapEx to OpEx   & Changing from capital expenditure to operating expense  \\ \hline

 \textit{After-sales experience}   & Ability to provide acceptable customer services  \\ \hline
 \textit{Market responsiveness}   & Ability to enhance the products’ after sales   \\ \hline
 \textit{Marketing execution}   & Ability to deliver the product that the
customer expected \\ \hline
\end{tabular}%
}
\end{table}
\subsection{Data Collection}
We collected two types of data: objective data and subjective data. The Amazon web service (AWS) quarterly revenue was considered as objective. The subjective data was collected in terms of AWS customer reviews.\\
\textbf{(1) Objective Data:\\}
IaaS is defined as the capability to provide computer resources, storage, and other fundamental computing resources to the customer. Accordingly, we consider the revenue generated by offering these types of cloud services. Amazon is considered as the leading cloud provider because it holds the largest market share in the IaaS segment. Therefore, we consider Amazon revenue as a cloud market performance. AWS revenue is reported for the whole segment (IaaS, PaaS, etc). In this paper, the AWS quarterly revenue growth \cite{AWS} from the 4th quarter of 2015 to the 4th quarter of 2018 was considered as an indicator of AWS performance in the market. Revenue growth was the increase in a company’s revenue from one period to the next. Revenue growth identifies over time trends in business. The AWS was calculated based on the following Equation \cite{Jayaratna}:
\begin{equation} \label{eq:growth}
AWS QRG_y = \frac{Revenue_q - Revenue_{q-1}}{Revenue_{q-1}}
\end{equation}
where $AWS QRG_y$ is the AWS quarterly revenue growth of quarter q, $Revenue_q$ is the revenue made in q, and $Revenue_{q-1}$ is the revenue made in the quarter before q.\\
\textbf{(2) Subjective Data:\\}
Social media is a rich source of customer opinion. Obvious differences can be seen among the customer perceptions collected from social media websites such as Twitter and Facebook and from customer review websites. However, social media websites also contain references to cloud services that are not based on experiences, such as references related to discussions or news on topics about newly-released features. In contrast, cloud customers’ review websites are exclusively focused on the customer experience. These review sites contain comprehensive information on users’ experiences. Moreover, any user can post comments about a cloud service/provider on social media, and the platforms do not verify whether a post is from a genuine cloud service customer. However, most of the popular review sites do go through this process before posting the feedback. Therefore, we can conclude that customer review websites provide more valuable information on cloud services than popular social media websites.

We examined six cloud social media, which are the most popular of those currently active. These are `G2 Crowd'\footnote{{https://www.g2crowd.com}}, `Trust Radius'\footnote{{https://www.trustradius.com}}, `Clutch'\footnote{{https://clutch.co/}}, `Gartner Peer Insights'\footnote{{https://www.gartner.com/reviews/home}}, `WhoIsHostingThis'\footnote{{https://www.whoishostingthis.com}} and `Spiceworks'\footnote{{https://www.spiceworks.com}}, all of which guarantee the genuineness of the reviews are based on the genuine customer experience. These sources contain reviews of a variety of cloud providers, including Amazon, Google, and Microsoft. The reviews are written by customers from different domains (i.e. education, health, insurance) and organisations of different scale (small, mid-sized and enterprise). We assumed that the customers’ reviews can be considered as a representative of the overall market customers because the number of customer reviews on these websites exceeds 1000.
\subsection{Perception Analysis on Cloud Aspects}
VADER sentiment analysis technique \cite{hutto2014vader} is a well-acknowledged sentiment detection tool. VADER takes into account
punctuation, capitalization, degree modifiers, and the use of contrastive conjunction. In this paper, we used VADER sentiment analysis to process the user reviews. Sentiment analysis was performed to find customers' perception on each cloud aspect separately. First, we built a vocabulary that contains the most frequent words to refer to each specific aspect. We used NVivo software\footnote{{https://www.qsrinternational.com/nvivo/home}}, which has been designed for qualitative and mixed-methods research, to build the vocabulary. We fed the customer reviews of each quarter to the NVivo software. Then, we found the most frequent words that related to the cloud aspects. For example, secure, secured, securely, security were repeated more than 40 times in 2016 4th quarter reviews. Therefore, we assumed that those words will fit into the security cloud aspect.

Next, the sentiments of the AWS user reviews of each quarter were analysed using VADER. VADER is based on lexicons of sentiment related-words. Every word in the lexicon is rated as to whether it is positive or negative. VADER hired many domain experts to manually rate the words. Each word in the lexicon is rated between -4 (extremely negative) and +4 (extremely positive). VADER checks a piece of text to see if any of the words in the text are present in the lexicon.  VADER produces four sentiment scores: positive, neutral, negative, and compound. The positive, neutral, and negative scores are ratios for the proportions of text that falls into those categories. The compound score is a metric that calculates the sum of all the lexicon ratings\footnote{{http://comp.social.gatech.edu/papers/}} which have been normalized between -1 (most extreme negative) and +1 (extremely positive). Finally, we considered the compound sentiment results as the overall perception of a particular cloud aspect. The customers' perception of cloud aspect A in quarter Q is defined as follows:

\begin{equation}\label{SENTEQ}
Perception_{A,Q}= \frac{Compound_{A,Q}}{Reviews_{A,Q}}
\end{equation}
where $Compound_{A,Q}$ is the compound sentiment value, and $Reviews_{A,Q}$ is the number of reviews referring to aspect $A$ in quarter $Q$.
\subsection{Sensitivity Analysis}
We designed the following machine learning models to evaluate cloud market performance based on the subjective metrics.\\
\textbf{(1) Linear Regression Analysis (LR):\\}
Linear regression is a linear approach used to ascertain the relationship between a dependent variable and one or more independent variables. In our approach, the independent variables are sets of cloud aspects, while the dependent variable is the AWS revenue growth. The linear regression has the following Equation:
\begin{equation}
Y = a + bX_1 + cX_2 + ... + mX_n
\end{equation}
where $Y$ is used to represent the AWS revenue growth, and $X_1$, $X_2$, ... $X_n$ represent the user perception of each cloud aspect.\\
\textbf{(2) Artificial Neural Network (ANN):\\}
ANN models are mathematical models inspired by the functioning of the nervous system. ANNs are based on learning which is a characteristic of adaptive systems which are capable of improving their performance on a problem as a function of previous experience \cite{luk2000study}. In this paper, we used a Multlayer Perceptron. The Levenberg-Marquardt backpropagation algorithm the most widely-used training algorithm for time series prediction \cite{more1978levenberg} was used to train the network. The input activation function was obtained using the following Equation \cite{more1978levenberg}:
\begin{equation}
   s = \sum w_{ij}x_j + b
\end{equation}
where $x_i$ is the user perception of each cloud aspect, $w_j$ is the weight, and b is the bias.

The output neuron uses the sigmoid activation function obtained with the following Equation \cite{more1978levenberg}:
\begin{equation}
f(s) = \frac{1}{1+e^{-s}}
\end{equation}
where $s$ is the sum of products (sop) between each user perception on each cloud aspect and its corresponding weight. $s$ is an input activation function.

The error function ascertains how close the predicted output is from the target output. The error function was obtained using the following Equation \cite{more1978levenberg}:
\begin{equation}
E= \frac{1}{2} \sum_{i=1}^N (O_i-T_i)
\end{equation}
where $N$ is the total number of output data, $O_i$ is the actual AWS revenue growth of $i^{th}$ data and $T_i$ is the predicted AWS revenue growth of $i^{th}$ data.\\
\textbf{(3) Support Vector Machine (SVM):\\}
The nu-SVR model, which is applicable for modelling continuous time series, was chosen. The nu-SVR model was found to be reliable and robust, even for models based on small training samples or data burdened by noise based on experience from applications in a different area \cite{zhu2007performance}. In our approach, we used the nu-SVR model with a radial basis function kernel (RBF) and which takes the user perception of cloud services and AWS revenue growth as the target. The RBF kernel can be defined as follows \cite{zhu2007performance}:

\begin{equation}
\exp ( \gamma\times\sum \mid u-v \mid^2 )
\end{equation}
where $\gamma$ is the parameter, $u$ is the actual AWS revenue growth, and $v$ is the predicted AWS revenue growth.
\section{Experiments}
A set of experiments was conducted to evaluate the efficiency of the proposed prediction models. First, we describe the dataset and how is it obtained. We then  compare the parameter of each model. Next, we  compare the prediction performance between four predictions models. We ran two experiments in order to analyse the impact of our new subjective metrics.  First, we used the user perceptions on the 13 cloud aspects identified in Jayaratna et al.'s \cite{Jayaratna}. Then, we used the user perceptions on the 16 cloud aspects in order to find the impact of the new subjective metrics. We evaluated all the proposed models in terms of Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and Theil’s U-statistics.
\subsection{Datasets}
We divided the data set into the sets of training data and testing data according to an approximate ratio of 2:1. AWS quarterly revenue growth and user perceptions of each cloud aspect from the 4th quarter of 2015 to the 4th quarter of 2017 are the training data. The data from the 2018 1st quarter to 2018 4th quarter are the testing data. Figure \ref{fig:awsgrowth} presents the AWS quarterly revenue growth data. We collected approximately 1100 user reviews  for analysis.
\begin{figure}[]
\centering
\includegraphics [width=0.8\linewidth]{AWS_r}
\caption{AWS quarterly revenue from 2015 4th quarter to 2018 4th quarter}\label{fig:awsgrowth}
\end{figure}
\subsection{Sentiment Analysis}
First, we identified the most frequent words from the customer reviews of each quarter. For example, After-sales aspect has ten words which are customer service, satisfaction, good service, after-sales,  client service,  product service,  troubleshooting,  assistance,  customer care and  support. Next, we categorized the customer reviews based on the 16 cloud aspects that were introduced in the previous section. Next, the sentiment of the AWS user reviews were analysed. We processed each review to get the sentiment results. We then calculated the user perception using Equation \ref{SENTEQ}. Table \ref{sent} shows the sentiment results and the user perceptions for After-sales for 2016 4th quarter as an example.
\begin{table}[]
\centering
\caption{Sentiment and user perceptions results for After-sales for 2016 Q4}
\label{sent}
\resizebox{\textwidth}{!}{%
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
\textbf{Review} & 1 &2  &3  &4  &5  &6 &7  &8  &9  &10  &11  & 12 \\ \hline
\textbf{Compound} &0.4215  &0.3612  &0.3182  &0.7092  &0.9196  &0.8876  &0.9681  &0.4091  &0.9702  &0.1027  &0.9715  &0.9829  \\ \hline
\textbf{User perception} & \multicolumn{12}{l|}{0.66848334} \\ \hline
\end{tabular}%
}
\end{table}
\vspace*{-10mm}
\subsection{Tuning Process for the Proposed Models}
We tested four models in the experiment, including an objective model (ARIMA) based on previous quarterly revenue growth and three subjective models based on previous quarterly revenue growth and customer perceptions. Here we introduced the optimal tuning result for each model.\\
\textbf{(1) ARIMA model:\\}
ARIMA model is a prediction model based on the objective metrics. Therefore, we considered the ARIMA model as our baseline. ARIMA stands for Autoregressive integrated moving average. ARIMA model is the most prominent methods in time series forecasting. ARIMA models have shown efficient capability to generate short-tram prediction \cite{tascikaraoglu2014review}.  In ARIMA model, the future value of a variable is a linear combination of past values and past errors, expressed as follows:
\begin{equation}
Y_t = \phi + \phi_1 Y_{t_{-_1}} + \phi_2 Y_{t_{-_2}} + ... + \phi_p Y_{t_{-_p}} +\varepsilon - \phi_1 \varepsilon_1{_{t_{-{_1}}}} - \phi_2 \varepsilon_2{_{t_{-{_2}}}} - ... - \phi_q \varepsilon_q{_{t_{-{_q}}}}
\end{equation}
where $Y_t$ is the AWS revenue growth, $\varepsilon$ is the random error at $t$, $\phi_i$ is the coefficients, $p$ and $q$ are integers that are often referred to as autoregressive and moving average, respectively.

A standard notation is used of ARIMA(p,d,q). The autoregressive (p), integrated(d) and moving average (q) parameters have to be effectively determined in order to construct the best ARIMA model for AWS revenue growth prediction. We tested the model on different parameters of (p) and (q) (i.e., ARIMA (1,0,0), (1,1,0), (2,1,0), (0,1,1), (2,0,0) and (3,0,0)). We found that ARIMA (1,0,0), (2,1,0) and (2,0,0) achieve the best performance. All the experiments were conducted using Python.

\textbf{\\(2) Linear Regression Analysis (LR):\\}
At first, we tested the 13 cloud aspects as the independent variables to identify the prediction model. The prediction model based on the 13 cloud aspects can be represented in the Equation \ref{model13}. Then,  we tested the 16 cloud aspects as the independent variables to identify the prediction model. The prediction model can be represented in the Equation \ref{model16}.
\begin{multline}\label{model13}
RG= - 0.151*Per_{IT}-0.069*Per_{Perf} + 0.075*Per_{Cost} \\- 0.138*Per_{control} -0.113*Per_{Ex}-0.156*Per_{Provider}+0.042*Per_{Security} + 0.519
\end{multline}
\begin{multline}\label{model16}
RG= -0.017*Per_{BC} + 0.054*Per_{Cost} - 0.065*Per_{control} -0.138*Per_{Ex}\\-0.132*Per_{Provider}-0.092*Per_{execution}-0.115*Per_{Security} + 0.515
\end{multline}
where $RG$ is the revenue growth, $Per_{BC}$ is the perception of business continuity, $Per_{Cost}$ is the perception of cost savings, $Per_{control}$ is the perception of lack of control, $Per_{Ex}$ is the perception of lack of expertise/resources, $Per_{Provider}$ is the perception of provider lock-in, $Per_{execution}$ is the perception of marketing execution, $Per_{Security}$ is the perception of security concerns, $Per_{IT}$ is the perception of IT staff efficiency  and $Per_{Perf}$ is the perception of higher performance.\\
\textbf{(3) Artificial Neural Network (ANN):\\}
We needed to establish the network parameters to build the neural network. The input data was randomly divided into training data (70\%), validation data (15\%) and testing data (15\%). The best validation performance for the 13 cloud aspects is obtained at epoch 2 and for the 16 cloud aspects is obtained at epoch 3. All experiments were conducted using the neural networks toolbox of MATLAB.\\

\textbf{(4) Support Vector Machine (SVM):\\}
Several levels of gamma constant of radial function was tested in the parameter settings experiments.
We chose four different values for gamma 0.01, 0.1, 1, and 5  \cite{englucasstock}. We found that the best performance is obtained when the value of gamma is 5 for the both cloud aspect combinations. All experiments were performed on windows using Chang and Lin’s library developed for SVM implementations \cite{CC01a}.

\subsection{Experimental Result and Discussion}
We predicted the AWS revenue growth form the 1st quarter of 2018 to the 4th quarter of 2018 using four different prediction models: ARIMA, LR, ANN, and SVM. Figure \ref{model:1} depicts the results of predicted revenue growth compared to the actual revenue growth. It can be seen that there is a significant gap between the actual and predicted revenue growth for LR with 13 and 16 cloud aspects prediction models. However, the other models were able to predict AWS revenue growth for all quarters.

All the experimental results prove the positive impact of the new set of subjective matrices and the prediction performance of the proposed models. All the models based on the 16 cloud factors outperform their counterparts based on the 13 cloud factors as described in Table \ref{per_allmodel}. We can confidently conclude that  \textit{After-seals}, \textit{Market responsiveness} and \textit{Market execution} improve the prediction performance. Additionally, the SVM prediction model with 16 cloud aspects produced the best results with the smallest error. The SVM prediction models outperformed ARIMA, ANN and LR models. The reason is that SVM implements the structural risk minimization principle, resulting in better generalization than the other techniques. In contrast, ANN needs a larger training set to perform more accurate predication, while ARIMA purely bases on the previous revenue data and disregards the impact of the subjective metrics on the avenue growth. LR preformed the worst due to the existence of nonlinear relationship between revenue growth and the subjective metrics.
\begin{figure}[]
\centering
\includegraphics[scale=0.45]{Model_prediction_result.png}
\caption{Actual vs. Predicted }\label{model:1}
\end{figure}
\begin{table}[]
\centering
\caption{Prediction performance of ARIMA, LR, ANN, and SVM}
\label{per_allmodel}
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{|l|l|l|l|}
\hline
\textbf{Prediction Model} &\textbf{MSE}  &\textbf{RMSE}  &\textbf{ Theil’s U-statistics} \\ \hline
 \textbf{ARIMA}&0.000583622  &0.024   &0.394573683  \\ \hline
 \textbf{LR - 13 Cloud Aspects}&0.071319997  &0.267058041 & 1.455854282 \\ \hline
 \textbf{LR - 16 Cloud Aspects}&0.004973533  &0.03526164  & 0.504616664  \\ \hline
 \textbf{ANN - 13 Cloud Aspects}&0.001027297  &0.0321  & 0.444061429  \\ \hline
 \textbf{ANN - 16 Cloud Aspects}&0.000493797  &0.0222  & 0.404017853  \\ \hline
 \textbf{SVM - 13 Cloud Aspects}&0.000143942  &0.011997583  & 0.354241995  \\ \hline
 \textbf{SVM - 16 Cloud Aspects}&\textbf{0.000117719}  &\textbf{0.010849839}  & \textbf{0.343687709} \\ \hline
\end{tabular}%
}
\end{table}
\section{Conclusion}
We explored a machine learning based approach to predict the cloud market performance using social media subjective metrics. We identified a set of comprehensive subjective metrics (including After-seals, Market responsiveness, and Market execution). We processed the user reviews collected from social media using VADER sentiment analysis, a popular sentiment analysis technique on each subjective metric. We applied three mainstream machine learning: linear regression, artificial neural network, and support vector machine. Experimental results show that the new set of subjective metrics have a positive impact on predicting cloud market performance. The proposed model is able to predict cloud market performance for both cloud consumers and cloud investors/providers. In the future work, we plan to apply deep learning models to asses cloud market performance.

\bibliographystyle{splncs04}

Social and Information Networks (cs.SI); Machine Learning (cs.LG)