Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Abstract: Task automation of surgical robot has the potentials to improve surgical efficiency. Recent reinforcement learning (RL) based approaches provide scalable solutions to surgical automation, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue is known as the exploration challenge, which can be alleviated by providing expert demonstrations to an RL agent. Yet, how to make effective use of demonstration data to improve exploration efficiency still remains an open challenge. In this work, we introduce Demonstration-guided EXploration (DEX), an efficient reinforcement learning algorithm that aims to overcome the exploration problem with expert demonstrations for surgical automation. To effectively exploit demonstrations, our method estimates expert-like behaviors with higher values to facilitate productive interactions, and adopts non-parametric regression to enable such guidance at states unobserved in demonstration data. Extensive experiments on $10$ surgical manipulation tasks from SurRoL, a comprehensive surgical simulation platform, demonstrate significant improvements in the exploration efficiency and task success rates of our method. Moreover, we also deploy the learned policies to the da Vinci Research Kit (dVRK) platform to show the effectiveness on the real robot. Code is available atthis https URL.

\section{Introduction}
Surgical robots nowadays assist surgeons to conduct minimally invasive interventions in clinical routine~\cite{d2021accelerating}. Automating surgical tasks has been increasingly desired to improve surgical efficiency, with many efforts being made on different tasks such as suturing~\cite{schwaner2021autonomous,srl_multistage,suture}, endoscope control~\cite{gao2022savanet,endoscope,pandya2014review}, tissue manipulation~\cite{saeidi2022autonomous,flexml,srl_reward} and pattern cutting~\cite{cutting1,nguyen2019new,nguyen2019manipulating}. Recently, reinforcement learning (RL) approaches have exhibited high scalability to learn diverse control policies and yielded promising performance in surgical automation~\cite{su2020reinforcement,srl_reward1,scheikl2021cooperative,segato2021inverse,tan2019robot,keller2020optical,d2022learning}, but typically require extensive data collection to solve a task if no prior knowledge is given. This issue, known as the exploration challenge, gives rise to the idea of providing expert knowledge from demonstration data to an RL agent~\cite{ramirez2022model}. Yet, how to make full effective of demonstrations to improve exploration efficiency still remains an open challenge.

One classical approach is to give higher sampling priority to demonstrated data over self-collected data in a simulator~\cite{levine2013guided,dqfd,ddpgfd}. While straightforward, the expert knowledge from demonstrations is not explicitly exploited during the exploration, making these methods still inefficient in robotic tasks. To provide explicit exploration guidance, one other approach offers RL agent with an additional reward function learned from demonstrations, which encourages the agent to stay close to expert samples~\cite{zhu,deepmimic,amp} by discerning expert-like and -unlike behaviors. However, these methods are not easily applicable to surgical tasks because they not only require careful task-specific design of environment reward function~\cite{tissue_lfd}, but also introduces the risk of local optima during policy learning.

\begin{figure}[t]
    \centering
    \vspace{0.3cm}
    \centerline{\includegraphics[width=1\linewidth]{fig/cover_page.pdf}}
    \vspace{-1.35cm}
    \caption{Our method DEX leverages demonstration data to efficiently guide the RL exploration in simulation tasks. The learned policies are successfully deployed to the real-world surgery subtasks on the dVRK platform.}
    \label{fig:overview}
    \vspace{0cm}
\end{figure}

To overcome such limitations, a promising approach is to directly regularize the robot policy to mimic the expert policy within the actor-critic framework~\cite{ddpgher,dapg,zhu2019dexterous,shah2021rrl,col}. Such guidance is typically operationalized by regularizing the actor loss with a penalty measuring the behavioral dissimilarity between the robot and the expert. Effective as these methods are, their exploration efficiency is still unsatisfactory due to two limitations. Firstly, these methods enforce little regularization on the critic, making the critic suffers from the notorious overestimation issue~\cite{td3,doubleql}. Without an accurate critic, the overestimated values of expert-unlike actions hinder the robot from exploring expert-like actions. Secondly, these methods provide effective guidance only when the current state is close to the ones in the demonstrations. This impedes their capability to guide exploration early in the learning process, as the robot with a mediocre policy is likely to visit states far from demonstrated ones. A stopgap is to cover as many possible demonstrated states as possible by collecting a large number of demonstrations~\cite{chiu2021bimanual}. Unfortunately, collecting many surgical expert data is often unaffordable due to the resources and ethical constraints.

In this work, we aim to improve the exploration efficiency of RL given a modest set of demonstration data for task automation of surgical robot. To tackle the overestimation issue of the critic, we propose a novel algorithmic framework, named expert-guided actor-critic, that regularizes both actor and critic with the action dissimilarity between agent and expert. As illustrated in Figure~\ref{fig:overview}, at a high level, with this regularization the critic lowers the value estimates of expert-unlike actions (colored with red), thus further encouraging the exploration on expert-like actions (colored with green). To guide exploration at states unobserved in demonstrations, we adopt a non-parametric regression model to robustly propagate the guidance (dashed arrow) from demonstration data to these states. Consequently, our method, named Demonstration-guided EXploration ({DEX}), is of high exploration efficiency with a limited amount of demonstration data. The empirical results on surgical automation tasks from SurRoL~\cite{surrol} show that~{DEX}~significantly outperforms prior learning-based methods. We also validate the transferability of learned policies on da Vinci surgical robots. The contributions of this work are summarized as follows:
\begin{itemize}
    \item We propose a novel actor-critic framework to mitigate the overestimation issue of the critic for encouraging explorations on expert-like actions in RL;
    \item We adopt non-parametric guidance propagation to guide exploration at demonstration-unobserved states;
    \item We empirically demonstrate that our method significantly outperforms prior RL-based approaches on the surgical robot learning tasks from SurRoL;
    \item We successfully deploy our trained policies on the real dVRK, indicating its great potential for task automation of surgical robot in the real world.
\end{itemize}

\section{Related Work}\label{sec:related_work}
\noindent\textbf{Surgical Robot Task Automation.} Considerable efforts have been made by researchers to design skill-oriented controllers for surgical task automation, including surgical suturing~\cite{leonard2014smart,schwaner2021autonomous}, endoscope control~\cite{endoscope,gao2022savanet}, tissue manipulation~\cite{attanasio2020autonomous,saeidi2022autonomous} and pattern cutting~\cite{patel2018using,cutting2}. However, these works follow a traditional paradigm that requires domain knowledge for designing control strategies and suffers from low generality, i.e., a controller designed for one task is often hard to generalize~\cite{ras}. In contrast, learning-based approaches, representatively RL, enable robots to flexibly learn to perform tasks from collected data. These methods thus do not require task-specific control strategies and have shown improved generalization capabilities in automating complex surgical tasks~\cite{su2020reinforcement,srl_reward1,scheikl2021cooperative,segato2021inverse,tan2019robot,keller2020optical,d2022learning}. Promising as it is, running existing RL methods in surgical tasks is still inefficient due to the exploration challenge, which is often alleviated through substantial reward engineering for each task. To this end, we propose a novel RL algorithm to solve diverse tasks efficiently by exploiting expert demonstrations for facilitating task automation in robotic surgery.

\noindent\textbf{Learning from Demonstrations.} Imitation learning (IL), also known as learning from demonstrations~\cite{argall2009survey}, is a typical learning-based framework that leverages demonstrations for policy learning. A well-known approach is behavior cloning~(BC,~\cite{bc}) which supervises the agent to imitate expert behavior. However, it suffers from the distribution shift problem and thus is of poor generalization performance~\cite{bc_compouding,osa2018algorithmic}. Some IL methods overcome this issue by using generative adversarial networks to match the trajectory distribution of demonstrations~\cite{gail, ding2019goal} or provide the agent with inferred reward function~\cite{meirl, airl}. But they require a great many online samples and demonstrations to ensure fair generalization capability and suffer from poor training stability~\cite{dac}. In contrast, our method goes beyond such limitations by utilizing the expert knowledge from demonstration data to expedite RL exploration, which combines advances of IL and RL and alleviates their respective drawbacks.

\noindent\textbf{Demonstration-Guided RL.}
A number of demonstration-guided RL approaches have been proposed to mitigate the exploration challenges of RL, including prioritizing expert samples over self-collected samples~\cite{levine2013guided,dqfd,ddpgfd} or learning extra reward functions~\cite{zhu,deepmimic,amp}. However, these methods either make no explicit use of demonstrations or still need careful reward engineering in surgical tasks. To overcome these issues, a promising approach is to leverage demonstrations to directly regularize the RL policy to explore expert-like behaviors~\cite{ddpgher,dapg,zhu2019dexterous,shah2021rrl,col}. It penalizes the behavioral dissimilarity between agent and expert with state-action pairs from demonstrations. Nonetheless, this approach often requires a large number of demonstrations to ensure high exploration efficiency due to the impediment of overestimated values and the incapability of offering guidance at states far from the demonstrated ones. Our method addresses these limitations through critic regularization in the proposed actor-critic framework and robust guidance propagation based on non-parametric regression.

\section{Method}
\begin{figure*}[t]
\centering
\centerline{\includegraphics[width=1\textwidth]{fig/framework.pdf}}
\caption{\textbf{RL with demonstrations-guided exploration.} We present the overall illustration of our proposed method DEX. It consists of two parts, a novel actor-critic based policy learning module that efficiently leverages demonstration date to guide the RL exploration (right), and a non-parametric module based on nearest-neighbor matching and locally weighted regression for robust guidance propagation at states far from demonstrated ones (upper left).}
\label{fig:framwork}
\vspace{-0cm}
\end{figure*}

We develop Demonstration-guided EXploration~(DEX), a novel exploration-efficient demonstration-guided RL algorithm for surgical subtask automation with limited demonstrations. Our method addresses the potential overestimation issue in existing methods based on our proposed actor-critic framework in Section~\ref{subsec:drrl}. To offer exploration guidance at states unobserved in demonstration data, our method propagates the guidance from demonstrations through a non-parametric module in Section~\ref{subsec:lwl}. Figure~\ref{fig:framwork} illustrates the overall framework of the proposed method DEX.

\noindent\textbf{Problem Formulation: }We consider an off-policy RL agent that interacts within an environment formulated by a Markov decision process. At time step $t$, the agent takes an action $a_t$ based on the current state $s_t$ and its deterministic policy $\pi$. The environment then rewards the agent with $r_t:=r(s_t,a_t)$ and transits it to the successor state $s_{t+1}$. After each transition, the agent stores the experience $(s_t, a_t, r_t, s_{t+1})$ into a replay buffer ${\mathcal{D}_A}$. Meanwhile, it also maintains a separate demonstration buffer ${\mathcal{D}_E}$, where experiences are collected through an unavailable expert policy $\pi^e$ in advance.

\subsection{Expert-Guided Actor-Critic Framework}\label{subsec:drrl}
Many existing actor-critic based methods learn optimal policy by training an actor that maximizes the expected return $\mathbb{E}_\pi[\sum_{t=0}^\infty\gamma^tr_t]$, which is estimated by a critic that approximates the Q-value function, where $\gamma\in(0,1]$ is a discount factor. Their common way to guide exploration is to jointly maximize the Q-value and the behavioral similarity between the agent and expert. Effective as it is, the inaccurate Q-value estimates of expert-unlike actions may dominate the guidance embodied as behavioral dissimilarity, and thus hamper the exploration, reminiscent of the regularization issue in offline RL literature~\cite{kostrikov2021offline,fujimoto2019off}. In other words, the agent is likely to take unproductive actions due to value misestimate, which hinders it from exploring expert-like actions.

To remedy this issue, we introduce a novel RL objective that augments the environment reward with the behavior gap between agent policy and expert policy:
\begin{align}\label{eq:obj}
    \max_{\pi} \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t(r_t-\alpha d(a_t, a^e_t)) \right], \;\; a^e_t:={\pi^e}(s_t),
\end{align}
where $\alpha$ is an exploration coefficient and the function $d(\cdot,\cdot)$ is a distance metric that measures the behavioral similarity between agent action and expert action. Correspondingly, we define the regularized Q-value function as $Q^\pi(s,a) := \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t(r_t-\gamma\alpha d(a_{t+1}, a^e_{t+1}))\big|s, a \right]\label{eq:qvalue}$. It assigns higher value estimates to actions similar to the expert action at any given state. It thus mitigates the effect of the inaccurate estimates of undesired actions by lowering its value through a dissimilarity regularization. Meanwhile, echoed with SAC~\cite{sac}, an appropriate choice of $\alpha$ steers the exploration-exploitation balance and benefits policy learning.

Based on the formulation, we introduce our proposed expert-guided actor-critic frameworks. The regularized value function (critic) $Q_\theta$, parameterized by $\theta$, minimizes the squared Bellman residual:
\begin{align}\label{eq:closs}
    \mathcal{L}_Q(\theta) = \mathbb{E}_{{\mathcal{D}_A}\cup{\mathcal{D}_E}}\left[ \left(r_t+\gamma V_{\bar{\theta}}(s_{t+1}) - Q_\theta(s_t,a_t)\right)^2\right],
\end{align}
where $V_{\bar{\theta}}$ is state value function served as Q-target whose parameters $\bar{\theta}$ are the exponential moving average of $\theta$ that stabilize the training process. The actor, parametrized by $\phi$, is trained by maximizing following the objective:
\begin{align}\label{eq:aloss}
    \mathcal{L}_\pi(\phi) = \mathbb{E}_{{\mathcal{D}_A}\cup{\mathcal{D}_E}}\left[Q_\theta(s_t, \pi_\phi(s_t))-\alpha d(\pi_\phi(s_t), a_t^e))\right].
\end{align}

Our framework bears some resemblances to behavior regularized actor-critic in offline RL domain~\cite{awac,brac}. At the online fine-tuning stage, their formulas target conservative policy updates by constraining the behavior gap between old and new policies. Different from their focus, our framework treats expert policy as a static reference policy and uses the behavior gap between agent and expert as a regularization term to accelerate the RL exploration more efficiently.

\subsection{Guidance Propagation from Limited Demonstrations}\label{subsec:lwl}
Prior methods provide the actor with exploration guidance only at states observed in demonstration data, since the expert action $a_t^e$ is available only therein. However, the agent is prone to visit regions uncovered by demonstrations at the initial stage of training, where the demonstrations are incapable of supervising the actor exploration. On the other hand, such an issue also limits the feasibility and efficacy of critic regularization. Only regularizing the demonstration-covered Q-value introduces an underestimation bias on the value estimate to these state-action pairs due to the negativity of the regularization term. This negates the original purpose of enforcing critic regularization. Propagating guidance from limited demonstrations is thus of high necessity to efficiently realize the proposed actor-critic framework.

One natural solution is to learn a parametric expert policy approximator $\hat{\pi}^e$ from demonstrations in an parametric manner, e.g., behavior cloning. Although using a parametric model already outperforms existing methods, which demonstrates the effectiveness of our framework, we empirically observe that the parametric propagated guidance, i.e., $\hat{\pi}(s)$, fairly differs from the ground-truth expert action when states are far from the demonstration ones. We thus resort to a non-parametric regression model that empirically propagates more robust guidance from limited demonstrations.

Specifically, we first sample a minibatch of states and actions from demonstration buffer ${\mathcal{D}_E}$. Then, given any state $s$, we search states in the minibatch and find $k$ nearest neighbors of $s$ based on the Euclidean distance. The selected $k$ states and their associated actions are denoted by $\{(s^{(i)}, a^{(i)})\}^k_{i=1}$, a set of transitions from ${\mathcal{D}_E}$. Subsequently, we approximate the expert policy through locally weighted regression~\cite{atkeson1997locally} with exponential kernel function:
\begin{align}\label{eq:lwl}
    \hat{\pi}^e(s) = \frac{\sum_{i=1}^k \mathrm{exp}\left(-\|s-s^{(i)}\|_2\right)\cdot a^{(i)}}{\sum_{i=1}^k \mathrm{exp}\left(-\|s-s^{(i)}\|_2 )\right)}.
\end{align}
The assumption here is that similar states share similar optimal actions, whose effectiveness and robustness have been demonstrated in some robotic learning tasks~\cite{vinn,fist,dime}. Unlike these IL methods, we aim to use the limited demonstration data to accelerate the exploration of RL and increase the generalization ability of policy.

\subsection{Implementation Details}
We implement our method with deep deterministic policy gradient (DDPG,~\cite{ddpg}) from OpenAI Baselines~\cite{baselines} and adopt their main hyperparamter settings. We choose DDPG to avoid the potential pathology of density-based methods~\cite{rudner2021pathologies}, though our implementation can be naturally extended to the stochastic setting.
Specifically, both actor and critic are parameterized as four fully-connected layers of hidden dimension 256 interpolated with \texttt{ReLU} activations, where actions are scaled to the range [-1,1] by a \texttt{Tanh} activation in the actor network. They are trained with ADAM~\cite{adam}. The exploration noise is set as Gaussian noise with a scale factor of 0.1,  For goal-conditioned environments, we adopt hindsight experience replay~\cite{her} with future sampling strategy on 80\% examples for both replay buffer and demonstration buffer. The distance metric is set as $L2$-norm for all tasks. The coefficient of exploration term $\alpha$ is set as $5$ for all tasks. The number of nearest neighbors $k$ used for expert policy estimation is set to $5$. More analyses on $\alpha$ and $k$ would be given in the ablation studies.
\section{Experiments}
In this section, we first conduct experiments on SurRoL~\cite{surrol}, a comprehensive simulation environment for task automation of surgical robots. Extensive evaluation and ablation analysis on 10 challenging surgical manipulation tasks demonstrate the effectiveness of our method DEX for surgical robot learning. Moreover, we transfer the learned policies to a dVRK platform for real-world robotic deployment.

\begin{figure}[t]
\centering
\centerline{\includegraphics[width=1\linewidth]{fig/task.pdf}}
\caption{\textbf{Task description of SurRoL.} We choose ten surgical tasks from SurRoL and divide them into three domains according to the type of manipulators. The arrows in each task represent the task flow as well as the execution process of the scripted expert policy provided by the platform.}
\label{fig:surrol}
\end{figure}

\input{table/main_results}

\subsection{Experimental Setup}\label{subsec:setup}

\noindent\textbf{Different Tasks on Surgical Manipulation.} To evaluate whether our method DEX can effectively handle different surgical manipulation scenarios, we conduct experiments on 10 challenging tasks from SurRoL. These 10 tasks widely cover three different domains according to the type of manipulator: (1) Single-handed patient-sided manipulator (\textbf{PSM}); (2) Bimanual PSM (\textbf{Bi-PSM}); and (3) Endoscopic camera manipulator (\textbf{ECM}). Figure~\ref{fig:surrol} illustrates the involved 10 manipulation tasks. All tasks are goal-conditioned with sparse reward functions that indicate task success, except for \emph{ActiveTrack}, whose reward function is dense that depends on the precision of object tracking. We use low-dimensional state representation that consists of object state (3D Cartesian positions and 6D pose) and robot proprioceptive state (jaw status and end-effector position), and Cartesian-space control as action space. Please refer to~\cite{surrol} for more details on the different surgical manipulation tasks. Besides, we sample 100 successful episodes of demonstration data for each task through the scripted expert policy provided by SurRoL.

\noindent\textbf{Evaluation Metrics.} Similar to~\cite{curl,drq}, we measure the manipulation performance of each task after a uniform 100k environment steps. The manipulation score for each task is linearly re-scaled to be within $[0,1]$ for comparisons across tasks as well as better visualization. Following~\cite{rliable}, we adopt interquartile mean~(IQM) and 95\% interval estimates via stratified bootstrap confidence intervals~(CIs) to measure the aggregate performance over each domain robustly.

\noindent\textbf{Baselines.} First of all, we compare our method with two state-of-the-art pure RL algorithms without using demonstrations: (1) \textbf{SAC}~\cite{sac} and (2) \textbf{DDPG}~\cite{ddpg}. Besides, we compare our method with three representative IL methods: (3) \textbf{BC}~\cite{bc} that supervises the agent to imitate expert actions from demonstrations. (4) \textbf{SQIL}~\cite{sqil} that reduces IL to RL by assigning positive rewards to demonstrations, a representative of the state-of-the-art IL algorithms based on SAC. (5) \textbf{VINN}~\cite{vinn} that directly uses the estimate of expert policy in Equation~\eqref{eq:lwl} as the agent policy. Moreover, we compare our method with four existing demonstration-guided RL algorithms: (6) \textbf{DDPGBC}~\cite{ddpgher} that guides DDPG actor with Q-filtered BC loss. (7) \textbf{AMP}~\cite{amp} that augments environment with GAIL~\cite{gail} reward based on SAC. (8) \textbf{CoL}~\cite{col} that initializes the policy with BC offline and integrates BC with DDPG in the online training stage. (9) \textbf{AWAC}~\cite{awac} that trains an offline RL with demonstrations and online fine-tunes it with a conservative constraint between current and old policies.

\input{table/main_demo}
\subsection{Main Results on SurRoL Platform}\label{subsec:main_results}
We present the aggregate and distributed performance of each method in Table~\ref{tab:main}. The results show that two pure RL methods achieve great performance in ECM domain, but poor performance in PSM and Bi-PSM. This implies that their exploration strategies are not applicable in complex surgical manipulation tasks, while our demonstration-depend exploration is able to handle these tasks. Similar to pure RL, three IL methods perform well in ECM domain, while their performance decreases dramatically in the PSM and Bi-PSM domain, which has a higher dimension of state and action space and requires more complex manipulation skills. This indicates that, given a limited number of demonstrations, the generalization ability of IL is severely deteriorated by the complexity of the task. Compared to IL, our method not only perform similarly great in ECM domain, but achieves \textbf{2.2x} IQM in PSM domain and, especially, \textbf{4.9x} in Bi-PSM domain. This demonstrates the advances of our method that leverages demonstrations to accelerate RL exploration.

Meanwhile, we also observe that three demonstration-guided RL methods, except for AMP which suffers from the instability issue of adversarial training~\cite{osa2018algorithmic}, perform favorably in ECM and PSM domains. This may attribute to their utilization of demonstration for RL exploration. However, these methods still get stuck in the complex PSM tasks, such as \emph{PegTransfer}, let alone the Bi-PSM domain that additionally demands coordination skills between two manipulators. Compared with them, our method fully leverages demonstrations to mitigate the guidance impediment of critic and guide the exploration efficiently at demonstration-uncovered states, leading to a prominent performance improvement of \textbf{0.39} in the Bi-PSM domain.

\begin{figure}[t]
\centering
\vspace{1mm}
\centerline{\includegraphics[width=1\linewidth]{fig/ablations.pdf}}\vspace{0mm}
\caption{\textbf{Ablation studies.} In (a), we propose two variants of {DEX}~that partially regularize the actor-critic to investigate the effectiveness of our proposed critic regularization. In (b), we ablate the choice of exploration coefficient $\alpha$. In (c), we test the influence of the number of nearest neighbors $k$ in non-parametric propagation. All results are IQMs with 95\% stratified bootstrap CIs (error bar) over 5 runs on PSM and Bi-PSM domains. }
\label{fig:ablations}
\vspace{-0.2mm}
\end{figure}

\begin{figure*}[t]
\centering
\centerline{\includegraphics[width=1\textwidth]{fig/dvrk.pdf}}
\caption{\textbf{Deployment on real dVRK.} We deploy the learned policies on the real dVRK platform (left) and present the generated trajectories of five representative tasks (right). The successful deployment demonstrates the transfer ability of our method from simulation to real-world surgical scenarios.}
\vspace{0cm}
\label{fig:dvrk}
\end{figure*}

\subsection{Ablation Studies}\label{subsec:abl}
\noindent\textbf{Effect of Demonstration Amount.} We further study the manipulation performance of each method with different amounts of demonstrations. We train agents with $10\%$, $25\%$, $50\%$, $75\%$ and $100\%$ of demonstrations, respectively. Table~\ref{tab:main_demo} presents the aggregate results over PSM and Bi-PSM domains. Our proposed method DEX consistently outperforms other competing methods with different amounts of demonstrations. This result indicates that with the same amount of demonstrations, our DEX can utilize demonstrations more efficiently to achieve a higher manipulation performance. More importantly, our proposed DEX exhibits a lower demand for the number of demonstrations for task automation. Even with merely $25\%$ of demonstrations, DEX has outperformed most competing methods that are trained with $100\%$ of demonstrations. This characteristic demonstrates that DEX is well suited for the challenging surgical scenarios where collecting large amounts of demonstrations is unaffordable.

\noindent\textbf{Effect of Critic Regularization.} While some existing methods only impose the guidance on actor~\cite{ddpgher,col,chiu2021bimanual}, our method guide both actor and critic to remedy the guidance impediments of the ordinary critic. In order to further investigate the effects of critic regularization on the manipulation performance, we propose two variants of {DEX}~that partially regularize the actor-critic modules in Figure~\ref{fig:ablations}(a). The results show that, compared with the ordinary no-guidance actor-critic (DEX-AC), the variant that only regularizes actor (DEX-RA) brings an improvement of 0.20 w.r.t. IQM,
and the proposed one which regularizes both actor and critic (DEX) brings 0.39. It demonstrates the impediment of inaccurate value estimates of ordinary critic and the effectiveness of our proposed critic regularization to alleviate this issue.

\noindent\textbf{Effect of Exploration Coefficient $\alpha$.} The exploration coefficient $\alpha$ determines the relative importance of the exploration term against environment reward. We investigate the effect of this parameter by gradually increasing the value of $\alpha$ from $0$ to $20$ and repeatedly train the agent with the remaining settings identical. Figure~\ref{fig:ablations}(b) present the experiment results. It shows that $\alpha$ around $5$ yields the best performance, indicating that an intermediate choice balances exploration and exploitation well. We also note that an adaptive coefficient (e.g., decay $\alpha$) may further balance the trade-off to avoid local minima and leave it as future work.

\noindent\textbf{Effect of Guidance Propagation.} To investigate the effect of guidance propagation, we propose a parametric variant (DEX-BC) that pre-trains a BC model to approximate the expert action at demonstration-uncovered states. The results show that, compared with no-propagation methods, DEX-BC still achieves a performance improvement of 0.3 IQM over PSM and Bi-PSM domains, though is 0.05 lower than DEX. It demonstrates not only the robustness of non-parametric guidance propagation but the insensitivity to the choice of propagation method in our actor-critic framework.

\noindent\textbf{Effect of $k$ for Guidance Propagation.} We investigate the influence of the number of nearest neighbors $k$ by gradually increasing its value from 1 to 9. The results in Figure~\ref{fig:ablations}(c) show that too large value of $k$ leads to a slight performance drop and an intermediate value around 5 works well, indicating that our method is not sensitive to the choice of $k$ within an appropriate interval.

\input{table/dvrk}
\subsection{Deployment on Real Robot Platform of dVRK}\label{subsec:real_dvrk}
To validate the transfer ability of our method, we train RL policies in SurRoL and deploy them on dVRK systems. We conduct real-world manipulation experiments on five representative tasks, namely \emph{GauzeRetrieve}, \emph{NeedlePick} and \emph{PegTransfer} in PSM domain, \emph{NeedleRegrasp} in Bi-PSM domain, and \emph{StaticTrack} in ECM domain. These tasks are selected to be diverse and comprehensive, requiring different levels of surgical manipulation skills. We train policies with 300k steps within SurRoL and select checkpoints with the best performance for deployment. We present experiment snapshots in Figure~\ref{fig:dvrk}. In three PSM tasks, we observe that agents successfully learn skills of reaching, picking, and placing to accomplish the task without collisions. In \emph{NeedleRegrasp}, the policy additionally acquires coordination skills between two manipulators compared with policies in the PSM domain. In \emph{StaticTrack}, the trained policy successfully learns endoscopic control with a high success rate. Meanwhile, we also compare our method with one imitation learning method BC and one competitive demonstration-guided RL method DDPGBC. The results in Table~\ref{tab:dvrk} show that our method achieves the highest success rates at all tasks with 20 trials, which demonstrates the great potential of our method for real-world surgical task automation.

\section{Conclusion}
\label{sec:conclusion}
We present DEX, a novel demonstration-guided RL method that narrows down exploration space by encouraging expert-like behaviors and enabling robust guidance when confronting states unobserved in demonstrations, to improve exploration efficiency with a modest set of demonstrations. We first demonstrate its performance improvement on 10 challenging surgical manipulation tasks compared with state-of-the-art methods on the SurRoL platform. We also deploy the trained policy to the dVRK platform to show its potential for transferring to real-world surgical automation scenarios.

Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)


Diffuse Thermal Emission from Very Hot Gas in Starburst Galaxies: Spectral Results

Abstract: New BeppoSAX observations of the nearby archetypical starburst galaxies (SBGs) NGC253 and M82 are presented. The main observational result is the unambiguous evidence that the hard (2-10 keV) component is (mostly) produced in both galaxies by thermal emission from a metal-poor (~ 0.1-0.3 solar), hot (kT \~ 6- 9 keV) and extended (see companion paper: Cappi et al. 1998) plasma. Possible origins of this newly discovered component are briefly discussed. A remarkable similarity with the (Milky Way) Galactic Ridge's X-ray emission suggests, nevertheless, a common physical mechanism.

\section{Main Interests}

The main interest of studying SBGs in the local universe
is that it is believed that the starburst phase was very frequent
during the early ($z \hbox{\raise0.5ex\hbox{$>\lower1.06ex\hbox{$\kern-1.07em{\sim}$}$}} 1$) evolution of galaxies (Madau et al. 1996).
Thus understanding the properties of local
SBGs means, presumably, understanding the properties of most galaxies of the
early universe.
Studies of SBGs can also help to shed light on key astrophysical issues like: i) the
AGN/starburst connections, since both phenomena require concentration of
material to the nuclear regions and are possibly linked to merging phenomena and/or
presence of molecular bars (Maiolino et al. 1998), ii) the chemical enrichment
of the intergalactic medium (IGM), because of the frequent presence  in SBGs
of chemically-enriched SB-driven outflows (Heckman 1997) and iii) the structure,
evolution and even
formation of galaxies through the effect of the ``feedback'' provided by massive
stars (Ikeuchi \& Norman 1991).

\vspace{-0.4cm}
\section{Current Observational Status}

Because of the short space available, we limit the following discussion to the
2--10 keV energy band (see Fabbiano 1989 for a review on the optical and soft
X-ray properties of SBGs).
In this energy range, the best-quality data available to date have been
essentially limited to the two brightest SBGs NGC 253 and M82 (but see recent
studies on lower-luminosity galaxies by Della Ceca et al. 1997). Data from
the $Einstein$ IPC, $EXOSAT$ ME, $Ginga$, $BBXRT$ and $ASCA$
(Ptak et al. 1997 and ref. therein; Tsuru et al. 1997, Moran \& Lehnert
1997) have clearly demonstrated the existence of a hard component well described by
either a thermal model (kT $\sim$ 6--9 keV) or a power-law model ($\Gamma$
$\sim$ 1.8--2.0). The absence in the data of a significant Fe-K line emission
has, however, always been puzzling and has induced several authors to propose
explanations alternative to the thermal emission, i.e. the presence of a LLAGN
(Ptak et al. 1997, Tsuru et al. 1997) or inverse-Compton (IC)
scattered emission from the interaction of IR photons with SNe-generated
relativistic e$^{-}$ (Moran \& Lehnert 1997).

In the following, we present new $BeppoSAX$ results for these two
SBGs which demonstrate that
such hard component is most probably emitted by a thermal plasma. In this paper,
spectral results showing the first evidences of Fe-K line emission (at $\sim$ 6.7 keV)
and high-energy steepening expected in the case of thermal emission are presented for both
NGC 253 and M82 (see Persic et al. 1998 for more details on the results for NGC 253).
In a companion paper (Cappi et al. 1998), we combine these spectral results
to the (unprecedented) spatial information of $BeppoSAX$ between 2--10 keV.

\vspace{-0.4cm}
\section{BeppoSAX spectra of NGC 253 and M82}

$BeppoSAX$ observed NGC 253 on Nov.29--Dec.2, 1996 for $\sim$ 50, 113 and 50 Ksec
and M82 on Dec.06--07, 1997 for $\sim$ 29, 85 and 30 Ksec, for the LECS, MECS and PDS
operating between 0.1--4 keV, 1.3--10 keV and 13--60 keV, respectively.
A Galactic column density of 1.28 $\times$ 10$^{20}$ cm$^{-2}$ for NGC 253 and
4.27 $\times$ 10$^{20}$ cm$^{-2}$ for M82 were added in all the following spectral
fits (Dickey \& Lockman 1990).
The spectral analysis was performed using version 10.00 of the XSPEC program.

\vspace{-0.5cm}
\begin{table}[hbt]
\begin{center}
{\bf Table1:} Fe K line parameters \\
\begin{tabular}{cccc}
\hline
Source 	& kT$_{\rm brem}$ 	& Obs. Energy 	& EW \\
	& keV		& keV		& eV \\
\hline
NGC 253 & 7.40$^{+0.18}_{-0.71}$	& 6.69$^{+0.07}_{-0.07}$ & 329$^{+89}_{-109}$ \\
M82	& 9.27$^{+0.41}_{-0.51}$ & 6.65$^{+0.22}_{-0.19}$& 60$^{+35}_{-30}$ \\
\hline
\end{tabular}
\end{center}
\end{table}

\vspace{-0.5cm}
Several continuum models (e.g. a power-law, a bremsstrahlung, a Raymond-Smith,
a multi-temperature Raymond-Smith, a Mewe-Kaastra-Liedahl model)
and combination of them with and without absorption,
with solar and variable abundances have been used to model the spectra (Fig. 1).
We have found that both spectra are best described in terms of a
two-component thermal model (Fig. 2). At energies
lower than $\sim$ 3 keV, emission lines (e.g. FeL, SiXV and SXV) characteristic
of a soft thermal component are detected, in agreement with $ASCA$ results.
At higher energies, both sources require
the continuum emission to steepen (thus excluding a power-law description) and, most
significantly, an Fe-K emission line at E $\sim$ 6.7 keV is detected in both
objects (see Table 1 and Fig. 1).
Interestingly, both hard components need to be absorbed in order not to
over-produce the continuum at E $\hbox{\raise0.5ex\hbox{$<\lower1.06ex\hbox{$\kern-1.07em{\sim}$}$}}$1 keV (as also found with $ASCA$, Ptak et al. 1997).

Best-fit results obtained by fitting both source spectra
with a two-temperature Mewe-Kaastra-Liedahl model with variable abundances
(``vmekal'' in XSPEC) are shown in Table 2 and Fig. 2.
Following Persic et al. (1998), in order to reduce the number of free parameters,
the abundances of He, C, and N were fixed at the solar value and the heavier
elements divided into two groups: Fe and Ni (most likely associated
to SNe I products), and the $\alpha$-elements O, Ne, Mg, Si, S, Ar and Ca
(most likely associated to SNe II products); elements in the same group
were constrained to have common abundances in solar units.
It should be noted that although some extra contribution from a non-thermal
(hard) component cannot be formaly excluded, we estimate that it should not
contribute more than $\sim$ 20\% of the total 2--10 keV flux.

The detection of the Fe K line allows us to reliably determine the Fe abundances
of the line-emitting matter: we find a value of $\sim$ 0.25 and $0.06$ solar for NGC 253
and M82, respectively, consistent with the sub-solar values (based on upper-limits)
predicted from previous hard X-ray observations (see \S 2). Although more insecure
because of the current uncertainties of theoretical models in computing the
Fe-L emissivity, similar Fe abundances are found for the soft component.
On the other side, $\alpha$-elements seem to have larger abundances ($\sim$ 1.7 solar
for NGC 253 and $\sim$ 2.4 solar for M82).

\vspace{-0.5cm}
\begin{table}[htb]
\begin{center}
{\bf Table 2:} Two-components (vmekal) thermal models \\
\vspace{-0.2cm}
\begin{tabular}{cccccccc}
&&&& \\
\hline
Source & kT$_{soft}$ & Ab$_{soft}$ &  $N_{\rm H}^{\rm hard}$ & kT$_{hard}$ &
Ab$_{hard}$ & $\chi^2/dof$ & log L \\
 & keV & $\alpha-$el. &  $\times 10^{22}$ & keV & $\alpha-$el. & & 2-10 keV\\
 & &  Fe, Ni & cm$^{-2}$ & & Fe, Ni & & erg/s \\
\hline
NGC253 & $0.88_{-0.13}^{+0.13}$ & $1.69_{-0.72}^{+0.02}$ & $1.30_{-0.45}^{+0.63}$ &
$5.60_{-0.60}^{+0.70}$ & $\leq 1.76$ & 244/237 & 1.6\\
& & $0.18_{-0.07}^{+0.29}$ & & & $0.27_{-0.08}^{+0.10}$ & &  \\
&&&&&& &  \\
 M82 & $0.70_{-0.05}^{+0.05}$ & $2.40_{-0.95}^{+1.31}$ & $0.51_{-0.17}^{+0.18}$
& $8.77_{-0.66}^{+0.53}$ & = Ab$_{soft}$ & 366/354 & 5.6 \\
& & $0.62_{-0.23}^{+0.40}$ & & & $0.07_{-0.03}^{+0.05}$ & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\vspace{-0.9cm}
\section{On the origin of the hard, thermal, component}

On one side, our results for the soft X-ray component are in good
agreement with previous X-ray results on SBGs (see \S 2), i.e. that it originates from
a SB-driven galactic superwind (see Persic et al. 1998 for more details).
On the other side, the origin of the hard thermal component is more puzzling.
As demonstrated in Cappi et al. (1998), it is also extended possibly flowing
out of the galaxy plane (as for the soft X-rays) in the case of NGC 253.
Among possible candidates, one could consider a collection of point sources
(e.g. X-ray binaries, SNRs, clusters of protostars) or truly diffuse emission
due to IC scattering of IR-optical
photons from relativistic e$^{-}$ or, finally, a very hot
($\sim$ 6--10 $\times$ 10$^7$ K) ISM phase. Alternatives to the hot ISM phase, however,
could hardly be reconciled with the observed spectrum because i) the average spectra of X-ray
binaries would predict a power-law continuum and Fe K line at 6.4 keV, ii) the average
spectra of SNRs would predict too softer (kT$\hbox{\raise0.5ex\hbox{$<\lower1.06ex\hbox{$\kern-1.07em{\sim}$}$}}$ 4 keV) spectra, iii) clusters
of protostars have too low luminosities (L$_{\rm 2-10 keV}$ $\sim$ 10$^{32}$ erg s$^{-1}$)
and iv) IC emission would predict a power-law spectrum.
The most natural explanation seems therefore a hot thermal diffuse emission, i.e.
a ``super super-wind'' most likely powered by the starburst.

It is interesting to note the striking analogy of the present results with
the $ASCA$ observations of the Galactic Ridge (GR) (compare our Fig. 2 with
Fig. 3 in Kaneda et al. 1997): also in this case, hot diffuse (and absorbed)
emission is detected at a similar temperature and, if scaled for a
$\sim$ 1/100 times lower Galactic star formation rate, with similar energetics. The only
difference being in the somewhat larger ($\sim$ 0.8 solar) metal abundances
deduced for the GR, these remarkable similarities strongly suggest a common
physical origin.

\vspace{-0.4truecm}
\section{Conclusions}

\vspace{-0.1cm}
The main conclusion from the above results is that the $BeppoSAX$ observations
of the two archetypical SBGs NGC 253 and M82 have revealed for the first
time strong evidence of a hot, really hot, thermal diffuse plasma
responsible for their hard (2--10 keV)
emission, and that this discovery
may have profound implications on AGN/starburst connections,
on the chemical enrichment of the IGM and on the formation and evolution of galaxies.

\begin{figure}
\vspace{-0.2cm}
\parbox{6.5truecm}
{\psfig{file=./cappi_fig1a.ps,width=6.5cm,height=5cm,angle=-90}}
\parbox{6.5truecm}
{\psfig{file=./cappi_fig1b.ps,width=6.5cm,height=5cm,angle=-90}}
\vspace{-0.2cm}
\caption[h]{Spectra of NGC 253 and M82 fitted with a single power-law model ($\Gamma$
$\sim$ 1.7) to illustrate the spectral complexities (emission lines and continuum curvature)}
\end{figure}
\begin{figure}
\vspace{-0.3cm}
\parbox{6.5truecm}
{\psfig{file=./cappi_fig2a.ps,width=6.5cm,height=5cm,angle=-90}}
\parbox{6.5truecm}
{\psfig{file=./cappi_fig2b.ps,width=6.5cm,height=5cm,angle=-90}}
\vspace{-0.2cm}
\caption[h]{Best-fit spectra of NGC 253 (left) and M 82 (right) with a
2T Mekal model (Table 2).}
\end{figure}

\vspace{-0.4cm}