RL Environment and Agent. The next task consisted in the implementation of the reinforcement learning agent
and the interface for interaction with the simulation.
The state space consisted of the expert and learner states, thus,
the environment's state \(\mathbf {s}\)  is defined by the tuple \(\mathbf {s}_t = \lbrace \mathbf {s}(\mathbf {q}_t, \dot{\mathbf {q}}_t), \hat{\mathbf {s}}(\hat{\mathbf {q}}_t, \dot{\hat{\mathbf {q}}}_t)\rbrace \) . The actions are given by the torque commands of the learner, \(\hat{\mathbf {a}}_t = \hat{\mathbf {\tau }}_t\) , subject to the torque limits for each corresponding joint. The control of the expert is not observable by the RL agent. For training and testing, multiple random trajectories of similar duration were recorded.
The step size was set to \(\Delta t=0.1\) s and a step consisted of the following transitions:
The agent executed an action in the environment by sending a torque command to the learner. The torques were then applied for the duration of the simulation time \(\Delta t\) . The simulation then paused and returned the next observed state \(s_{t+1}\)  together with the reward \(r_{t+1}\) , which was calculated from the state as the negative distance measure between the embodiment states.
To train the agent, we used Proximal Policy Optimization (PPO) [1]}, which is a state-of-the art actor-critic DRL algorithm. One big advantage of PPO is its robustness towards hyperparameter tuning. We employed the GPU-optimized version (PPO2) of the Stable-Baselines repository, based on OpenAI's implementations.
