State Representation Learning for Control: An Overview <s> SRL Formalism <s> From the Publisher: ::: In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. <s> BIB001 </s> State Representation Learning for Control: An Overview <s> SRL Formalism <s> Robot learning is critically enabled by the availability of appropriate state representations. We propose a robotics-specific approach to learning such state representations. As robots accomplish tasks by interacting with the physical world, we can facilitate representation learning by considering the structure imposed by physics; this structure is reflected in the changes that occur in the world and in the way a robot can effect them. By exploiting this structure in learning, robots can obtain state representations consistent with the aspects of physics relevant to the learning task. We name this prior knowledge about the structure of interactions with the physical world robotic priors. We identify five robotic priors and explain how they can be used to learn pertinent state representations. We demonstrate the effectiveness of this approach in simulated and real robotic experiments with distracting moving objects. We show that our method extracts task-relevant state representations from high-dimensional observations, even in the presence of task-irrelevant distractions. We also show that the state representations learned by our method greatly improve generalization in reinforcement learning. <s> BIB002
The nomenclature we use is very close to the one used in reinforcement learning literature BIB001 and is illustrated in Fig. 1 . We define an environment E where an agent performs actions a t ∈ A at time step t and where A is the action space (continuous or discrete). Each action makes the agent transition from a true states t tos t+1 which is unknown but assumed to exist. We call the true state spaceS. The agent obtains an observation of E from its sensors, denoted o t ∈ O where O is the observation space. Optionally, the agent may receive a reward r t . The reward is given ats t by a reward function designed to lead the agent to a certain behavior that solves a task. The reward is optional as learning a state representation don't aim to solve a task, but is often present as one of the goal of SRL may be to improve task learning performance. The SRL task is to learn a representation s t ∈ S of dimension K with characteristics similar to those ofs t without usings t . More formally, SRL learns a mapping φ of the history of observation to the current state s t = φ(o 1:t ). Note that actions a 1:t and rewards r 1:t can also be added to the parameters of φ BIB002 . In this paper, we are specifically interested in the particular setting in which this mapping is learned through proxy objectives without access to the true states t . This family of approaches is called unsupervised or self-supervised. Finally, we noteô t the reconstruction of o t (similarly forâ t andr t ), that will be used in various SRL approaches.