Efficient entity-based reinforcement learning
Recent deep reinforcement learning (DRL) successes rely on end-to-end
learning from fixed-size observational inputs (e.g. image, state-variables).
However, many challenging and interesting problems in decision making involve
observations or intermediary representations which are best described as a set
of entities: either the image-based approach would miss small but important
details in the observations (e.g. ojects on a radar, vehicles on satellite
images, etc.), the number of sensed objects is not fixed (e.g. robotic
manipulation), or the problem simply cannot be represented in a meaningful way
as an image (e.g. power grid control, or logistics). This type of structured
representations is not directly compatible with current DRL architectures,
however, there has been an increase in machine learning techniques directly
targeting structured information, potentially addressing this issue. We propose
to combine recent advances in set representations with slot attention and graph
neural networks to process structured data, broadening the range of
applications of DRL algorithms. This approach allows to address entity-based
problems in an efficient and scalable way. We show that it can improve training
time and robustness significantly, and demonstrate their potential to handle
structured as well as purely visual domains, on multiple environments from the
Atari Learning Environment and Simple Playgrounds.