@inproceedings{
wu2023agentcentric,
title={Agent-Centric State Discovery for Finite-Memory {POMDP}s},
author={Lili Wu and Ben Evans and Riashat Islam and Raihan Seraj and Yonathan Efroni and Alex Lamb},
booktitle={NeurIPS 2023 Workshop on Generalization in Planning},
year={2023},
url={https://openreview.net/forum?id=tKTTvWOzsZ}
}
Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory as well as negative results for alternative intuitive algorithms, such as encoding with only a forward-running sequence model. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.
Agent-Centric State Discovery for Finite-Memory POMDPs
Lili Wu Ben Evans Riashat Islam Raihan Seraj Yonathan Efroni Alex Lamb