MDPs can be continuous time, infinite horizon or partially observable. Infinite horizon MDPs do not have a goal state and the system runs for ever [1]}. The agent learns to maximize its total reward without expecting a goal state. In continuous time MDPs, unlike discrete time, the decisions are made at every point in time. The formulation of continuous time MDP is similar to discrete MDP, however, these problems are harder to solve in general [2]}. Partially Observable MDP (POMDP) is a kind of MDP where the agent has limited information about the environment due to different reasons such as limited sensors or uncertainty in the environment [3]}, [1]}. These variants of MDPs could also be modeled and solved by RL, although they need different considerations on defining the states which the agent has a belief about the environment rather than complete observation. In this article, we focus on discrete time fully observable MDPs.
