Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor @math , or in episodic settings, with @math . While this has proven effective for specific tasks with well-defined objectives (e.g., games), it has never been established that fixed discounting is suitable for general purpose use (e.g., as a model of human preferences). This paper characterizes rationality in sequential decision making using a set of seven axioms and arrives at a form of discounting that generalizes traditional fixed discounting. In particular, our framework admits a state-action dependent "discount" factor that is not constrained to be less than 1, so long as there is eventual long run discounting. Although this broadens the range of possible preference structures in continuous settings, we show that there exists a unique "optimizing MDP" with fixed @math whose optimal value function matches the true utility of the optimal policy, and we quantify the difference between value and utility for suboptimal policies. Our work can be seen as providing a normative justification for (a slight generalization of) Martha White's RL task formalism (2017) and other recent departures from the traditional RL, and is relevant to task specification in RL, inverse RL and preference-based RL.
koopmans1960stationary provided the first axiomatic development of discounted additive utility over time. This and several follow-up works are summarized and expanded upon by koopmans1972representations and meyer1976preferences . The applicability of these and other existing discounted additive utility frameworks to general purpose RL is limited in several respects. For instance, as remarked by sobel2013discounting , most axiomatic justifications for discounting have assumed deterministic outcomes, and only a handful of analyses address stochasticity @cite_5 @cite_19 @cite_18 . Naively packaging deterministic outcome streams into arbitrary lotteries (as suggested by meyer1976preferences , .3) is difficult to interpret in case of control (see our commentary in Subsection on the resolution of intra-trajectory uncertainty) and entirely hypothetical since the agent never makes such choices---compare our (slightly) less hypothetical original position'' approach in Subsection .
@cite_19 @cite_5 @cite_18
2041192133 2088108678
In the literature on multiperiod planning under uncertainty, it is generally postulated that preferences may be represented by a von Neumann-Morgenstern utility index that is additive over time. This paper accomplishes two objectives: First, an axiomatic basis is provided for a more general class of non-additive utility indices defined over infinite consumption streams. Second, this class of utility functions is applied to extend existing results (J. Econ. Theory 4 (1972), 479–513; J. Econ. Theory 11 (1975), 329–339) on the nature of optimal growth under uncertainty. Of particular interest are the existence and stability of a stochastic steady state. Although most applications of discounting occur in risky settings, the best-known axiomatic justifications are deterministic. This paper provides an axiomatic rationale for discounting in a stochastic framework. Consider a representation of time and risk preferences with a binary relation on a real vector space of vector-valued discrete-time stochastic processes on a probability space. Four axioms imply that there are unique discount factors such that preferences among stochastic processes correspond to preferences among present value random vectors. The familiar axioms are weak ordering, continuity and nontriviality. The fourth axiom, decomposition, is non-standard and key. These axioms and the converse of decomposition are assumed in previous axiomatic justifications for discounting with nonlinear intraperiod utility functions in deterministic frameworks. Thus, the results here provide the weakest known sufficient conditions for discounting in deterministic or stochastic settings. In addition to the four axioms, if there exists a von Neumann-Morgenstern utility function corresponding to the binary relation, then that function is risk neutral (i.e., affine). In this sense, discounting axioms imply risk neutrality. Copyright Springer Science+Business Media, LLC 2013