A Survey of solution techniques for the partially observed Markov decision process <s> Conclusions <s> A fluorescence spectrophotometer having a beam splitter placed between an exit slit of an excitation monochromator and a specimen. Part of the excitation radiation is conducted to a first light quantum meter by said beam splitter and a reference photomultiplier is provided for receiving fluorescence from said first light quantum meter. A second light quantum meter is placed at the position normally occupied by a specimen cell and first and second absorbing cells are disposed in front of said first and second light quantum meters, respectively. <s> BIB001 </s> A Survey of solution techniques for the partially observed Markov decision process <s> Conclusions <s> This paper formulates the optimal control problem for a class of mathematical models in which the system to be controlled is characterized by a finite-state discrete-time Markov process. The states of this internal process are not directly observable by the controller; rather, he has available a set of observable outputs that are only probabilistically related to the internal state of the system. The formulation is illustrated by a simple machine-maintenance example, and other specific application areas are also discussed. The paper demonstrates that, if there are only a finite number of control intervals remaining, then the optimal payoff function is a piecewise-linear, convex function of the current state probabilities of the internal Markov process. In addition, an algorithm for utilizing this property to calculate the optimal control policy and payoff function for any finite horizon is outlined. These results are illustrated by a numerical example for the machine-maintenance problem. <s> BIB002 </s> A Survey of solution techniques for the partially observed Markov decision process <s> Conclusions <s> This paper treats the discounted cost, optimal control problem for Markov processes with incomplete state information. The optimization approach for these partially observable Markov processes is a generalization of the well-known policy iteration technique for finding optimal stationary policies for completely observable Markov processes. The state space for the problem is the space of state occupancy probability distributions the unit simplex. The development of the algorithm introduces several new ideas, including the class of finitely transient policies, which are shown to possess piecewise linear cost functions. The paper develops easily implemented approximations to stationary policies based on these finitely transient policies and shows that the concave hull of an approximation can be included in the well-known Howard policy improvement algorithm with subsequent convergence. The paper closes with a detailed example illustrating the application of the algorithm to the two-state partially observable Markov process. <s> BIB003
We have examined several recently developed procedures for improving the tractability of the POMDP. Several of these have been convergence accelerating variants of, and approximations to, the SS algorithm, attesting to the seminal importance of results found in BIB002 BIB003 BIB001 . Finite grid approximations of X have been reviewed. The finite-memory suboptimal design results reported in section 9 attempt to look at the POMDP from a different perspective than that found in BIB002 , and the directions 9 for future research presented in section 10 diverge from dynamic programming as the basis for solution determination. Attempts have been made to indicate which algorithms may be the most useful for various types of POMDPs. We hope that this survey will further stimulate research to improve the tractability, and hence the usefulness, of the POMDP.