An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> Queues with a Single Server. Jackson Networks. Extensions to Queues with a Single Server. Baskett, Chandy, Muntz and Palacios Networks. Approximate Methods. Flows in Networks. G-Networks: Positive and Negative Customers, Signals and Batch Removal. G-Networks with Multiple Classes of Signals and Positive Customers. Discrete-Time Queueing Systems. Bibliography. Appendix. Index. <s> BIB001 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> Practical motivation some fundamental tools station balance - a down-to-Earth approach station balance - a more formal approach station and cluster balance for networks with limited clusters communications networks - source balance and insensitivity. <s> BIB002 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> 1. Discrete Part Manufacturing Systems. 2. Evolution of Manufacturing System Models: An Example. 3. Single Stage 'Produce-to-Order' Systems. 4. Single Stage 'Produce-to-Stock' Systems. 5. Flow Lines. 6. Transfer Lines. 7. Dynamic Job Shops. 8. Flexible Machining Systems. 9. Flexible Assembly Systems. 10. Multiple Cell Manufacturing Systems. 11. Unresolved Issues: Directions for Future Research. Appendix A: Standard Probability Distributions. Appendix B: Some Notions of Stochastic Ordering. Appendix C: Nonparametric Families of Distributions. <s> BIB003 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> considerations. Most of these open problems are easily stated and put into a theoretical framework, but they often require either intricate techniques on an ad hoc basis or deep mathematical methods. Often, both of these approaches have to be combined to tackle successfully the solution of quickly formulated problem. The development of queueing network theory, which provided application areas with solutions, formulas, and algorithms, commenced around 1950 in the area of Operations Research, with special emphasis on production, inventory, and transportation. The first breakthroughs were the works of Jackson [10] and Gordon and Newell [10]; both papers appeared in Operations Research. The second breakthrough in queueing network theory was already connected with Computer Science: the celebrated papers of Baskett, Chandy, Muntz, and Palacios [2], which appeared in the Journal of the Association for Computing Machinery, and of Kelly [11]. The two volumes of Kleinrock’s book [13, 14] appeared at the same time as [11]. From that time on, queueing network theory and its applications have become intimately connected with performance analysis of complex systems in Computer and Communication Sciences, both at the hardware and software levels. Today’s growth of production, manufacturing, and transportation, with information processing and communi <s> BIB004 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> 1 Jackson and Whittle Networks.- 1.1 Preliminaries on Networks and Markov Processes.- 1.2 Tandem Network.- 1.3 Definitions of Jackson and Whittle Processes.- 1.4 Properties of Service and Routing Rates.- 1.5 Equilibrium behavior.- 1.6 Production-Maintenance Network.- 1.7 Networks with Special Structures.- 1.8 Properties of Jackson Equilibrium Distributions.- 1.9 Convolutions for Single-Server Nodes.- 1.10 Throughputs and Expected Sojourn Times.- 1.11 Algorithms for Performance Parameters.- 1.12 Monte Carlo Estimation of Network Parameters.- 1.13 Properties of Whittle Networks.- 1.14 Exercises.- 1.15 Bibliographical Notes.- 2 Reversible Processes.- 2.1 Reversibility.- 2.2 Time Reversal.- 2.3 Invariant Measures.- 2.4 Construction of Reversible Processes.- 2.5 More Birth-Death Processes.- 2.6 Reversible Network Processes.- 2.7 Examples of Reversible Networks.- 2.8 Partition-Reversible Processes.- 2.9 Examples of Partition-Reversible Processes.- 2.10 Exercises.- 2.11 Bibliographical Notes.- 3 Miscellaneous Networks.- 3.1 Networks with Multiple Types of Units.- 3.2 Kelly Networks: Route-dependent Services.- 3.3 BCMP Networks: Class-Node Service Dependencies.- 3.4 Networks with Cox and General Service Times.- 3.5 Networks with Constraints.- 3.6 Networks with Blocking and Rerouting.- 3.7 Bottlenecks in Closed Jackson Networks.- 3.8 Modeling Whittle Networks by Locations of the Units.- 3.9 Partially Balanced Networks.- 3.10 Exercises.- 3.11 Bibliographical Notes.- 4 Network Flows and Travel Times.- 4.1 Point Process Notation.- 4.2 Extended Levy Formula for Markov Processes.- 4.3 Poisson Functionals of Markov Processes.- 4.4 Multivariate Compound Poisson Processes.- 4.5 Poisson Flows in Jackson and Whittle Networks.- 4.6 Palm Probabilities for Markov Processes.- 4.7 Sojourn and Travel Times of Markov Processes.- 4.8 Palm Probabilities of Jackson and Whittle Networks.- 4.9 Travel Times on Overtake-Free Routes.- 4.10 Exercises.- 4.11 Bibliographical Notes.- 5 Little Laws.- 5.1 Little Laws for Markovian Systems.- 5.2 Little Laws for General Queueing Systems.- 5.3 Preliminary Laws of Large Numbers.- 5.4 Utility Processes.- 5.5 Omnibus Little Laws.- 5.6 Little Laws for Regenerative Systems.- 5.7 Exercises.- 5.8 Bibliographical Notes.- 6 Stationary Systems.- 6.1 Preliminaries on Stationary Processes.- 6.2 Palm Probabilities.- 6.3 Campbell-Mecke Formulas for Palm Probabilities.- 6.4 Little Laws for Stationary Systems.- 6.5 Sojourn Times and Related Functionals.- 6.6 Travel Times for Stochastic Processes.- 6.7 Sojourn and Travel Times in Networks.- 6.8 Exercises.- 6.9 Bibliographical Notes.- 7 Networks with String Transitions.- 7.1 Definition of a String-Net.- 7.2 Invariant Measures of String-Nets.- 7.3 Traffic Equations, Partial Balance, and Throughputs.- 7.4 String-Nets with Unit-Vector Transitions.- 7.5 Networks with One-Stage Batch Transitions.- 7.6 Networks with Compound-Rate String Transitions.- 7.7 Networks with Multiple, Compound-Rate String Transitions.- 7.8 String-Nets with Two-Node Batch Transitions.- 7.9 Single Service Station With String Transitions.- 7.10 Bibliographical Notes.- 8 Quasi-Reversible Networks and Product Form Distributions.- 8.1 Quasi-Reversibility.- 8.2 Network to be Studied.- 8.3 Characterization of Product Form Distributions.- 8.4 Quasi-Reversibility and Biased Local Balance.- 8.5 Networks with Reversible Routing.- 8.6 Queueing Networks.- 8.7 Time-Reversals and Departure-Arrival Reversals.- 8.8 Networks with Multiclass Transitions.- 8.9 Exercises.- 8.10 Bibliographical Notes.- 9 Space-Time Poisson Models.- 9.1 Introductory Examples.- 9.2 Laplace Functionals of Point Processes.- 9.3 Transformations of Poisson Processes.- 9.4 Translations, Partitions, and Clusters.- 9.5 Service Systems with No Queueing.- 9.6 Network of M/G/? Service Stations.- 9.7 Particle Systems.- 9.8 Poisson Convergence of Space-Time Processes.- 9.9 Transformations into Large Spaces.- 9.10 Particle Flows in Large Spaces.- 9.11 Exercises.- 9.12 Bibliographical Notes.- 10 Spatial Queueing Systems.- 10.1 Preliminaries.- 10.2 Stationary Distributions and Ergodicity.- 10.3 Properties of Stationary Distributions and Examples.- 10.4 Throughputs and Expected Sojourn Times.- 10.5 Poisson Flows in Open Systems.- 10.6 Systems with Multiclass Units.- 10.7 Bibliographical Notes.- References. <s> BIB005 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> Preface. Part I: Queueing Network Models with Blocking and Applications. 1. Introduction. 2. Queueing Networks with Blocking. 3. Application Examples of Queueing Networks with Blocking. Part II: Analysis of Queueing Networks with Blocking. 4. Exact Analysis of Markovian Networks. 5. Exact Analysis of Special Networks. 6. Approximate and Bound Analysis. Part III: Properties of Networks with Blocking. 7. Equivalence, Insensitivity and Monotonicity Properties. 8. Buffer Allocation in Queueing Networks with Finite Capacities. References. Table of Symbols. Index. <s> BIB006 </s> An Overview for Markov Decision Processes in Queues and Networks <s> Introduction <s> Abstract This paper reviews the contributions and applications of queueing theory in the field of health care management problems. This review proposes a system of classification of health care areas which are examined with the assistance of queueing models. The categories described in the literature are expanded and a detailed taxonomy for subgroups is formulated. The goal is to provide sufficient information to analysts who are interested in using queueing theory to model a health care process and who want to locate the details of relevant models. <s> BIB007
One main purpose of this paper is to provide an overview for research on MDPs in queues and networks in the last six decades. Also, such a survey is first related to several other basic studies, such as, Markov processes, queueing systems, queueing networks, MDPs, sensitivity-based optimization, stochastic optimization, fluid and diffusion control. Van Dijk BIB002 , Gelenbe et al. BIB001 , Chao et al. BIB004 , Serfozo BIB005 , Chen and Yao  , Balsamo et al. BIB006 , Daduna [79] , Bolch et al.  and Boucherie and Van Dijk (Eds.)  . For applications of queueing networks, readers may refer to some excellent books, for example, manufacturing systems by Buzacott and Shanthikumar BIB003 , communication networks by Chang  , traffic networks by Garavello and Piccoli  , healthcare by Lakshmi and Iyer BIB007 , service management by Demirkan et al.  and others.