An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> The early growth of queueing theory was motivated by the design of telephone and other service systems. As the literature expanded, much of its growth consisted of mathematical theory suggested by the literature itself rather than by congested service systems. This self-sustaining theoretical growth has been coupled recently with a renewed pragmatic motivation. The result has been an increase in the volume of prescriptive research in queueing theory. Most of these recent contributions begin by embellishing a standard queueing model with a cost structure. Although they go on to address the question of optimal design or of optimal operation, one might imagine that including a cost structure is merely hanging bells and jangles on a standard model. Although that conjecture may be valid, the analysis of normative queueing models often differs fundamentally from the derivation of properties of descriptive models and it provides new insights into how congestion should be managed. <s> BIB001 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Research in the control of queueing systems has been going on at an ever-increasing rate in the last few years. Progress in this field has reached the point where one can make some general statements about “the queueing-control problem”. <s> BIB002 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This paper presents a classified bibliography organized into six sections-static design models, dynamic control models, models involving queue discipline control, miscellaneous models, tutorial and survey papers, and books that have specific portions devoted to design and control. Some of the above categories are further subdivided when appropriate, and explanations of the types of models in each category and subcategory precede the actual list of references. Papers that fall into more than one category are cross-referenced. <s> BIB003 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This paper presents an algorithm to compute an optimal s, S policy under standard assumptions stationary data, well-behaved one-period costs, discrete demand, full backlogging, and the average-cost criterion. The method is iterative, starting with an arbitrary, given s, S policy and converging to an optimal policy in a finite number of iterations. Any of the available approximations can thus be used as an initial solution. Each iteration requires only modest computations. Also, a lower bound on the true optimal cost can be computed and used in a termination test. Empirical testing suggests very fast convergence. <s> BIB004 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Socially optimal control of access to the landing queue of an airport is investigated in this paper. Semi-Markov decision process models, M / M /1 and M / E k /1, for several classes of customers are used to determine how access should be controlled. Only commercial jet aircraft are considered and they are divided up into five classes based on aircraft type. Data from the Greater Pittsburgh International Airport are used to determine the parameters of the service time distribution. Data from several published sources are used to develop the cost, reward, and arrival rate parameters for each of the classes. Results are developed and presented for both models. <s> BIB005 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Abstract This paper studies optimal dynamic load-distribution among parallel single-stage stations in flexible manufacturing systems and characterizes the form of the optimal policies as a function of the system state. Each work station maintains its own buffer and has independent input and output processes. At each decision epoch, a central Work-In-Process controller redistributes and routes jobs among all stations to minimize the total expected operating cost of the network system. The optimal policy for the system prescribes how many jobs will be sent to (taken from) each station so that optimal load balancing among all stations is implemented. The model takes into account holding and shortage costs for all the stations, which are assumed to be convex increasing functions, and routing costs as well. It is proved that the optimal action (number of jobs to be received) for a station is monotonically decreasing, as the number of jobs at the station grows, and is monotonically increasing, as the number of jobs at any other station grows. The resultant properties of the optimal policies for the model give fundamental guidelines on how optimal routing can be done among parallel stations by generalizing from a single-stage situation. Finally, a numerical example is presented to display the analytically proven properties of the optimal policies. <s> BIB006 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> The reorder point/reorder quantity policies, also referred to as r, Q policies, are widely used in industry and extensively studied in the literature. However, for a period of almost 30 years there has been no efficient algorithm for computing optimal control parameters for such policies. In this paper, we present a surprisingly simple and efficient algorithm for the determination of an optimal r*, Q* policy. The computational complexity of the algorithm is linear in Q*. For the most prevalent case of linear holding, backlogging and stockout penalty costs in addition to fixed order costs, the algorithm requires at most 6r* + 13Q* elementary operations additions, comparisons and multiplications, and hence, no more than 13 times the amount of work required to do a single evaluation of the long-run average cost function in the point r*, Q*. <s> BIB007 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> We review models for the optimal control of networks of queues. Our main emphasis is on models based on Markov decision theory and the characterization of the structure of optimal control policies. <s> BIB008 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> 1. Discrete Part Manufacturing Systems. 2. Evolution of Manufacturing System Models: An Example. 3. Single Stage 'Produce-to-Order' Systems. 4. Single Stage 'Produce-to-Stock' Systems. 5. Flow Lines. 6. Transfer Lines. 7. Dynamic Job Shops. 8. Flexible Machining Systems. 9. Flexible Assembly Systems. 10. Multiple Cell Manufacturing Systems. 11. Unresolved Issues: Directions for Future Research. Appendix A: Standard Probability Distributions. Appendix B: Some Notions of Stochastic Ordering. Appendix C: Nonparametric Families of Distributions. <s> BIB009 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Semi-Regenerative Decision Models Description of Basic Decision Model Rigorous Definitions and Assumptions Examples of Controlled Queues Optimization Problems Renewal Kernels of the Decision Model Special Classes of Strategies Sufficiency of Markov Strategies Dynamic Programming Discounting in Continuous Time Dynamic Programming Equation Bellman Functions Finite-Horizon Problem Infinite-Horizon Discounted-Cost Problem Random-Horizon Problem Average Cost Criterion Preliminaries: Weak Topology, Limit Passages Preliminaries: Taboo Probabilities, Limit Theorems for Markov Renewal Processes Notation, Recurrence-Communication Assumptions, Examples Existence of Optimal Policies Existence of Optimal Strategies: General Criterion Existence of Optimal Strategies: Sufficient Conditions Optimality Equation Constrained Average-Cost Problem Average-Cost Optimality as Limiting Case of Discounted-Cost Optimality Continuously Controlled Markov Jump Processes Facts About Measurability of Stochastic Processes Marked Point Processes and Random Measures The Predictable s-Algebra Dual Predictable Projections of Random Measures Definition of Controlled Markov Jump Process An M/M/1 Queue With Controllable Input and Service Rate Dynamic Programming Optimization Problems Structured Optimization Problems for Decision Processes Convex Regularization Submodular and Supermodular Functions Existence of Monotone Solutions for Optimization Problems Processes with Bounded Drift Birth and Death Processes Control of Arrivals The Model Description Finite-Horizon Discounted-Cost Problem Cost Functionals Infinite-Horizon Case with and without Discounting Optimal Dynamic Pricing Policy: Model Results Control of Service Mechanism Description of the System Static Optimization Problem Optimal Policies for the Queueing Process Service System with Two Interacting Servers Analysis of Optimality Equation Optimal Control in Models with Several Classes of Customers Description of Models and Processes Associated Controlled Processes Existence of Optimal Simple Strategies for the Systems with Alternating Priority Existence of Optimal Simple Strategy for the System with Feedback Equations for Stationary Distributions Stationary Characteristics of the Systems with Alternating Priority Stationary Characteristics of the System with Feedback Models with Alternating Priority: Linear Programming Problem Linear Programming Problem in the Model with Feedback Model with Periods of Idleness and Discounted-Cost Criterion Basic Formulas Construction of Optimal Modified Priority Discipline Bibliography Index Each chapter also includes an Introduction, and a Remarks and Exercises section <s> BIB010 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> We study a dynamic optimal routing problem, where a controller has to decide to which of two queues should arriving customers (representing packets, messages, call etc...) be sent. The service rate in each queue may depend on the state of the system, may change in time and is unknown to the controller. The goal of the controller is to design a strategy that guarantees the best performance under the worst case service conditions. The payoff is composed of a holding cost, an admission cost, and a cost that depends on the quality of the service. We consider both the finite and infinite horizon discounted cost and the expected average cost. The problem is studied in the framework of zero-sum Markov games where the server, called player 1, is assumed to play against the router, called player 2. Each player is assumed to have the information of all previous actions of both players as well as the current and past states of the system. We show that there exist pure optimal strategies for both players. A value iteration algorithm is used to establish properties of the value of the game, which are related to supermodularity and to convexity. This is then shown to imply the existence of optimal strategies described by <s> BIB011 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> We introduce the terms dynamic and static, respectively, to identify the prevailing approaches to the single-leg airline yield-management problem: those allowing customers of different fare classes to book concomitantly (dynamic), and those assuming that the demands for the different fare classes arrive separately in a predetermined order (static). We present a coherent frame-work linking these seemingly disparate models through the underlying dynamic program common to both. We develop a discrete-time Markov decision process formulation mirroring that of Janakiram et al. Transp. Sci. 33, 147Â 167 (1999) to solve the single-leg problem without cancellations, overbooking, or discounting. Borrowing a result from the queueing-control literature, we prove the concavity of the associated optimal value functions and, subsequently, the optimality of a booking limit policy. We then apply this same technique to the more influential papers from the single-leg literature, at once unifying the static and dynamic models and establishing the connection between the yield-management and queueing-control problems. Finally, we propose an omnibus formulation that yields the static and dynamic models as special cases. <s> BIB012 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This paper introduces a continuous-time, controllable Markov process model of a power-managed system. The system model is composed of the corresponding stochastic models of the service queue and the service provider. The system environment is modeled by a stochastic service request process. The problem of dynamic power management in such a system is formulated as a policy optimization problem and solved using an efficient "policy iteration" algorithm. Compared to previous work on dynamic power management, our formulation allows better modeling of the various system components, the power-managed system as a whole, and its environment. In addition it captures dependencies between the service queue and service provider status. Finally, the resulting power management policy is asynchronous, hence it is more power-efficient and more useful in practice. Experimental results demonstrate the effectiveness of our policy optimization algorithm compared to a number of heuristic (time-out and N-policy) algorithms. <s> BIB013 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Optimization Criteria. Finite Horizon Optimization. Infinite Horizon Discounted Cost Optimization. An Inventory Model. Average Cost Optimization for Finite State Spaces. Average Cost Optimization Theory for Countable State Spaces. Computation of Average Cost Optimal Policies for Infinite State Spaces. Optimization Under Actions at Selected Epochs. Average Cost Optimization of Continuous Time Processes. Appendices. Bibliography. Index. <s> BIB014 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> We develop a supply network model that takes as input the bill of materials, the (nominal) lead times, the demand and cost data, and the required customer service levels. In return, the model generates the base-stock level at each store--the stocking location for a part or an end-product, so as to minimize the overall inventory capital throughout the network and to guarantee the customer service requirements. The key ingredient of the model is a detailed, albeit approximate, analysis of theactual lead times at each store and the associated demand over such lead times, along with a characterization of the operation at each store via an inventory-queue model. The gradients are derived in explicit forms, and a conjugate gradient routine is used to search for the optimal solution. Several numerical examples are presented to validate the model and to illustrate its various features. <s> BIB015 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Systems and methods for locating an operative element within an interior body space use a locating probe, which includes at least one transmitting element to transmit an electric waveform output within at least a portion of the space. The systems and methods also use a sensing element, which is adapted to be carried by the operative element to sense a local electric waveform within the space. A processing element coupled to the sensing element generates a processed output that locates the sensing element relative to the locating probe based, at least in part, upon a differential comparison of the waveform output and the sensed local waveform. <s> BIB016 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Many real-life phenomena, such as computer systems, communication networks, manufacturing systems, supermarket checkout lines can be represented by means of queueing models. The queues with a controller may considerably improve the system performance by reducing queue lengths, or increasing the throughput, or diminishing the overhead. In this book, the essential work consists in designing controlled queueing systems with heterogeneous servers and investigation of their optimal control properties. This type of queues take into account many practical aspects in modelling real systems, e.g. network nodes with servers supplied by different types of processors as a consequence of system updates, nodes in telecommunication network with links of different capacities, etc, which should satisfy the demands for quality of service. For two types of optimization criterion (the model without penalties and with set-up costs) a class of controlled queues with Markov Arrival Process and heterogeneous Phase-Type service time distributions is defined. We derive some monotonicity relations for the dynamic programming value function and show that the optimal policy is of threshold type. <s> BIB017 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Radio resource management (RRM) plays a major role in Quality of Service (QoS) provisioning for wireless communication systems. The performance of RRM techniques has a direct impact on each user's individual performance and on the overall network performance. Arriving (new and handoff) calls are granted/denied access to the network by the call admission scheme (CAC) based on predefined criteria, taking the network loading conditions into consideration. CAC in wireless networks has been receiving a great deal of attention during the last two decades due to the growing popularity of wireless communications and the central role that CAC plays in QoS provisioning in terms of the signal quality, call blocking and dropping probabilities, packet delay and loss rate, and transmission rate. In the first and second generation of wireless systems, CAC has been developed for a single service environment. In the third generation and beyond wireless systems, multimedia services such as voice, video, data, and audio are to be offered with various QoS profiles. Hence, more sophisticated CAC schemes are developed to cope with these changes. This article provides a comprehensive survey of CAC schemes in modern wireless networks. <s> BIB018 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This paper studies the joint machine maintenance and product quality control problem of a finite horizon discrete time Markovian deteriorating, state unobservable batch production system. Unlike previous studies, both the timing of the sampling action and the sample size are directly included in the action space of our dynamic programming model of the system. We formulate the system as a partially observable Markov decision process and derive some properties of the optimal value function which enable us to efficiently search the optimal maintenance and quality control policy to minimize the expected total discounted system cost. <s> BIB019 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Abstract We consider a model consisting of a deteriorating installation (I) that transfers a raw material to a production unit and a buffer, which has been built between the I and the production unit to cope with unexpected failures of the I that may cause delays in production. The problem of the optimal preventive maintenance of the I is considered. It is assumed that the repair times follow some known continuous distributions. It seems intuitively reasonable that, for fixed buffer level, the optimal policy is of control-limit type, i.e. it initiates the preventive maintenance of the I if and only if the degree of its deterioration exceeds a critical level. An efficient semi-Markov decision algorithm, which operates on the class of control-limit policies, is developed. Numerical examples provide strong evidence that the algorithm converges to the optimal policy. <s> BIB020 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This book explores how to set the parameters of a queueing system, before putting it into operation. It considers various optimality criteria, including individual, social, class, and facility optimal solutions. The book first covers the optimal arrival rate model for a single-facility, single-class queue as well as dynamic algorithms for finding individually or socially optimal arrival rates and prices. It then examines several special cases of multiclass queues, presents models in which the service rate is a decision variable, and extends models and techniques to multifacility queueing systems. Focusing on networks of queues, the final chapters emphasize the qualitative properties of optimal solutions. <s> BIB021 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> This research examines the production control problem in two-station tandem queueing systems under time constraints. In these two-station tandem queueing systems, jobs must first be processed at the upstream station and then the downstream station. For each job, the sum of the waiting and processing time in the downstream queue is limited by an upper bound. This time constraint is called the process queue time constraint. When the process queue time constraint is violated, a significant scrap cost will be accrued. In this research, we develop a Markov decision model to study the production control problem under process queue time constraints. The objective is to minimise the sum of the expected long-run average inventory holding costs and scrap costs. According to the Markov decision model, an interesting exhaustive structure of the optimal production control policy is found. Based on this exhaustive structure, an efficient algorithm is developed to solve the production control problem numerically. The pe... <s> BIB022 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision problems under uncertainty as well as Reinforcement Learning problems. Written by experts in the field, this book provides a global view of current research using MDPs in Artificial Intelligence. It starts with an introductory presentation of the fundamental aspects of MDPs (planning in MDPs, Reinforcement Learning, Partially Observable MDPs, Markov games and the use of non-classical criteria). Then it presents more advanced research trends in the domain and gives some concrete examples using illustrative applications. <s> BIB023 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Markov Decision Processes (MDPs) are widely popular in Artificial Intelligence for modeling sequential decision-making scenarios with probabilistic dynamics. They are the framework of choice when designing an intelligent agent that needs to act for long periods of time in an environment where its actions could have uncertain outcomes. MDPs are actively researched in two related subareas of AI, probabilistic planning and reinforcement learning. Probabilistic planning assumes known models for the agent's goals and domain dynamics, and focuses on determining how the agent should behave to achieve its objectives. On the other hand, reinforcement learning additionally learns these models based on the feedback the agent gets from the environment. This book provides a concise introduction to the use of MDPs for solving probabilistic planning problems, with an emphasis on the algorithmic perspective. It covers the whole spectrum of the field, from the basics to state-of-the-art optimal and approximation algorithms. We first describe the theoretical foundations of MDPs and the fundamental solution techniques for them. We then discuss modern optimal algorithms based on heuristic search and the use of structured representations. A major focus of the book is on the numerous approximation schemes for MDPs that have been developed in the AI literature. These include determinization-based approaches, sampling techniques, heuristic functions, dimensionality reduction, and hierarchical representations. Finally, we briefly introduce several extensions of the standard MDP classes that model and solve even more complex planning problems. Table of Contents: Introduction / MDPs / Fundamental Algorithms / Heuristic Search Algorithms / Symbolic Algorithms / Approximation Algorithms / Advanced Notes <s> BIB024 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> We study the convergence of Markov decision processes, composed of a large number of objects, to optimization problems on ordinary differential equations. We show that the optimal reward of such a Markov decision process, which satisfies a Bellman equation, converges to the solution of a continuous Hamilton-Jacobi-Bellman (HJB) equation based on the mean field approximation of the Markov decision process. We give bounds on the difference of the rewards and an algorithm for deriving an approximating solution to the Markov decision process from a solution of the HJB equations. We illustrate the method on three examples pertaining, respectively, to investment strategies, population dynamics control and scheduling in queues. They are used to illustrate and justify the construction of the controlled ODE and to show the advantage of solving a continuous HJB equation rather than a large discrete Bellman equation. <s> BIB025 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Together with an explosive growth of the mobile applications and emerging of cloud computing concept, mobile cloud computing (MCC) has been introduced to be a potential technology for mobile services. MCC integrates the cloud computing into the mobile environment and overcomes obstacles related to the performance (e.g., battery life, storage, and bandwidth), environment (e.g., heterogeneity, scalability, and availability), and security (e.g., reliability and privacy) discussed in mobile computing. This paper gives a survey of MCC, which helps general readers have an overview of the MCC including the definition, architecture, and applications. The issues, existing solutions, and approaches are presented. In addition, the future research directions of MCC are discussed. Copyright © 2011 John Wiley & Sons, Ltd. <s> BIB026 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> The performance of medium access control (MAC) depends on both spatial locations and traffic patterns of wireless agents. In contrast to conventional MAC policies, we propose a MAC solution that adapts to the prevailing spatial and temporal opportunities. The proposed solution is based on a decentralized partially observable Markov decision process (DEC-POMDP), which is able to handle wireless network dynamics described by a Markov model. A DEC-POMDP takes both sensor noise and partial observations into account, and yields MAC policies that are optimal for the network dynamics model. The DEC-POMDP MAC policies can be optimized for a freely chosen goal, such as maximal throughput or minimal latency, with the same algorithm. We make approximate optimization efficient by exploiting problem structure: the policies are optimized by a factored DEC-POMDP method, yielding highly compact state machine representations for MAC policies. Experiments show that our approach yields higher throughput and lower latency than CSMA/CA based comparison methods adapted to the current wireless network configuration. <s> BIB027 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Cloud technology connects a network of virtualized computers that are dynamically provisioned as computing resources, based on negotiated agreements between service providers and users. It delivers information technology resources in diverse forms of service, and the explosion of Cloud services on the Internet brings new challenges in Cloud service discovery and selection. To address these challenges, a range of studies has been carried out to develop advanced techniques that will assist service users to choose appropriate services. In this paper, we survey state-of-the-art Cloud service selection approaches, which are analyzed from the following five perspectives: decision-making techniques; data representation models; parameters and characteristics of Cloud services; contexts, purposes. After comparing and summarizing the reviewed approaches from these five perspectives, we identify the primary research issues in contemporary Cloud service selection. This survey is expected to bring benefits to both researchers and business agents. <s> BIB028 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are used to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs. <s> BIB029 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Dynamic power management (DPM) plays a significant role to save power consumption effectively in both the design and operational phases of computer-based systems. It is well known that the state-dependent control policy by monitoring energy states in each component or the whole system is efficient for power saving in server systems whose system state, such as transaction request, can be completely observed. In this paper, we consider an optimal power-aware design in a cluster system and formulate the DPM problem by means of the Markov decision process. We derive the dynamic programming equation for the optimal control policy, which maximizes the expected reward per unit electrical power, which is called the power effectiveness, and give the policy iteration algorithm to determine the optimal control policy sequentially. In numerical experiments, we show the optimal control policy for an example of a cluster system with two service nodes, where the arrival stream of the transaction request is described as a Markov modulated Poisson process. In addition, based on the access data of an enterprise system, the optimal power-aware control for the cluster system and its effectiveness is examined. <s> BIB030 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> In this study, we consider a stochastic inventory system in which the objective of the manufacturer is to maximize the long-run average profit by dynamically offering the selling price and switching the production on or off. The demand process is non-homogeneous Poisson with a price-dependent arrival rate. System costs consist of switching costs, inventory holding and backlogging costs. We show that an $(s,S,{\mathbf p})$ policy is optimal. Moreover, we characterize the structural properties of the optimal profit function and pricing strategy, and show that the optimal price is a quasiconcave function of the inventory level when the production is off and is a quasiconvex function when the production is on. <s> BIB031 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (c) Queueing dynamic control <s> Big networks express various large-scale networks in many practical areas such as computer networks, internet of things, cloud computation, manufacturing systems, transportation networks, and healthcare systems. This paper analyzes such big networks, and applies the mean-field theory and the nonlinear Markov processes to set up a broad class of nonlinear continuous-time block-structured Markov processes, which can be applied to deal with many practical stochastic systems. Firstly, a nonlinear Markov process is derived from a large number of interacting big networks with symmetric interactions, each of which is described as a continuous-time block-structured Markov process. Secondly, some effective algorithms are given for computing the fixed points of the nonlinear Markov process by means of the UL-type RG-factorization. Finally, the Birkhoff center, the Lyapunov functions and the relative entropy are used to analyze stability or metastability of the big network, and several interesting open problems are proposed with detailed interpretation. We believe that the results given in this paper can be useful and effective in the study of big networks. <s> BIB032
In 1967, Miller  and Ryokov and Lembert  were the first to apply MDPs to considering dynamic control of queues and networks. Those two works opened a novel interesting research direction: MDPs in queues and networks. For MDPs of queues and networks, we refer readers to three excellent books by Kitaev and Rykov BIB010 , Sennott BIB014 and Stidham BIB021 . In MDPs of queues and networks, so far there have been some best survey papers, for instance, Crabill et al.  BIB003 , Sobel BIB001 , Stidham and Prabhu BIB002 , Rykov   , Kumar [184] , Stidham and Weber BIB008 , Stidham BIB016 and Brouns  . For some Ph.D thesises by using MDPs of queues and networks, reader may refer to, such as, Farrell  , Abdel-Gawad  , Bartroli [25] , Farrar  , Veatch  , Altman BIB011 , Atan  and Efrosinin BIB017 . Now, MDPs of queues and networks play an important role in dynamic control of many practical stochastic networks, for example, inventory control BIB031 BIB007 BIB004 , supply chain management BIB015 , maintenance and quality BIB020 BIB019 , manufacturing systems BIB009 BIB006 , production lines BIB022 , communication networks BIB029  BIB027 , wireless and mobile networks BIB018 BIB026 , cloud service BIB028 , healthcare  , airport management BIB012 BIB005 , energy-efficient management BIB030 BIB013 and artificial intelligence BIB024 BIB023 . With rapid development of Internet of Things (IoT), big data, cloud computing, blockchain and artificial intelligence, it is necessary to discuss MDPs of queues and networks under an intelligent environment. From the detailed survey on MDPs of queues and networks, this paper suggests a future research under an intelligent environment from three different levels as follows: 1. Networks with several nodes: Analyzing MDPs of policy-based Markov processes with block structure, for example, QBD processes, Markov processes of GI/M/1 type, and Markov processes of M/G/1 type, and specifically, discuss their sensitivitybased optimization. MDPs of practical big networks, such as   blockchain systems, sharing economics, intelligence healthcare and so forth. 3. Networks with a lot of clusters: studying MDPs of practical big networks by means of the mean-field theory, e.g., see Gast and Gaujal  , Gast et al. BIB025 and Li BIB032 .