An Overview for Markov Decision Processes in Queues and Networks <s> MDPs of Queueing Networks with Special Structure <s> We derive convexity results and related properties for the value functions of tandem queuing systems. The results for standard multiserver queues are new. For completeness, we also prove and generalize existing results on tandems of controllable queues. The results can be used to compare queuing systems. This is done for systems with and without batch arrivals and for systems with different numbers of on–off sources. <s> BIB001 </s> An Overview for Markov Decision Processes in Queues and Networks <s> MDPs of Queueing Networks with Special Structure <s> We consider a two-station tandem queue loss model where customers arrive to station 1 according to a Poisson process. A gatekeeper who has complete knowledge of the number of customers at both stations decides to accept or reject each arrival. A cost c1 is incurred if a customer is rejected, while if an admitted customer finds that station 2 is full at the time of his service completion at station 1, he leaves the system and a cost c2 is incurred. Assuming exponential service times at both stations, an arbitrary but finite buffer size at station 1 and a buffer size of one at station 2, we show that the optimal admission control policy for minimizing the long-run average cost per unit time has a simple structure. Depending on the value of c2 compared to a threshold value c*, it is optimal to admit a customer at the time of his arrival either only if the system is empty or as long as there is space at station 1. We also provide the closed-form expression of c*, which depends on the service rates at both stations, the arrival rate and c1. <s> BIB002 </s> An Overview for Markov Decision Processes in Queues and Networks <s> MDPs of Queueing Networks with Special Structure <s> Motivated by various applications in logistics, road traffic and production management, we investigate two versions of a tandem queueing model in which the service rate of the first queue can be controlled. The objective is to keep the mean number of jobs in the second queue as low as possible, without compromising the total system delay (i.e. avoiding starvation of the second queue). The balance between these objectives is governed by a linear cost function of the queue lengths. In the first model, the server in the first queue can be either switched on or off, depending on the queue lengths of both queues. This model has been studied extensively in the literature. Obtaining the optimal control is known to be computationally intensive and time consuming. We are particularly interested in the scenario that the first queue can operate at larger service speed than the second queue. This scenario has received less attention in literature. We propose an approximation using an efficient mathematical analysis of a near-optimal threshold policy based on a matrix-geometric solution of the stationary probabilities that enables us to compute the relevant stationary measures more efficiently and determine an optimal choice for the threshold value. <s> BIB003
In this section, we provide an overview for MDPs of queueing networks with special structure, for example, multi-station tandem queues, multi-station parallel queues, polling queues, fork-join queues and so on. Koole BIB001 , Zhang and Ayhan BIB002 and Leeuwen and Núnez-Queija BIB003 .