An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> Two of the optimality criteria most commonly considered to compare job assignment policies, namely, the stochastic order and the average total completion time criterion, are discussed. It is shown that the latter, while being weaker than the former, still provides a useful criterion when policies are to be evaluated in terms of average performance measures such as the system average response time. Some of the most relevant results available in the literature on job assignment to parallel first come, first served (FCFS) servers are summarized. The assignment problem is discussed for a parallel system of processor sharing (PS) queues with exponential interarrival times and exponential and identical service requirements. The optimality of the join-the-shortcut-queue (JSQ) assignment policy for this case is proven. Feasible policies for general service time distribution are discussed. An approach allowing a meaningful comparison of servers in the perspective of job assignment is proposed. A comparison of the assignment problem for FCFS and PS parallel systems is presented. > <s> BIB001 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> Zero-sum stochastic games model situations where two persons, called players, control some dynamic system, and both have opposite objectives. One player wishes typically to minimize a cost which has to be paid to the other player. Such a game may also be used to model problems with a single controller who has only partial information on the system: the dynamic of the system may depend on some parameter that is unknown to the controller, and may vary in time in an unpredictable way. A worst-case criterion may be considered, where the unknown parameter is assumed to be chosen by “nature” (called player 1), and the objective of the controller (player 2) is then to design a policy that guarantees the best performance under worst-case behaviour of nature. The purpose of this paper is to present a survey of stochastic games in queues, where both tools and applications are considered. The first part is devoted to the tools. We present some existing tools for solving finite horizon and infinite horizon discounted Markov games with unbounded cost, and develop new ones that are typically applicable in queueing problems. We then present some new tools and theory of expected average cost stochastic games with unbounded cost. In the second part of the paper we present a survey on existing results on worst-case control of queues, and illustrate the structural properties of best policies of the controller, worst-case policies of nature, and of the value function. Using the theory developed in the first part of the paper, we extend some of the above results, which were known to hold for finite horizon costs or for the discounted cost, to the expected average cost. <s> BIB002 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> The purpose of this paper is to investigate situations of non-cooperative dynamic control of queueing systems by two agents, having different objectives. The main part of the paper is devoted to analyzing a problem of an admission and a service (vacation) control. The admission controller has to decide whether to allow arrivals to occur. Once the queue empties, the server goes on vacation, and controls the vacations duration (according to the state and past history of the queue). The immediate costs for each controller are increasing in the number of customers, but no convexity assumptions are made. The controllers are shown to have a stationary equilibrium policy pair, for which each controller uses a stationary threshold type policy with randomization in at most one state. We then investigate a problem of a non-zero sum stochastic game between a router into several queues, and a second controller that allocates some extra service capacity to one of the queues. We establish the equilibrium of a policy pair for which the router uses the intuitive “Join the shortest queue” policy. <s> BIB003 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> In this paper we will consider two-person zero-sum games and derive a general approach for solving them. We apply this approach to a queueing problem. In section 1 we will introduce the model and formulate the Key-theorem. In section 2 we develop the theory that we will use in section 3 to prove the Key-theorem. This includes a general and useful result in Lemma 2.1 on the sufficiency of stationary policies. <s> BIB004 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> We study a dynamic optimal routing problem, where a controller has to decide to which of two queues should arriving customers (representing packets, messages, call etc...) be sent. The service rate in each queue may depend on the state of the system, may change in time and is unknown to the controller. The goal of the controller is to design a strategy that guarantees the best performance under the worst case service conditions. The payoff is composed of a holding cost, an admission cost, and a cost that depends on the quality of the service. We consider both the finite and infinite horizon discounted cost and the expected average cost. The problem is studied in the framework of zero-sum Markov games where the server, called player 1, is assumed to play against the router, called player 2. Each player is assumed to have the information of all previous actions of both players as well as the current and past states of the system. We show that there exist pure optimal strategies for both players. A value iteration algorithm is used to establish properties of the value of the game, which are related to supermodularity and to convexity. This is then shown to imply the existence of optimal strategies described by <s> BIB005 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> A supermarket game is considered with N FCFS queues with unit exponential service rate and global Poisson arrival rate Nλ. Upon arrival each customer chooses a number of queues to be sampled uniformly at random and joins the least loaded sampled queue. Customers are assumed to have cost for both waiting and sampling, and they want to minimize their own expected total cost. We study the supermarket game in a mean field model that corresponds to the limit as N converges to infinity in the sense that (i) for a fixed symmetric customer strategy, the joint equilibrium distribution of any fixed number of queues converges as N → ∞ to a product distribution determined by the mean field model and (ii) a Nash equilibrium for the mean field model is an e-Nash equilibrium for the finite N model with N sufficiently large. It is shown that there always exists a Nash equilibrium for λ < 1 and the Nash equilibrium is unique with homogeneous waiting cost for λ≤1/2. Furthermore, we find that the action of sampling more que... <s> BIB006 </s> An Overview for Markov Decision Processes in Queues and Networks <s> (3) MDPs of parallel queues <s> Game theoretic analysis of queueing systems is an important research direction of queueing theory. In this paper, we study the service rate control problem of closed Jackson networks from a game theoretic perspective. The payoff function consists of a holding cost and an operating cost. Each server optimizes its service rate control strategy to maximize its own average payoff. We formulate this problem as a non-cooperative stochastic game with multiple players. By utilizing the problem structure of closed Jackson networks, we derive a difference equation which quantifies the performance difference under any two different strategies. We prove that no matter what strategies the other servers adopt, the best response of a server is to choose its service rates on the boundary. Thus, we can limit the search of equilibrium strategy profiles from a multidimensional continuous polyhedron to the set of its vertex. We further develop an iterative algorithm to find the Nash equilibrium. Moreover, we derive the social optimum of this problem, which is compared with the equilibrium using the price of anarchy. The bounds of the price of anarchy of this problem are also obtained. Finally, simulation experiments are conducted to demonstrate the main idea of this paper. <s> BIB007
Weber  , Bonomi BIB001 , Menich The competitive MDPs are called to be stochastic games. Altman and Hordijk BIB002 studied the zero-sum Markov game and applied the obtained results to the worst-case optimal control of queueing networks. Altman BIB003 studied non-zero stochastic games and applied their results to admission, service and routing control in queueing networks. Altman BIB005 proposed a Markov game approach for analyzing the optimal routing of a queueing network. Hordijk et al. BIB004 studied a multi-chain stochastic game which is applied to the worst case admission control in a queueing network. Xu and Hajek BIB006 studied the game problem of supermarket models. Xia BIB007 applied the stochastic games to analyzing the service rate control of a closed queueing network.