In this paper, we study the problem of dynamic channel allocation for URLLC traffic in a multiuser multi-channel wireless network where urgent packets have to be successfully received in a timely manner. We formulate the problem as a finite-horizon Markov Decision Process with a stochastic constraint related to the QoS requirement, defined as the packet loss rate for each user. We propose a novel weighted formulation that takes into account both the total expected reward (number of successfully decoded packets) and the risk which we define as the QoS requirement violation. First, we use the value iteration algorithm to find the optimal policy, which assumes a perfect knowledge of the controller of all the parameters, namely the channel statistics. We then propose a Q-learning algorithm where the controller learns the optimal policy without having knowledge of neither the CSI nor the channel statistics. We illustrate the performance of our algorithms with numerical studies.
Most risk-sensitive approaches consist in analyzing higher order statistics than the average metric such as the variance of the reward @cite_19 @cite_2 @cite_0 @cite_17 . For instance, a risk-sensitive reinforcement learning is studied in @cite_5 in millimeter-wave communications to optimize both the bandwidth and transmit power. The authors consider a utility (data rate) that incorporates both the average and the variance to capture the tail distribution of the rate, useful for the reliability requirement of URLLC traffic. The authors do not exploit frequency diversity.
@cite_0 @cite_19 @cite_2 @cite_5 @cite_17
2181918151 2786608596 2963856199 2490752264
This thesis investigates risk-sensitive sequential decision-making problems in an uncertain environment. We rst introduce the axiomatic concept of valuation functions that generalize known concepts of risk measures in mathematical nance to cover most of the existing risk related models in various elds, in particular, behavioral economics and cognitive neuroscience. By applying this concept to Markov processes, we construct valuation maps and develop thereby a uni ed framework for incorporating risk into Markov decision processes on general spaces. Within the framework, we study mainly two types of in nite-horizon risk-sensitive criteria, discounted and average valuations, and solve the associated optimization problems by value iteration. For the discounted case, we propose a new discount scheme, which is di erent from the conventional form but consistent with existing literature, while for the average criterion, we state Lyapunov-type stability conditions that generalize known conditions for Markov chains to ensure the existence of solutions to the optimality equation and a geometric convergence rate for the value iteration. Applying a set of valuation functions, called utility-based shortfall, we derive a family of model-free risk-sensitive reinforcement learning algorithms for solving the optimization problems corresponding to risk-sensitive valuations. In addition, we nd that when appropriate utility functions are chosen, agents’ behaviors express key features of human behavior as predicted by prospect theory, for example, di erent risk preferences for gains and losses, as well as the shape of subjective probability curves. As a proof of principle for the applicability of the new algorithms, we apply them to two tasks, 1) to quantify human behavior in a sequential investment task and 2) to perform risk control in simulated algorithmic trading of stocks. In the rst task, the risk-sensitive variant provides a signi cantly better t to the behavioral data and it leads to an interpretation of the subject’s responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a signi cant correlation of the risk-sensitive temporal di erence error with BOLD signal change in the ventral striatum. In the second task, our algorithm outperforms the risk-neutral reinforcement learning algorithm by keeping the trading cost at a substantially low level at the spot when the 2010 Flash Crash happened, and signi cantly reducing the risk over the whole test period. In this letter, we investigate the problem of providing gigabit wireless access with reliable communication in 5G millimeter-wave (mmWave) massive multiple-input multiple-output networks. In contrast to the classical network design based on average metrics, we propose a distributed risk-sensitive reinforcement learning-based framework to jointly optimize the beamwidth and transmit power, while taking into account the sensitivity of mmWave links due to blockage. Numerical results show that our proposed algorithm achieves more than 9 Gbps of user throughput with a guaranteed probability of 90 , whereas the baselines guarantee less than 7.5 Gbps. More importantly, there exists a rate-reliability-network density tradeoff, in which as the user density increases from 16 to 96 per km2, the fraction of users that achieves 4 Gbps is reduced by 11.61 and 39.11 in the proposed and the baseline models, respectively. In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales--a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application. Energy harvesting point-to-point communications are considered. The transmitter harvests energy from the environment and stores it in a finite battery. It is assumed that the transmitter has always data to transmit and the harvested energy is used exclusively for data transmission. As in practical scenarios prior knowledge about the energy harvesting process might not be available, we assume that at each time instant only information about the current state of the transmitter is available, i.e., harvested energy, battery level and channel coefficient. We model the scenario as a Markov decision process and we implement reinforcement learning at the transmitter to find a power allocation policy that aims at maximizing the throughput. To overcome the limitations of traditional reinforcement learning algorithms, we apply the concept of function approximation and we propose a set of binary functions to approximate the expected throughput given the state of the transmitter. Numerical results show that the performance of the proposed approach, which requires only causal knowledge of the energy harvesting process and channel coefficients, has only a small degradation compared to the optimum case which requires perfect non-causal knowledge. Additionally, the proposed approach outperforms naive policies that assume only causal knowledge at the transmitter.