In this paper, we study the problem of dynamic channel allocation for URLLC traffic in a multiuser multi-channel wireless network where urgent packets have to be successfully received in a timely manner. We formulate the problem as a finite-horizon Markov Decision Process with a stochastic constraint related to the QoS requirement, defined as the packet loss rate for each user. We propose a novel weighted formulation that takes into account both the total expected reward (number of successfully decoded packets) and the risk which we define as the QoS requirement violation. First, we use the value iteration algorithm to find the optimal policy, which assumes a perfect knowledge of the controller of all the parameters, namely the channel statistics. We then propose a Q-learning algorithm where the controller learns the optimal policy without having knowledge of neither the CSI nor the channel statistics. We illustrate the performance of our algorithms with numerical studies.
Most risk-sensitive approaches consist in analyzing higher order statistics than the average metric such as the variance of the reward @cite_19 @cite_2 @cite_0 @cite_17 . For instance, a risk-sensitive reinforcement learning is studied in @cite_5 in millimeter-wave communications to optimize both the bandwidth and transmit power. The authors consider a utility (data rate) that incorporates both the average and the variance to capture the tail distribution of the rate, useful for the reliability requirement of URLLC traffic. The authors do not exploit frequency diversity.
@cite_0 @cite_19 @cite_2 @cite_5 @cite_17
2181918151 2786608596 2963856199 2490752264 2141203641 2101915445 2157758004 2614764375 2547508527 1975741370 2912139568 2740650368 2146025745 2143687524 2063449729 2965267010 2295341737 2033976720 2963418374 1964809923
This thesis investigates risk-sensitive sequential decision-making problems in an uncertain environment. We rst introduce the axiomatic concept of valuation functions that generalize known concepts of risk measures in mathematical nance to cover most of the existing risk related models in various elds, in particular, behavioral economics and cognitive neuroscience. By applying this concept to Markov processes, we construct valuation maps and develop thereby a uni ed framework for incorporating risk into Markov decision processes on general spaces. Within the framework, we study mainly two types of in nite-horizon risk-sensitive criteria, discounted and average valuations, and solve the associated optimization problems by value iteration. For the discounted case, we propose a new discount scheme, which is di erent from the conventional form but consistent with existing literature, while for the average criterion, we state Lyapunov-type stability conditions that generalize known conditions for Markov chains to ensure the existence of solutions to the optimality equation and a geometric convergence rate for the value iteration. Applying a set of valuation functions, called utility-based shortfall, we derive a family of model-free risk-sensitive reinforcement learning algorithms for solving the optimization problems corresponding to risk-sensitive valuations. In addition, we nd that when appropriate utility functions are chosen, agents’ behaviors express key features of human behavior as predicted by prospect theory, for example, di erent risk preferences for gains and losses, as well as the shape of subjective probability curves. As a proof of principle for the applicability of the new algorithms, we apply them to two tasks, 1) to quantify human behavior in a sequential investment task and 2) to perform risk control in simulated algorithmic trading of stocks. In the rst task, the risk-sensitive variant provides a signi cantly better t to the behavioral data and it leads to an interpretation of the subject’s responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a signi cant correlation of the risk-sensitive temporal di erence error with BOLD signal change in the ventral striatum. In the second task, our algorithm outperforms the risk-neutral reinforcement learning algorithm by keeping the trading cost at a substantially low level at the spot when the 2010 Flash Crash happened, and signi cantly reducing the risk over the whole test period. In this letter, we investigate the problem of providing gigabit wireless access with reliable communication in 5G millimeter-wave (mmWave) massive multiple-input multiple-output networks. In contrast to the classical network design based on average metrics, we propose a distributed risk-sensitive reinforcement learning-based framework to jointly optimize the beamwidth and transmit power, while taking into account the sensitivity of mmWave links due to blockage. Numerical results show that our proposed algorithm achieves more than 9 Gbps of user throughput with a guaranteed probability of 90 , whereas the baselines guarantee less than 7.5 Gbps. More importantly, there exists a rate-reliability-network density tradeoff, in which as the user density increases from 16 to 96 per km2, the fraction of users that achieves 4 Gbps is reduced by 11.61 and 39.11 in the proposed and the baseline models, respectively. In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales--a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application. Energy harvesting point-to-point communications are considered. The transmitter harvests energy from the environment and stores it in a finite battery. It is assumed that the transmitter has always data to transmit and the harvested energy is used exclusively for data transmission. As in practical scenarios prior knowledge about the energy harvesting process might not be available, we assume that at each time instant only information about the current state of the transmitter is available, i.e., harvested energy, battery level and channel coefficient. We model the scenario as a Markov decision process and we implement reinforcement learning at the transmitter to find a power allocation policy that aims at maximizing the throughput. To overcome the limitations of traditional reinforcement learning algorithms, we apply the concept of function approximation and we propose a set of binary functions to approximate the expected throughput given the state of the transmitter. Numerical results show that the performance of the proposed approach, which requires only causal knowledge of the energy harvesting process and channel coefficients, has only a small degradation compared to the optimum case which requires perfect non-causal knowledge. Additionally, the proposed approach outperforms naive policies that assume only causal knowledge at the transmitter. In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance-related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application. This paper presents a detailed study of average reward reinforcement learning, an undiscounted optimality framework that is more appropriate for cyclical tasks than the much better studied discounted framework. A wide spectrum of average reward algorithms are described, ranging from synchronous dynamic programming methods to several (provably convergent) asynchronous algorithms from optimal control and learning automata. A general sensitive discount optimality metric called n-discount-optimality is introduced, and used to compare the various algorithms. The overview identifies a key similarity across several asynchronous algorithms that is crucial to their convergence, namely independent estimation of the average reward and the relative values. The overview also uncovers a surprising limitation shared by the different algorithms: while several algorithms can provably generate gain-optimal policies that maximize average reward, none of them can reliably filter these to produce bias-optimal (or T-optimal) policies that also maximize the finite reward to absorbing goal states. This paper also presents a detailed empirical study of R-learning, an average reward reinforcement learning method, using two empirical testbeds: a stochastic grid world domain and a simulated robot environment. A detailed sensitivity analysis of R-learning is carried out to test its dependence on learning rates and exploration levels. The results suggest that R-learning is quite sensitive to exploration strategies, and can fall into sub-optimal limit cycles. The performance of R-learning is also compared with that of Q-learning, the best studied discounted RL method. Here, the results suggest that R-learning can be fine-tuned to give better performance than Q-learning in both domains. This paper examines the application of reinforcement learning to a wireless communication problem. The problem requires that channel utility be maximized while simultaneously minimizing battery usage. We present a solution to this multi-criteria problem that is able to significantly reduce power consumption. The solution uses a variable discount factor to capture the effects of battery usage. In this paper, we provide an analytical framework to analyze heterogeneous downlink millimeter-wave (mm-wave) cellular networks consisting of @math tiers of randomly located base stations (BSs), where each tier operates in an mm-wave frequency band. Signal-to-interference-plus-noise ratio (SINR) coverage probability is derived for the entire network using tools from stochastic geometry. The distinguishing features of mm-wave communications, such as directional beamforming, and having different path loss laws for line-of-sight and non-line-of-sight links are incorporated into the coverage analysis by assuming averaged biased-received power association and Nakagami fading. By using the noise-limited assumption for mm-wave networks, a simpler expression requiring the computation of only one numerical integral for coverage probability is obtained. Also, the effect of beamforming alignment errors on the coverage probability analysis is investigated to get insight on the performance in practical scenarios. Downlink rate coverage probability is derived as well to get more insights on the performance of the network. Moreover, the effect of deploying low-power smaller cells and the impact of biasing factor on energy efficiency is analyzed. Finally, a hybrid cellular network operating in both mm-wave and @math -wave frequency bands is addressed. We present an algorithm for the statistical learning setting with a bounded exp-concave loss in @math dimensions that obtains excess risk @math with probability at least @math . The core technique is to boost the confidence of recent in-expectation @math excess risk bounds for empirical risk minimization (ERM), without sacrificing the rate, by leveraging a Bernstein condition which holds due to exp-concavity. We also show that with probability @math the standard ERM method obtains excess risk @math . We further show that a regret bound for any online learner in this setting translates to a high probability excess risk bound for the corresponding online-to-batch conversion of the online learner. Lastly, we present two high probability bounds for the exp-concave model selection aggregation problem that are quantile-adaptive in a certain sense. The first bound is a purely exponential weights type algorithm, obtains a nearly optimal rate, and has no explicit dependence on the Lipschitz continuity of the loss. The second bound requires Lipschitz continuity but obtains the optimal rate. Abstract With regard to future service robots, unsafe exceptional circumstances can occur in complex systems that are hardly to foresee. In this paper, the assumption of having no knowledge about the environment is investigated using reinforcement learning as an option for learning behavior by trial-and-error. In such a scenario, action-selection decisions are made based on future reward predictions for minimizing costs in reaching a goal. It is shown that the selection of safety-critical actions leading to highly negative costs from the environment is directly related to the exploration exploitation dilemma in temporal-difference learning. For this, several exploration policies are investigated with regard to worst- and best-case performance in a dynamic environment. Our results show that in contrast to established exploration policies like e-Greedy and Softmax, the recently proposed VDBE-Softmax policy seems to be more appropriate for such applications due to its robustness of the exploration parameter for unexpected situations. In reinforcement learning, building policies of high-quality is challenging when the feature space of states is small and the training data is limited. Directly transferring data or knowledge from an agent to another agent will not work due to the privacy requirement of data and models. In this paper, we propose a novel reinforcement learning approach to considering the privacy requirement and building Q-network for each agent with the help of other agents, namely federated reinforcement learning (FRL). To protect the privacy of data and models, we exploit Gausian differentials on the information shared with each other when updating their local models. In the experiment, we evaluate our FRL framework in two diverse domains, Grid-world and Text2Action domains, by comparing to various baselines. We consider a single cell wireless uplink in which randomly arriving devices transmit their payload to a receiver. Given SNR per user, payload size per device, a fixed latency constraint T, total available bandwidth W, i.e., total symbol resources is given by N = TW. The total bandwidth W is evenly partitioned into B bins. Each time slot of duration T is split into a maximum number of retransmission attempts M. Hence, the N resources are partitioned into N MB resources each bin per retransmission. We characterize the maximum average rate or number of Poisson arrivals that can successfully complete the random access procedure such that the probability of outage is sufficiently small. We analyze the proposed setting for i) noise-limited regime and ii) interference-limited regime. We show that in the noise-limited regime the devices share the resources, and in the interference-limited regime, the resources split such that devices do not experience any interference. We then incorporate Rayleigh fading to model the channel power gain distribution. Although the variability of the channel causes a drop in the number of arrivals that can successfully complete the random access phase, similar scaling results extend to the Rayleigh fading case. This paper considers the design of efficient strategies that allow cognitive users to choose frequency bands to sense and access among multiple bands with unknown parameters. First, the scenario in which a single cognitive user wishes to opportunistically exploit the availability of frequency bands is considered. By adopting tools from the classical bandit problem, optimal as well as low complexity asymptotically optimal solutions are developed. Next, the multiple cognitive user scenario is considered. The situation in which the availability probability of each channel is known is first considered. An optimal symmetric strategy that maximizes the total throughput of the cognitive users is developed. To avoid the possible selfish behavior of the cognitive users, a game-theoretic model is then developed. The performance of both models is characterized analytically. Then, the situation in which the availability probability of each channel is unknown a priori is considered. Low-complexity medium access protocols, which strike an optimal balance between exploration and exploitation in such competitive environments, are developed. The operating points of these low-complexity protocols are shown to converge to those of the scenario in which the availability probabilities are known. Finally, numerical results are provided to illustrate the impact of sensing errors and other practical considerations. In cognitive radio networks, a cognitive source node requires two essential phases to complete a cognitive transmission process: the phase of spectrum sensing with a certain time duration (also referred to as spectrum sensing overhead) to detect a spectrum hole and the phase of data transmission through the detected spectrum hole. In this paper, we focus on the outage probability analysis of cognitive transmissions by considering the two phases jointly to examine the impact of spectrum sensing overhead on system performance. A closed-form expression of an overall outage probability that accounts for both the probability of no spectrum hole detected and the probability of a channel outage is derived for cognitive transmissions over Rayleigh fading channels. We further conduct an asymptotic outage analysis in high signal-to-noise ratio regions and obtain an optimal spectrum sensing overhead solution to minimize the asymptotic outage probability. Besides, numerical results show that a minimized overall outage probability can be achieved through a tradeoff in determining the time durations for the spectrum hole detection and data transmission phases. In this paper, we also investigate the use of cognitive relay to improve the outage performance of cognitive transmissions. We show that a significant improvement is achieved by the proposed cognitive relay scheme in terms of the overall outage probability. We determine the optimal adaptive rate and power control strategies to maximize the total throughput in a multirate code-division multiple-access system. The total throughput of the system provides a meaningful baseline in the form of an upper bound to the throughput achievable with additional restrictions imposed on the system to guarantee fairness. Peak power and instantaneous bit energy-to-noise spectral density constraints are assumed at the transmitter with matched filter detection at the receiver. Our results apply to frequency selective fading in so far as the bit energy-to-equivalent noise power spectral density ratio definition can be used as the quality-of-service metric. The bit energy-to-equivalent noise power spectral density ratio metric coincides with the bit-error rate metric under the assumption that the processing gains and the number of users are high enough so that self-interference can be neglected. We first obtain results for the case where the rates available to each user are unrestricted, and we then consider the more practical scenario where each user has a finite discrete set of rates. An upper bound to the maximum average throughput is obtained and evaluated for Rayleigh fading. Suboptimal low-complexity schemes are considered to illustrate the performance tradeoffs between optimality and complexity. We also show that the optimum rate and power adaptation scheme with unconstrained rates is in fact just a rate adaptation scheme with fixed transmit powers, and it performs significantly better than a scheme that uses power adaptation alone. This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages. In this paper, we study hierarchical codebook design for channel estimation in millimeter-wave (mmWave) communications with a hybrid precoding structure. Due to the limited saturation power of the mmWave power amplifier, we consider the per-antenna power constraint (PAPC). We first propose a metric, termed generalized detection probability (GDP), to evaluate the quality of an arbitrary codeword . This metric not only enables an optimization approach for mmWave codebook design, but also can be used to compare the performance of two different codewords codebooks. To the best of our knowledge, GDP is the first such metric, particularly for mmWave codebook design. We then propose a heuristic approach to design a hierarchical codebook exploiting beam widening with the multi-RF-chain sub-array (BMW-MS) technique. To obtain crucial parameters of BMW-MS, we provide two solutions, namely, a low-complexity search (LCS) solution to optimize the GDP metric and a closed-form (CF) solution to pursue a flat beam pattern. Performance comparisons show that BMW-MS LCS and BMW-MS CF achieve very close performances, and they outperform the existing alternatives under the PAPC. This thesis considers three complications that arise from applying reinforcement learning to a real-world application. In the process of using reinforcement learning to build an adaptive electronic market-maker, we find the sparsity of data, the partial observability of the domain, and the multiple objectives of the agent to cause serious problems for existing reinforcement learning algorithms. We employ importance sampling (likelihood ratios) to achieve good performance in partially observable Markov decision processes with few data. Our importance sampling estimator requires no knowledge about the environment and places few restrictions on the method of collecting data. It can be used efficiently with reactive controllers, finite-state controllers, or policies with function approximation. We present theoretical analyses of the estimator and incorporate it into a reinforcement learning algorithm. Additionally, this method provides a complete return surface which can be used to balance multiple objectives dynamically. We demonstrate the need for multiple goals in a variety of applications and natural solutions based on our sampling method. The thesis concludes with example results from employing our algorithm to the domain of automated electronic market-making. Thesis Supervisor: Tomaso Poggio Title: Professor of Brain and Cognitive Science Future 5G cellular networks supporting delay-sensitive, low-latency communications could employ random access communication to reduce the overhead compared to scheduled access techniques used in 4G networks. We consider a wireless communication system where multiple devices transmit payloads of a given fixed size in a random access fashion over shared radio resources to a common receiver. We allow retransmissions and assume Chase combining at the receiver. The radio resources are partitioned in the time and frequency dimensions, and we determine the optimal partition granularity to maximize throughput, subject to given constraints on latency and outage. In the regime of high and low signal-to-noise ratio (SNR), we derive explicit expressions for the granularity and throughput, first using a Shannon capacity approximation and then using finite block length analysis. Numerical results show that the throughput scaling results are applicable over a range of SNRs. The proposed analytical framework can provide insights for resource allocation strategies in reliable and delay-sensitive random access systems and in specific 5G use cases for massive, short packet uplink access. This paper explores a class of empirical Bayes methods for level-dependent threshold selection in wavelet shrinkage. The prior considered for each wavelet coefficient is a mixture of an atom of probability at zero and a heavy-tailed density. The mixing weight, or sparsity parameter, for each level of the transform is chosen by marginal maximum likelihood. If estimation is carried out using the posterior median, this is a random thresholding procedure; the estimation can also be carried out using other thresholding rules with the same threshold. Details of the calculations needed for implementing the procedure are included. In practice, the estimates are quick to compute and there is software available. Simulations on the standard model functions show excellent performance, and applications to data drawn from various fields of application are used to explore the practical performance of the approach. By using a general result on the risk of the corresponding marginal maximum likelihood approach for a single sequence, overall bounds on the risk of the method are found subject to membership of the unknown function in one of a wide range of Besov classes, covering also the case of f of bounded variation. The rates obtained are optimal for any value of the parameter p in (0, ∞], simultaneously for a wide range of loss functions, each dominating the Lq norm of the σth derivative, with σ ≥ 0 and 0 < q ≤ 2. Attention is paid to the distinction between sampling the unknown function within white noise and sampling at discrete points, and between placing constraints on the function itself and on the discrete wavelet transform of its sequence of values at the observation points. Results for all relevant combinations of these scenarios are obtained. In some cases a key feature of the theory is a particular boundary-corrected wavelet basis, details of which are discussed. Overall, the approach described seems so far unique in combining the properties of fast computation, good theoretical properties and good performance in simulations and in practice. A key feature appears to be that the estimate of sparsity adapts to three different zones of estimation, first where the signal is not sparse enough for thresholding to be of benefit, second where an appropriately chosen threshold results in substantially improved estimation, and third where the signal is so sparse that the zero estimate gives the optimum accuracy rate.