This paper investigates the use of deep reinforcement learning (DRL) in the design of a "universal" MAC protocol referred to as Deep-reinforcement Learning Multiple Access (DLMA). The design framework is partially inspired by the vision of DARPA SC2, a 3-year competition whereby competitors are to come up with a clean-slate design that "best share spectrum with any network(s), in any environment, without prior knowledge, leveraging on machine-learning technique". While the scope of DARPA SC2 is broad and involves the redesign of PHY, MAC, and Network layers, this paper's focus is narrower and only involves the MAC design. In particular, we consider the problem of sharing time slots among a multiple of time-slotted networks that adopt different MAC protocols. One of the MAC protocols is DLMA. The other two are TDMA and ALOHA. The DRL agents of DLMA do not know that the other two MAC protocols are TDMA and ALOHA. Yet, by a series of observations of the environment, its own actions, and the rewards - in accordance with the DRL algorithmic framework - a DRL agent can learn the optimal MAC strategy for harmonious co-existence with TDMA and ALOHA nodes. In particular, the use of neural networks in DRL (as opposed to traditional reinforcement learning) allows for fast convergence to optimal solutions and robustness against perturbation in hyper- parameter settings, two essential properties for practical deployment of DLMA in real wireless networks.
RL was employed to develop channel access schemes for cognitive radios @cite_4 @cite_9 @cite_7 @cite_10 and wireless sensor networks @cite_11 @cite_15 . Unlike this paper, these works do not leverage the recent advances in DRL.
@cite_4 @cite_7 @cite_9 @cite_15 @cite_10 @cite_11
1810054459 2123927904 2101640072 2071707134
Cognitive Radio (CR) is a next-generation wireless communication system that enables unlicensed users to exploit underutilized licensed spectrum to optimize the utilization of the overall radio spectrum. A Distributed Cognitive Radio Network (DCRN) is a distributed wireless network established by a number of unlicensed users in the absence of fixed network infrastructure such as a base station. Context awareness and intelligence are the capabilities that enable each unlicensed user to observe and carry out its own action as part of the joint action on its operating environment for network-wide performance enhancement. These capabilities can be applied in various application schemes in CR networks such as Dynamic Channel Selection (DCS), congestion control, and scheduling. In this paper, we apply Reinforcement Learning (RL), including single-agent and multi-agent approaches, to achieve context awareness and intelligence. Firstly, we show that the RL approach achieves a joint action that provides better network-wide performance in respect to DCS in DCRNs. The multi-agent approach is shown to provide higher levels of stability compared to the single-agent approach. Secondly, we show that RL achieves high level of fairness. Thirdly, we show the effects of network density and various essential parameters in RL on the network-wide performance. We consider a decentralized cognitive radio network in which autonomous secondary users seek spectrum opportunities in licensed spectrum bands. We assume that the primary users' channel occupancy follows a Markovian evolution, and formulate the spectrum sensing problem as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP). We develop a distributed Reinforcement Learning (RL) algorithm that allows each autonomous cognitive radio to distributively learn its own spectrum sensing policy. The resulting decentralized sensing policy enables secondary users to non-cooperatively reach an equilibrium that leads to high utilization of idle channels while minimizing the collisions among secondary cognitive radios. Moreover, we propose a decentralized channel access policy that permits controlling, with high accuracy, the collision probability with primary users. Our numerical results validate the robustness of this collision probability control as the sensing noise changes. They also show the efficiency of the proposed learning algorithm in improving the spectrum utilization. A Markov chain analysis for spectrum access in licensed bands for cognitive radios is presented and forced termination probability, blocking probability and traffic throughput are derived. In addition, a channel reservation scheme for cognitive radio spectrum handoff is proposed. This scheme allows the tradeoff between forced termination and blocking according to QoS requirements. Numerical results show that the proposed scheme can greatly reduce forced termination probability at a slight increase in blocking probability Cognitive radio is viewed as a novel approach for improving the utilization of a precious natural resource: the radio electromagnetic spectrum. The cognitive radio, built on a software-defined radio, is defined as an intelligent wireless communication system that is aware of its environment and uses the methodology of understanding-by-building to learn from the environment and adapt to statistical variations in the input stimuli, with two primary objectives in mind: spl middot highly reliable communication whenever and wherever needed; spl middot efficient utilization of the radio spectrum. Following the discussion of interference temperature as a new metric for the quantification and management of interference, the paper addresses three fundamental cognitive tasks. 1) Radio-scene analysis. 2) Channel-state estimation and predictive modeling. 3) Transmit-power control and dynamic spectrum management. This work also discusses the emergent behavior of cognitive radio.