This paper investigates the use of deep reinforcement learning (DRL) in the design of a "universal" MAC protocol referred to as Deep-reinforcement Learning Multiple Access (DLMA). The design framework is partially inspired by the vision of DARPA SC2, a 3-year competition whereby competitors are to come up with a clean-slate design that "best share spectrum with any network(s), in any environment, without prior knowledge, leveraging on machine-learning technique". While the scope of DARPA SC2 is broad and involves the redesign of PHY, MAC, and Network layers, this paper's focus is narrower and only involves the MAC design. In particular, we consider the problem of sharing time slots among a multiple of time-slotted networks that adopt different MAC protocols. One of the MAC protocols is DLMA. The other two are TDMA and ALOHA. The DRL agents of DLMA do not know that the other two MAC protocols are TDMA and ALOHA. Yet, by a series of observations of the environment, its own actions, and the rewards - in accordance with the DRL algorithmic framework - a DRL agent can learn the optimal MAC strategy for harmonious co-existence with TDMA and ALOHA nodes. In particular, the use of neural networks in DRL (as opposed to traditional reinforcement learning) allows for fast convergence to optimal solutions and robustness against perturbation in hyper- parameter settings, two essential properties for practical deployment of DLMA in real wireless networks.
RL was employed to develop channel access schemes for cognitive radios @cite_4 @cite_9 @cite_7 @cite_10 and wireless sensor networks @cite_11 @cite_15 . Unlike this paper, these works do not leverage the recent advances in DRL.
@cite_4 @cite_7 @cite_9 @cite_15 @cite_10 @cite_11
1976294375 2163905491 1810054459 2051376838 2123927904
An Aloha-like spectrum access scheme without negotiation is considered for multiuser and multichannel cognitive radio systems. To avoid collisions incurred by the lack of coordination, each secondary user learns how to select channels according to its experience. Multiagent reinforcement leaning (MARL) is applied for the secondary users to learn good strategies of channel selection. Specifically, the framework of Q-learning is extended from single user case to multiagent case by considering other secondary users as a part of the environment. The dynamics of the Q-learning are illustrated using a Metrick-Polak plot, which shows the traces of Q-values in the two-user case. For both complete and partial observation cases, rigorous proofs of the convergence of multiagent Q-learning without communications, under certain conditions, are provided using the Robins-Monro algorithm and contraction mapping, respectively. The learning performance (speed and gain in utility) is evaluated by numerical simulations. Wireless cognitive radio (CR) is a newly emerging paradigm that attempts to opportunistically transmit in licensed frequencies, without affecting the pre-assigned users of these bands. To enable this functionality, such a radio must predict its operational parameters, such as transmit power and spectrum. These tasks, collectively called spectrum management, is difficult to achieve in a dynamic distributed environment, in which CR users may only take local decisions, and react to the environmental changes. In this paper, we introduce a multi-agent reinforcement learning approach based spectrum management. Our approach uses value functions to evaluate the desirability of choosing different transmission parameters, and enables efficient assignment of spectrums and transmit powers by maximizing long-term reward. We then investigate various real-world scenarios, and compare the communication performance using different sets of learning parameters. We also apply Kanerva-based function approximation to improve our approach's ability to handle large cognitive radio networks and evaluate its effect on communication performance. We conclude that our reinforcement learning based spectrum management can significantly reduce the interference to the licensed users, while maintaining a high probability of successful transmissions in a cognitive radio ad hoc network. Cognitive Radio (CR) is a next-generation wireless communication system that enables unlicensed users to exploit underutilized licensed spectrum to optimize the utilization of the overall radio spectrum. A Distributed Cognitive Radio Network (DCRN) is a distributed wireless network established by a number of unlicensed users in the absence of fixed network infrastructure such as a base station. Context awareness and intelligence are the capabilities that enable each unlicensed user to observe and carry out its own action as part of the joint action on its operating environment for network-wide performance enhancement. These capabilities can be applied in various application schemes in CR networks such as Dynamic Channel Selection (DCS), congestion control, and scheduling. In this paper, we apply Reinforcement Learning (RL), including single-agent and multi-agent approaches, to achieve context awareness and intelligence. Firstly, we show that the RL approach achieves a joint action that provides better network-wide performance in respect to DCS in DCRNs. The multi-agent approach is shown to provide higher levels of stability compared to the single-agent approach. Secondly, we show that RL achieves high level of fairness. Thirdly, we show the effects of network density and various essential parameters in RL on the network-wide performance. This paper presents a novel Medium Access Control (MAC) protocol named ALOHA and Q-Learning based MAC with Informed Receiving (ALOHA-QIR) for Wireless Sensor Networks (WSNs). This scheme is implemented by combining frame based Slotted ALOHA and Q-Learning. ALOHA based schemes have the benefits of simplicity, low computation and overheads, but suffer collisions from the blind transmission strategy. By considering the previous transmission experience, Q-Learning offers nodes certain intelligence to access slots with a lower probability of collision. During data transmissions, receivers are informed by transmitters about their future slot selections so that the receivers can turn off their radio in other slots to save energy. Compared with basic Slotted ALOHA, ALOHA-QIR obtains significant improvements in throughput, delay and energy efficiency. We consider a decentralized cognitive radio network in which autonomous secondary users seek spectrum opportunities in licensed spectrum bands. We assume that the primary users' channel occupancy follows a Markovian evolution, and formulate the spectrum sensing problem as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP). We develop a distributed Reinforcement Learning (RL) algorithm that allows each autonomous cognitive radio to distributively learn its own spectrum sensing policy. The resulting decentralized sensing policy enables secondary users to non-cooperatively reach an equilibrium that leads to high utilization of idle channels while minimizing the collisions among secondary cognitive radios. Moreover, we propose a decentralized channel access policy that permits controlling, with high accuracy, the collision probability with primary users. Our numerical results validate the robustness of this collision probability control as the sensing noise changes. They also show the efficiency of the proposed learning algorithm in improving the spectrum utilization.