Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q-network (DQN) enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop a RL based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.
In general, the related research works on EH based systems could be categorized into two classes based on the availability of the knowledge about energy arrivals. The first class comprises offline approaches that require complete non-causal knowledge of the considered stochastic system, which are usually adopted to derive the performance upper bounds @cite_21 @cite_11 @cite_22 . In particular, the optimal uplink resource allocation was investigated in @cite_11 for the scenario where two EH users first harvested energy from the wireless signals and then cooperatively sent information to the access point. Also, the optimal packet scheduling over multiple access channels was studied in @cite_22 , with the goal of minimizing the time by which all packets from both users are delivered to the destination.
@cite_21 @cite_22 @cite_11
2590740354 2168521983 2760327456
We focus on energy harvesting (EH) two-hop communications since they are the essential building blocks of more complicated multi-hop networks. The scenario consists of three nodes, where an EH transmitter wants to send data to a receiver through an EH relay. The harvested energy is used exclusively for data transmission and we address the problem of how to efficiently use it. As in practical scenarios, we assume only causal knowledge at the EH nodes, i.e., in each time interval, the transmitter and the relay know their own current and past amounts of incoming energy, battery levels, data buffer levels and channel coefficients for their own transmit channels. Our goal is to find transmission policies which aim at maximizing the throughput considering that the EH nodes fully cooperate with each other to exchange their causal knowledge during a signaling phase. We model the problem as a Markov game and propose a multi-agent reinforcement learning algorithm to find the transmission policies. Furthermore, we show the trade-off between the achievable throughput and the signaling required, and provide convergence guarantees for the proposed algorithm. Results show that even when the signaling overhead is taken into account, the proposed algorithm outperforms other approaches that do not consider cooperation among the nodes. In this paper, we investigate the optimal packet scheduling problem in a two-user multiple access communication system, where the transmitters are able to harvest energy from the nature. Under a deterministic system setting, we assume that the energy harvesting times and harvested energy amounts are known before the transmission starts. For the packet arrivals, we assume that packets have already arrived and are ready to be transmitted at the transmitter before the transmission starts. Our goal is to minimize the time by which all packets from both users are delivered to the destination through controlling the transmission powers and transmission rates of both users. We first develop a generalized iterative backward waterfilling algorithm to characterize the maximum departure region of the transmitters for any given deadline T. Then, based on the sequence of maximum departure regions at energy arrival instants, we decompose the transmission completion time minimization problem into convex optimization problems and solve the overall problem efficiently. This paper investigates the optimal resource allocation in wireless powered communication network with user cooperation, where two single-antenna users first harvest energy from the signals transmitted by a multi-antenna hybrid access point (H-AP) and then cooperatively send information to the H-AP using their harvested energy. To explore the system information transmission performance limit, an optimization problem is formulated to maximize the weighted sum-rate (WSR) by jointly optimizing energy beamforming vector, time assignment, and power allocation. Besides, another optimization problem is also formulated to minimize the total transmission time for given amount of data required to be transmitted at the two sources. Because both problems are non-convex, we first transform them to be convex by using proper variable substitutions and then apply semi-definite relaxation to solve them. We theoretically prove that our proposed methods guarantee the global optimum of both problems. Simulation results show that system WSR and transmission time can be significantly enhanced by using energy beamforming and user cooperation. It is observed that when the total amount of information of two users is fixed, with the increase of the information amount of the user relatively farther away from the H-AP, the transmission time of the user cooperation scheme decreases while that of the direct transmission increases. Besides, the effects of user position on the system performances are also discussed, which provides some useful insights.