Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q-network (DQN) enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop a RL based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.
Wireless access control strategies for EH nodes are usually proposed to make the full use of energy and maintain a perpetual lifetime. However, the uncertainty of ambient energy availability poses new challenge to sustainable perpetual operations @cite_4 . Thus, battery level prediction in such EH based systems is also worth investigating since a high battery prediction accuracy could potentially benefit the communication performance. For example, a novel solar energy prediction algorithm with Q-learning based on the weather-conditioned moving average (WCMA) algorithm was proposed in @cite_23 . Unfortunately, this algorithm is restricted to one type of energy sources and suffers from high computation complexity. @cite_24 , an online energy prediction model for multi-source EH wireless sensor networks was proposed, which leverages the past observations to forecast the future energy availability. Nevertheless, this energy prediction model requires that the EH dynamics should be known in advance. On the other hand, instead of studying the access control and battery prediction problems separately in EH based IoT applications, it has great significance to design a joint scheme that feeds the energy prediction results to the access control design, which could lead to better overall system performances. This is the focus of this paper.
@cite_24 @cite_4 @cite_23
2073749694 1988876637 2005435577 2508041822
A multi-access wireless network with N transmitting nodes, each equipped with an energy harvesting (EH) device and a rechargeable battery of finite capacity, is studied. At each time slot (TS) a node is operative with a certain probability, which may depend on the availability of data, or the state of its channel. The energy arrival process at each node is modelled as an independent two-state Markov process, such that, at each TS, a node either harvests one unit of energy, or none. At each TS a subset of the nodes is scheduled by the access point (AP) . The scheduling policy that maximises the total throughput is studied assuming that the AP does not know the states of either the EH processes or the batteries. The problem is identified as a restless multi-armed bandit (RMAB) problem, and an upper bound on the optimal scheduling policy is found. Under certain assumptions regarding the EH processes and the battery sizes, the optimality of the myopic policy (MP) is proven. For the general case, the performance of MP is compared numerically to the upper bound. Energy harvesting (EH) technology has emerged as a promising energy-supplier to unattended wireless systems. In the commonly used harvest-store-use (HSU) scheme, harvested energy is always stored in a battery before its subsequent use. The existence of storage loss in practical battery systems, however, unavoidably reduces the energy efficiency. In this paper, we therefore propose the use of a more efficient harvest-use-store (HUS) architecture for point-to-point data transmission, where the harvested energy is prioritized for use in data transmission while its balance debt is stored in or extracted from the storage device. We derive the optimal energy polices, under the criterion of throughput maximization, for the HUS architecture on static and block fading channels, and investigate the properties of the resulting power allocation pattern. The optimization is done in the Lagrangian framework, uncovering the special structure of the optimal power pattern and obtaining a closed-form solution conditioned on the knowledge of the block locations for zero battery level. A dynamic programming (DP) based algorithm is developed for locating such blocks in the optimal power patterns. Numerical results are presented to demonstrate the properties of the proposed HUS architecture and its superior performance over the existing schemes. This paper addresses the problem of finding outage- optimal power control policies for wireless energy harvesting sensor (EHS) nodes with automatic repeat request (ARQ)-based packet transmissions. The power control policy of the EHS specifies the transmission power for each packet transmission attempt, based on all the information available at the EHS. In particular, the acknowledgement (ACK) or negative acknowledgement (NACK) messages received provide the EHS with partial information about the channel state. We solve the problem of finding an optimal power control policy by casting it as a partially observable Markov decision process (POMDP). We study the structure of the optimal power policy in two ways. First, for the special case of binary power levels at the EHS, we show that the optimal policy for the underlying Markov decision process (MDP) when the channel state is observable is a threshold policy in the battery state. Second, we benchmark the performance of the EHS by rigorously analyzing the outage probability of a general fixed-power transmission scheme, where the EHS uses a predetermined power level at each slot within the frame. Monte Carlo simulation results illustrate the performance of the POMDP approach and verify the accuracy of the analysis. They also show that the POMDP solutions can significantly outperform conventional ad hoc approaches. Traditional wireless sensor networks (WSNs) face the problem of a limited-energy source, typically batteries, resulting in the need for careful and effective utilization of the energy source. However, inevitable energy depletion will eventually disturb the operation of a WSN. Energy harvesting (EH) technology is acquiring particular interest, because it has the potential to provide a continuous energy supply in battery-powered WSNs. Solar energy is the most effective environmental energy for EH-WSNs because of its high energy intensity, which comes from a non-controllable source. Therefore, the prediction of future energy availability is a critical issue, as the amount of the harvestable energy may vary over time. In this paper, a novel solar energy prediction algorithm with Q-learning, called Q-learning-based solar energy prediction (QL-SEP), is proposed. Q-learning is an effective way of predicting future actions based on past observations. The distinctive feature of QL-SEP is that not only past days’ observations but also the current weather conditions are considered for prediction. The performance of QL-SEP is simulated in this paper using real-world measurements obtained from a solar panel in comparison with the state-of-art approaches.