Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q-network (DQN) enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop a RL based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.
In general, the related research works on EH based systems could be categorized into two classes based on the availability of the knowledge about energy arrivals. The first class comprises offline approaches that require complete non-causal knowledge of the considered stochastic system, which are usually adopted to derive the performance upper bounds @cite_21 @cite_11 @cite_22 . In particular, the optimal uplink resource allocation was investigated in @cite_11 for the scenario where two EH users first harvested energy from the wireless signals and then cooperatively sent information to the access point. Also, the optimal packet scheduling over multiple access channels was studied in @cite_22 , with the goal of minimizing the time by which all packets from both users are delivered to the destination.
@cite_21 @cite_22 @cite_11
2614312052 2952044966 2073749694 1990808825
Energy harvesting (EH) two-hop communications are considered. The transmitter and the relay harvest energy from the environment and use it exclusively for transmitting data. A data arrival process is assumed at the transmitter. At the relay, a finite data buffer is used to store the received data. We consider a realistic scenario in which the EH nodes have only local causal knowledge, i.e., at any time instant, each EH node only knows the current value of its EH process, channel state, and data arrival process. Our goal is to find a power allocation policy to maximize the throughput at the receiver. We show that because the EH nodes have local causal knowledge, the two-hop communication problem can be separated into two point-to-point problems. Consequently, independent power allocation problems are solved at each EH node. To find the power allocation policy, reinforcement learning with linear function approximation is applied. Moreover, to perform function approximation two feature functions which consider the data arrival process are introduced. Numerical results show that the proposed approach has only a small degradation as compared to the offline optimum case. Furthermore, we show that with the use of the proposed feature functions a better performance is achieved compared to standard approximation techniques. We consider the optimal online packet scheduling problem in a single-user energy harvesting wireless communication system, where energy is harvested from natural renewable sources, making future energy arrivals instants and amounts random in nature. The most general case of arbitrary energy arrivals is considered where neither the future energy arrival instants or amount, nor their distribution is known. The problem considered is to adaptively change the transmission rate according to the causal energy arrival information, such that the time by which all packets are delivered is minimized. We assume that all bits have arrived and are ready at the source before the transmission begins. For a minimization problem, the utility of an online algorithm is tested by finding its competitive ratio or competitiveness that is defined to be the maximum of the ratio of the gain of the online algorithm with the optimal offline algorithm over all input sequences. We derive a lower and upper bound on the competitive ratio of any online algorithm to minimize the total transmission time in an energy harvesting system. The upper bound is obtained using a lazy' transmission policy that chooses its transmission power to minimize the transmission time assuming that no further energy arrivals are going to occur in future. The lazy transmission policy is shown to be strictly two-competitive. We also derive an adversarial lower bound that shows that competitive ratio of any online algorithm is at least 1.325. A multi-access wireless network with N transmitting nodes, each equipped with an energy harvesting (EH) device and a rechargeable battery of finite capacity, is studied. At each time slot (TS) a node is operative with a certain probability, which may depend on the availability of data, or the state of its channel. The energy arrival process at each node is modelled as an independent two-state Markov process, such that, at each TS, a node either harvests one unit of energy, or none. At each TS a subset of the nodes is scheduled by the access point (AP) . The scheduling policy that maximises the total throughput is studied assuming that the AP does not know the states of either the EH processes or the batteries. The problem is identified as a restless multi-armed bandit (RMAB) problem, and an upper bound on the optimal scheduling policy is found. Under certain assumptions regarding the EH processes and the battery sizes, the optimality of the myopic policy (MP) is proven. For the general case, the performance of MP is compared numerically to the upper bound. Communication over a broadband fading channel powered by an energy harvesting transmitter is studied. Assuming non-causal knowledge of energy data arrivals and channel gains, optimal transmission schemes are identified by taking into account the energy cost of the processing circuitry as well as the transmission energy. A constant processing cost for each active sub-channel is assumed. Three different system objectives are considered: i) throughput maximization, in which the total amount of transmitted data by a deadline is maximized for a backlogged transmitter with a finite capacity battery; ii) energy maximization, in which the remaining energy in an infinite capacity battery by a deadline is maximized such that all the arriving data packets are delivered; iii) transmission completion time minimization, in which the delivery time of all the arriving data packets is minimized assuming infinite size battery. For each objective, a convex optimization problem is formulated, the properties of the optimal transmission policies are identified, and an algorithm which computes an optimal transmission policy is proposed. Finally, based on the insights gained from the offline optimizations, low-complexity online algorithms performing close to the optimal dynamic programming solution for the throughput and energy maximization problems are developed under the assumption that the energy data arrivals and channel states are known causally at the transmitter.