Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q-network (DQN) enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop a RL based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.
A multi-access wireless network with N transmitting nodes, each equipped with an energy harvesting (EH) device and a rechargeable battery of finite capacity, is studied. At each time slot (TS) a node is operative with a certain probability, which may depend on the availability of data, or the state of its channel. The energy arrival process at each node is modelled as an independent two-state Markov process, such that, at each TS, a node either harvests one unit of energy, or none. At each TS a subset of the nodes is scheduled by the access point (AP) . The scheduling policy that maximises the total throughput is studied assuming that the AP does not know the states of either the EH processes or the batteries. The problem is identified as a restless multi-armed bandit (RMAB) problem, and an upper bound on the optimal scheduling policy is found. Under certain assumptions regarding the EH processes and the battery sizes, the optimality of the myopic policy (MP) is proven. For the general case, the performance of MP is compared numerically to the upper bound. The application of wireless charging technology in traditional battery-powered wireless sensor networks (WSNs) grows rapidly recently. Although previous studies indicate that the technology can deliver energy reliably, it still faces regulatory mandate to provide high power density without incurring health risks. In particular, in clustered WSNs there exists a mismatch between the high energy demands from cluster heads and the relatively low energy supplies from wireless chargers. Fortunately, solar energy harvesting can provide high power density without health risks. However, its reliability is subject to weather dynamics. In this paper, we propose a hybrid framework that combines the two technologies - cluster heads are equipped with solar panels to scavenge solar energy and the rest of nodes are powered by wireless charging. We divide the network into three hierarchical levels. On the first level, we study a discrete placement problem of how to deploy solar-powered cluster heads that can minimize overall cost and propose a distributed @math -approximation algorithm for the placement. Then, we extend the discrete problem into continuous space and develop an iterative algorithm based on the Weiszfeld algorithm. On the second level, we establish an energy balance in the network and explore how to maintain such balance for wireless-powered nodes when sunlight is unavailable. We also propose a distributed cluster head re-selection algorithm. On the third level, we first consider the tour planning problem by combining wireless charging with mobile data gathering in a joint tour. We then propose a polynomial-time scheduling algorithm to find appropriate hitting points on sensors’ transmission boundaries for data gathering. For wireless charging, we give the mobile chargers more flexibility by allowing partial recharge when energy demands are high. The problem turns out to be a Linear Program. By exploiting its particular structure, we propose an efficient algorithm that can achieve near-optimal solutions. Our extensive simulation results demonstrate that the hybrid framework can reduce battery depletion by 20 percent and save vehicles’ moving cost by 25 percent compared to previous works. By allowing partial recharge, battery depletion can be further reduced at a slightly increased cost. The results also suggest that we can reduce the number of high-cost mobile chargers by deploying more low-cost solar-powered sensors. We investigate performance limits of a multiple access communication system with energy harvesting nodes where the utility function is taken to be the long-term average sum-throughput. We assume a causal structure for energy arrivals and study the problem in the continuous time regime. For this setting, we first characterize a storage dam model that captures the dynamics of a battery with energy harvesting and variable transmission power. Using this model, we next establish an upper bound on the throughput problem as a function of battery capacity. We also formulate a nonlinear optimization problem to determine optimal achievable power policies for transmitters. Applying a calculus of variations technique, we then derive Euler-Lagrange equations as necessary conditions for optimum power policies in terms of a system of coupled partial integro-differential equations. Based on a Gauss-Seidel algorithm, we devise an iterative algorithm to solve these equations. We also propose a fixed-point algorithm for the symmetric multiple access setting in which the statistical descriptions of energy harvesters are identical. To further support our iterative algorithms, along with the analysis, comprehensive numerical results are also obtained. A time-slotted multiple access wireless system with N transmitting nodes, each equipped with an energy harvesting (EH) device and a rechargeable battery of finite capacity, is studied. The energy arrival process at each node is modeled as an independent two-state Markov process, such that a node either harvests one unit of energy, or none, at each time slot (TS). The access point (AP) schedules a subset of K nodes to transmit over K orthogonal channels at each TS. The maximum total throughput is studied for a backlogged system without the knowledge of the EH processes and nodes' battery states at the AP. The problem is identified as a partially observable Markov decision process, and the optimal policy for the general model is studied numerically. Under certain assumptions regarding the EH processes and the battery sizes, the optimal scheduling policy is characterized explicitly, and is shown to be myopic.