Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals. We show such a reward function and use policy-gradient approaches to learn capable policies, leading to behavior more appropriate for IoT nodes with less manual design effort, increasing the level of autonomy in IoT.
RL has also been used for power allocation in energy-harvesting communication systems. @cite_11 used the SARSA algorithm with linear function approximation to learn a power allocation policy in two-hop communication. The objective of their policy is to maximize throughput of a communication system, but they designed a reward function based on the total power assigned to transmissions, which they claimed was correlated. They simulated a communication environment and approximated the state space with discrete features that indicate battery level and its constraints, harvested energy over an hour, characteristics of the communication link, data arrival process, and the data buffers at the communicating nodes. Similarly, @cite_13 , used an actor-critic method with linear function approximation to learn approximation for both policy and value function. They used a Gaussian policy to generate continuous values of bounded packet rates and summarized the state space by continuous values of the current residual energy. The objective is to maximize transmitted packet rate while sustaining perpetual operation; hence they designed a reward function to be a multiplication of normalized residual energy and the packet rate.
@cite_13 @cite_11
2786137637 2614312052
A promising solution for achieving autonomous wireless sensor networks is to enable each node to harvest energy in its environment. To address the time-varying behavior of energy sources, each node embeds an energy manager responsible for dynamically adapting the power consumption of the node in order to maximize the quality of service while avoiding power failures. A novel energy management algorithm based on reinforcement learning (RLMan) is proposed in this paper. By continuously exploring the environment, RLMan adapts its energy management policy to time-varying environment, regarding both the harvested energy and the energy consumption of the node. Linear function approximations are used to achieve very low computational and memory footprint, making RLMan suitable for resource-constrained systems, such as wireless sensor nodes. Moreover, RLMan only requires the state of charge of the energy storage device to operate, which makes it practical to implement. Exhaustive simulations using real measurements of indoor light and outdoor wind show that RLMan outperforms current state-of-the-art approaches, by enabling almost 70 gain regarding the average packet rate. Moreover, RLMan is more robust to variability of the node energy consumption. Energy harvesting (EH) two-hop communications are considered. The transmitter and the relay harvest energy from the environment and use it exclusively for transmitting data. A data arrival process is assumed at the transmitter. At the relay, a finite data buffer is used to store the received data. We consider a realistic scenario in which the EH nodes have only local causal knowledge, i.e., at any time instant, each EH node only knows the current value of its EH process, channel state, and data arrival process. Our goal is to find a power allocation policy to maximize the throughput at the receiver. We show that because the EH nodes have local causal knowledge, the two-hop communication problem can be separated into two point-to-point problems. Consequently, independent power allocation problems are solved at each EH node. To find the power allocation policy, reinforcement learning with linear function approximation is applied. Moreover, to perform function approximation two feature functions which consider the data arrival process are introduced. Numerical results show that the proposed approach has only a small degradation as compared to the offline optimum case. Furthermore, we show that with the use of the proposed feature functions a better performance is achieved compared to standard approximation techniques.