Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals. We show such a reward function and use policy-gradient approaches to learn capable policies, leading to behavior more appropriate for IoT nodes with less manual design effort, increasing the level of autonomy in IoT.
@cite_17 used a SARSA( @math ) RL algorithm to develop adaptive power management for a solar-energy harvesting sensor node. To simulate a sensor node, they used a scaled-up version of a real sensor powered by a battery and a solar panel, and used solar radiation data to calculate hourly harvested energy. They designed a reward function based the distance from energy neutrality, defined as the difference between the current level of energy and the optimum battery level, which they calculated to be @math of the battery's maximum level. They trained an agent in episodes of 24 hours and rewarded it at the end of a training episode. The state space consisted of discretized information about the battery level, the distance from energy neutrality, the harvested energy, and the weather forecast, which they approximate by calculating the total amount of energy harvested in a particular day.
@cite_17
2756881949 2122021409 2902867577 2140780201
In this paper, we present an adaptive power manager for solar energy harvesting sensor nodes. We use a simplified model consisting of a solar panel, an ideal battery and a general sensor node with variable duty cycle. Our power manager uses Reinforcement Learning (RL), specifically SARSA(λ) learning, to train itself from historical data. Once trained, we show that our power manager is capable of adapting to changes in weather, climate, device parameters and battery degradation while ensuring near-optimal performance without depleting or overcharging its battery. Our approach uses a simple but novel general reward function and leverages the use of weather forecast data to enhance performance. We show that our method achieves near perfect energy neutral operation (ENO) with less than 6 root mean square deviation from ENO as compared to more than 23 deviation that occur when using other approaches. Harvesting energy from the environment is feasible in many applications to ameliorate the energy limitations in sensor networks. In this paper, we present an adaptive duty cycling algorithm that allows energy harvesting sensor nodes to autonomously adjust their duty cycle according to the energy availability in the environment. The algorithm has three objectives, namely (a) achieving energy neutral operation, i.e., energy consumption should not be more than the energy provided by the environment, (b) maximizing the system performance based on an application utility model subject to the above energy-neutrality constraint, and (c) adapting to the dynamics of the energy source at run-time. We present a model that enables harvesting sensor nodes to predict future energy opportunities based on historical data. We also derive an upper bound on the maximum achievable performance assuming perfect knowledge about the future behavior of the energy source. Our methods are evaluated using data gathered from a prototype solar energy harvesting platform and we show that our algorithm can utilize up to 58 more environmental energy compared to the case when harvesting-aware power management is not used. With the advent of the Internet of Things (IoT), an increasing number of energy harvesting methods are being used to supplement or supplant battery based sensors. Energy harvesting sensors need to be configured according to the application, hardware, and environmental conditions to maximize their usefulness. As of today, the configuration of sensors is either manual or heuristics based, requiring valuable domain expertise. Reinforcement learning (RL) is a promising approach to automate configuration and efficiently scale IoT deployments, but it is not yet adopted in practice. We propose solutions to bridge this gap: reduce the training phase of RL so that nodes are operational within a short time after deployment and reduce the computational requirements to scale to large deployments. We focus on configuration of the sampling rate of indoor solar panel based energy harvesting sensors. We created a simulator based on 3 months of data collected from 5 sensor nodes subject to different lighting conditions. Our simulation results show that RL can effectively learn energy availability patterns and configure the sampling rate of the sensor nodes to maximize the sensing data while ensuring that energy storage is not depleted. The nodes can be operational within the first day by using our methods. We show that it is possible to reduce the number of RL policies by using a single policy for nodes that share similar lighting conditions. A reinforcement learning (RL) method for Quality of Service (QoS)-aware power management (PM) of an energy harvesting wireless sensor network, named QoS-aware RLPM, is proposed in this paper. The RL environment for each sensor node is represented by the observable measurements of harvesting energy and residual energy stored in the battery. To achieve QoS-awareness, the proposed QoS-aware RLPM attempts to satisfy various QoS demands by adjusting the duty-cycle rate of sensor node according to the observable measurements. The outcomes of these interactions are evaluated by rewards that express how well the duty-cycle rate adjustments satisfy QoS requests under energy neutrality criteria [1]. The QoS-aware RLPM learns the adjustment strategy of the duty-cycle rate by interacting with the environment and at the same time the QoS of the sensor network is maintained. Experiment results show that the proposed method satisfies the QoS requests under the energy neutrality criteria and performs better than the adaptive duty-cycling method of [1].