Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals. We show such a reward function and use policy-gradient approaches to learn capable policies, leading to behavior more appropriate for IoT nodes with less manual design effort, increasing the level of autonomy in IoT.
Several works have already adopted RL within various contexts of IoT. An example is a dynamic power manager for energy-harvesting wireless sensor networks @cite_6 . Here used the Q-learning algorithm to train an agent to choose an action from four levels of duty cycles. Their environment state space is based on the distance from energy neutrality (i.e., the difference between harvested and consumed energy), the harvested energy, and the current battery level, while the reward function is based on the distance from energy neutrality and the current energy storage level. They extended this work in @cite_19 to include quality of service (QoS), both in the state space and the reward function. Additionally, in @cite_4 , they included meeting a requested throughput along with the energy level in the reward function and added penalizing terms for overcharging, deep-discharge, and depletion of the energy storage. Furthermore, in @cite_15 and @cite_18 , they used fuzzy decision processes to model an energy harvesting node as a fuzzy environment and used a modified Q-learning with fuzzy reward to train an RL agent.
@cite_18 @cite_4 @cite_6 @cite_19 @cite_15
1645543104 2163190245 2140780201 2002130617
This paper considers a scenario that a wireless sensor node is powered by harvesting energy from ambient solar energy and required the sustainable operation. Fuzzy inference system with reinforcement learning is used in this study for the dynamic energy management of the node in the energy harvesting environment. By interacting with the given environment, the proposed method adjusts the duty cycle of the sensing task in the node according to the proposed method. The outcomes of these interactions are evaluated by fuzzy inference system in terms of reward that express how well the duty-cycle adjustments in satisfying the given requirement of energy neutrality. Experimental results show that the proposed method obtained better convergence and less root mean square deviation off energy neutral level in comparing with other methods. In this paper, a reinforcement learning-based throughput on demand (ToD) provisioning dynamic power management method (RLTDPM) is proposed for sustaining perpetual operation and satisfying the ToD requirements for today's energy harvesting wireless sensor node (EHWSN). The RLTDPM monitors the environmental state of the EHWS and adjusts their operational duty cycle under criteria of energy neutrality to meet the demanded throughput. Outcomes of these observation-adjustment interactions are then evaluated by feedback reward that represents how well the ToD requests are met; subsequently, the observation-adjustment-evaluation process, so-called reinforcement learning, continues. After the learning process, the RLTDPM is able to autonomously adjust the duty cycle for satisfying the ToD requirement, and in doing so, sustain the perpetual operation of the EHWSN. Simulations of the proposed RLTDPM on a wireless sensor node powered by a battery and solar cell for image sensing tasks were performed. Experimental results demonstrate that the achieved demanded throughput is improved 10.7 for the most stringent ToD requirement, while the residual battery energy of the RLTDPM is improved 7.4 compared with an existing DPM algorithm for EHWSN with image sensing purpose. A reinforcement learning (RL) method for Quality of Service (QoS)-aware power management (PM) of an energy harvesting wireless sensor network, named QoS-aware RLPM, is proposed in this paper. The RL environment for each sensor node is represented by the observable measurements of harvesting energy and residual energy stored in the battery. To achieve QoS-awareness, the proposed QoS-aware RLPM attempts to satisfy various QoS demands by adjusting the duty-cycle rate of sensor node according to the observable measurements. The outcomes of these interactions are evaluated by rewards that express how well the duty-cycle rate adjustments satisfy QoS requests under energy neutrality criteria [1]. The QoS-aware RLPM learns the adjustment strategy of the duty-cycle rate by interacting with the environment and at the same time the QoS of the sensor network is maintained. Experiment results show that the proposed method satisfies the QoS requests under the energy neutrality criteria and performs better than the adaptive duty-cycling method of [1]. This paper considers a scenario that wireless sensor node is powered by harvesting energy with the characteristics of ambiguity and uncertainty. Reinforcement learning with fuzzy reward (RLFR) is used in this study for the dynamic power management of such energy harvesting wireless sensor nodes. By interacting with the given environment, the RLFR adjusts the duty-cycle in data sensing task according to the variable incoming energy related signals. The outcomes of these interactions are evaluated by fuzzy reward that express how well the duty-cycle adjustments in satisfying given requirement of energy neutrality. Simulation results show that the RLFR not only satisfies the sensing requirement in maintaining energy neutrality, but it also achieves better energy utilization in terms of residual battery energy in comparing with another existing dynamic power management method.