Urban wastewater sector is being pushed to optimize processes in order to reduce energy consumption without compromising its quality standards. Energy costs can represent a significant share of the global operational costs (between 50 and 60 ) in an intensive energy consumer. Pumping is the largest consumer of electrical energy in a wastewater treatment plant. Thus, the optimal control of pump units can help the utilities to decrease operational costs. This work describes an innovative predictive control policy for wastewater variable-frequency pumps that minimize electrical energy consumption, considering uncertainty forecasts for wastewater intake rate and information collected by sensors accessible through the Supervisory Control and Data Acquisition system. The proposed control method combines statistical learning (regression and predictive models) and deep reinforcement learning (Proximal Policy Optimization). The following main original contributions are produced: i) model-free and data-driven predictive control; ii) control philosophy focused on operating the tank with a variable wastewater set-point level; iii) use of supervised learning to generate synthetic data for pre-training the reinforcement learning policy, without the need to physically interact with the system. The results for a real case-study during 90 days show a 16.7 decrease in electrical energy consumption while still achieving a 97 reduction in the number of alarms (tank level above 7.2 meters) when compared with the current operating scenario (operating with a fixed set-point level). The numerical analysis showed that the proposed data-driven method is able to explore the trade-off between number of alarms and consumption minimization, offering different options to decision-makers.
Considering the revised literature, the present paper produces the following original contributions. Applies a model-free and data-driven control approach based in RL, in contrast to the use of meta-heuristics @cite_14 @cite_9 @cite_34 or fuzzy logic control @cite_19 . The control philosophy is focused in operating the tank with a variable water level, instead of controlling the frequency increase decrease rate like in @cite_19 . The vector of system states from RL is modified to include probabilistic forecasts of wastewater intake and implement a predictive control of the pumping system, which was never proposed in the literature. Finally, a real-world implementation of the RL control method is made possible by applying data-mining algorithms to construct models from data and generate synthetic data for pre-training the RL algorithm, without the need to physically interact with the system. This also represents an original contribution compared to other control problems like @cite_24 @cite_23 .
@cite_14 @cite_9 @cite_24 @cite_19 @cite_23 @cite_34
1925816294 2130751706 2163932785 2152004941
With the goal to generate more scalable algorithms with higher efficiency and fewer open parameters, reinforcement learning (RL) has recently moved towards combining classical techniques from optimal control and dynamic programming with modern learning techniques from statistical estimation theory. In this vein, this paper suggests to use the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton-Jacobi-Bellman (HJB) equations, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The update equations have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Our new algorithm demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a simulated 12 degree-of-freedom robot dog illustrates the functionality of our algorithm in a complex robot learning scenario. We believe that Policy Improvement with Path Integrals (PI2) offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL based on trajectory roll-outs. This article investigates the vulnerabilities of Supervisory Control and Data Acquisition (SCADA) systems which monitor and control the modern day irrigation canal systems. This type of monitoring and control infrastructure is also common for many other water distribution systems. We present a linearized shallow water partial differential equation (PDE) system that can model water flow in a network of canal pools which are equipped with lateral offtakes for water withdrawal and are connected by automated gates. The knowledge of the system dynamics enables us to develop a deception attack scheme based on switching the PDE parameters and proportional (P) boundary control actions, to withdraw water from the pools through offtakes. We briefly discuss the limits on detectability of such attacks. We use a known formulation based on low frequency approximation of the PDE model and an associated proportional integral (PI) controller, to create a stealthy deception scheme capable of compromising the performance of the closed-loop system. We test the proposed attack scheme in simulation, using a shallow water solver; and show that the attack is indeed realizable in practice by implementing it on a physical canal in Southern France: the Gignac canal. A successful field experiment shows that the attack scheme enables us to steal water stealthily from the canal until the end of the attack. Received 28 May 2002; revised 18 September 2002; accepted 24 September 2002; published 14 February 2003. [1] When an inverse problem is solved to estimate an unknown function such as the hydraulic conductivity in an aquifer or the contamination history at a site, one constraint is that the unknown function is known to be everywhere nonnegative. In this work, we develop a statistically rigorous method for enforcing function nonnegativity in Bayesian inverse problems. The proposed method behaves similarly to a Gaussian process with a linear variogram (i.e., unrestricted Brownian motion) for parameter values significantly greater than zero. The method uses the method of images to define a prior probability density function based on reflected Brownian motion that implicitly enforces nonnegativity. This work focuses on problems where the unknown is a function of a single variable (e.g., time). A Markov chain Monte Carlo (MCMC) method, specifically, a highly efficient Gibbs sampler, is implemented to generate conditional realizations of the unknown function. The new method is applied to the estimation of the trichloroethylene (TCE) and perchloroethylene (PCE) contamination history in an aquifer at Dover Air Force Base, Delaware, based on concentration profiles obtained from an underlying aquitard. INDEX TERMS: 1831 Hydrology: Groundwater quality; 1869 Hydrology: Stochastic processes; 3260 Mathematical Geophysics: Inverse theory; KEYWORDS: stochastic inverse modeling, contaminant source identification, inference under constraints, Markov chain Monte Carlo (MCMC), Gibbs sampling, Bayesian inference A data-driven model for scheduling pumps in a wastewater treatment process is proposed. The objective is to minimize the cost of pump operations and maintenance. A neural network algorithm is applied to model performance of the pumps using the data collected at a municipal wastewater treatment plant. The discrete-state Markov process is utilized to develop a model of maintenance decisions. The developed pump performance and maintenance models are integrated into a scheduling model. A hierarchical particle swarm optimization algorithm is designed to solve the proposed scheduling model. The concepts developed in this paper are illustrated with two case studies.