Simulation Based Stochastic Optimal Control Design and Its Application to Building Control Problems
The goal of this paper is to study the potential applicability and performance of stochastic approximationbased optimal control designs and its application to office building HVAC (Heating, Ventilation, and Air conditioning) control systems. We consider stochastic systems whose behavio r cannot be easily described by mathematical tools. For example, a real world building thermal dynamics is affected by occupant interactions, and its probability density function depends on the state variables. In this case, existing approaches, for instance, stochastic model predictive control methods, cannot be applied to design optimal controllers. To overcome this difficulty, we construct a simulation oracle, approximate the gradient of the cost function, and apply a gradient descent type algorithm to design an optimal control policy. We assess the performance of the proposed approach through a simulation study of building HVAC systems. I. I NTRODUCTION Optimal control designs for stochastic systems have been an active area of research during the last decades. A classic al approach is the linear quadratic Gaussian (LQG) control, which computes optimal control policy for linear timeinvariant (LTI) systems with Gaussian random disturbances Another main research direction is stochastic model predic tive controls (MPCs) For example, under Gaussian multiplicative disturbances, a stochastic MPC is formulated as online semidefinite programming problems (SDPs) in [1] However, Gaussian disturbances cannot describe more complicated behavior of real world systems. For this reason, it is of great importance to develop optimal control designs for systems with more generic stochastic disturbances to meet practical needs. For example, systems with disturbances/uncertainties that depend on Markov chains arise in many applications, for instance, the vehicle pathplanning [2] [3] macroeconomic model [4] economic models of government expenditure [5] vehicle controls with driver's behavior models [6] [7] occupant models based on Markov chains for building climate control [8] [9] and hybrid electric vehicle powertrain management [10] However such complicated stochastic systems cannot be formulated as easily solvable optimization problems in general. One tractable approach is the scenario based (or sample based approximation approaches [2] [3] [11] [14] Its signific ant advantage is the ability to cope with generic probability This material is based upon work supported by the National Sci ence Foundation under Grant No. 1539527 D. Lee and J. Hu are with the Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 4 7906, USA lee1923@purdue.edu, jianghai@purdue.edu. S. Lee and P. Karava are with the Department of Civil Engineeri g, Purdue University, Purdue University, West Lafayette, IN 4 7906, USA lee1904@purdue.edu, pkarava@purdue.edu. distributions as long as a sufficient number of random samples can be obtained. The approach was successfully applied to robot path planning problems in [2] [3] and the aircraft conflict detection in [11] Unfortunately, there are still more challenging practical situations for which the scenario based control design scheme s are not applicable. For example, consider the case that probabilities of random disturbances/uncertainties depe nd on environmental factors such as the current state variables of the control system. The probabilities then depend on the control policy and the corresponding design parameters as well. In this case, state samples generated by using fixed controller parameters cannot reflect changes of the controller parameters themselves. Therefore, the scenari obased approaches cannot be used. Such cases arise in many applications. For example, stochastic systems with distur bances modelled by Markov chains were considered in [10] [15] for hybrid electric vehicle powertrain management pro blems, where transition probabilities of the Markov chain depend on the state of the dynamic system. One possible approach to solve these optimal control problems is to use the approximate dynamic programming (ADP) [16] [18] (or reinforcement learning [18] from the machine learning context) For instance, [10] used an infinite horizon ADP method, while [15] used a finite horizon ADP method. Although the ADP is one viable approach to deal with stochastic systems mentioned above, it also has drawbacks. First, the infinite horizon ADP cannot capture time varyin g and periodic behavior of dynamic systems. This restricts the applicability of the infinite horizon ADP to real world control problems. On the other hand, the finite horizon ADP is suitable for dealing with time varying and periodic natu re of the system. However, it has high computational cost. Contrary to the infinite horizon case, the finite horizon AD P should approximateN number of value functions, where N is the time horizon. This means that when N is relatively large and if the system does not have the Markov property, then the numbers of feature functions and parameters to approximateN value functions [17] increase quickly, making the problem numerically inefficient or intractable. On the other hand, an important application is building climate control, whose main goal is to balance between the energy consumption and occupants' comfort in work environments. There has been a great amount of research interest in energy consumption and comfort management in buildings [19] One of the main difficulties in building control problems is the presence of uncertainties and disturbances, such as weather and occupant interactions, that cannot be exactly predicted in general. For this reason, stochastic control methods have been used. For example, stochastic MPCs have been studied in [20] [23] and occupant models based on Markov chains have been developed in [8] [9] for building controls. Especially, the role of occupants is significant in the thermal dynamics of building spaces [9] [24] [25] However, the Markov chain models cannot describe more realistic occupant behaviors in gener al because they are more complicated functions of environmental variables. In building environment research, advan ced occupant thermal preference models have been developed, e.g. [26] [28] where occupant's thermal preferences are expressed as probability density functions that depend on environmental factors, for example, the indoor air temperature. Note that the thermal preferences induce occupant's actions that perturb the thermal dynamics of building space s. Thus, a building space with occupants is a stochastic system whose probabilistic behavior depends on the state variable s. Therefore, the aforementioned difficulties arise, i.e. ex cept for ADPs, there is no efficient approach to design optimal controls. Motivated by these discussions, the goal of this paper is to study an efficient method to approximately solve optimal control problems for aforementioned stochastic systems an d to assess its applicability/peroformance. The main contri butions of the paper are two fold: First, we apply the smoothin g and stochastic approximation procedure studied in [29] [30] to stochastic optimal control problems where the cost function is non differentiable and non convex. Second, th e proposed method is applied to building HVAC (Heating, Ventilation, and Air conditioning) system control problem s, and the validity of the proposed approach is evaluated. For the first contribution, a gradient descent type algorith m is applied to the cost function J) minimization of the finitehorizon optimal control problem without forming it as an MPC or dynamic programming problem. The main difficulty is that there is no explicit analytic/numerical formulatio n f the gradientJ as well as the cost functionJ itself. In addition, we assume that the cost function is non convex and non smooth; hence, even the gradient does not exist. To solve it, we follow the convolutional function smoothing an d stochastic gradient approximation procedure studied in [2 9] [30] which found their applications in the yield optimization [31] and circuit design [32] More recent advances of the procedure can be found in [33] for convex optimizations and in [34] for more rigorous and general analysis of the approach. The smoothed function is an approximation of the original one, and described by an expectation of the original function with respect to a random variable that perturbs its optimization parameters. Benefits are two fol d: the smoothed approximation is differentiable, and its grad ient can be stochastically approximated by sample means. For the second contribution, we apply the recent occupant thermal preference model constructed based on a novel Bayesian modelling approach in [28] The occupant's feelin g of cold, comfort, and hot is described by a probability densi ty function which varies according to the indoor air temperature. The occupant's feeling or cognition induces actions t hat perturb thermal dynamics; however, the connection between the cognition and actions has not been fully studied so far. For simulation, we assume a simple and reasonable stochasti map between the cognition and actions, e.g. occupant's overriding on the current temperature set point. II. PRELIMINARIES The adopted notation is as follows: N and N+ sets of nonnegative and positive integers, respectively; R: set of real numbers;R+ set of nonnegative real numbers; R+ set of positive real numbers; R: n dimensional Euclidean space; R set of all n x m real matrices;A transpose of matrix A; any norm of a vector or a matrix; for a set S, |S| cardinality of the setS; E{ expectation operator; N (x, S) Gaussian distribution with mean x and covariance matrix S. In this paper, we consider the discrete time stochastic system x(k 1) f(x(k) u(k) w(k) x(0) z R, (1) wherek {1, 2, N 1} is the finite time step, x(k) R n is the state,u(k) R