We consider a smart-grid connecting several agents, modeled as stochastic dynamical systems, who may be electricity consumers producers. At each discrete time instant, which may represent a 15 minute interval, each agent may consume generate some quantity of electrical energy. The Independent System Operator (ISO) is given the task of assigning consumptions generations to the agents so as to maximize the sum of the utilities accrued to the agents, subject to the constraint that energy generation equals consumption at each time. This task of coordinating generation and demand has to be accomplished by the ISO without the agents revealing their system states, dynamics, or utility cost functions. We show how and when a simple iterative procedure converges to the optimal solution. The ISO iteratively obtains electricity bids by the agents, and declares the tentative market clearing prices. In response to these prices, the agents submit new bids. On the demand side, the solution yields an optimal demand response for dynamic and stochastic loads. On the generation side, it provides the optimal utilization of stochastically varying renewables such as solar wind, and generation with fossil fuel based generation with dynamic constraints such as ramping rates. Thereby we solve a decentralized stochastic control problem, without agents sharing any information about their system models, states or utility functions.
We show below that the ISO Problem is an example of a decentralized control system with non-classical observation patterns in which signaling can successfully result in globally optimum performance. The agents need not reveal their observations, state values, their individual system dynamics or their individual cost-functions. We construct concrete signaling schemes which encode-decode the information required in order to recover the same performance as that of centralized control. From the economics side, this work is an extension of general equilibrium theory @cite_2 . To the authors' knowledge there does not appear to be any similar result for coordinating multiple LQG systems or the efficiency of the simplified signaling.
@cite_2
1511433378
We consider a multi-agent system where, at every time instant, many users choose to use one of multiple resources, whose performance depends on the number of concurrent users. In addition, a central agent has up-to-date knowledge of the congestion across all resources. What signals should the central agent broadcast to the individual users in order to reduce the total congestion? We model users with varying levels of aversion to risk, and a heterogeneous population of users over these levels. We consider signaling schemes that communicate for each resource an interval of possible congestion values, instead of scalar values. We show how to optimise over these intervals with respect to the social cost, under constraints that these intervals be consistent with past observations. Moreover, under mild assumptions, the resulting distribution of users across resources converges. We consider linear quadratic Gaussian (LQG) games in large population systems where the agents evolve according to nonuniform dynamics and are coupled via their individual costs. A state aggregation technique is developed to obtain a set of decentralized control laws for the individuals which possesses an epsiv-Nash equilibrium property. A stability property of the mass behavior is established, and the effect of inaccurate population statistics on an isolated agent is also analyzed by variational techniques. The classic Vickrey-Clarke-Groves (VCG) mechanism ensures incentive compatibility, i.e., that truth-telling of all agents is a dominant strategy, for a static one-shot game. However, in a dynamic environment that unfolds over time, the agents' intertemporal payoffs depend on the expected future controls and payments, and a direct extension of the VCG mechanism is not sufficient to guarantee incentive compatibility. In fact, it does not appear to be feasible to construct mechanisms that ensure the dominance of dynamic truth-telling for agents comprised of general stochastic dynamic systems. The contribution of this paper is to show that such a dynamic stochastic extension does exist for the special case of Linear-Quadratic-Gaussian (LQG) agents with a careful construction of a sequence of layered payments over time. We propose a layered version of a modified VCG mechanism for payments that decouples the intertemporal effect of current bids on future payoffs, and prove that truth-telling of dynamic states forms a dominant strategy if system parameters are known and agents are rational. An important example of a problem needing such optimal dynamic coordination of stochastic agents arises in power systems where an Independent System Operator (ISO) has to ensure balance of generation and consumption at all time instants, while ensuring social optimality (maximization of the sum of the utilities of all agents). Addressing strategic behavior is critical as the price-taking assumption on market participants may not hold in an electricity market. Agents, can lie or otherwise game the bidding system. The challenge is to determine a bidding scheme between all agents and the ISO that maximizes social welfare, while taking into account the stochastic dynamic models of agents, since renewable energy resources such as solar wind are stochastic and dynamic in nature, as are consumptions by loads which are influenced by factors such as local temperatures and thermal inertias of facilities. We aim to reduce the social cost of congestion in many smart city applications. In our model of congestion, agents interact over limited resources after receiving signals from a central agent that observes the state of congestion in real time. Under natural models of agent populations, we develop new signalling schemes and show that by introducing a non-trivial amount of uncertainty in the signals, we reduce the social cost of congestion, i.e., improve social welfare. The signalling schemes are efficient in terms of both communication and computation, and are consistent with past observations of the congestion. Moreover, the resulting population dynamics converge under reasonable assumptions. A smart grid connects several electricity consumers producers, e.g., wind solar storage farms, fossil-fuel plants, industrial commercial loads, or load-serving aggregators, all modeled as stochastic dynamical systems. In each time period, each consumes supplies some electrical energy. Each such agent's utility is the benefit accrued from its consumption or the negative of its generation cost. The social welfare, the sum of all these utilities, is the total benefit accrued from all consumption minus the total cost of generation. The independent system operator is charged with maximizing the social welfare subject to total generation equalling consumption in each time period, but without the agents revealing their system states, dynamic models, utility functions, or uncertainties. It has to announce prices after interacting with agents via bid-price interactions. This paper examines the case where the agents respond in a compliant price-taking manner. It is shown that there is an iterative bid-price interaction, where agents respond to price announcements by complying to the requirement to announce their optimal responses according to their true stochastic model, or, in the case where the agents are linear quadratic Gaussian (LQG) systems, according to deterministic versions of their true stochastic models, that leads to the same global maximum value of social welfare attainable if all agents had pooled their information. In the important LQG case, the bid-price iteration is dramatically simple, exchanging only real-valued vectors of future prices and consumptions generations at each time step. Agents need not even know of the existence of other agents. DC power flow equations can also be incorporated. The results may be of broader interest vis-a-vis general equilibrium theory of economics for stochastic dynamic agents. So far, most equilibrium concepts in game theory require that the rewards and actions of the other agents are known and or observed by all agents. However, in real life problems, agents are generally faced with situations where they only have partial or no knowledge about their environment and the other agents evolving in it. In this context, all an agent can do is reasoning about its own payoffs and consequently, cannot rely on classical equilibria through deliberation, which requires full knowledge and observability of the other agents. To palliate to this difficulty, we introduce the satisfaction principle from which an equilibrium can arise as the result of the agents' individual learning experiences. We define such an equilibrium and then we present different algorithms that can be used to reach it. Finally, we present experimental results that show that using learning strategies based on this specific equilibrium, agents will generally coordinate themselves on a Pareto-optimal joint strategy, that is not always a Nash equilibrium, even though each agent is individually rational, in the sense that they try to maximize their own satisfaction. Stochastic dynamic control systems relate in a prob- abilistic fashion the space of control signals to the space of corresponding future states. Consequently, stochastic dynamic systems can be interpreted as an information channel between the control space and the state space. In this work we study this control-to-state informartion capacity of stochastic dynamic systems in continuous-time, when the states are observed only partially. The control-to-state capacity, known as empowerment, was shown in the past to be useful in solving various Artificial Intelligence & Control benchmarks, and was used to replace problem-specific utilities. The higher the value of empowerment is, the more optional future states an agent may reach by using its controls inside a given time horizon. The contribution of this work is that we derive an efficient solution for computing the control-to-state information capacity for a linear, partially-observed Gaussian dynamic control system in continuous time, and discover new relationships between control-theoretic and information-theoretic properties of dynamic systems. Particularly, using the derived method, we demonstrate that the capacity between the control signal and the system output does not grow without limits with the length of the control signal. This means that only the near-past window of the control signal contributes effectively to the control-to-state capacity, while most of the information beyond this window is irrelevant for the future state of the dynamic system. We show that empowerment depends on a time constant of a dynamic system. Software technology is undergoing a transition form monolithic systems, constructed according to a single overall design, into conglomerates of semiautonomous, heterogeneous, and independently designed subsystems, constructed and managed by different organizations, with little, if any, knowledge of each other. Among the problems inherent in such conglomerates, none is more serious than the difficulty to control the activities of the disparate agents operating in it, and the difficulty for such agents to coordinate their activities with each other. We argue that the nature of coordination and control required for such systems calls for the following principles to be satisfied: (1) coordination policies need to be enforced: (2) the enforcement needs to be decentralized; and (3) coordination policies need to be formulated explicitly—rather than being implicit in the code of the agents involved—and they should be enforced by means of a generic, broad spectrum mechanism; and (4) it should be possible to deploy and enforce a policy incrementally, without exacting any cost from agents and activities not subject to it. We describe a mechansim called law-governed interaction (LGI), currently implemented by the Moses toolkit, which has been designed to satisfy these principles. We show that LGI is at least as general as a conventional centralized coordination mechanism (CCM), and that it is more scalable, and generally more efficient, then CCM. Motivated in part by the “rational inattention” framework of information-constrained decision-making by economic agents, we have recently introduced a general model for average-cost optimal control of Markov processes subject to mutual information constraints [1]. The optimal information-constrained control problem reduces to an infinite-dimensional convex program and admits a decomposition based on the Bellman error, which is the object of study in approximate dynamic programming. In this paper, we apply our general theory to an information-constrained variant of the scalar linear-quadratic-Gaussian (LQG) control problem. We give an upper bound on the optimal steady-state value of the quadratic performance objective and present explicit constructions of controllers that achieve this bound. We show that the obvious certainty-equivalent control policy is suboptimal when the information constraints are very severe, and exhibit another policy that performs better in this low-information regime. In the two extreme cases of no information (open-loop) and perfect information, these two policies coincide with the optimum. Plug-in electric vehicles (PEVs) can be considered flexible loads, as the time when they are charged can be shifted to a certain extent without impacting the drivers' mobility. An aggregator coordinating these flexible resources aims to minimize the costs of charging, subject to individual PEV constraints, imposed by battery and charging infrastructure characteristics, as well as driving patterns. Since driving behavior cannot be perfectly forecasted, this problem is stochastic. In this paper, we propose a decentralized control algorithm to coordinate charging, based on the Alternating Direction Method of Multipliers (ADMM). In this setup, the aggregator and PEVs find the global solution by individually solving local optimization problems. The solution is found iteratively, whereby information between the PEVs and the aggregator is exchanged at each iteration. When the objective function of the charging optimization problem and the PEV constraints are convex, the algorithm converges to the global optimum. To take driving behavior uncertainty into account, the scheme considers several scenarios of driving patterns for each vehicle. A receding time horizon optimization is used, whereby at each new stage the representation of the fleet is updated with new scenarios consistent with the current observations. The local optimization problems, which can be solved very fast, could be solved in parallel for each PEV and scenario using decentralized computing, making the approach suitable for large-scale problems. A numerical example shows how this method can be applied to flatten the system load. Global games are games of incomplete information whose type space is determined by the players each observing a noisy signal of the underlying state. With strategic complementarities, global games often have a unique, dominance solvable equilibrium, allowing analysis of a number of economic models of coordination failure. For symmetric binary action global games, equilibrium strategies in the limit (as noise becomes negligible) are simple to characterize in terms of 'diffuse' beliefs over the actions of others. We describe a number of economic applications that fall in this category. We also explore the distinctive roles of public and private information in this setting, review results for general global games, discuss the relationship between global games and a literature on higher order beliefs in game theory and describe the relationship to local interaction games and dynamic games with payoff shocks. Brain-computer interfaces (BCIs) use brain signals to convey a user's intent. Some BCI approaches begin by decoding kinematic parameters of movements from brain signals, and then proceed to using these signals, in absence of movements, to allow a user to control an output. Recent results have shown that electrocorticographic (ECoG) recordings from the surface of the brain in humans can give information about kinematic parameters (e.g., hand velocity or finger flexion). The decoding approaches in these demonstrations usually employed classical classification regression algorithms that derive a linear mapping between brain signals and outputs. However, they typically only incorporate little prior information about the target kinematic parameter. In this paper, we show that different types of anatomical constraints that govern finger flexion can be exploited in this context. Specifically, we incorporate these constraints in the construction, structure, and the probabilistic functions of a switched non-parametric dynamic system (SNDS) model. We then apply the resulting SNDS decoder to infer the flexion of individual fingers from the same ECoG dataset used in a recent study. Our results show that the application of the proposed model, which incorporates anatomical constraints, improves decoding performance compared to the results in the previous work. Thus, the results presented in this paper may ultimately lead to neurally controlled hand prostheses with full fine-grained finger articulation. When designing optimal controllers for any system, it is often the case that the true state of the system is unknown to the controller. Imperfect state information must be taken into account in the controller’s design in order to preserve its optimality. The same is true when performing reachability calculations. To estimate the probability that the state of a stochastic system reaches, or stays within, some set of interest in a given time horizon, it is necessary to find a controller that drives the system to that set with maximum probability, given the controller’s knowledge of the true state of the system. To date, little work has been done on stochastic reachability calculations with partially observable states. The work that has been done relies on converting the reachability optimization problem to one with an additive cost function, for which theoretical results are well known. Our approach is to preserve the multiplicative cost structure when deriving a sufficient statistic that reduces the problem to one of perfect state information. Our transformation includes a change of measure that simplifies the distribution of the sufficient statistic conditioned on its previous value. We develop a dynamic programming recursion for the solution of the equivalent perfect information problem, proving that the recursion is valid, an optimal solution exists, and results in the same solution as to the original problem. We also show that our results are equivalent to those for the reformulated additive cost problem, and so such a reformulation is not required. This paper develops a strategy to coordinate the charging of autonomous plug-in electric vehicles (PEVs) using concepts from non-cooperative games. The foundation of the paper is a model that assumes PEVs are cost-minimizing and weakly coupled via a common electricity price. At a Nash equilibrium, each PEV reacts optimally with respect to a commonly observed charging trajectory that is the average of all PEV strategies. This average is given by the solution of a fixed point problem in the limit of infinite population size. The ideal solution minimizes electricity generation costs by scheduling PEV demand to fill the overnight non-PEV demand “valley”. The paper's central theoretical result is a proof of the existence of a unique Nash equilibrium that almost satisfies that ideal. This result is accompanied by a decentralized computational algorithm and a proof that the algorithm converges to the Nash equilibrium in the infinite system limit. Several numerical examples are used to illustrate the performance of the solution strategy for finite populations. The examples demonstrate that convergence to the Nash equilibrium occurs very quickly over a broad range of parameters, and suggest this method could be useful in situations where frequent communication with PEVs is not possible. The method is useful in applications where fully centralized control is not possible, but where optimal or near-optimal charging patterns are essential to system operation. Reinforcement Learning was originally developed for Markov Decision Processes (MDPs). It allows a single agent to learn a policy that maximizes a possibly delayed reward signal in a stochastic stationary environment. It guarantees convergence to the optimal policy, provided that the agent can sufficiently experiment and the environment in which it is operating is Markovian. However, when multiple agents apply reinforcement learning in a shared environment, this might be beyond the MDP model. In such systems, the optimal policy of an agent depends not only on the environment, but on the policies of the other agents as well. These situations arise naturally in a variety of domains, such as: robotics, telecommunications, economics, distributed control, auctions, traffic light control, etc. In these domains multi-agent learning is used, either because of the complexity of the domain or because control is inherently decentralized. In such systems it is important that agents are capable of discovering good solutions to the problem at hand either by coordinating with other learners or by competing with them. This chapter focuses on the application reinforcement learning techniques in multi-agent systems. We describe a basic learning framework based on the economic research into game theory, and illustrate the additional complexity that arises in such systems. We also described a representative selection of algorithms for the different areas of multi-agent reinforcement learning research. Following the foundational work that established basic ideas for optimum distributed defection schemes using multiple sensors (as reviewed in Part I of this two-part review), further work on distributed detection has developed many useful and interesting extensions of the basic concepts. These more recent developments parallel those that arose from the early work on centralized, classical signal detection, resulting in new ideas of asymptotically optimum nonparametric, robust, and sequential centralized detection. Recent developments on these topics in the setting of distributed signal detection are reviewed in the present paper. Results in these directions are important in practice because they allow cases of modeling uncertainty to be addressed, and they provide more efficient detection schemes by optimizing more general performance criteria. The classic optimal transportation problem consists in finding the most cost-effective way of moving masses from one set of locations to another, minimizing its transportation cost. The formulation of this problem and its solution have been useful to understand various mathematical, economical, and control theory phenomena, such as, e.g., Witsenhausen's counterexample in stochastic control theory, the principal-agent problem in microeconomic theory, location and planning problems, etc. In this work, we incorporate the effect of network congestion to the optimal transportation problem and we are able to find a closed form expression for its solution. As an application of our work, we focus on the mobile association problem in cellular networks (the determination of the cells corresponding to each base station). In the continuum setting, this problem corresponds to the determination of the locations at which mobile terminals prefer to connect (by also considering the congestion they generate) to a given base station rather than to other base stations. Two types of problems have been addressed: a global optimization problem for minimizing the total power needed by the mobile terminals over the whole network (global optimum), and a user optimization problem, in which each mobile terminal chooses to which base station to connect in order to minimize its own cost (user equilibrium). This work combines optimal transportation with strategic decision making to characterize both solutions. We present a new approach to motion planning under sensing and motion uncertainty by computing a locally optimal solution to a continuous partially observable Markov decision process (POMDP). Our approach represents beliefs (the distributions of the robot's state estimate) by Gaussian distributions and is applicable to robot systems with non-linear dynamics and observation models. The method follows the general POMDP solution framework in which we approximate the belief dynamics using an extended Kalman filter and represent the value function by a quadratic function that is valid in the vicinity of a nominal trajectory through belief space. Using a belief space variant of iterative LQG (iLQG), our approach iterates with second-order convergence towards a linear control policy over the belief space that is locally optimal with respect to a user-defined cost function. Unlike previous work, our approach does not assume maximum-likelihood observations, does not assume fixed estimator or control gains, takes into account obstacles in the environment, and does not require discretization of the state and action spaces. The running time of the algorithm is polynomial (O[n6]) in the dimension n of the state space. We demonstrate the potential of our approach in simulation for holonomic and non-holonomic robots maneuvering through environments with obstacles with noisy and partial sensing and with non-linear dynamics and observation models. The operation of large dynamic systems such as urban traffic networks remains a challenge in control engineering to a great extent due to their sheer size, intrinsic complexity, and nonlinear behavior. Recently, control engineers have looked for unconventional means for modeling and control of complex dynamic systems, in particular the technology of multi-agent systems whose appeal stems from their composite nature, flexibility, and scalability. This paper contributes to this evolving technology by proposing a framework for multi-agent control of linear dynamic systems, which decomposes a centralized model predictive control problem into a network of coupled, but small sub-problems that are solved by the distributed agents. Theoretical results ensure convergence of the distributed iterations to a globally optimal solution. The framework is applied to the signaling split control of traffic networks. Experiments conducted with simulation software indicate that the multi-agent framework attains performance comparable to conventional control. The main advantages of the multi-agent framework are its graceful extension and localized reconfiguration, which require adjustments only in the control strategies of the agents in the vicinity. In this paper, we investigate opacity of discrete event systems in a decentralized framework with several agents, each of them performing its observation of the system. We consider two cases, one without coordination among agents and one with coordination. Both cases are useful because many systems used today are distributed over a network, some with agents coordinating among themselves and some without. We introduce general definitions of decentralized opacity for both cases. The definitions are based on languages. Therefore, they are flexible and can include other properties of discrete event systems as special cases. In particular, we show that co-observability used in supervisory control is a special case of decentralized opacity. We illustrate the usefulness of decentralized opacity by applying it in solving an interesting security problem in computer systems.