BACKGROUND

This specification relates to control systems used to control the operation of industrial plants.

An industrial plant can be any facility that processes materials (e.g., chemically, mechanically, electrically, or a combination thereof) to generate a processed output. Examples of industrial plants include smelting plants, paper mills, and oil refineries. A control system of an industrial plant can select control actions to be performed to control the operation of the industrial plant.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains an industrial plant controller that controls the operation of an industrial plant.

According to a first aspect there is provided a method, performed by one or more data processing apparatus, for training an industrial plant controller that controls operation of an industrial plant. The method includes generating training data using an industrial plant simulation model that simulates operation of the industrial plant. Generating the training data includes, at each of multiple time steps: processing, using the industrial plant simulation model, (i) a current state vector characterizing a simulated state of the industrial plant at the current time step, and (ii) a control action to be performed at the current time step; generating, using the industrial plant simulation model, a subsequent state vector characterizing the simulated state of the industrial plant after the control action is performed; and determining a reward received at the current time step based on at least the subsequent state vector characterizing the simulated state of the industrial plant after the control action is performed. The industrial plant controller is trained by a reinforcement learning technique using the training data. The industrial plant controller is configured to process an input comprising a state vector characterizing a state of the industrial plant in accordance with industrial plant controller parameters to generate an action selection policy output that defines a control action to be performed to control the operation of the industrial plant.

In some implementations, the training includes adjusting values of the industrial plant controller parameters to increase a measure of cumulative reward received by performing control actions defined by action selection policy outputs generated by the industrial plant controller.

In some implementations, the training data is generated using multiple instances of the industrial plant simulation model running in parallel.

In some implementations, generating the training data further includes, at one or more particular time steps: adjusting the current state vector to simulate occurrence of an event affecting the operation of the industrial plant.

In some implementations, the event includes an equipment failure in the industrial plant.

In some implementations, at each particular time step, the event is determined by sampling from a probability distribution over a predetermined set of possible events, wherein the possible events include a non-event that does not affect the operation of the industrial plant.

In some implementations, the rewards received at the time steps characterize how effectively the control actions performed at the time steps accomplish certain tasks.

In some implementations, the method further includes: determining whether the industrial plant controller passes one or more certification tests, where a certification test assesses whether the industrial plant controller can effectively control the operation of the industrial plant by generating control actions in accordance with current values of the industrial plant controller parameters; and using the industrial plant controller to control the operation of the industrial plant in response to determining that the industrial plant controller passes the certification tests.

In some implementations, the method further includes using the industrial plant controller to control the operation of the industrial plant, including, at each of multiple given time steps: obtaining a state vector characterizing a state of the industrial plant at the given time step; processing an input including the state vector characterizing the state of the industrial plant at the given time step using the industrial plant controller to generate an action selection policy output; and determining a control action to be performed at the given time step based on the action selection policy output.

In some implementations, the action selection policy output includes a respective score for each control action in a predetermined set of possible control actions.

In some implementations, determining a control action to be performed based on the action selection policy output includes selecting a control action with a highest score.

In some implementations, the industrial plant controller includes one or more neural networks, and the industrial plant controller parameters include weight values of the one or more neural networks.

According to a second aspect there is provided a system including: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, where the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to train an industrial plant controller that controls operation of an industrial plant by performing operations including the operations of the previously described method.

According to a third aspect there is provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to train an industrial plant controller that controls operation of an industrial plant by performing operations including the operations of the previously described method.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

The training system described in this specification can train an industrial plant controller used to control the operation of an industrial plant using training data generated by one or more simulation systems that numerically simulate the operation of the industrial plant. The training system can generate large quantities of training data by, for example, running multiple simulation systems in parallel and running the simulations systems faster than the “real-time” speed at which the industrial plant actually operates. The large quantities of training data that can be generated by the simulation systems greatly exceeds the limited quantities of real-world (i.e., non-simulated) training data that could be obtained by logging data characterizing the real-world operation of the industrial plant. The training system can use the large quantities of training data generated using the simulation systems to train the controller to control the industrial plant more effectively than if the training system used only real-world training data.

The training system described in this specification can train an industrial plant controller to respond effectively to a large variety of events that affect the operation of the industrial plant (e.g., equipment failures or input material changes), without having to actually experience those events. In particular, the training system can simulate the occurrence of many thousands of events, which could affect the operation of the industrial plant using the simulation systems, and thereafter train the controller to respond effectively to the occurrence of these events. In contrast, human operators of industrial plants may be ill-prepared to respond effectively to certain events that affect the operation of the industrial plant. For example, certain events may occur relatively rarely during real-world operation of the industrial plant, so human operators may lack experience in responding to these events. However, the training system described in this specification can train the controller to respond effectively to these rare events by exposing the controller to them many thousands of times in different simulations. More specifically, the simulations can enable training on events that would render an industrial plant inoperable, which is not feasible using real-world data.

The training system described in this specification can generate highly diverse sets of training data by simulating the operation of the industrial plant when control actions are selected in accordance with an “exploration strategy” (e.g., where some control actions are selected randomly). In this manner, the training system enables the controller to “explore” the space of possible control actions and their simulated consequences on the industrial plant before the controller is deployed to control the real-world operation of the industrial plant. If the controller were directly trained to control the operation of the industrial plant without the benefit of simulated training data, the controller could not implement an exploration strategy in selecting control actions since poorly chosen control actions could damage the industrial plant or cause it to operate unsafely.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

