<jupyter_start><jupyter_text>Tutorial 08: Creating Custom Environments 创建自定义环境This tutorial walks you through the process of creating custom environments in Flow. Custom environments contain specific methods that define the problem space of a task, such as the state and action spaces of the RL agent and the signal (or reward) that the RL algorithm will optimize over. By specifying a few methods within a custom environment, individuals can use Flow to design traffic control tasks of various types, such as optimal traffic light signal timing and flow regulation via mixed autonomy traffic (see the figures below). Finally, these environments are compatible with OpenAI Gym.本教程将带您完成在Flow中创建自定义环境的过程。自定义环境包含定义任务的问题空间的特定方法，例如RL代理的状态和操作空间，以及RL算法将优化的信号(或奖励)。通过在自定义环境中指定一些方法，个人可以使用流来设计各种类型的交通控制任务，例如最优的交通灯信号定时和混合自主交通的流量调节(见下图)。最后，这些环境与OpenAI健身房是兼容的。The rest of the tutorial is organized as follows: in section 1 walks through the process of creating an environment for mixed autonomy vehicle control where the autonomous vehicles perceive all vehicles in the network, and section two implements the environment in simulation.本教程的其余部分组织如下:第1节介绍了创建混合自主车辆控制环境的过程，其中自主车辆感知网络中的所有车辆，第2节在仿真中实现了该环境。 1. Creating an Environment Class 创建一个环境类In this tutorial we will create an environment in which the accelerations of a handful of vehicles in the network are specified by a single centralized agent, with the objective of the agent being to improve the average speed of all vehicle in the network. In order to create this environment, we begin by inheriting the base environment class located in *flow.envs*:在本教程中，我们将创建一个环境，其中网络中少数车辆的加速由一个集中的代理指定，代理的目标是提高网络中所有车辆的平均速度。为了创建这样的环境，我们从继承位于*flow.envs*中的基本环境类开始:<jupyter_code># import the base environment class
from flow.envs import Env

# define the environment class, and inherit properties from the base environment class
class myEnv(Env):
    pass<jupyter_output><empty_output><jupyter_text>`Env` provides the interface for running and modifying a SUMO simulation. Using this class, we are able to start sumo, provide a network to specify a configuration and controllers, perform simulation steps, and reset the simulation to an initial configuration.“Env”提供了运行和修改sumo模拟的接口。使用这个类，我们可以启动sumo，提供指定配置和控制器的网络，执行模拟步骤，并将模拟重置为初始配置。By inheriting Flow's base environment, a custom environment for varying control tasks can be created by adding the following functions to the child class: 通过继承Flow的基环境，可以通过在子类中添加以下函数来创建用于变化控制任务的自定义环境:* **action_space**动作空间* **observation_space**观测空间* **apply_rl_actions**RL应用空间* **get_state**获取状态* **compute_reward**计算奖励值Each of these components are covered in the next few subsections. 1.1 ADDITIONAL_ENV_PARAMSThe features used to parametrize components of the state/action space as well as the reward function are specified within the `EnvParams` input, as discussed in tutorial 1. Specifically, for the sake of our environment, the `additional_params` attribute within `EnvParams` will be responsible for storing information on the maximum possible accelerations and decelerations by the autonomous vehicles in the network. Accordingly, for this problem, we define an `ADDITIONAL_ENV_PARAMS` variable of the form:用于参数化状态/动作空间组件的特性以及奖励功能在“EnvParams”输入中指定，如教程1中所述。具体来说，为了保护我们的环境，‘EnvParams’中的‘additional_params’属性将负责存储网络中自动驾驶车辆最大可能的加速和减速信息。因此，对于这个问题，我们定义了表单的‘ADDITIONAL_ENV_PARAMS’变量:<jupyter_code>ADDITIONAL_ENV_PARAMS = {
    "max_accel": 1,
    "max_decel": 1,
}<jupyter_output><empty_output><jupyter_text>All environments presented in Flow provide a unique `ADDITIONAL_ENV_PARAMS` component containing the information needed to properly define some environment-specific parameters. We assume that these values are always provided by the user, and accordingly can be called from `env_params`. For example, if we would like to call the "max_accel" parameter, we simply type:Flow中提供的所有环境都提供了一个惟一的‘ADDITIONAL_ENV_PARAMS’组件，其中包含正确定义某些特定于环境的参数所需的信息。我们假设这些值总是由用户提供的，因此可以从' env_params '中调用。例如，如果我们想调用“max_accel”参数，我们只需输入:    max_accel = env_params.additional_params["max_accel"] 1.2 action_space 动作空间The `action_space` method defines the number and bounds of the actions provided by the RL agent. In order to define these bounds with an OpenAI gym setting, we use several objects located within *gym.spaces*. For instance, the `Box` object is used to define a bounded array of values in $\mathbb{R}^n$.“action_space”方法定义了RL代理提供的操作的数量和界限。为了定义OpenAI健身房设置的这些边界，我们使用了位于*gym.spaces*内的几个对象。例如，“Box”对象用于定义$\mathbb{R}^n$中的有界值数组。<jupyter_code>from gym.spaces.box import Box<jupyter_output><empty_output><jupyter_text>In addition, `Tuple` objects (not used by this tutorial) allow users to combine multiple `Box` elements together.此外，“Tuple”对象(本教程中没有使用)允许用户将多个“Box”元素组合在一起。<jupyter_code>from gym.spaces import Tuple<jupyter_output><empty_output><jupyter_text>Once we have imported the above objects, we are ready to define the bounds of our action space. Given that our actions consist of a list of n real numbers (where n is the number of autonomous vehicles) bounded from above and below by "max_accel" and "max_decel" respectively (see section 1.1), we can define our action space as follows:一旦导入了上述对象，就可以定义操作空间的边界了。假设我们的动作是由n个实数组成的列表(其中n是自动驾驶车辆的数量)，从上到下分别由“max_accel”和“max_decel”约束(参见1.1节)，我们可以这样定义我们的动作空间:<jupyter_code>class myEnv(myEnv):

    @property
    def action_space(self):
        num_actions = self.initial_vehicles.num_rl_vehicles
        accel_ub = self.env_params.additional_params["max_accel"]
        accel_lb = - abs(self.env_params.additional_params["max_decel"])

        return Box(low=accel_lb,
                   high=accel_ub,
                   shape=(num_actions,))<jupyter_output><empty_output><jupyter_text>1.3 observation_space 观察空间The observation space of an environment represents the number and types of observations that are provided to the reinforcement learning agent. For this example, we will be observe two values for each vehicle: its position and speed. Accordingly, we need a observation space that is twice the size of the number of vehicles in the network.环境的观察空间表示提供给强化学习代理的观察的数量和类型。对于本例，我们将观察每个车辆的两个值:位置和速度。因此，我们需要的观测空间是网络中车辆数量的两倍。<jupyter_code>class myEnv(myEnv):  # update my environment class

    @property
    def observation_space(self):
        return Box(
            low=0,
            high=float("inf"),
            shape=(2*self.initial_vehicles.num_vehicles,),
        )<jupyter_output><empty_output><jupyter_text>1.4 apply_rl_actions 应用Rl动作The function `apply_rl_actions` is responsible for transforming commands specified by the RL agent into actual actions performed within the simulator. The vehicle kernel within the environment class contains several helper methods that may be of used to facilitate this process. These functions include:函数' apply_rl_actions '负责将RL代理指定的命令转换为在模拟器中执行的实际操作。environment类中的vehicle内核包含几个辅助方法，可以用来促进这个过程。这些功能包括:* **apply_acceleration** (list of str, list of float) -> None: converts an action, or a list of actions, into accelerations to the specified vehicles (in simulation)* **apply_lane_change** (list of str, list of {-1, 0, 1}) -> None: converts an action, or a list of actions, into lane change directions for the specified vehicles (in simulation)* **choose_route** (list of str, list of list of str) -> None: converts an action, or a list of actions, into rerouting commands for the specified vehicles (in simulation)For our example we consider a situation where the RL agent can only specify accelerations for the RL vehicles; accordingly, the actuation method for the RL agent is defined as follows:在我们的例子中，我们考虑这样一种情况:RL代理只能为RL车辆指定加速;因此，RL agent的驱动方法定义如下:<jupyter_code>class myEnv(myEnv):  # update my environment class

    def _apply_rl_actions(self, rl_actions):
        # the names of all autonomous (RL) vehicles in the network
        rl_ids = self.k.vehicle.get_rl_ids()

        # use the base environment method to convert actions into accelerations for the rl vehicles
        self.k.vehicle.apply_acceleration(rl_ids, rl_actions)<jupyter_output><empty_output><jupyter_text>1.5 get_state 获取状态The `get_state` method extracts features from within the environments and provides then as inputs to the policy provided by the RL agent. Several helper methods exist within flow to help facilitate this process. Some useful helper method can be accessed from the following objects:“get_state”方法从环境中提取特性，然后作为RL代理提供的策略的输入。flow中存在几个帮助方法来帮助简化这个过程。一些有用的帮助方法可以从以下对象访问:* **self.k.vehicle**: provides current state information for all vehicles within the network为网络中的所有车辆提供当前状态信息* **self.k.traffic_light**: provides state information on the traffic lights提供交通信号灯的状态信息* **self.k.network**: information on the network, which unlike the vehicles and traffic lights is static网络上的信息，这与车辆和红绿灯是静态的* More accessor objects and methods can be found within the Flow documentation at: http://berkeleyflow.readthedocs.io/en/latest/In order to model global observability within the network, our state space consists of the speeds and positions of all vehicles (as mentioned in section 1.3). This is implemented as follows:为了在网络中建立全局可观测性模型，我们的状态空间由所有车辆的速度和位置组成(如第1.3节所述)。实施办法如下:<jupyter_code>import numpy as np

class myEnv(myEnv):  # update my environment class

    def get_state(self, **kwargs):
        # the get_ids() method is used to get the names of all vehicles in the network
        ids = self.k.vehicle.get_ids()

        # we use the get_absolute_position method to get the positions of all vehicles
        pos = [self.k.vehicle.get_x_by_id(veh_id) for veh_id in ids]

        # we use the get_speed method to get the velocities of all vehicles
        vel = [self.k.vehicle.get_speed(veh_id) for veh_id in ids]

        # the speeds and positions are concatenated to produce the state
        return np.concatenate((pos, vel))<jupyter_output><empty_output><jupyter_text>1.6 compute_reward 计算奖励值The `compute_reward` method returns the reward associated with any given state. These value may encompass returns from values within the state space (defined in section 1.5) or may contain information provided by the environment but not immediately available within the state, as is the case in partially observable tasks (or POMDPs).' compute_reward '方法返回与任何给定状态相关联的奖励。这些值可能包含状态空间(在第1.5节中定义)中的值的返回，或者可能包含环境提供的信息，但是不能立即在状态中使用，就像部分可观察任务(或POMDPs)中的情况一样。For this tutorial, we choose the reward function to be the average speed of all vehicles currently in the network. In order to extract this information from the environment, we use the `get_speed` method within the Vehicle kernel class to collect the current speed of all vehicles in the network, and return the average of these speeds as the reward. This is done as follows:在本教程中，我们选择奖励函数作为当前网络中所有车辆的平均速度。为了从环境中提取这些信息，我们在车辆内核类中使用' get_speed '方法来收集网络中所有车辆的当前速度，并返回这些速度的平均值作为奖励。具体做法如下:<jupyter_code>import numpy as np

class myEnv(myEnv):  # update my environment class

    def compute_reward(self, rl_actions, **kwargs):
        # the get_ids() method is used to get the names of all vehicles in the network
        ids = self.k.vehicle.get_ids()

        # we next get a list of the speeds of all vehicles in the network
        speeds = self.k.vehicle.get_speed(ids)

        # finally, we return the average of all these speeds as the reward
        return np.mean(speeds)<jupyter_output><empty_output><jupyter_text>2. Testing the New Environment 测试新环境 2.1 Testing in SimulationNow that we have successfully created our new environment, we are ready to test this environment in simulation. We begin by running this environment in a non-RL based simulation. The return provided at the end of the simulation is indicative of the cumulative expected reward when jam-like behavior exists within the netowrk. 现在我们已经成功地创建了新的环境，我们准备在模拟中测试这个环境。我们首先在一个非基于rl的模拟中运行这个环境。在模拟结束时提供的回报指示了在netowrk中存在类似于jam的行为时累积的预期回报。<jupyter_code>from flow.controllers import IDMController, ContinuousRouter
from flow.core.experiment import Experiment
from flow.core.params import SumoParams, EnvParams, \
    InitialConfig, NetParams
from flow.core.params import VehicleParams
from flow.networks.ring import RingNetwork, ADDITIONAL_NET_PARAMS

sim_params = SumoParams(sim_step=0.1, render=True)

vehicles = VehicleParams()
vehicles.add(veh_id="idm",
             acceleration_controller=(IDMController, {}),
             routing_controller=(ContinuousRouter, {}),
             num_vehicles=22)

env_params = EnvParams(additional_params=ADDITIONAL_ENV_PARAMS)

additional_net_params = ADDITIONAL_NET_PARAMS.copy()
net_params = NetParams(additional_params=additional_net_params)

initial_config = InitialConfig(bunching=20)

flow_params = dict(
    exp_tag='ring',
    env_name=myEnv,  # using my new environment for the simulation
    network=RingNetwork,
    simulator='traci',
    sim=sim_params,
    env=env_params,
    net=net_params,
    veh=vehicles,
    initial=initial_config,
)

# number of time steps
flow_params['env'].horizon = 1500
exp = Experiment(flow_params)

# run the sumo simulation
_ = exp.run(1)<jupyter_output><empty_output><jupyter_text>2.2 Training the New Environment 培训新环境Next, we wish to train this environment in the presence of the autonomous vehicle agent to reduce the formation of waves in the network, thereby pushing the performance of vehicles in the network past the above expected return.接下来，我们希望在自主车辆代理存在的情况下训练这种环境，以减少网络中波浪的形成，从而使网络中车辆的性能超过上述预期收益。The below code block may be used to train the above environment using the Proximal Policy Optimization (PPO) algorithm provided by RLlib. In order to register the environment with OpenAI gym, the environment must first be placed in a separate ".py" file and then imported via the script below. Then, the script immediately below should function regularly.下面的代码块可以使用RLlib提供的Proximal Policy Optimization (PPO)算法来训练上述环境。为了注册OpenAI健身房的环境，环境必须首先放在一个单独的。py”。然后通过下面的脚本导入。然后，下面的脚本应该正常工作。<jupyter_code>#############################################################
####### Replace this with the environment you created #######
#############################################################
from flow.envs import AccelEnv as myEnv<jupyter_output><empty_output><jupyter_text>**Note**: We do not recommend training this environment to completion within a jupyter notebook setting; however, once training is complete, visualization of the resulting policy should show that the autonomous vehicle learns to dissipate the formation and propagation of waves in the network.**注**:我们不建议在这种环境下进行的培训是在木星笔记本设置中完成的;然而，一旦训练完成，结果策略的可视化应该表明，自主车辆学会了在网络中消散波的形成和传播。<jupyter_code>import json
import ray
from ray.rllib.agents.registry import get_agent_class
from ray.tune import run_experiments
from ray.tune.registry import register_env

from flow.networks.ring import RingNetwork, ADDITIONAL_NET_PARAMS
from flow.utils.registry import make_create_env
from flow.utils.rllib import FlowParamsEncoder
from flow.core.params import SumoParams, EnvParams, InitialConfig, NetParams
from flow.core.params import VehicleParams, SumoCarFollowingParams
from flow.controllers import RLController, IDMController, ContinuousRouter


# time horizon of a single rollout
HORIZON = 1500
# number of rollouts per training iteration
N_ROLLOUTS = 20
# number of parallel workers
N_CPUS = 2


# We place one autonomous vehicle and 22 human-driven vehicles in the network
vehicles = VehicleParams()
vehicles.add(
    veh_id="human",
    acceleration_controller=(IDMController, {
        "noise": 0.2
    }),
    car_following_params=SumoCarFollowingParams(
        min_gap=0
    ),
    routing_controller=(ContinuousRouter, {}),
    num_vehicles=21)
vehicles.add(
    veh_id="rl",
    acceleration_controller=(RLController, {}),
    routing_controller=(ContinuousRouter, {}),
    num_vehicles=1)

flow_params = dict(
    # name of the experiment
    exp_tag="stabilizing_the_ring",

    # name of the flow environment the experiment is running on
    env_name=myEnv,  # <------ here we replace the environment with our new environment

    # name of the network class the experiment is running on
    network=RingNetwork,

    # simulator that is used by the experiment
    simulator='traci',

    # sumo-related parameters (see flow.core.params.SumoParams)
    sim=SumoParams(
        sim_step=0.1,
        render=True,
    ),

    # environment related parameters (see flow.core.params.EnvParams)
    env=EnvParams(
        horizon=HORIZON,
        warmup_steps=750,
        clip_actions=False,
        additional_params={
            "target_velocity": 20,
            "sort_vehicles": False,
            "max_accel": 1,
            "max_decel": 1,
        },
    ),

    # network-related parameters (see flow.core.params.NetParams and the
    # network's documentation or ADDITIONAL_NET_PARAMS component)
    net=NetParams(
        additional_params=ADDITIONAL_NET_PARAMS.copy()
    ),

    # vehicles to be placed in the network at the start of a rollout (see
    # flow.core.params.VehicleParams)
    veh=vehicles,

    # parameters specifying the positioning of vehicles upon initialization/
    # reset (see flow.core.params.InitialConfig)
    initial=InitialConfig(
        bunching=20,
    ),
)


def setup_exps():
    """Return the relevant components of an RLlib experiment.

    Returns
    -------
    str
        name of the training algorithm
    str
        name of the gym environment to be trained
    dict
        training configuration parameters
    """
    alg_run = "PPO"

    agent_cls = get_agent_class(alg_run)
    config = agent_cls._default_config.copy()
    config["num_workers"] = N_CPUS
    config["train_batch_size"] = HORIZON * N_ROLLOUTS
    config["gamma"] = 0.999  # discount rate
    config["model"].update({"fcnet_hiddens": [3, 3]})
    config["use_gae"] = True
    config["lambda"] = 0.97
    config["kl_target"] = 0.02
    config["num_sgd_iter"] = 10
    config['clip_actions'] = False  # FIXME(ev) temporary ray bug
    config["horizon"] = HORIZON

    # save the flow params for replay
    flow_json = json.dumps(
        flow_params, cls=FlowParamsEncoder, sort_keys=True, indent=4)
    config['env_config']['flow_params'] = flow_json
    config['env_config']['run'] = alg_run

    create_env, gym_name = make_create_env(params=flow_params, version=0)

    # Register as rllib env
    register_env(gym_name, create_env)
    return alg_run, gym_name, config


alg_run, gym_name, config = setup_exps()
ray.init(num_cpus=N_CPUS + 1)
trials = run_experiments({
    flow_params["exp_tag"]: {
        "run": alg_run,
        "env": gym_name,
        "config": {
            **config
        },
        "checkpoint_freq": 20,
        "checkpoint_at_end": True,
        "max_failures": 999,
        "stop": {
            "training_iteration": 200,
        },
    }
})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07543241511851377, 'fraction_numerical': 0.005658765748451847, 'mean_word_length': 4.971628944851769, 'pattern_counts': {'":': 16, '<': 50, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '9487', 'license': 'MIT', 'n_tokens_mistral': 6450, 'n_tokens_neox': 6570, 'n_words': 1850, 'path': 'tutorials/tutorial08_environments.ipynb', 'repo_name': '2855239858/CenLight-Traffic-Grid-Signal-Optimization-viaAction-and-State-Decomposition', 'size': '18732'}