<jupyter_start><jupyter_text>TrainingThe API provides both a [Unity ml-agents](https://github.com/Unity-Technologies/ml-agents) and an [OpenAI Gym](https://github.com/openai/gym) interface. We include training examples for both in [the examples folder](./examples); the former uses ml-agents' own training library which is optimised for the environment, the latter uses [OpenAI baselines](https://github.com/openai/baselines).In this notebook we show you how to run the `animal-ai` trainers which are optimized for training on the AnimalAI environment. It's a powerful modular library you can tinker with in order to implement your own algorithms. We strongly recommend that you have a look at its various parts (described at the end of this tutorial) should you wish to make modifications. Can your agent self control? - Part IIIf you haven't done so already, go through the environement tutorial where we decribe the problem of self-control in animals. We created a curriculum which includes increasingly difficult levels in which the agent must retrieve food, while being introduced to objects similar to those in the final experiment, without encountering the exact testing configuration(s).Having created a curriculum in the previous notebook, we now need to configure the training environment. The `animalai-train` library provides all the tools you'll need to train using PPO or SAC - we'll be using the former here.First, we need to set all the hyperparameters of our model, which is done by creating a yaml file as follows:<jupyter_code>with open("configurations/training_configurations/train_ml_agents_config_ppo.yaml") as f:
    print(f.read())<jupyter_output>AnimalAI:
    trainer: ppo
    epsilon: 0.2
    lambd: 0.95
    learning_rate: 3.0e-4
    learning_rate_schedule: linear
    memory_size: 128
    normalize: false
    sequence_length: 64
    summary_freq: 10000
    use_recurrent: false
    vis_encode_type: simple
    time_horizon: 128
    batch_size: 64
    buffer_size: 2024
    hidden_units: 256
    num_layers: 1
    beta: 1.0e-2
    max_steps: 1.0e7
    num_epoch: 3
    reward_signals:
        extrinsic:
            strength: 1.0
            gamma: 0.99
        curiosity:
            strength: 0.01
            gamma: 0.99
            encoding_size: 256<jupyter_text>If you're already familiar with RL algorithms in general, these parameters should be fairly self-explanatory. Nonetheless, you can have a look at [this page](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-Configuration-File.md) for explanations of the parameters specified for both PPO and SAC.You then need to configure the trainer, which is just a named tuple defining parameters such as:- the paths to the environment and your configuration file (above) - how many environments to launch in parallel and with how many agent per environment- the path to your curriculum- and many more!This is all done as follows:<jupyter_code>import warnings
warnings.filterwarnings('ignore')
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

from mlagents.trainers.trainer_util import load_config;
from animalai_train.run_options_aai import RunOptionsAAI;
from animalai_train.run_training_aai import run_training_aai;


trainer_config_path = (
    "configurations/training_configurations/train_ml_agents_config_ppo.yaml"
)
environment_path = "env/AnimalAI"
curriculum_path = "configurations/curriculum"
run_id = "self_control_curriculum"
base_port = 5005
number_of_environments = 4
number_of_arenas_per_environment = 8

args = RunOptionsAAI(
    trainer_config=load_config(trainer_config_path),
    env_path=environment_path,
    run_id=run_id,
    base_port=base_port,
    num_envs=number_of_environments,
    curriculum_config=curriculum_path,
    n_arenas_per_env=number_of_arenas_per_environment,
)<jupyter_output><empty_output><jupyter_text>Once this is done we're pretty much just left with a one liner! The training library isn't verbose, but you can monitor training via Tensorboard. The first few lines just load tensorboard, once it is launched and you can see the orange window below, just click on the refresh button in the top right of Tensorboard - graphs will appear after a few training steps._Note_: in case you don't want to wait for the model to train, you can jump ahead to the next step as we provide a pre-trained model for inference.<jupyter_code>import os
# logging.getLogger('tensorflow').disabled = True

logs_dir = "summaries/"
os.makedirs(logs_dir, exist_ok=True)
%load_ext tensorboard
%tensorboard --logdir {logs_dir}

run_training_aai(0, args)<jupyter_output><empty_output><jupyter_text>You can see the lessons increasing as the agent gets better at each level. That's pretty much it for training using the provided library.Lastly, let's assess how well our agent trained with transparent walls present can perform on the transparent cylinder task. To do so we can load the model and run it in inference<jupyter_code>from animalai.envs.arena_config import ArenaConfig

#Comment the row below to load your trained model
run_id = "self_control_curriculum_pre_trained"

args = RunOptionsAAI(
    trainer_config=load_config(trainer_config_path),
    env_path=environment_path,
    run_id=run_id,
    base_port=base_port+3,
    load_model=True,
    train_model=False,
    arena_config=ArenaConfig("configurations/arena_configurations/cylinder_task.yml")
)
run_training_aai(0, args)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06199854121079504, 'fraction_numerical': 0.011487964989059081, 'mean_word_length': 4.822717622080679, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '311032', 'license': 'Apache-2.0', 'n_tokens_mistral': 1631, 'n_tokens_neox': 1547, 'n_words': 586, 'path': 'examples/training_tutorial.ipynb', 'repo_name': 'southpawac/AnimalAI-Olympics', 'size': '5484'}