<jupyter_start><jupyter_text>Here we have simple function for evaluations where you can change hyperparameters and run with multiprocessing: \If you see wierd multiprocessing errors try to run cell again. Here we will run experiment with `gamma=0.95`<jupyter_code>processes = 3 # Number of threads
evaluations = 3 # Number of evaluations to run

def evaluate(_):
    model = A2C(
        iterations=200, # Number of training iterations
        gamma=0.95, 
        batch_size=50,
        stats_freq=20, 
        tensorboard_dir="tensorboard", 
        tensorboard_comment="A2C",
        actor_lr=3e-3,
        critic_lr=3e-4,
        critic_num_target_updates=10,
        num_critic_updates_per_target=10,
        normalize_adv=True,
        return_done = None,
    )
    model.train()
    return model.stats_logger.stats

with mp.Pool(processes) as p:
    outputs = p.map(evaluate, range(evaluations))<jupyter_output><empty_output><jupyter_text>Next will be `gamma=0.99`<jupyter_code>processes = 3 # Number of threads
evaluations = 3 # Number of evaluations to run

def evaluate(_):
    model = A2C(
        iterations=200, # Number of training iterations
        gamma=0.99, 
        batch_size=50,
        stats_freq=10, 
        tensorboard_dir="tensorboard", 
        tensorboard_comment="A2C",
        actor_lr=3e-3,
        critic_lr=3e-4,
        critic_num_target_updates=10,
        num_critic_updates_per_target=10,
        normalize_adv=True,
    )
    model.train()
    return model.stats_logger.stats

with mp.Pool(processes) as p:
    outputs_2 = p.map(evaluate, range(evaluations))<jupyter_output><empty_output><jupyter_text>Plotting the results:<jupyter_code>plot_evals_list(outputs, color="blue")
plot_evals_list(outputs_2, color="green")
plt.legend(["g=0.95", "g=0.99"], loc='upper left')<jupyter_output><empty_output><jupyter_text>Tasks:In example above only 3 seeds was run. For your experiments use at least 10.1. Plot how different values of `gamma` [0.9, 0.95, 0.99] impact training.2. Check whether normalization of advantages  (`normalize_adv`) improves performance.3. Explore impact of frequent critic updates setting following parameters:    * `critic_num_target_updates=1` and `num_critic_updates_per_target=1`    * `critic_num_target_updates=100` and `num_critic_updates_per_target=1`    * `critic_num_target_updates=1` and `num_critic_updates_per_target=100`    * `critic_num_target_updates=10` and `num_critic_updates_per_target=10`    Prepare plots to illustrate all experiments.  BonusSimple rendering example (based on https://gym.openai.com/). \First we train model:<jupyter_code>model = A2C(
    iterations=20, # Number of training iterations
    gamma=0.95, 
    batch_size=200,
    stats_freq=10, 
    tensorboard_dir="tensorboard", 
    tensorboard_comment="A2C",
    actor_lr=3e-3,
    critic_lr=3e-4,
    critic_num_target_updates=10,
    num_critic_updates_per_target=10,
    normalize_adv=True,
)
model.train()<jupyter_output>2020-12-21 13:09:42.376 MainProcess INFO  Iteration    0	 Running return:          22.22222222
2020-12-21 13:09:42.377 MainProcess INFO  Average iteration is 0.227799 seconds
2020-12-21 13:09:44.722 MainProcess INFO  Iteration   10	 Running return:          28.69517586
2020-12-21 13:09:44.723 MainProcess INFO  Average iteration is 0.23031930000000003 seconds
2020-12-21 13:09:46.918 MainProcess INFO  Task finished at 20th iteration. Running return is 22.962453189542384<jupyter_text>Then we iterate through our environment:<jupyter_code>import gym

num_frames = 100

env = gym.make("CartPole-v1")
observation = env.reset()
for _ in range(num_frames):
    env.render()
    action = model.action(observation) # your agent here (this takes random actions)
    observation, reward, done, info = env.step(action)

    if done:
        observation = env.reset()
env.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09509043927648579, 'fraction_numerical': 0.06770025839793281, 'mean_word_length': 4.100131752305665, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '173124', 'license': 'MIT', 'n_tokens_mistral': 1462, 'n_tokens_neox': 1298, 'n_words': 298, 'path': 'notebooks/01-A2C.ipynb', 'repo_name': 'raznem/rlex', 'size': '3870'}