<jupyter_start><jupyter_text>TeBag-RL - Text-based Adventure Game Reinforcement Learning               Run in Google Colab                View source on GitHub   This is the jupyter notebook for our project for the TUM seminar "Applied Deep Learning for Natural Language Processing". We attempted to tackle the difficult combination of NLP with reinforcement learning in form of Deep Q-Learning for text-based adventures games, such as the classic [Zork](https://en.wikipedia.org/wiki/Zork).We are using the [TextWorld](https://github.com/microsoft/TextWorld) environment with a custom PyEnvironment wrapper to use it with TensorFlow. In addition, we utilize custom reward functions and a biased reply buffer accept/reject sampling method. Two agents are tested and trained, one ultiziing an [NNLM](https://tfhub.dev/google/nnlm-en-dim128-with-normalization/2) pre-trained embedding and one with a pre-trained [smallBert](https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1) model. More details can be found in the accompanying materials.  General Setup Installing necessary python packages on either local machine or Google Colab:<jupyter_code>%%capture
!pip install tf-agents
!pip install textworld
!pip install tensorflow-text<jupyter_output><empty_output><jupyter_text>Google Colab set-up with files in Google Drive:<jupyter_code>%%capture
from google.colab import drive
drive.mount("/content/drive")
import os
PROJECT_PATH = "/content/drive/MyDrive/TeBaG-RL/"
os.chdir(PROJECT_PATH)<jupyter_output><empty_output><jupyter_text>***Or alternatively*** importing from GitHub onto (Google Colab) machine:<jupyter_code>%%capture
!git clone https://github.com/floriandonhauser/TeBaG-RL.git
import os
PROJECT_NAME = "TeBaG-RL/"
os.chdir(PROJECT_NAME)
PROJECT_PATH = os.getcwd()<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>from resources import DEFAULT_PATHS
from tf_train_loop import TWTrainer

%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Generate gamesGenerate simple debug game and large dataset of train and eval games.<jupyter_code>os.chdir(PROJECT_PATH + "/scripts/")<jupyter_output><empty_output><jupyter_text>This cell will create **a single** debug game. This is at least necessary to run anaything.<jupyter_code>%%shell
bash ./make_debug_game.sh<jupyter_output><empty_output><jupyter_text>**Only** run this if necessary. Depending on system, this process **will take hours**.<jupyter_code>%%capture
%%shell
bash ./make_allgames.sh
os.chdir(PROJECT_PATH)<jupyter_output><empty_output><jupyter_text>Play a game yourself<jupyter_code>os.chdir("resources/")
!tw-play game_th_lvl2_simple.ulx
os.chdir(PROJECT_PATH)<jupyter_output><empty_output><jupyter_text>Test environmentWe wrote a unit test for the environment creation, as we needed to write a a TensorFlow wrapper for the TextWorld game environment. This also enabled us to add custom rewards and punishments for different scenarios.The environment test function uses TensorFlow buil-in util methods and a random agent. The printed output shows the input command "Doing: ___",  the full environment state that we work with (only "obs", "description" and "inventory" are passed to the agent) and the resulting reward.<jupyter_code>from tests import test_environment_creation
test_environment_creation()<jupyter_output><empty_output><jupyter_text>Run automatic vocab generation We've implemented aa feature in the environment wrapper to check at each time step whether an interactable entity within the TextWorld game is in the current agent object vocabulary. All missing entities are stored and can be appended to the object vocuabulary file at the end of a game cycle.This enables automatic vocabulary generation for a set of training games. The implemented method below utilizes a random agent, however, it needs the full generated training set in the "Generate Games" section above.<jupyter_code>from environments import run_auto_vocab
run_auto_vocab()<jupyter_output><empty_output><jupyter_text>Train Set rewards for training.* **"win_lose_value"**: Value to be rewarded/punished for winning/losing the current game* **"max_loop_pun"**: Punishmend for having the same state in the buffered set of hashed last states to avoid rewarding loops. (E.g. agent would drop item, go to another room, go back and pick up item while being rewarded for using items and changing the environment.)* **"change_reward"**: Reward for changing either inventory (using/taking an item) or or the environment (exploring or opening an object).* **"useless_act_pun"**: Punishment for using a non-recoqnizable command (Env-Return: "I don't understand that." or "I do not see such an object here." etc.)* **"cmd_in_adm"**: Positive reward if current executed command is in set of admissible commands allowed from the environment. This should encourage environment object linking to commands, even though that command is not the right option.<jupyter_code>REWARD_DICT = {
    "win_lose_value": 100,
    "max_loop_pun": 0,
    "change_reward": 1,
    "useless_act_pun": 1,
    "cmd_in_adm": 1,
}<jupyter_output><empty_output><jupyter_text>Activate TensorBoard logging<jupyter_code>pathdir = DEFAULT_PATHS["path_logdir"]
%load_ext tensorboard
%tensorboard --logdir $pathdir<jupyter_output><empty_output><jupyter_text>Overfit on single debug game with each agent Try overfitting on one debug game (correct command "take x" will immediately win or lose the game).Depending on whether random agent finds correct WIN command, number of iterations will be enough or not.<jupyter_code>DEFAULT_HP = {
    "learning_rate": 1e-4,
    "initial_collect_steps": 3000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    "batch_size": 128,
    "num_eval_episodes": 1,
    "game_gen_buffer": 25,
    "num_eval_games": 5,
}
trainer = TWTrainer(
    reward_dict=REWARD_DICT,
    hpar=DEFAULT_HP,
    debug=False,
    biased_buffer=True,
    # embedding into fc is default policy
    agent_label="FCPolicy",
)
eval_scores = trainer.train(
    num_iterations=2000,
    log_interval=100,
    eval_interval=100,
    game_gen_interval=500,
    plot_avg_ret=True,
)
DEFAULT_HP = {
    "learning_rate": 1e-4,
    "initial_collect_steps": 3000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    # large values lead to OOM with bert policy
    "batch_size": 64,
    "num_eval_episodes": 1,
    "game_gen_buffer": 25,
    "num_eval_games": 5,
}
trainer = TWTrainer(
    reward_dict=REWARD_DICT, 
    hpar=DEFAULT_HP,
    debug=False,
    biased_buffer=True,
    # embedding into fc is default policy
    # agent_label="FCPolicy",
    agent_label="BertPolicy",
)
eval_scores = trainer.train(
    num_iterations=2000,
    log_interval=100,
    eval_interval=100,
    game_gen_interval=500,
    plot_avg_ret=True,
)<jupyter_output><empty_output><jupyter_text>Train on 10 training games from the same level (level 2) with (simple) Embedding-FC policy agent.<jupyter_code>DEFAULT_HP = {
    "learning_rate": 4.8247e-05,
    "initial_collect_steps": 30000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    "batch_size": 128,
    "num_eval_episodes": 1,
    "game_gen_buffer": 10,
    "num_eval_games": 10,
}

trainer = TWTrainer(
    env_dir="train_games_lvl2",
    reward_dict=REWARD_DICT,
    hpar=DEFAULT_HP,
    debug=False,
    # !!!!!
    biased_buffer=False,
    # embedding into fc is default policy
    agent_label="FCPolicy",
    # agent_label="BertPolicy",
)

eval_scores = trainer.train(
    num_iterations=10000,
    log_interval=250,
    eval_interval=500,
    game_gen_interval=1000000,
    rndm_fill_replay=True,
    plot_avg_ret=True,
)<jupyter_output><empty_output><jupyter_text>Train on 10 training games from the same level (level 2) with Bert policy agent.<jupyter_code>DEFAULT_HP = {
    "learning_rate": 4.8247e-05,
    "initial_collect_steps": 30000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    "batch_size": 64,
    "num_eval_episodes": 1,
    "game_gen_buffer": 10,
    "num_eval_games": 10,
    "num_test_games": 50,
}

trainer = TWTrainer(
    env_dir="train_games_lvl2",
    reward_dict=REWARD_DICT,
    hpar=DEFAULT_HP,
    debug=False,
    biased_buffer=True,
    agent_label="BertPolicy",
)

eval_scores = trainer.train(
    num_iterations=10000,
    log_interval=250,
    eval_interval=250,
    game_gen_interval=1000000,
    rndm_fill_replay=True,
    plot_avg_ret=True,
    test_agent=True,
)<jupyter_output><empty_output><jupyter_text>Train on all 1000 training games from the same level (level 2) with NNLM-FC policy agent.<jupyter_code>DEFAULT_HP = {
    "learning_rate": 4.8247e-05,
    "initial_collect_steps": 50000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    "batch_size": 128,
    "num_eval_episodes": 1,
    "game_gen_buffer": 25,
    "num_eval_games": 10,
    "num_test_games": 50,
}

trainer = TWTrainer(
    env_dir="train_games_lvl2",
    reward_dict=REWARD_DICT,
    hpar=DEFAULT_HP,
    debug=False,
    biased_buffer=True,
    agent_label="FCPolicy",
)

eval_scores = trainer.train(
    num_iterations=50000,
    log_interval=250,
    eval_interval=500,
    game_gen_interval=1000,
    rndm_fill_replay=True,
    plot_avg_ret=True,
    test_agent=True,
)<jupyter_output><empty_output><jupyter_text>Train on all generated  levels with Bert-FC policy agent This would be the training loop to utilize multiple levels (with increasing difficulty) of the TreasureHunter type game genereted by TextWorld.They differ in the number of rooms that need to be visited (including dead ends) and the number of other items being available in the environment.<jupyter_code>DEFAULT_HP = {
    "learning_rate": 4.8247e-05,
    "initial_collect_steps": 30000,
    "collect_steps_per_iteration": 1,
    "replay_buffer_max_length": 100000,
    "batch_size": 64,
    "num_eval_episodes": 1,
    "game_gen_buffer": 10,
    "num_eval_games": 10,
}

trainer = TWTrainer(
    env_dir="train_games_lvl2",
    reward_dict=REWARD_DICT,
    hpar=DEFAULT_HP,
    debug=False,
    biased_buffer=True,
    agent_label="BertPolicy",
)

eval_scores = trainer.train(
    num_iterations=10000,
    log_interval=250,
    eval_interval=500,
    game_gen_interval=1000,
    rndm_fill_replay=True,
    plot_avg_ret=True,
)

print(f"Changing to next lvl: 3 \n")

trainer.change_env_dir(f"train_games_lvl3")
eval_scores = trainer.train(
    num_iterations=10000,
    log_interval=250,
    eval_interval=500,
    game_gen_interval=1000,
    continue_training=True,
    rndm_fill_replay=True, 
    plot_avg_ret=True,
)

print(f"Changing to next lvl: 4 \n")

trainer.change_env_dir(f"train_games_lvl4")
eval_scores = trainer.train(
    num_iterations=12000,
    log_interval=250,
    eval_interval=500,
    game_gen_interval=1000,
    continue_training=True,
    rndm_fill_replay=True,
    plot_avg_ret=True,
)<jupyter_output><empty_output><jupyter_text>Hyper parameter searchSimple hyper parameter search using the Optuna python package. It also comes with very handy visualization tools illustrating correlations and importance between different hypter parameters.To test for stable and good training, the objective value is defined as the average score for the last few iterations for the games in the current buffer.<jupyter_code>%%capture
!pip install optuna
import optuna
import numpy as np
from optuna.visualization import plot_contour
from optuna.visualization import plot_slice
from optuna.visualization import plot_optimization_history
from optuna.visualization import plot_param_importances
def objective(trial):
    """"""

    REWARD_DICT = {
        "win_lose_value": 100,
        "max_loop_pun": 0,
        "change_reward": 1,
        "useless_act_pun": 1,
        "cmd_in_adm": 1,
    }

    DEFAULT_HP = {
        "learning_rate": trial.suggest_loguniform("lr", 1e-5, 1e-3,
        "initial_collect_steps": 30000,
        "collect_steps_per_iteration": 1,
        "replay_buffer_max_length": 100000,
        #CAREFUL: OOM - Can you handle more than 64?
        "batch_size": trial.suggest_int("batch_size", 32, 64),
        "num_eval_episodes": 1,
        "game_gen_buffer": 10,
        "num_eval_games": 5,
    }

    trainer = TWTrainer(
        env_dir="train_games_lvl2",
        reward_dict=REWARD_DICT,
        hpar=DEFAULT_HP,
        debug=False,
        biased_buffer=True,
        agent_label="BertPolicy",
    )

    eval_scores = trainer.train(
        num_iterations=20000,
        log_interval=250,
        eval_interval=250,
        game_gen_interval=1000000,
        rndm_fill_replay=True,
        plot_avg_ret=True,
    )
    print(DEFAULT_HP)
    print(eval_scores)

    return np.mean(eval_scores[1][-5:])

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=30)
print(study.best_trial)
plot_optimization_history(study)
plot_contour(study)
plot_slice(study)
plot_param_importances(study)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09019096556484393, 'fraction_numerical': 0.027992944244190507, 'mean_word_length': 4.82402858418937, 'pattern_counts': {'":': 68, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '584227', 'license': 'MIT', 'n_tokens_mistral': 4648, 'n_tokens_neox': 4224, 'n_words': 1106, 'path': 'tf_TextWorld_RL.ipynb', 'repo_name': 'floriandonhauser/TeBaG-RL', 'size': '13039'}