<jupyter_start><jupyter_text><a href="https://colab.research.google.com/github/christianhidber/easyagents/blob/master/jupyter_notebooks/intro_logging.ipynb"    target="_parent">     Investigate an agents api calls, set seeds and fix juypter output cell clearing issues  Install packages (gym, tfagents, tensorflow,....)  suppress package warnings, in colab: load additional packages for rendering<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import sys
import warnings

warnings.filterwarnings('ignore')
if 'google.colab' in sys.modules:
    !apt-get update >/dev/null
    !apt-get install xvfb >/dev/null
    !pip install pyvirtualdisplay >/dev/null    
    
    from pyvirtualdisplay import Display
    Display(visible=0, size=(960, 720)).start() 
else:
    #  for local installation
    sys.path.append('..')<jupyter_output><empty_output><jupyter_text>install easyagents<jupyter_code>import sys
if 'google.colab' in sys.modules:
    !pip install easyagents >/dev/null<jupyter_output><empty_output><jupyter_text>Agent logging Use the log.Agent() callback to investigate how easyagents interacts with a backend:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([log.Agent(), duration.Fast()], default_plots=False)<jupyter_output>backend_name             tfagents 
TFPyEnvironment          ( suite_gym.load( "CartPole-v0", discount=1.0) ) 
AdamOptimizer            () 
ActorDistributionNetwork () 
ValueNetwork             () 
PpoAgent                 () 
tf_agent.initialize      () 
TFUniformReplayBuffer    () 
DynamicEpisodeDriver     () 
TFPyEnvironment          ( suite_gym.load( "CartPole-v0", discount=1) ) 
-----                    iteration    0 of 10        ----- 
collect_driver.run       () 
replay_buffer.gather_all () 
tf_agent.train           (experience=...) 
                         loss=4509.4  [actor=0.1     critic=4509.3 ] 
replay_buffer.clear      () 
-----                    iteration    1 of 10        ----- 
collect_driver.run       () 
replay_buffer.gather_all () 
tf_agent.train           (experience=...) 
                         loss=4121.6  [actor=0.1     critic=4121.5 ] 
replay_buffer.clear      () 
-----                    iteration    2 of 10        ----- 
collect_driver.run       () 
repla[...]<jupyter_text>Plots clear the jupyter cell ouput before each update thereby clearing the log output as well, thus we turned them off.Typically each call to the backend api during training is logged. Note that the logging starts with 'tfagents' the default backend for the PpoAgent.We then see a sequence of calls performing the Agent initialisation before we enter the train loop.Api calls during play or evaluation are not logged.Let's take a look at the tensorforce backend:**tensorforce and keras are suspended pending tensorflow 2.0 support**<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

#ppoAgent = PpoAgent('CartPole-v0', backend='tensorforce')
#ppoAgent.train([log.Agent(), duration.Fast()], default_plots=False)<jupyter_output><empty_output><jupyter_text>While in tensorforce we also first do a sequence of agent and policy. Note that in contrast to tfagents we do notbuild up actor and critic policy networks but instead pass a network specification to the Agent.create call.Moreover tensorforce implements already the train loop through its Runner class. Thus we only see 1 call to runner.run instead of the many api calls for tfagents.  SeedingTo set a seed use:<jupyter_code>import easyagents

easyagents.agents.seed = 0<jupyter_output><empty_output><jupyter_text>Once set, the seed is applied before each call to train. Let's validate this using our log.Agent callback:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([log.Agent(), duration.Fast()], default_plots=False)<jupyter_output>backend_name             tfagents 
tf.random.set_seed       (seed=0) 
numpy.random.seed        (0) 
random.seed              (0) 
TFPyEnvironment          ( suite_gym.load( "CartPole-v0", discount=1.0) ) 
AdamOptimizer            () 
ActorDistributionNetwork () 
ValueNetwork             () 
PpoAgent                 () 
tf_agent.initialize      () 
TFUniformReplayBuffer    () 
DynamicEpisodeDriver     () 
TFPyEnvironment          ( suite_gym.load( "CartPole-v0", discount=1) ) 
-----                    iteration    0 of 10        ----- 
collect_driver.run       () 
replay_buffer.gather_all () 
tf_agent.train           (experience=...) 
                         loss=8528.5  [actor=0.0     critic=8528.5 ] 
replay_buffer.clear      () 
-----                    iteration    1 of 10        ----- 
collect_driver.run       () 
replay_buffer.gather_all () 
tf_agent.train           (experience=...) 
                         loss=5304.3  [actor=0.1     critic=5304.3 ] 
replay_buffer.clear      () [...]<jupyter_text>Note that at the very beginning the calls to set the seeds for tensorflow, numpy and python. Iteration & Duration loggingUse the log.Iteration() callback to log the training progress:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([log.Iteration(), duration.Fast()], default_plots=False)<jupyter_output>iteration 0  of 10       episodes_done=0   steps_done=0     rewards=(50.0,50.0,50.0) steps=(50.0,50.0,50.0) 
iteration 1  of 10       episodes_done=3   steps_done=59    loss=8528.5  [actor=0.0     critic=8528.5 ] 
iteration 2  of 10       episodes_done=6   steps_done=114   loss=5304.3  [actor=0.1     critic=5304.3 ] 
iteration 3  of 10       episodes_done=9   steps_done=178   loss=8645.1  [actor=0.0     critic=8645.0 ] 
iteration 4  of 10       episodes_done=12  steps_done=244   loss=9818.4  [actor=0.0     critic=9818.3 ] rewards=(50.0,50.0,50.0) steps=(50.0,50.0,50.0) 
iteration 5  of 10       episodes_done=15  steps_done=288   loss=4630.6  [actor=0.1     critic=4630.6 ] 
iteration 6  of 10       episodes_done=18  steps_done=361   loss=11265.8 [actor=0.0     critic=11265.8] 
iteration 7  of 10       episodes_done=21  steps_done=476   loss=26230.5 [actor=0.0     critic=26230.4] 
iteration 8  of 10       episodes_done=24  steps_done=526   loss=5172.6  [actor=0.0     critic=5172.6 ] rewa[...]<jupyter_text>The first line shows the result of an initial policy evaluation depicting the policy's performance before any training has happend. Policy evaluation happens every `num_iterations_between_eval` iterations and spans over`num_episodes_per_eval` episodes. For every evaluation period the result is logged again. The `steps_done`value is the number of training steps (excluding the steps taken during evaluation). To see the training duration configuration use the log.Duration() callback:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([log.Duration(), duration.Fast()], default_plots=False)<jupyter_output>duration                 num_iterations=100 num_episodes_per_iteration=10 num_max_steps_per_episode=500 num_iterations_between_plot=2 num_iterations_between_eval=5 num_episodes_per_eval=10<jupyter_text>Gym steps loggingUse the log.Step() callback to investigate how the agent interacts with the gym environment:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, log

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([log.Step(), duration.Fast()], default_plots=False)<jupyter_output>[CartPole-v0 9:0  :1  ] train iteration=0  step=0   play  episode=0  step=1     sum_of_rewards=1.0     reward=1.0   done=False action=1 observation=[-0.04363321  0.24146826  0.01284913 -0.30946528]
[CartPole-v0 9:0  :2  ] train iteration=0  step=0   play  episode=0  step=2     sum_of_rewards=2.0     reward=1.0   done=False action=0 observation=[-0.03880385  0.04616562  0.00665982 -0.01275795]
[CartPole-v0 9:0  :3  ] train iteration=0  step=0   play  episode=0  step=3     sum_of_rewards=3.0     reward=1.0   done=False action=1 observation=[-0.03788053  0.24119143  0.00640466 -0.30333221]
[CartPole-v0 9:0  :4  ] train iteration=0  step=0   play  episode=0  step=4     sum_of_rewards=4.0     reward=1.0   done=False action=0 observation=[-0.03305671  0.04597879  0.00033802 -0.0086363 ]
[CartPole-v0 9:0  :5  ] train iteration=0  step=0   play  episode=0  step=5     sum_of_rewards=5.0     reward=1.0   done=False action=1 observation=[-3.21371306e-02  2.41095889e-01  1.65294467e-04 -3.01212554[...]<jupyter_text>For each call to the gym environments step method you get a log entry, along with the action taken and currentobservation. Each entry starts with [{gym_env_id} {instance_id}:{episode_in_instance}:{step_in_episode}]followed by the id of the current training iteration as well as the current iteration step count.If in a evaluation period you get the same statistics for the current evaluation episode.You may easily implement other log callbacks to produce statistics specific to your problem domain. Fixing jupyter output cell clearingIt seems that jupyter / matplotlib backend changes its behaviour of outputing the current figure of an evaluated cell (if you can help here, please let use know by [creating an issue](https://github.com/christianhidber/easyagents/issues/new/choose)).Nonetheless you may directly control easyagents jupyter ouput cell clearing behaviour through the plot.Clear()callback:<jupyter_code>from easyagents.agents import PpoAgent
from easyagents.callbacks import duration, plot

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([plot.Clear(on_train=False,on_play=False), duration.Fast()])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09395292207792208, 'fraction_numerical': 0.056513798701298704, 'mean_word_length': 3.5996266915538966, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '392704', 'license': 'MIT', 'n_tokens_mistral': 3403, 'n_tokens_neox': 3132, 'n_words': 840, 'path': 'jupyter_notebooks/intro_logging.ipynb', 'repo_name': 'doMConSwiss/easyagents', 'size': '9856'}