<jupyter_start><jupyter_text>1. `envGym``envGym` is a wrapper around the OpenAI Gym environment. It has several methds that can be useful for using the environment. In this notebook, we shall explore this environment.<jupyter_code>from lib.envs import envGym
from time import sleep
import numpy as np
import torch<jupyter_output><empty_output><jupyter_text>1.1. The `Env1D()`  context managerThe `Env` class exposes a context manager that will allow this environment to generate an OpenAI environment within. The first time this context manager is entered, this creates a current state `self.state`. This can be reset with the `self.reset()` method. This introduces a new input parameter `N` that will allow you to specify how many earlier states you want to consider to determine a single state.<jupyter_code>name = 'Breakout-ramNoFrameskip-v4'
with envGym.Env1D(name, showEnv=False, N=2) as env:
    print(f'Initial environment state:\n {env.state}')
    env.reset()
    print(f'Initial environment state after reset:\n {env.state}')<jupyter_output>Initial environment state:
 deque([array([ 63,  63,  63,  63,  63,  63, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 192, 192, 192, 192, 192, 192, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 240,   0,   0,
       255,   0,   0, 240,   0,   5,   0,   0,   6,   0,  70, 182, 134,
       198,  22,  38,  54,  70,  88,   6, 146,   0,   8,   0,   0,   0,
         0,   0,   0, 241,   0, 242,   0, 242,  25, 241,   5, 242,   0,
         0, 255,   0, 228,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   8,   0, 255, 255, 255, 255, 255, 255, 255,
         0,   0,   5,   0,   0, 186, 214, 117, 246, 219, 242], dtype=uint8), array([ 63,  63,  63,  63,  63,  63, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 192, 192, 192, 192, 192, 192, 255, 255, 255,
       255, 255, 255, 25[...]<jupyter_text>1.2. The `self.env` propertyThe `self.env` property points to the OpenAI Gym environment. Use this for any of the OpenAI Gym methods. However, it is best not to use the internal environment directly. It would be much more preferabe to update this environment to create a new method for this specific class.<jupyter_code>name = 'Breakout-ramNoFrameskip-v4'
with envGym.Env1D(name, showEnv=False) as env:
    print(f'Check environments action space: {env.env.action_space}')
    env.env.render()
    sleep(2)
    
env.env.close()<jupyter_output>Check environments action space: Discrete(4)<jupyter_text>1.3. Stepping and playingThe `Env` class provides two main methods to interact with the environment - a `self.step(policy)` and an `selef.episode(policy, maxSteps)`. For making this environment compatible with the `envUnity` environment, which can simulate more than a single actor at a single time, this policy is a function that should return a number of actions, one for each actor. For this reason, the result of a policy should always be a list of actions. For the `envGym` environment, this will mean a list with a single action. 1.3.1. Let us take a couple of stepsNote that the _Breakout_ takes a `Discrete(4)` pytorch tensor. Note allso that this returrns a result per actor (only one in this case). We shall specify a random action. It is worth noting that the size of the state is twice the size that will be obtained from using `Env` rather than `Env1`<jupyter_code>name = 'Breakout-ramNoFrameskip-v4'

policy = lambda m: [torch.randint(0, 4, (1,))]
with envGym.Env1D(name, N=2, showEnv=False) as env:
    
    print('Taking the first step')
    result = env.step(policy)
    print(result)
    
    print('\n\nTaking the second step')
    result = env.step(policy)[0]
    state, action, reward, nextState, done = result
    print(f'''
    state     : \n{state}
    nextState : \n{nextState}
    action    : {action}
    done      : {done}
    ''')<jupyter_output>Taking the first step
[(array([ 63,  63,  63,  63,  63,  63, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 192, 192, 192, 192, 192, 192, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 240,   0,   0,
       255,   0,   0, 240,   0,   5,   0,   0,   6,   0,  70, 182, 134,
       198,  22,  38,  54,  70,  88,   6, 146,   0,   8,   0,   0,   0,
         0,   0,   0, 241,   0, 242,   0, 242,  25, 241,   5, 242,   0,
         0, 255,   0, 228,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   8,   0, 255, 255, 255, 255, 255, 255, 255,
         0,   0,   5,   0,   0, 186, 214, 117, 246, 219, 242,  63,  63,
        63,  63,  63,  63, 255, 255, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
       255, 255, 192, 192, 192, 192, 192, 192, 255, 255, 255, 255, 255,
       255, 255, 255, 255, 255, 255, 255[...]<jupyter_text>1.3.1. Let us play an entire episodeNote that the _Breakout_ takes a `Discrete(4)` pytorch tensor. Note allso that this returrns a result per actor (only one in this case). We shall specify a randoom action ...<jupyter_code>name = 'Breakout-ramNoFrameskip-v4'

policy = lambda m: [torch.randint(0, 4, (1,))]
with envGym.Env1D(name, N=2, showEnv=False) as env:
    
    result = env.episode(policy, 10)[0]
    for r in result:
        state, action, reward, nextState, done = r
        print(f'reward = {reward}, action = {action}, done = {done}')
        print((nextState - state).sum())
    
    print(f'final state:\n{state}')
    print(f'final nextState-state:\n{nextState-state}')<jupyter_output>reward = 0.0, action = 3, done = False
772
reward = 0.0, action = 1, done = False
2360
reward = 0.0, action = 2, done = False
3116
reward = 0.0, action = 3, done = False
2559
reward = 0.0, action = 2, done = False
2558
reward = 0.0, action = 2, done = False
2807
reward = 0.0, action = 1, done = False
2057
reward = 0.0, action = 1, done = False
1799
reward = 0.0, action = 2, done = False
2046
reward = 0.0, action = 2, done = False
2045
final state:
[ 63  63  63  63  63  63 255 255 255 255 255 255 255 255 255 255 255 255
 255 255 255 255 255 255 255 255 255 255 255 255 192 192 192 192 192 192
 255 255 255 255 255 255 255 255 255 255 255 255 255 240   0   0 255   0
   0 240   0   5   0   0   6   0  70 182 134 198  22  38  54  70  82   4
 150   0   2   1   0   0   0   0   0 241   0 242   0 242  25 241   5 242
   8   0 255   0 220  65   0   0   0 135   0 119   0   1   0 255   0   0
   8   0 255 255 255 255 255 255 255   0   0   4   0   0 186 214 117 246
 219 242  63  63  63  63  63  63 255 [...]
{'fraction_non_alphanumeric': 0.11757754800590842, 'fraction_numerical': 0.18242245199409157, 'mean_word_length': 2.7121710526315788, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '26683', 'license': 'MIT', 'n_tokens_mistral': 3418, 'n_tokens_neox': 2429, 'n_words': 1020, 'path': 'notebooks/.ipynb_checkpoints/Environments - envGym.Env1D-checkpoint.ipynb', 'repo_name': 'sankhaMukherjee/RLalgos', 'size': '6770'}