<jupyter_start><jupyter_text>Welcome to Reinforcement Learning Reinforcement Learning is a framework for tackling **sequential decision problems**: what to do next in order to maximize a reward (which might be delayed), on a changing universe (which might react to our actions).Concrete examples include:- Game playing: which actions are critical to win a game? This could be [Atari](http://karpathy.github.io/2016/05/31/rl/) or [Go](https://en.wikipedia.org/wiki/AlphaZero).- Learning in a "small world": what actions maximize pleasure / minimize pain?- [Treatment of chronic diseases](https://www.ncbi.nlm.nih.gov/pubmed/19731397): how to evolve the treatment for a disease that creates resistance?The unifying theme on the problems above can be abstracted as follows: - An **agent** receives a signal from the environment, selected by Nature. - The agent executes an **action**. - Given the agents' action, Nature assigns a reward and draws a new state, which is announced to the agent. - The situation repeats until a terminal criterion is reached. We will use the OpenAI Gym [(https://github.com/openai/gym)](https://github.com/openai/gym).  environment for this. It consists of a number of Atari environments that we can use for experimenting. If you haven't please install the library OpenAI gym (`pip install gym`).To test your installation, run the following script:```pythonimport gymenv = gym.make('CartPole-v0')env.reset()for _ in range(1000):    env.render()    env.step(env.action_space.sample())  take a random action```    You should see a window that opens with a car and a pole, and will most likely close quickly.  A walk on the Frozen Lake  We will start with a very simple environment, called `FrozenLake`.<jupyter_code>!pip install -q tqdm
#from tqdm import tqdm
import gym

#create a single game instance
env = gym.make("FrozenLake-v0")<jupyter_output><empty_output><jupyter_text>Here, **S** is the initial state, and your aim is to reach **G**, without falling into the holes, **H**. The squares marked with **F** are frozen, which means you can step on them.  **Note:** The environment is non-deterministic, you can slip in the ice and end up in a different state.How to use the environment? - **reset()** returns the initial state / first observation.- **render()** returns the current environment state. - **step(a)** returns what happens after action a:     - *new observation*: the new state.     - *reward*: the reward corresponding to that action in that state.     - *is done*: binary flag, True if the game is over.      - *info*: Some auxiliary stuff, which we can ignore now.<jupyter_code>print("The initial state: ", env.reset())
print(" and it looks like: ")
env.render()
print("Now let's take an action: ")
env.reset()
new_state, reward, done, _ = env.step(1)
env.render()


idx_to_action = {
    0:"<", #left
    1:"v", #down
    2:">", #right
    3:"^" #up
}<jupyter_output>Now let's take an action: 
  (Down)
[41mS[0mFFF
FHFH
FFFH
HFFG<jupyter_text>A **policy** is a function from states to actions. It tells us what we should do on each state. In this case, an array of size 16 with entries 0,1,2 or 3 determines a **deterministic** policy, whereas an array of size 16x4 with entries between 0 and 1 and where each row sums 1 determines a **stochastic** policy. For simplicity, we will implement policies as dictionaries for `FrozenLake`.<jupyter_code>import numpy as np
n_states = env.observation_space.n
n_actions = env.action_space.n

# Initialize random_policy:
def init_random_policy():
    random_policy  = {}
    for state in range(n_states):
        random_policy[state] = np.random.choice(n_actions)
    return random_policy<jupyter_output><empty_output><jupyter_text>We need now to define a function to evaluate our policy.<jupyter_code>def evaluate(env, policy, max_episodes=100): 
    tot_reward = 0
    for ep in range(max_episodes):
        state = env.reset()
        done = False
        ep_reward = 0
        
        # Reward per episode
        while not done:
            action = policy[state]
            new_state, reward, done, _ = env.step(action)
            ep_reward += reward
            state = new_state
            if done:
                tot_reward += ep_reward
    return tot_reward/(max_episodes)<jupyter_output><empty_output><jupyter_text>Looking for the best policy: Random searchAs a very first example, let's try to find our policy by pure random search: we will play for some time and keep track of the best actions we can do on each state.`FrozenLake` defines "solving" as getting average reward of 0.78 over 100 consecutive trials<jupyter_code>best_policy = None
best_score = -float('inf')

# Random search
for i in range(1,10000): #tip: you can use tqdm(range(1,10000)) for a progress bar
    policy = init_random_policy()
    score = evaluate(env,policy,100)
    if score > best_score:
        best_score = score
        best_policy = policy
    if i%5000 == 0:
        print("Best score:", best_score)
print("Best policy:")
#print(best_policy)<jupyter_output>Best score: 0.65
Best policy:<jupyter_text>Now let's see the policy in action:<jupyter_code>def play(env,policy, render=False):
    s = env.reset()
    d = False
    while not d:
        a = policy[s]
        print("*"*10)
        print("State: ",s)
        print("Action: ",idx_to_action[a])
        s, r, d, _ = env.step(a)
        if render:
            env.render()
        if d:
            print(r)<jupyter_output><empty_output><jupyter_text>Let's create a small function to print a nicer policy:<jupyter_code>def print_policy(policy):
    lake = "SFFFFHFHFFFHHFFG"
    arrows = [idx_to_action[policy[i]] 
               if lake[i] in 'SF' \
              else '*' for i in range(n_states)]
    for i in range(0,16,4):
        print(''.join(arrows[i:i+4]))<jupyter_output><empty_output><jupyter_text>We can call then use the functions above to render the optimal policy. Note that the policy might not give the optimal action: recall that there is some noise involved (you can slip on the ice) which is responsible of a non-optimal action looking like optimal.<jupyter_code>print_policy(best_policy)

#play(env,best_policy)<jupyter_output><v<v
<*v*
^v<*
*>v*<jupyter_text>Using a different policyLet's try some different implementation of a random policy, which will be more useful later on.<jupyter_code># theta = 0.25*np.ones((n_states,n_actions))
def random_parameter_policy(theta):
    theta = theta/np.sum(theta, axis=1, keepdims=True) # ensure that the array is normalized
    policy  = {}
    probs = {}
    for state in range(n_states):
        probs[state] = np.array(theta[state,:])
        policy[state] = np.random.choice(n_actions, p = probs[state])
    return policy
best_policy = None
best_score = -float('inf')
alpha = 1e-2

theta = 0.25*np.ones((n_states,n_actions))

# Random search
for i in range(1,10000): 
    policy = random_parameter_policy(theta)
    score = evaluate(env,policy,100)
    if score > best_score:
        best_score = score
        best_policy = policy
    theta = theta + alpha*np.random.rand(n_states,n_actions)
    if i%5000 == 0:
        print("Best score:", best_score)
#print("Best policy:")
#print(best_policy)
#print("Best score:", best_score)<jupyter_output>Best score: 0.69
{'fraction_non_alphanumeric': 0.10472787245739418, 'fraction_numerical': 0.014980758658603628, 'mean_word_length': 3.8513333333333333, 'pattern_counts': {'":': 1, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '46190', 'license': 'MIT', 'n_tokens_mistral': 2320, 'n_tokens_neox': 2182, 'n_words': 812, 'path': 'Day 1/01 welcome.ipynb', 'repo_name': 'jpmaldonado/packt-rl', 'size': '7276'}