<jupyter_start><jupyter_text>Guessing Gamesmachine learning, one step at a timeStep 9. Exploration v. Exploitation **9. Exploration v. exploitation**_Reinforcment learning_ is a balancing act between __exploration__ and __exploitation__:- __Exploration__: where am I? what actions can I take? what happens when I take an action?- __Exploitation__: can I use my past knowledge of penalties and rewards to go in the right direction?We _explored_ the maze by randomly examining it many times. How can we _exploit_ those results to find our way quickly and easily?Let's go back to building our __q-table__:<jupyter_code>import numpy as np
from maze import Maze
maze = Maze()

# run the maze lots of times; take note of every result in a q-table
q = np.zeros((4,4,4))
for n in range(3000):
    state = maze.reset()
    done = False
    while not done:
        action = maze.sample_n()                     # return a random action (0,1,2,3)  
        new_state, reward, done = maze.step(action)  # takes a step based upon the random action
        q[state[0]][state[1]][action] += reward      # makes note of the resulting transition
        state = new_state                            # ...and switches to the new state
        
Maze.print_q(q)<jupyter_output><empty_output><jupyter_text>Around half the attempts ended right away, by going out of bounds (by stepping either north or west). Doesn't that seem a little wasteful?Or put another way: for any given state, if an action has resulted in penalties in the past, can we avoid that action in the future?We would like to use the q table to find our way, based on past results... but first! we need the awesome power of _argmax_:<jupyter_code>import numpy as np

# here is an array of rewards...
a = [-100,-200,0,-50]

# I wish there was a function to tell me 
# the index of the entry that has the 
# maximum value... in this case, what is 
# the index of the entry with a value of 
# zero, indicating no penalties?

# enter argmax! which return the index of 
# the maximum value. wow. just wow.
print(np.argmax(a))<jupyter_output><empty_output><jupyter_text>OK good. Now that we have a fully populated __q-table__ and the awesome power of _argmax_, let's traverse the maze:<jupyter_code># here is how we might take the first step...
state = maze.reset()

row = state[0]
col = state[1]
action = np.argmax(q[row][col])         # it's easier to use 0,1,2,3 instead of N,S,E,W for our action...
state, reward, done = maze.step(action) # ...those values are interchangeable when calling step(action)
print(maze.action_space()[action], state, reward, done)
print(maze)<jupyter_output><empty_output><jupyter_text>That's a start... we will always take a first step that avoids a penalty.How about taking 10 steps?<jupyter_code>state = maze.reset()
for n in range(10):
    row = state[0]
    col = state[1]
    action = np.argmax(q[row][col])
    state, reward, done = maze.step(action)
    print(maze.action_space()[action], state, reward, done)<jupyter_output><empty_output><jupyter_text>Well... we avoided penalties. Unfortunately, we probably got caught in a loop, and did not go anywhere.In fact, the q-table might be __all full of penalties__ at this stage. Let's create another q-table, explore the maze until we stumble across the exit once, and plot the average value for each __state__:<jupyter_code>%matplotlib inline
import numpy as np
from maze import Maze

maze = Maze()
q = np.zeros((4,4,4))
completed = False
while not completed:
    state = maze.reset()
    done = False
    while not done:
        action = maze.sample_n()                     # use numeric form (sample_n)                   
        new_state, reward, done = maze.step(action)
        q[state[0]][state[1]][action] += reward      # the action is a number, so it fits right into q[][][]
        state = new_state                           
    completed = True if reward > 0 else False        # stop exploring when we get a positive reward

Maze.plot(q)<jupyter_output><empty_output><jupyter_text>The initial __state__ (0,0) is in the lower left corner... in terms of getting to positive values for rewards, we have to find our way out of a pretty deep hole.What if we tried to balance __exploration__ and __exploitation__?For example, as a first attempt: what if we moved at random, but discarded moves that have penalties? (or more formally: what if we discard __transitions__ that have negative __q-values__)? ***Exercises***- traverse the maze using random actions, but avoid actions with negative q-values- note! that won't work unless you allow the code, as provided, to populate a q-table in advance- see the guidance in the code sample<jupyter_code>import numpy as np
from maze import Maze
'''
=== DO NOT CHANGE CODE STARTING HERE ====== THOU SHALT NOT PASS ================================
'''
maze = Maze()

# build a q-table
q = np.zeros((4,4,4))
for n in range(10000):
    state = maze.reset()
    done = False
    while not done:
        action = maze.sample_n()                     # return a random action (0,1,2,3)  
        new_state, reward, done = maze.step(action)  # takes a step based upon the random action
        q[state[0]][state[1]][action] += reward      # makes note of the resulting transition
        state = new_state                            # ...and switches to the new state
'''
=== AND ENDING HERE ======================= THOU SHALT NOT PASS ================================
'''

###############################################################################
#                                                                             #
#  YOUR CODE GOES HERE...                                                     #
#     ...something like                                                       #
#             state = maze.reset()                                            #
#             done = False                                                    #
#             while not Done:                                                 #
#                 get a sample action                                         #
#                 while the q-value for the action is negative                #
#                     get a different sample action                           #
#                 use maze.step(action) to take step & update state           #
#                                                                             #
###############################################################################

print(maze)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11737664725712535, 'fraction_numerical': 0.008887526815813668, 'mean_word_length': 2.440695835529784, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '474203', 'license': 'MIT', 'n_tokens_mistral': 1799, 'n_tokens_neox': 1687, 'n_words': 726, 'path': 'Machine Learning, 09. Exploration v. Exploitation.ipynb', 'repo_name': 'junior-ai-intro/guess-learn-jrnlyras19', 'size': '6526'}