<jupyter_start><jupyter_text>Guessing Gamesmachine learning, one step at a timeStep 14. Training ***14. Training***Remember the maze? When we figured out how to __explore__ versus __exploit__ the experience represented in a __q-table__? And remember we decided to __discount__ future rewards?Let's build those elements into an algorithm that _trains itself_ to play tic-tac-toe.__And here's the big idea: there is nothing specific to tic-tac-toe in the algorithm. It may as well be solving a maze, or some other problem. The algorithm just explores & exploits based on states, transitions, and resulting rewards. _It doesn't even know the rules of the game._ __Here is an alogrithm that can learn to play, or solve the maze, or lots of things (it will take several minutes to run -- and will stop when it has trained itself to play the game):<jupyter_code>import numpy as np
from tictactoe import *

env = Game()                # the environment happens to be a game, but could be a maze or a puzzle...

discount = 0.9              # these two variables control how we store rewards or penalties in our
explore = 0.01              # q-table, just like in the lessons about the maze.

#######################################################################################################
#                                                                                                     #
# SUPER IMPORTANT POINT: we don't even need to know the dimensions of the problem (the total number   #
#                        of states of actions... we can ask the environment to tell us those limits)  #
#                                                                                                     #
q = np.zeros((Game.state_space(), Game.action_space()))                                               #                                       
#                                                                                                     #
#######################################################################################################

attempts = 0                # keep track of how many times we have tried to solve the problem
not_yet_trained = True      # and keep trying until we are 'trained', see below

while not_yet_trained:
    total_reward = 0
    attempts += 1
    for n in range(10):                                    # play 10 games, keep track of the total reward
        state = env.reset()
        done = False
        ###############################################################################################
        #                                                                                             #
        # The actual q-learning is exactly like the example with the maze                             #
        #                                                                                             #
        while not done:                                                                               #
            if np.random.random() < explore:                                                          #
                action = env.sample()                                                                 #
            else:                                                                                     #
                action = np.argmax(q[state])                                                          #
            new_state, reward, done = env.step(action)                                                #
            q[state][action] += reward + discount * np.amax(q[new_state])                             #
            state = new_state                                                                         #
        #                                                                                             #
        ###############################################################################################      
        total_reward += reward
    not_yet_trained = True if total_reward < 0 else False  # if we never lose in 10 games, we are 'trained'
    print(attempts, total_reward)                          # print a progress report
        
env.replay()                                               # print a replay of the last game<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13689639006094703, 'fraction_numerical': 0.004453820909517112, 'mean_word_length': 1.1239422598307616, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 48, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '523504', 'license': 'MIT', 'n_tokens_mistral': 788, 'n_tokens_neox': 731, 'n_words': 327, 'path': 'Machine Learning, 14. Training.ipynb', 'repo_name': 'junior-ai-intro/guess-learn-jrnlyras19', 'size': '4266'}