<jupyter_start><jupyter_text>--- Load the Policy and Value Models<jupyter_code>latest_policy_model = tf.train.latest_checkpoint("./models/PolicyNet_1.0/")
print(latest_policy_model)
policy_model = PolicyModel(board_size=19, n_blocks=10, 
                    n_layers=3, n_filters=32, 
                    activation='relu')
policy_model.load_weights(latest_policy_model)
latest_value_model = tf.train.latest_checkpoint("./models/ValueNet_3.0/")
print(latest_value_model)
value_model = ValueModel(19, 10, 3, 32, 'relu')
value_model.load_weights(latest_value_model)<jupyter_output>./models/ValueNet_3.0/cp-0001.ckpt<jupyter_text>--- Self-Play<jupyter_code>sp = SelfPlay(policy=policy_model, board_size=19)<jupyter_output><empty_output><jupyter_text>You can create a single episode. The ```traj``` return value is the trajectory, also referred to as *stones* or *moves*. They come in board coordinates ---<jupyter_code>game, traj, terminated = sp.create_episode(limit=50, n_choices=10, greedy_bias=300, m2b='board')
episodes = sp.create_episodes(3, m2b='board')
print(episodes[1])
board = GomokuBoard(N=19, heuristics=Heuristics(kappa=3.0), stones=episodes[0], 
                    disp_width=10)
board.display()<jupyter_output><empty_output><jupyter_text>--- Value Function FittingNow we create a dataset for fitting the value function. The labels are:$$y_{i,t} = r(s_{i,t}, a_{i,t}) + \hat V_{\phi}^{\pi}(s_{i,t+2})$$Note the index $t+2$ in the value function. We're training by fitting to the next sample of the same player.  Generating Samples ```tf.keras``` can create ```tf.data``` datasets from generators. We use class ```A2C_SampleGeneratorFactory``` to create a fresh sample using the current policy.<jupyter_code>from wgomoku import A2C_SampleGeneratorFactory
factory = A2C_SampleGeneratorFactory(19, value_model, gamma=.9, cut_off=None)<jupyter_output><empty_output><jupyter_text>Understandig the *shift*The following depiction should help understand method ```_generator``` of class ```A2C_SampleGeneratorFactory```S0, S1, S2, S3, S4, S5, S6, S7, S8    sample positionsV2, V3, V4, V5, V6, V7, V8, +1, -1    shifted to the left, padded with -1, 1 and multiplied by gamma                                     V0, V1, V2, V3, V4, V5, V6, V7, V8You can easily convince yourself that the target definitions for the value update and the policy update can be taken from this setup.  EpisodesQuick-check: Episodes end with value -1<jupyter_code>episodes = sp.create_episodes(3, m2b='padded')
stones = [gt.m2b(move, 19)
    for move in np.array(episodes[0])-[1,1]]

board = GomokuBoard(N=19, heuristics=Heuristics(kappa=3.0), stones=stones, 
                    disp_width=10)
board.display()
generator = factory.create_generator(episodes, kind='values')
iterator = generator()<jupyter_output><empty_output><jupyter_text>Quick checkRun this cell until the value is -1 and then compare to the image above.<jupyter_code>s, v = next(iterator)
display_sample(s)
print(v)<jupyter_output>[[ 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2]
 [ 2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0 -1  0 -1  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  1  0  1 -1  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0 -1  0  1  0  1  1  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0 -1  0  1 -1  1  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  1  1 -1 -1 -1  0  1  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0 -1  0 -1 -1  1  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0 -1  0  0  1  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0 -1  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2]
 [ 2  0  0[...]<jupyter_text>--- The Serious Curriculum<jupyter_code>GAMMA = 0.9
N_EPOCHS = 5
N_EPISODES = 4
BATCH_SIZE = 256
VALUE_MODEL_DIR = "./models/ValueNet_3.0/"
POLICY_MODEL_DIR = "./models/PolicyNet_1.0/"<jupyter_output><empty_output><jupyter_text>--- Load the models from their lates checkpoints<jupyter_code>latest_value_model = tf.train.latest_checkpoint(VALUE_MODEL_DIR)
print(latest_value_model)
value_model = ValueModel(19, 10, 3, 32, 'relu')
value_model.load_weights(latest_value_model)

# The Critic is getting ready
factory = A2C_SampleGeneratorFactory(19, value_model, gamma=GAMMA, cut_off=None)
latest_policy_model = tf.train.latest_checkpoint(POLICY_MODEL_DIR)
print(latest_policy_model)
policy_model = PolicyModel(board_size=19, n_blocks=10, 
                    n_layers=3, n_filters=32, 
                    activation='relu')
policy_model.load_weights(latest_policy_model)

# So is the Actor
sp = SelfPlay(policy=policy_model, board_size=19)
sdh = SampleDataHelper(N, representation='NxNx1B', cut_off=12)
def new_value_dataset(n_episodes, n_epochs, batch_size):
    """
    Create a dataset of (state, target) pairs
    target = V(s_t+2)
    """
    episodes = sp.create_episodes(n_episodes, m2b='padded')
    generator = factory.create_generator(episodes, kind='values')
    inputs = tf.data.Dataset.from_generator(
        generator, output_types=(tf.int32, tf.float32))

    inputs = inputs.shuffle(100).repeat(n_epochs).batch(batch_size)
    return inputs, episodes


def new_policy_dataset(episodes, n_epochs, batch_size):
    """
    create a data set of (state, action, advantage) triples, given existing
    episodes
    advantage = V(s_t+2) - V(s_t)
    """
    generator = factory.create_generator(episodes, kind='advantages')
    inputs = tf.data.Dataset.from_generator(
        generator, output_types=(tf.int32, tf.int32, tf.float32))

    inputs = inputs.shuffle(100).repeat(n_epochs).batch(batch_size)
    return inputs<jupyter_output><empty_output><jupyter_text>--- 1) Sample a little batch (N episodes x 8 symmetries)<jupyter_code>value_inputs, episodes = new_value_dataset(
    n_episodes=N_EPISODES,
    n_epochs=N_EPOCHS, 
    batch_size=BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>--- 2) Train for some epochs<jupyter_code>optimizer = tf.keras.optimizers.SGD(learning_rate=5e-3)
value_model.compile(optimizer=optimizer, loss='mean_squared_error')

value_model.fit(value_inputs)<jupyter_output>14/14 [==============================] - 14s 986ms/step - loss: 0.1059<jupyter_text>--- 3) Examine the value network qualityAfter a while you may observe more and more of the alternating pattern sneaking in from the end of the episodes. That's a sign that the *Critic* is getting ready.<jupyter_code>samples = sdh.traj_to_samples(episodes[0], -1, GAMMA)
print(np.shape(samples[0]))
values = value_model.call(samples[0])
values[-10:]<jupyter_output>(22, 21, 21, 2)<jupyter_text>--- 4) Prepare the dataset for policy training$$\hat{A}^\pi(s_i, a_i) = r(s_i, a_i) + \gamma\hat{V}_\phi^\pi(s_{i+2}) - \hat{V}_\phi^\pi(s_i)$$<jupyter_code>policy_inputs = new_policy_dataset(
    episodes = episodes,
    n_epochs=N_EPOCHS, 
    batch_size=BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>--- Intermezzo: What's in the dataset?<jupyter_code>generator = factory.create_generator(episodes, kind='advantages')
batch=next(iter(generator()))<jupyter_output><empty_output><jupyter_text>Keras accepts a third column in the dataset that represents the sample weights. This is a perfect place to position our *Critic*'s advice, i.e. the advantages.<jupyter_code>np.shape(batch[0]), np.divmod(batch[1], 19), batch[2]<jupyter_output><empty_output><jupyter_text>--- 5) Train the policy network<jupyter_code>loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
policy_model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=1e-2), loss=loss)
policy_model.fit(policy_inputs)<jupyter_output>14/14 [==============================] - 18s 1s/step - loss: -0.0569<jupyter_text>Note that the *loss* function can go below zero due to the potentially negative weights. --- Closing the loop<jupyter_code>N_EPOCHS = 10
N_EPISODES = 16
BATCH_SIZE = 256
NUM_CYCLES = 5
for n in range(NUM_CYCLES):

    print("Cycle %s" % n)
    # Sample
    value_inputs, episodes = new_value_dataset(
        n_episodes=N_EPISODES,
        n_epochs=N_EPOCHS, 
        batch_size=BATCH_SIZE)
    
    # Fit value function
    value_model.fit(value_inputs)
    
    # Get policy data
    policy_inputs = new_policy_dataset(
        episodes = episodes,
        n_epochs=N_EPOCHS, 
        batch_size=BATCH_SIZE)
    
    # Fit policy function
    policy_model.fit(policy_inputs)<jupyter_output>Cycle 0
167/167 [==============================] - 58s 347ms/step - loss: 0.0877
162/162 [==============================] - 59s 366ms/step - loss: 0.0025
Cycle 1
167/167 [==============================] - 58s 349ms/step - loss: 0.1122
162/162 [==============================] - 60s 369ms/step - loss: 0.0012
Cycle 2
139/139 [==============================] - 49s 356ms/step - loss: 0.1180
134/134 [==============================] - 51s 379ms/step - loss: -0.0247
Cycle 3
158/158 [==============================] - 56s 351ms/step - loss: 0.1029
153/153 [==============================] - 57s 372ms/step - loss: -0.0074
Cycle 4
128/128 [==============================] - 46s 362ms/step - loss: 0.1223
123/123 [==============================] - 47s 383ms/step - loss: 0.0469<jupyter_text>checkpoint_path = './models/PolicyNet_1.0/cp-{iteration:04d}.ckpt/'policy_model.save_weights(checkpoint_path.format(iteration=6)) checkpoint_path = './models/ValueNet_3.0/cp-{iteration:04d}.ckpt/'value_model.save_weights(checkpoint_path.format(iteration=6))<jupyter_code>game, traj, terminated = sp.create_episode(limit=80, n_choices=10, 
                                           greedy_bias=300, m2b='board')
board = GomokuBoard(N=19, heuristics=Heuristics(kappa=3.0), stones=traj, 
                    disp_width=10)
board.display()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14786069651741293, 'fraction_numerical': 0.07164179104477612, 'mean_word_length': 3.709934395501406, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '434355', 'license': 'Apache-2.0', 'n_tokens_mistral': 4137, 'n_tokens_neox': 3815, 'n_words': 965, 'path': 'A2C_Curriculum.ipynb', 'repo_name': 'smurve/DeepGomoku', 'size': '10050'}