<jupyter_start><jupyter_text>Now we have enough data<jupyter_code>import main, pickle
random_board_pairs = []
with open("2021-09-09-mcts-v3-boards10-random.pickle", "rb") as f:
    try:
        while True:
            random_board_pairs.append(pickle.load(f))
    except EOFError:
        pass
len(random_board_pairs)
game_board_pairs = []
with open("2021-09-12-mcts-v3-boards11-games.pickle", "rb") as f:
    try:
        while True:
            game_board_pairs.append(pickle.load(f))
    except EOFError:
        pass
len(game_board_pairs)
random_inputs = []
random_outputs = []
for p in random_board_pairs:
    random_inputs.append(p[0].get_flipped_3d_array())
    random_outputs.append(float(p[1]))
game_inputs = []
game_outputs = []
for p in game_board_pairs:
    game_inputs.append(p[0].get_flipped_3d_array())
    game_outputs.append(float(p[1]))
import numpy as np
game_inputs = np.array(game_inputs)
random_inputs = np.array(random_inputs)
game_outputs = np.array(game_outputs)
random_outputs = np.array(random_outputs)
random_mark = int(len(random_outputs)*0.8)
game_mark = int(len(game_outputs)*0.8)
print(random_mark, game_mark)
train_inputs = np.concatenate((random_inputs[0:random_mark] , game_inputs[0:game_mark] 
                               , random_inputs[random_mark:] ,  game_inputs[game_mark:])) 
train_outputs = np.concatenate(( random_outputs[0:random_mark],
                                 game_outputs[0:game_mark],
                                random_outputs[random_mark:] ,
                                game_outputs[game_mark:] ))
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow as tf<jupyter_output><empty_output><jupyter_text>Maybe I need pooling?? Or an extra Conv2d layer?  Model3 has an extra conv2d layer<jupyter_code>model3 = keras.Sequential()
model3.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(19, 15, 2)))
model3.add(layers.Conv2D(64, (3, 3), activation='relu'))
model3.add(layers.Conv2D(64, (3, 3), activation='relu'))
model3.add(layers.Flatten())
model3.add(layers.Dense(256, activation='relu'))
model3.add(layers.Dense(1, activation='relu'))
model3.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 17, 13, 32)        608       
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 15, 11, 64)        18496     
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 13, 9, 64)         36928     
_________________________________________________________________
flatten (Flatten)            (None, 7488)              0         
_________________________________________________________________
dense (Dense)                (None, 256)               1917184   
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 257       
========================================================[...]<jupyter_text>I have fewer parameters? Because the last conv layer made the array narrower than it was tall, so I didn't need as much dense?<jupyter_code>model3.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model2.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.1007 - val_loss: 0.0646
Epoch 2/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0521 - val_loss: 0.0523
Epoch 3/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0477 - val_loss: 0.0491
Epoch 4/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0441 - val_loss: 0.0503
Epoch 5/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0418 - val_loss: 0.0490
Epoch 6/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0392 - val_loss: 0.0468
Epoch 7/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0372 - val_loss: 0.0497
Epoch 8/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0350 - val_loss: 0.0485
Epoch 9/10
1136/1136 [==============================] - 23s 20ms/step - loss: 0.0327 - val_loss: 0.0459
Epoch 10/10
1136/1136 [==============================] - 23s 20m[...]<jupyter_text>When training v4 (and the others, I think), the CV loss would be about 0.029. Now, when training v5, it's about 0.047. That's 62% higher. Maybe that's a good sign—the data is more complicated and matches the real world better, so the neural net has to struggle to learn it.Maybe that's a bad sign—the data has a lot of random noise, so the net has to struggle to learn it Aiming for below 0.047 CV loss<jupyter_code>model3.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 35s 30ms/step - loss: 0.1009 - val_loss: 0.0562
Epoch 2/10
1136/1136 [==============================] - 34s 30ms/step - loss: 0.0521 - val_loss: 0.0528
Epoch 3/10
1136/1136 [==============================] - 35s 31ms/step - loss: 0.0462 - val_loss: 0.0440
Epoch 4/10
1136/1136 [==============================] - 52s 46ms/step - loss: 0.0425 - val_loss: 0.0439
Epoch 5/10
1136/1136 [==============================] - 40s 35ms/step - loss: 0.0404 - val_loss: 0.0431
Epoch 6/10
1136/1136 [==============================] - 42s 37ms/step - loss: 0.0386 - val_loss: 0.0435
Epoch 7/10
1136/1136 [==============================] - 43s 38ms/step - loss: 0.0367 - val_loss: 0.0440
Epoch 8/10
1136/1136 [==============================] - 45s 40ms/step - loss: 0.0354 - val_loss: 0.0401
Epoch 9/10
1136/1136 [==============================] - 45s 39ms/step - loss: 0.0342 - val_loss: 0.0407
Epoch 10/10
1136/1136 [==============================] - 46s 40m[...]<jupyter_text>Hmm, so CV loss is 0.041. 41e-2 is better than 47e-2, but I want to aim for 29e-2Let's add pooling between the Conv2Ds, like this: https://www.tensorflow.org/tutorials/images/cnn<jupyter_code>model4 = keras.Sequential()
model4.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(19, 15, 2)))
model4.add(layers.MaxPooling2D((2, 2)))
model4.add(layers.Conv2D(64, (3, 3), activation='relu'))
model4.add(layers.Conv2D(64, (3, 3), activation='relu'))

model4.summary()
model4.add(layers.Flatten())
model4.add(layers.Dense(256, activation='relu'))
model4.add(layers.Dense(1, activation='relu'))
model4.summary()<jupyter_output>Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_8 (Conv2D)            (None, 17, 13, 32)        608       
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 8, 6, 32)          0         
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 6, 4, 64)          18496     
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 4, 2, 64)          36928     
_________________________________________________________________
flatten_1 (Flatten)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 256)               131328    
______________________________________________________[...]<jupyter_text>Pooling makes the model smaller<jupyter_code>model4.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model4.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)
model4.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0423 - val_loss: 0.0545
Epoch 2/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0411 - val_loss: 0.0533
Epoch 3/10
1136/1136 [==============================] - 8s 7ms/step - loss: 0.0402 - val_loss: 0.0522
Epoch 4/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0387 - val_loss: 0.0529
Epoch 5/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0378 - val_loss: 0.0537
Epoch 6/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0371 - val_loss: 0.0537
Epoch 7/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0358 - val_loss: 0.0541
Epoch 8/10
1136/1136 [==============================] - 7s 7ms/step - loss: 0.0350 - val_loss: 0.0551
Epoch 9/10
1136/1136 [==============================] - 7s 6ms/step - loss: 0.0343 - val_loss: 0.0548
Epoch 10/10
1136/1136 [==============================] - 8s 7ms/step - loss: 0.033[...]<jupyter_text>So this seems stuck at 53e-2<jupyter_code>model3.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 43s 38ms/step - loss: 0.0318 - val_loss: 0.0423
Epoch 2/10
1136/1136 [==============================] - 51s 45ms/step - loss: 0.0307 - val_loss: 0.0400
Epoch 3/10
1136/1136 [==============================] - 45s 40ms/step - loss: 0.0295 - val_loss: 0.0402
Epoch 4/10
1136/1136 [==============================] - 50s 44ms/step - loss: 0.0285 - val_loss: 0.0402
Epoch 5/10
1136/1136 [==============================] - 56s 49ms/step - loss: 0.0272 - val_loss: 0.0423
Epoch 6/10
1136/1136 [==============================] - 51s 45ms/step - loss: 0.0265 - val_loss: 0.0411
Epoch 7/10
1136/1136 [==============================] - 50s 44ms/step - loss: 0.0256 - val_loss: 0.0403
Epoch 8/10
1136/1136 [==============================] - 47s 42ms/step - loss: 0.0248 - val_loss: 0.0410
Epoch 9/10
1136/1136 [==============================] - 56s 50ms/step - loss: 0.0241 - val_loss: 0.0402
Epoch 10/10
1136/1136 [==============================] - 48s 43m[...]<jupyter_text>And this is still stuck at 41e-2 model5 has 4 conv layers<jupyter_code>model5 = keras.Sequential()
model5.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(19, 15, 2)))
model5.add(layers.Conv2D(64, (3, 3), activation='relu'))
model5.add(layers.Conv2D(64, (3, 3), activation='relu'))
model5.add(layers.Conv2D(64, (3, 3), activation='relu'))

model5.summary()
model5.add(layers.Flatten())
model5.add(layers.Dense(256, activation='relu'))
model5.add(layers.Dense(1, activation='relu'))
model5.summary()<jupyter_output>Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_11 (Conv2D)           (None, 17, 13, 32)        608       
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 15, 11, 64)        18496     
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 13, 9, 64)         36928     
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 11, 7, 64)         36928     
_________________________________________________________________
flatten_2 (Flatten)          (None, 4928)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 256)               1261824   
______________________________________________________[...]<jupyter_text>A little smaller than model3, but much larger than model 4<jupyter_code>model5.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model5.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 40s 35ms/step - loss: 0.1210 - val_loss: 0.0666
Epoch 2/10
1136/1136 [==============================] - 41s 36ms/step - loss: 0.0532 - val_loss: 0.0498
Epoch 3/10
1136/1136 [==============================] - 55s 48ms/step - loss: 0.0461 - val_loss: 0.0493
Epoch 4/10
1136/1136 [==============================] - 54s 47ms/step - loss: 0.0434 - val_loss: 0.0439
Epoch 5/10
1136/1136 [==============================] - 58s 51ms/step - loss: 0.0407 - val_loss: 0.0430
Epoch 6/10
1136/1136 [==============================] - 53s 46ms/step - loss: 0.0386 - val_loss: 0.0407
Epoch 7/10
1136/1136 [==============================] - 58s 51ms/step - loss: 0.0380 - val_loss: 0.0395
Epoch 8/10
1136/1136 [==============================] - 63s 56ms/step - loss: 0.0366 - val_loss: 0.0397
Epoch 9/10
1136/1136 [==============================] - 51s 45ms/step - loss: 0.0344 - val_loss: 0.0390
Epoch 10/10
1136/1136 [==============================] - 60s 53m[...]<jupyter_text>Hmm. This got the CV error down to 39e-2, but then it jumped back up to 43e-2 model6 has 3 conv layers, but the final one is a little larger so that it has the same number of parameters as model2<jupyter_code>model6 = keras.Sequential()
model6.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(19, 15, 2)))
model6.add(layers.Conv2D(64, (3, 3), activation='relu'))
model6.add(layers.Conv2D(90, (3, 3), activation='relu'))
model6.add(layers.Flatten())
model6.add(layers.Dense(256, activation='relu'))
model6.add(layers.Dense(1, activation='relu'))
model6.summary()
model6.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model6.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)
model6.fit(train_inputs, train_outputs, epochs=3, validation_split = 0.2)
model7 = keras.Sequential()
model7.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(19, 15, 2)))
model7.add(layers.Conv2D(64, (3, 3), activation='relu'))
model7.add(layers.Conv2D(90, (3, 3), activation='relu'))
model7.add(layers.Flatten())
model7.add(layers.Dense(256, activation='relu'))
model7.add(layers.Dropout(0.5))
model7.add(layers.Dense(1, activation='relu'))
model7.summary()
model7.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model7.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)
model8=keras.Sequential()
model8.add(layers.Dense(256, activation='relu',input_shape=(19, 15, 2)))
model8.add(layers.Dropout(0.5))
model8.add(layers.Dense(1, activation='relu'))
model8.summary()
model7.compile(loss='mean_absolute_error',optimizer=tf.keras.optimizers.Adam(0.001))
model7.fit(train_inputs, train_outputs, epochs=10, validation_split = 0.2)<jupyter_output>Epoch 1/10
1136/1136 [==============================] - 40s 35ms/step - loss: 0.1167 - val_loss: 0.0547
Epoch 2/10
1136/1136 [==============================] - 41s 36ms/step - loss: 0.0652 - val_loss: 0.0495
Epoch 3/10
1136/1136 [==============================] - 46s 41ms/step - loss: 0.0563 - val_loss: 0.0455
Epoch 4/10
1136/1136 [==============================] - 55s 49ms/step - loss: 0.0523 - val_loss: 0.0449
Epoch 5/10
1136/1136 [==============================] - 48s 42ms/step - loss: 0.0493 - val_loss: 0.0502
Epoch 6/10
1136/1136 [==============================] - 53s 47ms/step - loss: 0.0482 - val_loss: 0.0416
Epoch 7/10
1136/1136 [==============================] - 51s 45ms/step - loss: 0.0466 - val_loss: 0.0437
Epoch 8/10
1136/1136 [==============================] - 63s 55ms/step - loss: 0.0454 - val_loss: 0.0434
Epoch 9/10
1136/1136 [==============================] - 52s 46ms/step - loss: 0.0442 - val_loss: 0.0423
Epoch 10/10
1136/1136 [==============================] - 52s 46m[...]
{'fraction_non_alphanumeric': 0.24548460951411855, 'fraction_numerical': 0.12598575426100228, 'mean_word_length': 4.435534047701348, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '438876', 'license': 'MIT', 'n_tokens_mistral': 6474, 'n_tokens_neox': 5109, 'n_words': 1124, 'path': '2021-09-16-training-v6.ipynb', 'repo_name': 'TheodoreEhrenborg/phutball', 'size': '15724'}