<jupyter_start><jupyter_text>noteThere are quite some fluctuations in how good we can train the network; just rerunning it again with the same parameters can give better results in some cases.There are various techniques to make the training more reliable, chapter 3 of the Nielsen's book talks about them.Yet, we have already seen the basic idea behind the neural networks (and machine learning in general): instead of designing an algorithm for a very specific task (like classifying handwritten digits), we generate a structure which can "learn" thealgorithm from a lot of example data provided (so-called supervised learning). This approach is "universal" in the sense that it can be applied to a lot of different problems, and in many cases even yields better results than writing a "code" in the traditional sense, e.g. it is nearly impossible to write a code recognising handwritten digits from image data.<jupyter_code>net = Network([784, 30, 10])
net.SGD(training_data, 15, 10, 0.5, test_data=test_data) # same parameters<jupyter_output>Epoch  0:     8455 / 10000 = 84.5%
Epoch  1:     8825 / 10000 = 88.2%
Epoch  2:     8953 / 10000 = 89.5%
Epoch  3:     9060 / 10000 = 90.6%
Epoch  4:     9124 / 10000 = 91.2%
Epoch  5:     9162 / 10000 = 91.6%
Epoch  6:     9213 / 10000 = 92.1%
Epoch  7:     9225 / 10000 = 92.2%
Epoch  8:     9244 / 10000 = 92.4%
Epoch  9:     9251 / 10000 = 92.5%
Epoch  10:     9282 / 10000 = 92.8%
Epoch  11:     9293 / 10000 = 92.9%
Epoch  12:     9306 / 10000 = 93.1%
Epoch  13:     9321 / 10000 = 93.2%
Epoch  14:     9326 / 10000 = 93.3%<jupyter_text>try again with a deeper network and more neurons<jupyter_code>net = Network([784, 50, 30, 10]) # (training takes a long time!)
net.SGD(training_data, 15, 10, 0.5, test_data=test_data)<jupyter_output>Epoch  0:     8383 / 10000 = 83.8%
Epoch  1:     8899 / 10000 = 89.0%
Epoch  2:     9055 / 10000 = 90.5%
Epoch  3:     9153 / 10000 = 91.5%
Epoch  4:     9219 / 10000 = 92.2%
Epoch  5:     9250 / 10000 = 92.5%
Epoch  6:     9302 / 10000 = 93.0%
Epoch  7:     9320 / 10000 = 93.2%
Epoch  8:     9341 / 10000 = 93.4%
Epoch  9:     9370 / 10000 = 93.7%
Epoch  10:     9373 / 10000 = 93.7%
Epoch  11:     9387 / 10000 = 93.9%
Epoch  12:     9386 / 10000 = 93.9%
Epoch  13:     9419 / 10000 = 94.2%
Epoch  14:     9408 / 10000 = 94.1%<jupyter_text>actually hard to improve...  Training with Monte Carlo data generate and load Monte carlo data<jupyter_code>generate = False
try:
    training_data_ising, _, test_data_ising = data_loader.load_data_wrapper("mcIsing.pkl.gz")
except IOError: 
    # file not existent: need to generate data first
    generate = True
    
if generate:
    print("need to generate")
    import generate_mc_data
    data = generate_mc_data.generate_data()
    generate_mc_data.save_data(data, filename="mcIsing.pkl.gz")
    # retry loading
    training_data_ising, _, test_data_ising = data_loader.load_data_wrapper("mcIsing.pkl.gz")
# plot a few of the images in the test data
L = 28
img_label_data = test_data_ising[:9]
fig, axes = plt.subplots(3, 3, True, True, figsize=(10, 10))
for ((img, label), ax) in zip(img_label_data, axes.flatten()):
    ax.imshow(np.reshape(img, [L, L]))
    ax.set_title("label = "+str(label))
plt.show()<jupyter_output><empty_output><jupyter_text>associate labels to temperatures|label|temp. T|   |-----|-------|-----------------------------------|0    |3      |disordered|1    |2.3    |close to critical, already ordered for the given system size|2    |1.5    |ordered, few fluctuations train a network<jupyter_code>net = Network([784, 30, 3])
net.SGD(training_data_ising, 15, 10, 0.5, test_data=test_data_ising)<jupyter_output>Epoch  0:     3078 / 6000 = 51.3%
Epoch  1:     3412 / 6000 = 56.9%
Epoch  2:     3532 / 6000 = 58.9%
Epoch  3:     3476 / 6000 = 57.9%
Epoch  4:     3672 / 6000 = 61.2%
Epoch  5:     3749 / 6000 = 62.5%
Epoch  6:     3719 / 6000 = 62.0%
Epoch  7:     3838 / 6000 = 64.0%
Epoch  8:     3843 / 6000 = 64.0%
Epoch  9:     3899 / 6000 = 65.0%
Epoch  10:     3972 / 6000 = 66.2%
Epoch  11:     4025 / 6000 = 67.1%
Epoch  12:     4062 / 6000 = 67.7%
Epoch  13:     4092 / 6000 = 68.2%
Epoch  14:     4108 / 6000 = 68.5%<jupyter_text>note that we come above 66%, i.e. we classify at least some images of each temperature correctly!<jupyter_code>net = Network([784, 10, 10, 10, 3])
net.SGD(training_data_ising, 15, 30, 0.5, test_data=test_data_ising)<jupyter_output>Epoch  0:     3896 / 6000 = 64.9%
Epoch  1:     4143 / 6000 = 69.0%
Epoch  2:     4289 / 6000 = 71.5%
Epoch  3:     4492 / 6000 = 74.9%
Epoch  4:     4629 / 6000 = 77.2%
Epoch  5:     4962 / 6000 = 82.7%
Epoch  6:     5176 / 6000 = 86.3%
Epoch  7:     5232 / 6000 = 87.2%
Epoch  8:     5276 / 6000 = 87.9%
Epoch  9:     5282 / 6000 = 88.0%
Epoch  10:     5329 / 6000 = 88.8%
Epoch  11:     5297 / 6000 = 88.3%
Epoch  12:     5334 / 6000 = 88.9%
Epoch  13:     5360 / 6000 = 89.3%
Epoch  14:     5365 / 6000 = 89.4%<jupyter_text>a deeper network with the same number of neurons is significantly better!<jupyter_code>net = Network([784, 50, 30, 20, 3]) # and maybe more neurons?
net.SGD(training_data_ising, 15, 30, 0.5, test_data=test_data_ising)<jupyter_output><empty_output><jupyter_text>Bonus: can we distinguish more temperatures?<jupyter_code>import generate_mc_data
temps = [3., 2.5, 2.0, 1.5]
data = generate_mc_data.generate_data(temps=temps, N_training=5000)
train_mc_moreT, _, test_mc_more_T = data_loader.load_data_wrapper(data)
net = Network([784, 50, 30, len(temps)]) # higher number of neurons & deep!
net.SGD(train_mc_moreT, 15, 10, 0.2, test_data=test_mc_more_T)
# plot a few of the images in the test data along with the output of the network
L = 28
img_label_data = test_mc_more_T[213:213+9] #some of the images
fig, axes = plt.subplots(3, 3, True, True, figsize=(10, 10))
for ((img, label), ax) in zip(img_label_data, axes.flatten()):
    ax.imshow(np.reshape(img, [L, L]))
    predict = np.argmax(net.feedforward(img)) # check what the network predicts
    ax.set_title("T={T:.1f}, predict={predict:.1f}".format(T=temps[label], predict=temps[predict]))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12571242468653313, 'fraction_numerical': 0.14704445530043966, 'mean_word_length': 3.122147651006711, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '161734', 'license': 'MIT', 'n_tokens_mistral': 2930, 'n_tokens_neox': 2306, 'n_words': 723, 'path': '4_machine_learning/sol11_neural_networks.ipynb', 'repo_name': 'jhauschild/lecture_comp_methods', 'size': '6141'}