<jupyter_start><jupyter_text>Распознавание рукописных цифр MNIST<jupyter_code>from pathlib import Path

import numpy as np

from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import utils

from PIL import Image

from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Подготовка данных Загрузка данных MNIST<jupyter_code>(x_train_org, y_train_org), (x_test_org, y_test_org) =\
    mnist.load_data()
print(x_train_org.shape)<jupyter_output>(60000, 28, 28)<jupyter_text>Контрольное отображение цифры из набора<jupyter_code>n = 143
plt.imshow(Image.fromarray(x_train_org[n]).convert('RGB'))

plt.title(str(y_train_org[n]))
plt.axis('off')

plt.show();<jupyter_output><empty_output><jupyter_text>Отобразить примеры цифр из набора<jupyter_code>digits = np.unique(y_train_org)

_, axes = plt.subplots(2, 5, figsize=(12, 5))
axes = axes.flatten()

for digit, ax in zip(digits, axes):
    ax.imshow(
        Image.fromarray(x_train_org[y_train_org == digit][0]).convert('RGB')
    )

    ax.set_title(str(digit))
    ax.set_axis_off()

plt.show();<jupyter_output><empty_output><jupyter_text>Изменить формат входных изображений<jupyter_code>x_train = x_train_org.reshape(-1, 28*28).astype(np.float32) / 255
x_test = x_test_org.reshape(-1, 28*28).astype(np.float32) / 255<jupyter_output><empty_output><jupyter_text>Преобразовать ответы в унитарный код (one-hot)<jupyter_code>y_train = utils.to_categorical(y_train_org, 10)
y_test = utils.to_categorical(y_test_org, 10)
print(y_train.shape)<jupyter_output>(60000, 10)<jupyter_text>Создание нейронной сети<jupyter_code>model = Sequential()
model.add(
    Dense(
        800,
        input_dim=28*28,
        activation='relu'
    )
)
model.add(
    Dense(
        400,
        activation='relu'
    )
)
model.add(
    Dense(
        10,
        activation='softmax'
    )
)<jupyter_output><empty_output><jupyter_text>Компиляция нейронной сети<jupyter_code>model.compile(
    loss='categorical_crossentropy',
    optimizer=Adam(),
    metrics=['accuracy']
)

print(model.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 800)               628000    
_________________________________________________________________
dense_1 (Dense)              (None, 400)               320400    
_________________________________________________________________
dense_2 (Dense)              (None, 10)                4010      
=================================================================
Total params: 952,410
Trainable params: 952,410
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>Обучение нейронной сети<jupyter_code>epochs = 15
history = model.fit(
    x_train,
    y_train,
    batch_size=128,
    epochs=epochs,
    verbose=1
)<jupyter_output>Epoch 1/15
469/469 [==============================] - 6s 12ms/step - loss: 0.2104 - accuracy: 0.9382
Epoch 2/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0767 - accuracy: 0.9764
Epoch 3/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0474 - accuracy: 0.9849
Epoch 4/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0338 - accuracy: 0.9893
Epoch 5/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0265 - accuracy: 0.9912
Epoch 6/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0213 - accuracy: 0.9931
Epoch 7/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0181 - accuracy: 0.9941
Epoch 8/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0150 - accuracy: 0.9949
Epoch 9/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0141 - accuracy: 0.9953
Epoch 10/15
469/469 [==============================] - 6s 13ms/step - loss: 0.0126 - accura[...]<jupyter_text>Отобразить график изменения loss и accuracy в процессе обучения<jupyter_code>_, axes = plt.subplots(1, 2, figsize=(12, 4))

axes[0].plot(
    history.epoch,
    history.history['loss'],
    color='steelblue'
)
axes[0].set_ylabel('loss')
axes[0].set_xlabel('epoch')
axes[0].grid(True)

axes[1].plot(
    history.epoch,
    history.history['accuracy'],
    color='salmon'
)
axes[1].set_ylabel('accuracy')
axes[1].set_xlabel('epoch')
axes[1].grid(True)

plt.show();<jupyter_output><empty_output><jupyter_text>Получить loss и accuracy для тестовой выборки<jupyter_code>score = model.evaluate(x_test, y_test)
print(f"Test loss: {score[0]}\nTest accuracy: {score[1]}")<jupyter_output>313/313 [==============================] - 1s 3ms/step - loss: 0.0779 - accuracy: 0.9821
Test loss: 0.07786726206541061
Test accuracy: 0.9821000099182129<jupyter_text>Сохранить и загрузить веса нейронной сети<jupyter_code>path = Path.home() / 'aiu-01-mnist-ultra-light-model.h5'

model.save_weights(path)
model.load_weights(path)<jupyter_output><empty_output><jupyter_text>Распознавание рукописных цифр<jupyter_code>n_rec = 387

plt.imshow(Image.fromarray(x_test_org[n_rec]).convert('RGB'))

plt.title(str(y_test_org[n_rec]))
plt.axis('off')

plt.show();<jupyter_output><empty_output><jupyter_text>Выбрать изображение из тестовой выборки<jupyter_code>x = x_test[n_rec]
print(x.shape)<jupyter_output>(784,)<jupyter_text>Добавить дополнительную размерность<jupyter_code>x = np.expand_dims(x, axis=0)
print(x.shape)<jupyter_output>(1, 784)<jupyter_text>Распознать пример<jupyter_code>prediction = model.predict(x)
print(f"{prediction}, {sum(prediction[0])}")<jupyter_output>[[1.6661818e-10 6.3051289e-08 9.9999976e-01 1.5023822e-07 7.1624577e-14
  5.6112784e-12 6.4197106e-11 4.6665324e-08 3.5383817e-08 6.5899299e-13]], 1.0000000571572294<jupyter_text>Интерпретировать результат предсказания<jupyter_code>print(np.argmax(prediction))<jupyter_output>2<jupyter_text>Вывести правильное значение для сравнения<jupyter_code>print(y_test_org[n_rec])<jupyter_output>2
{'fraction_non_alphanumeric': 0.19285829178060157, 'fraction_numerical': 0.08428502493163906, 'mean_word_length': 5.490605427974947, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '650620', 'license': 'MIT', 'n_tokens_mistral': 2722, 'n_tokens_neox': 2388, 'n_words': 363, 'path': 'notebooks/01-mnist/ultra-light.ipynb', 'repo_name': 'new-divos/aiu-tasks', 'size': '6217'}