<jupyter_start><jupyter_text>Распознавание марок машин *Добейтесь точности не менее 85% на проверочной выборке на базе трех иномарок.*Размер проверочной выборки - 20%.<jupyter_code>import logging
from pathlib import Path
from typing import Optional

import numpy as np

import tensorflow as tf
from tensorflow.keras import utils
from tensorflow.keras.callbacks import History
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (
    BatchNormalization,
    Conv2D,
    Dense,  
    Dropout, 
    Flatten,
    MaxPooling2D
)
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing import image
from tensorflow.python.keras.preprocessing.image import ImageDataGenerator

from PIL import Image 

from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Установить уровень диагностических сообщений для TensorFlow.<jupyter_code>os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
logging.getLogger('tensorflow').setLevel(logging.FATAL)<jupyter_output><empty_output><jupyter_text>Отобразить список доступных устройств.<jupyter_code>tf.config.experimental.list_physical_devices('GPU')<jupyter_output><empty_output><jupyter_text>Задать количество эпох для обучения.<jupyter_code>EPOCHS = 45<jupyter_output><empty_output><jupyter_text>Задать размер пакета.<jupyter_code>BATCH_SIZE = 25<jupyter_output><empty_output><jupyter_text>Определить размеры изображения.<jupyter_code>IMG_WIDTH = 96  # Ширина изображения
IMG_HEIGHT = 54  # Высота изображения<jupyter_output><empty_output><jupyter_text>Определить вспомогательную функцию для визуализации процесса обучения.<jupyter_code>def plot_history(
    history: History, 
    title: str, 
    axes: Optional[np.ndarray] = None
) -> None:
    if axes is None or not axes.all():
        _, axes = plt.subplots(1, 2, figsize=(12, 5))
        show_plot = True
    else:
        show_plot = False

    axes[0].plot(
        history.epoch,
        history.history['loss'],
        color='steelblue',
        label='train set'
    )
    axes[0].plot(
        history.epoch,
        history.history['val_loss'],
        color='darkblue',
        label='validation set'
    )
    axes[0].set_title(f"{title} (изменение loss)")
    axes[0].set_ylabel('loss')
    axes[0].set_xlabel('epoch')
    axes[0].set_xticks(range(0, len(history.epoch) + 1, 5))
    axes[0].grid(True)
    axes[0].legend()

    axes[1].plot(
        history.epoch,
        history.history['accuracy'],
        color='salmon',
        label='train set'
    )
    axes[1].plot(
        history.epoch,
        history.history['val_accuracy'],
        color='darkred',
        label='validation set'
    )
    axes[1].set_title(f"{title} (изменение accuracy)")
    axes[1].set_ylabel('accuracy')
    axes[1].set_xlabel('epoch')
    axes[1].set_xticks(range(0, len(history.epoch) + 1, 5))
    axes[1].grid(True)
    axes[1].legend()

    if show_plot:
        plt.show()<jupyter_output><empty_output><jupyter_text>Задать путь для сохранения моделей нейронных сетей.<jupyter_code>out_path = Path.home() / 'aiu-03-cars-ultra-pro-models'
if not out_path.exists():
    Path.mkdir(out_path, parents=True)<jupyter_output><empty_output><jupyter_text>Подготовка данных Определить путь к набору данных.<jupyter_code>root_path = Path.cwd().parent.parent / 'data' / 'images' / '03-cars'<jupyter_output><empty_output><jupyter_text>Создать генератор изображений:<jupyter_code>datagen = ImageDataGenerator(
    rescale=1. / 255, 
    rotation_range=10, 
    width_shift_range=0.1, 
    height_shift_range=0.1, 
    zoom_range=0.1, 
    horizontal_flip=True, 
    fill_mode='nearest', 
    validation_split=0.2
)<jupyter_output><empty_output><jupyter_text>Определить генераторы для обучающей и проверочной выборок.<jupyter_code># Обучающая выборка
train_generator = datagen.flow_from_directory(
    root_path / 'train',
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=BATCH_SIZE, 
    class_mode='categorical',
    shuffle=True, 
    subset='training'
)

# Проверочная выборка
validation_generator = datagen.flow_from_directory(
    root_path / 'train', 
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=BATCH_SIZE,
    class_mode='categorical',  
    shuffle=True, 
    subset='validation' 
)<jupyter_output>Found 2196 images belonging to 3 classes.
Found 547 images belonging to 3 classes.<jupyter_text>Определить генератор для тестовой выборки.<jupyter_code># Тестовая выборка
test_generator = datagen.flow_from_directory(
    root_path / 'val', 
    target_size=(IMG_WIDTH, IMG_HEIGHT),
    batch_size=BATCH_SIZE,
    class_mode='categorical',  
    shuffle=True, 
    subset='validation' 
)<jupyter_output>Found 136 images belonging to 3 classes.<jupyter_text>Вывести для примера изображения по каждому классу.<jupyter_code>_, axes = plt.subplots(1, 3, figsize=(25, 5))

classes = set()
plot_number = 0
for i in (root_path / 'train').glob('**/*'):
    img_path = i.absolute()
    if not img_path.is_file():
        continue

    class_name = img_path.parent.name

    if class_name not in classes:
        classes.add(class_name)

        axes[plot_number].imshow(
            image.load_img(img_path, target_size=(IMG_HEIGHT, IMG_WIDTH))
        )
        axes[plot_number].set_title(class_name)
        axes[plot_number].set_axis_off()

        plot_number += 1
    else:
        continue

plt.show();<jupyter_output><empty_output><jupyter_text>Создание и обучение эталонной модели сверточной нейронной сети Создать и скомпилировать эталонную модель.<jupyter_code>reference_net = Sequential()

reference_net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu', 
        input_shape=(IMG_WIDTH, IMG_HEIGHT, 3)
    )
)
reference_net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
reference_net.add(MaxPooling2D(pool_size=(3, 3)))

reference_net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
reference_net.add(Dropout(0.2))

reference_net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
reference_net.add(MaxPooling2D(pool_size=(3, 3)))
reference_net.add(Dropout(0.2))

reference_net.add(
    Conv2D(
        512, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)

reference_net.add(
    Conv2D(
        1024, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
reference_net.add(MaxPooling2D(pool_size=(3, 3)))
reference_net.add(Dropout(0.2))
 
reference_net.add(Flatten())
reference_net.add(Dense(2048, activation='relu'))
reference_net.add(Dense(4096, activation='relu'))
reference_net.add(
    Dense(
        len(train_generator.class_indices), 
        activation='softmax'
    )
)

reference_net.compile(
    loss='categorical_crossentropy', 
    optimizer=Adam(lr=0.0001), 
    metrics=['accuracy']
)
print(reference_net.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 96, 54, 256)       7168      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 96, 54, 256)       590080    
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 32, 18, 256)       0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 18, 256)       590080    
_________________________________________________________________
dropout (Dropout)            (None, 32, 18, 256)       0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 32, 18, 256)       590080    
________________________________________________________[...]<jupyter_text>Обучить нейронную сеть.<jupyter_code>history = reference_net.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // BATCH_SIZE,
    validation_data=validation_generator, 
    validation_steps=validation_generator.samples // BATCH_SIZE,
    epochs=EPOCHS,
    verbose=1
)<jupyter_output>Epoch 1/45
87/87 [==============================] - 7s 84ms/step - loss: 1.0229 - accuracy: 0.4592 - val_loss: 0.9051 - val_accuracy: 0.5486
Epoch 2/45
87/87 [==============================] - 7s 76ms/step - loss: 0.9184 - accuracy: 0.5398 - val_loss: 0.8981 - val_accuracy: 0.5543
Epoch 3/45
87/87 [==============================] - 7s 76ms/step - loss: 0.8928 - accuracy: 0.5596 - val_loss: 0.8908 - val_accuracy: 0.5714
Epoch 4/45
87/87 [==============================] - 7s 75ms/step - loss: 0.8680 - accuracy: 0.5753 - val_loss: 0.8436 - val_accuracy: 0.6171
Epoch 5/45
87/87 [==============================] - 7s 75ms/step - loss: 0.8376 - accuracy: 0.5910 - val_loss: 0.8452 - val_accuracy: 0.6400
Epoch 6/45
87/87 [==============================] - 7s 76ms/step - loss: 0.8066 - accuracy: 0.6099 - val_loss: 0.8012 - val_accuracy: 0.6229
Epoch 7/45
87/87 [==============================] - 7s 77ms/step - loss: 0.7968 - accuracy: 0.6269 - val_loss: 0.7454 - val_accuracy: 0.6762
Epoch 8/45
87[...]<jupyter_text>Отобразить прогресс обучения нейронной сети.<jupyter_code>plot_history(history, "Эталонная модель")<jupyter_output><empty_output><jupyter_text>Сохранить архитектуру нейронной сети.<jupyter_code>reference_net.save(out_path / 'aiu-03-cars-ultra-pro-model-reference')<jupyter_output><empty_output><jupyter_text>Получить значения loss и accuracy для тестовой выборки.<jupyter_code>score = reference_net.evaluate(test_generator, verbose=0)

print(f"Test loss: {score[0]:.5f}\nTest accuracy: {score[1]:.5f}")<jupyter_output>Test loss: 0.52744
Test accuracy: 0.82353<jupyter_text>Создание и обучение сверточной сети с требуемыми характеристиками Создать и скомпилировать модель.<jupyter_code>net = Sequential()

net.add(BatchNormalization(input_shape=(IMG_WIDTH, IMG_HEIGHT, 3)))
net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu', 
        input_shape=(IMG_WIDTH, IMG_HEIGHT, 3)
    )
)
net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
net.add(MaxPooling2D(pool_size=(3, 3)))

net.add(BatchNormalization())
net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
net.add(Dropout(0.2))

net.add(
    Conv2D(
        256, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
net.add(MaxPooling2D(pool_size=(3, 3)))
net.add(Dropout(0.2))

net.add(
    Conv2D(
        512, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)

net.add(
    Conv2D(
        2048, 
        (3, 3), 
        padding='same', 
        activation='relu'
    )
)
net.add(MaxPooling2D(pool_size=(3, 3)))
net.add(Dropout(0.2))
 
net.add(Flatten())
net.add(Dense(2048, activation='relu'))
net.add(Dropout(0.2))
net.add(Dense(4096, activation='relu'))
net.add(Dropout(0.2))
net.add(
    Dense(
        len(train_generator.class_indices), 
        activation='softmax'
    )
)

net.compile(
    loss='categorical_crossentropy', 
    optimizer=Adam(lr=0.0001), 
    metrics=['accuracy']
)
print(net.summary())<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization (BatchNo (None, 96, 54, 3)         12        
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 96, 54, 256)       7168      
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 96, 54, 256)       590080    
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 32, 18, 256)       0         
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 18, 256)       1024      
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 32, 18, 256)       590080    
______________________________________________________[...]<jupyter_text>Обучить нейронную сеть.<jupyter_code>history = net.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // BATCH_SIZE,
    validation_data=validation_generator, 
    validation_steps=validation_generator.samples // BATCH_SIZE,
    epochs=EPOCHS,
    verbose=1
)<jupyter_output>Epoch 1/45
87/87 [==============================] - 8s 93ms/step - loss: 1.0010 - accuracy: 0.4758 - val_loss: 1.0870 - val_accuracy: 0.5448
Epoch 2/45
87/87 [==============================] - 8s 90ms/step - loss: 0.8800 - accuracy: 0.5675 - val_loss: 1.0657 - val_accuracy: 0.5943
Epoch 3/45
87/87 [==============================] - 8s 88ms/step - loss: 0.8252 - accuracy: 0.6085 - val_loss: 0.9766 - val_accuracy: 0.6533
Epoch 4/45
87/87 [==============================] - 8s 89ms/step - loss: 0.7778 - accuracy: 0.6421 - val_loss: 0.9479 - val_accuracy: 0.5771
Epoch 5/45
87/87 [==============================] - 8s 90ms/step - loss: 0.7604 - accuracy: 0.6490 - val_loss: 0.8474 - val_accuracy: 0.6171
Epoch 6/45
87/87 [==============================] - 8s 91ms/step - loss: 0.7047 - accuracy: 0.6771 - val_loss: 0.8160 - val_accuracy: 0.6076
Epoch 7/45
87/87 [==============================] - 8s 92ms/step - loss: 0.6662 - accuracy: 0.7089 - val_loss: 0.8188 - val_accuracy: 0.6095
Epoch 8/45
87[...]<jupyter_text>Отобразить прогресс обучения нейронной сети.<jupyter_code>plot_history(history, "Требуемая модель")<jupyter_output><empty_output><jupyter_text>Сохранить архитектуру нейронной сети.<jupyter_code>net.save(out_path / 'aiu-03-cars-ultra-pro-model')<jupyter_output><empty_output><jupyter_text>Получить значения loss и accuracy для тестовой выборки.<jupyter_code>score = net.evaluate(test_generator, verbose=0)

print(f"Test loss: {score[0]:.5f}\nTest accuracy: {score[1]:.5f}")<jupyter_output>Test loss: 0.91873
Test accuracy: 0.77941
{'fraction_non_alphanumeric': 0.15239092495637, 'fraction_numerical': 0.05884816753926701, 'mean_word_length': 3.8251936679016505, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '556037', 'license': 'MIT', 'n_tokens_mistral': 5610, 'n_tokens_neox': 5090, 'n_words': 869, 'path': 'notebooks/03-cars/ultra-pro.ipynb', 'repo_name': 'new-divos/aiu-tasks', 'size': '14325'}