<jupyter_start><jupyter_text>使用卷积神经网络的垃圾分类模型<jupyter_code>import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras
import os
import globalConfig.config as config
import pathlib<jupyter_output><empty_output><jupyter_text>查看 GPU 支持<jupyter_code>gpus = tf.config.list_physical_devices(device_type='GPU')
cpus = tf.config.list_physical_devices(device_type='CPU')
print(gpus, cpus)<jupyter_output>[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]<jupyter_text>准备原始图片 参数：<jupyter_code>dataDir = pathlib.Path(config.model2Classi.dataDir)<jupyter_output><empty_output><jupyter_text>分割数据集 图片统一缩放到 180 x 180 x colorChannel每个样本集取默认值 32 参数：<jupyter_code>imgHeight = config.model2Classi.imgHeight
imgWidth = config.model2Classi.imgWidth
batchSize = config.model2Classi.batchSize
trainDataset = tf.keras.utils.image_dataset_from_directory(
    dataDir,
    validation_split=0.2,
    subset="training",
    seed=123,
    image_size=(imgHeight, imgWidth),
    batch_size=batchSize
)
valDataset = tf.keras.utils.image_dataset_from_directory(
    dataDir,
    validation_split=0.2,
    subset="validation",
    seed=123,
    image_size=(imgHeight, imgWidth),
    batch_size=batchSize
)
classNames = trainDataset.class_names
f'labels {classNames}'<jupyter_output>Found 2513 files belonging to 2 classes.
Using 2011 files for training.
Found 2513 files belonging to 2 classes.
Using 502 files for validation.<jupyter_text>以第一个样本集为例<jupyter_code>plt.figure(figsize=(22, 12))
for images, labels in trainDataset.take(1):
  for i in range(1, 33):
    ax = plt.subplot(4, 8, i)
    plt.imshow(images[i - 1].numpy().astype("uint8"))
    plt.title(classNames[labels[i - 1]])<jupyter_output><empty_output><jupyter_text>将数据集缓存在内存<jupyter_code>trainDataset = trainDataset.cache().shuffle(1000).prefetch(buffer_size=tf.data.AUTOTUNE)
valDataset = valDataset.cache().prefetch(buffer_size=tf.data.AUTOTUNE)<jupyter_output><empty_output><jupyter_text>模型创建<jupyter_code>modelConfiguration = [ keras.layers.InputLayer(input_shape=(imgHeight, imgWidth, 3)) ]<jupyter_output><empty_output><jupyter_text>数据增强层这一层用来对数据进行随机的平移、旋转和缩放，增强模型的泛化能力，这是减少过拟合措施的一部分。<jupyter_code>modelConfiguration.extend([
    keras.layers.RandomFlip("horizontal"),
    keras.layers.RandomRotation(0.1),
    keras.layers.RandomZoom(0.1),
])<jupyter_output><empty_output><jupyter_text>输入规范化层将每个 colorChannel [0, 255] -> [0, 1]另外，输入的数据集已经在 `tf.keras.utils.image_dataset_from_directory` 做过预处理缩放，所以不需要再加入缩放层<jupyter_code>modelConfiguration.append(keras.layers.Rescaling(1. / 255))
# modelConfiguration.append(tf.keras.layers.Resizing(imgHeight, imgWidth))<jupyter_output><empty_output><jupyter_text>卷积、池化层包含三个卷积核大小为 3 的二维卷积块，在每个二维卷积块处理后紧邻一个最大池化层来提取合并特征。<jupyter_code>modelConfiguration.extend([
    keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
    keras.layers.MaxPooling2D(),
    keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
    keras.layers.MaxPooling2D(),
    keras.layers.Conv2D(64, 3, padding='same', activation='relu'),
    keras.layers.MaxPooling2D(),
])<jupyter_output><empty_output><jupyter_text>过渡层从最后一个最大池化层处理后，我们输出的三维特征在一维展平但在此之前，我们向网络中引入 dropout 正则化，让每次训练时随机忽略一部分神经元。换句话讲，这些神经元在正向传播时对下游的启动影响被忽略，反向传播时也不会更新权重。这可以让网络对某个神经元的权重变化更不敏感，增加泛化能力，这是减少过拟合措施的一部分。<jupyter_code>modelConfiguration.extend([
    keras.layers.Dropout(0.2),
    keras.layers.Flatten()
])<jupyter_output><empty_output><jupyter_text>全连接层然后经过一个 128 个单元的全连接网络层，进行对输出特征的分类，最终在一个包含 label 数量的全连接层输出结果向量，作为分类结果。<jupyter_code>modelConfiguration.extend([
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(len(classNames))
])<jupyter_output><empty_output><jupyter_text>编译模型<jupyter_code>model = keras.models.Sequential(modelConfiguration)
model.compile(
    optimizer='adam',
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 random_flip (RandomFlip)    (None, 180, 180, 3)       0         
                                                                 
 random_rotation (RandomRota  (None, 180, 180, 3)      0         
 tion)                                                           
                                                                 
 random_zoom (RandomZoom)    (None, 180, 180, 3)       0         
                                                                 
 rescaling (Rescaling)       (None, 180, 180, 3)       0         
                                                                 
 conv2d (Conv2D)             (None, 180, 180, 16)      448       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 90, 90, 16)       0[...]<jupyter_text>训练优化方法选择 Adam，它是一种基于一阶和二阶矩自适应估计的随机梯度下降方法。损失函数通过预测和标签之间的交叉熵损失提供梯度，由 `SparseCategoricalCrossentropy` 调用提供。 参数：<jupyter_code>epochs=config.model2Classi.epochs
history = model.fit(
    trainDataset,
    validation_data=valDataset,
    epochs=epochs
)<jupyter_output>Epoch 1/10
63/63 [==============================] - 16s 123ms/step - loss: 0.5556 - accuracy: 0.7926 - val_loss: 0.3387 - val_accuracy: 0.9104
Epoch 2/10
63/63 [==============================] - 7s 109ms/step - loss: 0.3027 - accuracy: 0.8817 - val_loss: 0.2555 - val_accuracy: 0.8884
Epoch 3/10
63/63 [==============================] - 8s 121ms/step - loss: 0.2310 - accuracy: 0.9100 - val_loss: 0.1827 - val_accuracy: 0.9263
Epoch 4/10
63/63 [==============================] - 8s 121ms/step - loss: 0.2234 - accuracy: 0.9070 - val_loss: 0.1652 - val_accuracy: 0.9422
Epoch 5/10
63/63 [==============================] - 7s 104ms/step - loss: 0.2186 - accuracy: 0.9175 - val_loss: 0.1602 - val_accuracy: 0.9303
Epoch 6/10
63/63 [==============================] - 8s 131ms/step - loss: 0.2048 - accuracy: 0.9184 - val_loss: 0.1334 - val_accuracy: 0.9442
Epoch 7/10
63/63 [==============================] - 7s 111ms/step - loss: 0.1834 - accuracy: 0.9299 - val_loss: 0.1239 - val_accuracy: 0.9442
Epoch[...]<jupyter_text>训练表现 查看在训练集和验证集上的精确度和损失表现<jupyter_code>acc = history.history['accuracy']
valAcc = history.history['val_accuracy']

loss = history.history['loss']
valLoss = history.history['val_loss']

epochsRange = range(epochs)

plt.figure(figsize=(8, 8))

plt.subplot(1, 2, 1)
plt.plot(epochsRange, acc, label='training')
plt.plot(epochsRange, valAcc, label='validation')
plt.legend(loc='lower right')
plt.title('accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochsRange, loss, label='training')
plt.plot(epochsRange, valLoss, label='validation')
plt.legend(loc='upper right')
plt.title('loss')
plt.show()<jupyter_output><empty_output><jupyter_text>可以看出，在我们使用了数据增强和 dropout 层后，验证集的准确率和损失很好地收敛在了训练集附近，模型的拟合效果不错。  持久化神经网络权值<jupyter_code>modelConfig = config.model2Classi
if not os.path.exists(modelConfig.savedDir):
    os.makedirs(modelConfig.savedDir)

model.save_weights(os.path.join(modelConfig.savedDir, modelConfig.savedName))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1562023562023562, 'fraction_numerical': 0.05252945252945253, 'mean_word_length': 4.381058911260253, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '587592', 'license': 'WTFPL', 'n_tokens_mistral': 3120, 'n_tokens_neox': 3003, 'n_words': 364, 'path': 'notebookModel2Classi.ipynb', 'repo_name': 'Drincann/cnn-trash-classification', 'size': '7215'}