<jupyter_start><jupyter_text>实战 Kaggle 比赛：CIFAR-10 分类（Keras 版）本章将使用 Keras 动手实战 CIFAR-10 图像分类，该任务是一个多分类任务。该任务的网页地址是 https://www.kaggle.com/c/cifar-10 。思考：将 CIFAR-10 封装为 HDF5（可参考：https://www.cnblogs.com/q735613050/p/9244223.html）> 本章快报： 8.1 数据处理从 Data Description（https://www.kaggle.com/c/cifar-10/data ）可获知：人们熟知的 CIFAR-10 数据包含 $60\,000$ 张 $32\times 32$ 的彩色图片。该数据总共有 $10$ 个类别，每个类别下均有 $6000$ 张图片。其中，训练集和测试集占比为 $5:1$。Kaggle 比赛“CIFAR-10 - Object Recognition in Images”则提供了更多的数据。- train.7z：和官方提供的一致- test.7z：增加了 $290,000$ 张图片- trainLabels.csv：训练集的标签由于 Keras 的局限性（自定义数据迭代器很不方便），需要将数据集进行解压。先载入一些必备包：<jupyter_code>import zipfile  # 处理压缩文件
import os

import pandas as pd  # 处理 csv 文件
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>解压已经下载好的数据：<jupyter_code>base_dir = r'E:\Data\Kaggle'
#from utils.kaggle.helper import unzip
#dataDir = unzip(base_dir, dataDir)  # 解压 all.zip 
dataDir = os.path.join(base_dir, 'Cifar10')<jupyter_output><empty_output><jupyter_text>解压后的数据存在 `.7z` 文件，故而需要下载 7-Zip（https://sparanoid.com/lab/7z/ ）来继续解压数据。解压好之后，继续对数据继续处理：<jupyter_code>class CSVCat:
    def __init__(self, root, name):
        self.csv2dict(root, name)  # name 是 .csv 文件

    @staticmethod
    def read_csv(root, name):
        return pd.read_csv(os.path.join(root,
                                        name))  # 从本地读取标签信息，格式 (id, label)

    def csv2dict(self, root, name):
        rec = CSVCat.read_csv(root, name).to_records()  # 将 CSV 转换为 Records
        self.cat_dict = {}  # 格式为 {'cat':[id1, id2, ...], ...}
        for _, p, class_name in rec:
            self.cat_dict[class_name] = self.cat_dict.get(class_name,
                                                          []) + [p]  # 列表加法
        self.class_names = tuple(self.cat_dict.keys())  # 获取类别名称列表

    def split(self, test_size=.3):
        import random
        train_dict = {}
        val_dict = {}
        test_size = .3
        for class_name, id_list in self.cat_dict.items():
            random.shuffle(id_list)
            n = len(id_list)
            test_num = int(n * test_size)
            val_dict[class_name], train_dict[
                class_name] = id_list[:test_num], id_list[test_num:]
        return train_dict, val_dict<jupyter_output><empty_output><jupyter_text>CSVCat 被封装进了 zipimage 模块中。下面将利用该类来划分数据集并获取类别名称列表。<jupyter_code>cat = CSVCat(dataDir, 'trainLabels.csv') # 实例化
class_names = cat.class_names  # 获取类别名称列表
train_dict, val_dict = cat.split()  # 划分数据集<jupyter_output><empty_output><jupyter_text>为了代码的简洁，下面先定义两个辅助函数：<jupyter_code>def mkdir_if_not_exist(root, dir_name):
    # 在 root 下生成目录
    _dir = os.path.join(root, dir_name) # 拼出分完整目录名
    if not os.path.exists(_dir):  # 是否存在目录，如果没有创建
        os.makedirs(_dir)
    return _dir

def copyfile(original_dir, obj_dir, fnames):
    import shutil
    # 将 original_dir 目录下的 fnames 复制到 obj_dir
    for fname in fnames:
        src = os.path.join(original_dir, fname)
        dst = os.path.join(obj_dir, fname)
        shutil.copyfile(src, dst)   # 复制<jupyter_output><empty_output><jupyter_text>`mkdir_if_not_exist` 函数用来创建目录，而 `copyfile` 则从源数据中划分出训练集和验证集。<jupyter_code>baseDir = mkdir_if_not_exist(dataDir, 'data')  # 根目录
oriDir = os.path.join(dataDir, 'train')  # 源目录

'''
for label, id_list in train_dict.items():
    trainDir = mkdir_if_not_exist(baseDir, f'train/{label}')
    fnames = [f'{p}.png' for p in id_list]
    copyfile(oriDir, trainDir, fnames)  # 从源数据复制训练数据
    
for label, id_list in val_dict.items():
    valDir = mkdir_if_not_exist(baseDir, f'val/{label}')
    fnames = [f'{p}.png' for p in id_list] 
    copyfile(oriDir, valDir, fnames)  # 从源数据复制验证数据
'''<jupyter_output><empty_output><jupyter_text>下面可以利用 Keras 的 `ImageDataGenerator` 来直接读取数据，并加入一些数据增强的处理：<jupyter_code>from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
      rescale=1./255,  # 将数据的数值归一化到 [0, 1]
      rotation_range=40,  # 图像随机旋转的角度
      width_shift_range=0.2,  # 图像在水平方向平移的范围
      height_shift_range=0.2,  # 图像在垂直方向平移的范围
      shear_range=0.2,        # 随机错切变换的角度
      zoom_range=0.2,       # 随机缩放的范围
      horizontal_flip=True,  # 随机水平翻转
      fill_mode='nearest')   # 填充新创建像素

test_datagen = ImageDataGenerator(rescale=1./255) # 注意，不能增强验证数据

train_dir = os.path.join(baseDir, 'train')   # 训练数据所在路径
validation_dir = os.path.join(baseDir, 'val')  # 验证数据所在路径
train_generator = train_datagen.flow_from_directory(
    train_dir, batch_size=20, target_size=(32, 32), classes=class_names) # 将所有图像的大小调整为 32 x 32
validation_generator = test_datagen.flow_from_directory(
    validation_dir, target_size=(32, 32), batch_size=20, classes=class_names)<jupyter_output>C:\ProgramData\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>8.2 使用预训练的基网络来训练模型使用预训练基网络的方法有：- 特征提取（feature extraction）- 微调模型（fine-tuning）<jupyter_code>from keras.applications.vgg16 import VGG16
conv_base = VGG16(include_top=False, weights='imagenet', classes=10) # 卷积基<jupyter_output><empty_output><jupyter_text>参数：- `include_top`: 是否包括顶层的全连接层- `weights`：指定模型初始化的权重检查点- `input_shape`：网络的输入张量的 shape- `classes`：类别个数 我们可以看看**卷积基**（在诸如 imagenet 上训练过的网络的除去其顶层全连接部分）的详细结构：<jupyter_code>conv_base.summary()
for inputs, labels in train_generator:
    train_features = conv_base.predict(inputs)
    break
def extract_feaures(data_generator):
    features = []
    labels = []
    for input_batch, label_batch in data_generator:
        features.append(conv_base.predict(input_batch))
        labels.append(label_batch)
    return features, labels
train_features, train_labels = extract_feaures(train_generator)
validation_features, validation_labels = extract_feaures(validation_generator)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1170177198261451, 'fraction_numerical': 0.02022734871280508, 'mean_word_length': 4.166666666666667, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '149724', 'license': 'MIT', 'n_tokens_mistral': 2728, 'n_tokens_neox': 2865, 'n_words': 398, 'path': 'Notebook/CIFAR-10_keras.ipynb', 'repo_name': 'XinetAI/CVX', 'size': '5982'}