<jupyter_start><jupyter_text>MindInsight的模型溯源和数据溯源体验  概述在调参的场景下，需要多次调整模型超参并进行多次训练，在这个过程，往往需要手动记录每次训练使用参数以及训练结果。为此，MindSpore提供了自动记录模型参数，训练信息，以及训练结果评估指标的功能，并通过MindInsight进行可视化展示。本次体验会从MindInsight的数据记录，可视化效果，如何方便用户在模型调优，数据调优上做一次整体流程的体验。下面按照MindSpore的训练数据模型的正常步骤进行，当使用`SummaryCollector`进行数据保存操作时，会增加相应的说明，本次体验的整体流程如下：1. 数据集的准备，这里使用的是MNIST数据集。2. 构建一个网络，这里使用LeNet网络。3. 搭建及运行训练网络和测试网络。4. 启动MindInsight服务。5. 模型溯源的使用。调整模型参数多次训练并存储数据，并使用MindInsight的模型溯源功能对不同优化参数下训练产生的模型作对比，了解MindSpore中的各类优化对训练过程的影响及如何调优训练过程。6. 数据溯源的使用。调整数据参数多次训练并存储数据，并使用MindInsight的数据溯源功能对不同数据集下训练产生的模型进行对比分析，了解如何调优。本次体验将使用快速入门案例作为基础用例，将MindInsight的模型溯源和数据溯源的数据记录功能加入到案例中，快速入门案例的源码请参考：。  训练的数据集下载  准备 方法一：从以下网址下载，并将数据包解压缩后放至Jupyter的工作目录下：训练数据集：{"", ""}测试数据集：{"", ""}我们用下面代码查询jupyter的工作目录。<jupyter_code>import os
os.getcwd()<jupyter_output><empty_output><jupyter_text>训练数据集放在----`Jupyter工作目录+\MNIST_Data\train\`，此时`train`文件夹内应该包含两个文件，`train-images-idx3-ubyte`和`train-labels-idx1-ubyte` 测试数据集放在----`Jupyter工作目录+\MNIST_Data\test\`，此时`test`文件夹内应该包含两个文件，`t10k-images-idx3-ubyte`和`t10k-labels-idx1-ubyte`  方法二：直接执行下面代码，会自动进行训练集的下载与解压，但是整个过程根据网络好坏情况会需要花费几分钟时间。<jupyter_code>import urllib.request   
from urllib.parse import urlparse
import gzip 

def unzip_file(gzip_path):
    """unzip dataset file
    Args:
        gzip_path (str): Dataset file path
    """
    open_file = open(gzip_path.replace('.gz',''), 'wb')
    gz_file = gzip.GzipFile(gzip_path)
    open_file.write(gz_file.read())
    gz_file.close()
    
def download_dataset():
    """Download the dataset from http://yann.lecun.com/exdb/mnist/."""
    print("******Downloading the MNIST dataset******")
    train_path = "./MNIST_Data/train/" 
    test_path = "./MNIST_Data/test/"
    train_path_check = os.path.exists(train_path)
    test_path_check = os.path.exists(test_path)
    if not train_path_check and not test_path_check:
        os.makedirs(train_path)
        os.makedirs(test_path)
    train_url = {"http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz", "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz"}
    test_url = {"http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz", "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz"}
    
    for url in train_url:
        url_parse = urlparse(url)
        # split the file name from url
        file_name = os.path.join(train_path,url_parse.path.split('/')[-1])
        if not os.path.exists(file_name.replace('.gz', '')):
            file = urllib.request.urlretrieve(url, file_name)
            unzip_file(file_name)
            os.remove(file_name)
            
    for url in test_url:
        url_parse = urlparse(url)
        # split the file name from url
        file_name = os.path.join(test_path,url_parse.path.split('/')[-1])
        if not os.path.exists(file_name.replace('.gz', '')):
            file = urllib.request.urlretrieve(url, file_name)
            unzip_file(file_name)
            os.remove(file_name)

download_dataset()<jupyter_output><empty_output><jupyter_text>这样就完成了数据集的下载解压缩工作。  数据集处理 数据集处理对于训练非常重要，好的数据集可以有效提高训练精度和效率。在加载数据集前，我们通常会对数据集进行一些处理。我们定义一个函数`create_dataset`来创建数据集。在这个函数中，我们定义好需要进行的数据增强和处理操作：1. 定义数据集。2. 定义进行数据增强和处理所需要的一些参数。3. 根据参数，生成对应的数据增强操作。4. 使用`map`映射函数，将数据操作应用到数据集。5. 对生成的数据集进行处理。具体的数据集操作可以在MindInsight的数据溯源中进行可视化分析。<jupyter_code>import mindspore.dataset.vision.c_transforms as CV
import mindspore.dataset.transforms.c_transforms as C
from mindspore.dataset.vision import Inter
from mindspore.common import dtype as mstype
import mindspore.dataset as ds

def create_dataset(data_path, batch_size=32, repeat_size=1,
                   num_parallel_workers=1):
    """ create dataset for train or test
    Args:
        data_path (str): Data path
        batch_size (int): The number of data records in each group
        repeat_size (int): The number of replicated data records
        num_parallel_workers (int): The number of parallel workers
    """
    # define dataset
    mnist_ds = ds.MnistDataset(data_path)

    # define some parameters needed for data enhancement and rough justification
    resize_height, resize_width = 32, 32
    rescale = 1.0 / 255.0
    shift = 0.0
    rescale_nml = 1 / 0.3081
    shift_nml = -1 * 0.1307 / 0.3081

    # according to the parameters, generate the corresponding data enhancement method
    resize_op = CV.Resize((resize_height, resize_width), interpolation=Inter.LINEAR)
    # if you need to use SummaryCollector to extract image data, do not use the following normalize operator operation
    rescale_nml_op = CV.Rescale(rescale_nml, shift_nml)
    
    rescale_op = CV.Rescale(rescale, shift)
    hwc2chw_op = CV.HWC2CHW()
    type_cast_op = C.TypeCast(mstype.int32)

    # using map method to apply operations to a dataset
    mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns="label", num_parallel_workers=num_parallel_workers)
    mnist_ds = mnist_ds.map(operations=resize_op, input_columns="image", num_parallel_workers=num_parallel_workers)
    mnist_ds = mnist_ds.map(operations=rescale_op, input_columns="image", num_parallel_workers=num_parallel_workers)
    mnist_ds = mnist_ds.map(operations=rescale_nml_op, input_columns="image", num_parallel_workers=num_parallel_workers)
    mnist_ds = mnist_ds.map(operations=hwc2chw_op, input_columns="image", num_parallel_workers=num_parallel_workers)
    
    # process the generated dataset
    buffer_size = 10000
    mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)  # 10000 as in LeNet train script
    mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)
    mnist_ds = mnist_ds.repeat(repeat_size)

    return mnist_ds<jupyter_output><empty_output><jupyter_text>定义LeNet5网络<jupyter_code>from mindspore.ops import operations as P
import mindspore.nn as nn
from mindspore.common.initializer import TruncatedNormal

def conv(in_channels, out_channels, kernel_size, stride=1, padding=0):
    """Conv layer weight initial."""
    weight = weight_variable()
    return nn.Conv2d(in_channels, out_channels,
                     kernel_size=kernel_size, stride=stride, padding=padding,
                     weight_init=weight, has_bias=False, pad_mode="valid")

def fc_with_initialize(input_channels, out_channels):
    """Fc layer weight initial."""
    weight = weight_variable()
    bias = weight_variable()
    return nn.Dense(input_channels, out_channels, weight, bias)

def weight_variable():
    """Weight initial."""
    return TruncatedNormal(0.02)

class LeNet5(nn.Cell):
    """Lenet network structure."""
    def __init__(self):
        super(LeNet5, self).__init__()
        self.batch_size = 32 
        self.conv1 = conv(1, 6, 5)
        self.conv2 = conv(6, 16, 5)
        self.fc1 = fc_with_initialize(16 * 5 * 5, 120)
        self.fc2 = fc_with_initialize(120, 84)
        self.fc3 = fc_with_initialize(84, 10)
        self.relu = nn.ReLU()
        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
        self.flatten = nn.Flatten()

    def construct(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.max_pool2d(x)
        x = self.conv2(x)
        x = self.relu(x)
        x = self.max_pool2d(x)
        x = self.flatten(x)
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.relu(x)
        x = self.fc3(x)
        return x<jupyter_output><empty_output><jupyter_text>记录数据及启动训练 MindSpore 提供 `SummaryCollector` 进行记录训练过程中的信息。通过 `SummaryCollector` 的 `collect_specified_data` 参数，可以自定义记录指定数据。在本次体验中，我们将记录训练数据与数据集预处理的操作，我们将 `collect_specified_data` 中的 `collect_train_lineage`, `collect_eval_lineage`, `collect_dataset_graph` 设置成 `True`。SummaryCollector的更多用法，请参考[API文档](https://www.mindspore.cn/api/zh-CN/master/api/python/mindspore/mindspore.train.html?highlight=collectormindspore.train.callback.SummaryCollector)。<jupyter_code>from mindspore.train.callback import SummaryCollector
from mindspore.nn.metrics import Accuracy
from mindspore import context
from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits

if __name__=="__main__":
    context.set_context(mode=context.GRAPH_MODE, device_target = "GPU")
    lr = 0.01
    momentum = 0.9 
    epoch_size = 10
    mnist_path = "./MNIST_Data"
    
    net_loss = SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')
    repeat_size = 1
    # create the network
    network = LeNet5()

    # define the optimizer
    net_opt = nn.Momentum(network.trainable_params(), lr, momentum)
    config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10)
    ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet", config=config_ck)
    model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()})
    
    collect_specified_data = {"collect_eval_lineage": True, "collect_train_lineage": True， "collect_dataset_graph": True}
    summary_collector = SummaryCollector(summary_dir="./summary_base/quick_start_summary01", collect_specified_data=collect_specified_data, keep_default_action=False) 

    # Start to train
    ds_train = create_dataset(os.path.join(mnist_path, "train"), 32, repeat_size)
    model.train(epoch_size, ds_train, callbacks=[ckpoint_cb, summary_collector], dataset_sink_mode=True)

    print("============== Starting Testing ==============")
    # load the saved model for evaluation
    param_dict = load_checkpoint("checkpoint_lenet-10_1875.ckpt")
    # load parameter to the network
    load_param_into_net(network, param_dict)
    # load testing dataset
    ds_eval = create_dataset(os.path.join(mnist_path, "test"))
    acc = model.eval(ds_eval, callbacks=[summary_collector], dataset_sink_mode=True)
    print("============== Accuracy:{} ==============".format(acc))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11401695265660533, 'fraction_numerical': 0.01571221831713872, 'mean_word_length': 4.438448566610456, 'pattern_counts': {'":': 5, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '304987', 'license': 'Apache-2.0', 'n_tokens_mistral': 4084, 'n_tokens_neox': 4141, 'n_words': 560, 'path': 'tutorials/notebook/mindinsight/mindinsight_model_lineage_and_data_lineage.ipynb', 'repo_name': 'akineeic/docs', 'size': '9674'}