<jupyter_start><jupyter_text>高级数据集管理`Ascend` `GPU` `CPU` `进阶` `数据准备`[![在线运行](https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tb2RlbGFydHMvcXVpY2tfc3RhcnQvbWluZHNwb3JlX2RhdGEuaXB5bmI=&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c)&emsp;[![下载Notebook](https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/mindspore_data.ipynb)&emsp;[![下载样例代码](https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/tutorials/zh_cn/mindspore_data.py)&emsp;[![查看源文件](https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/master/tutorials/source_zh_cn/intermediate/data.ipynb)MindSpore可以加载常见的数据集或自定义的数据集，这部分功能在初级教程中进行了部分介绍。加载自定义数据集有两种途径：- 通过`GeneratorDataset`对象加载，使用方法可参考[初级教程-自定义数据集](https://www.mindspore.cn/tutorials/zh-CN/master/dataset.html%E8%87%AA%E5%AE%9A%E4%B9%89%E6%95%B0%E6%8D%AE%E9%9B%86)。- 将数据集转换为MindRecord，即MindSpore数据格式，通过读取MindRecord文件进行加载数据。如果用户想要获得更好的性能体验，可以将数据集转换为MindRecord，从而方便地加载到MindSpore中进行训练。MindRecord的性能优化如下：- 实现多变的用户数据统一存储、访问，训练数据读取更加简便。- 数据聚合存储，高效读取，且方便管理、移动。- 高效的数据编解码操作，对用户透明、无感知。- 可以灵活控制分区的大小，实现分布式训练。常见数据集转换MindRecord可参考官方编程指南中的[MindSpore数据格式转换](https://www.mindspore.cn/docs/programming_guide/zh-CN/master/dataset_conversion.htmlMindSpore%E6%95%B0%E6%8D%AE%E6%A0%BC%E5%BC%8F%E8%BD%AC%E6%8D%A2)，自定义数据集转换可参考下文。MindRecord的目标是归一化用户的数据集，并进一步通过`MindDataset`实现数据的读取，用于训练过程。下面对这两步进行说明。 自定义数据集转换为MindRecord首先，下载需要处理的图片数据`transform.jpg`作为待处理的原始数据。创建文件夹目录`./datasets/convert_dataset_to_mindrecord/data_to_mindrecord/`用于存放所有的转换数据集。创建文件夹目录`./datasets/convert_dataset_to_mindrecord/images/`用于存放下载下来的图片数据。以下示例代码完成图片下载和文件夹的创建，并将图片移动到指定位置。<jupyter_code>import os
import requests
import tarfile
import zipfile

requests.packages.urllib3.disable_warnings()

def download_dataset(url, target_path):
    """下载并解压数据集"""
    if not os.path.exists(target_path):
        os.makedirs(target_path)
    download_file = url.split("/")[-1]
    if not os.path.exists(download_file):
        res = requests.get(url, stream=True, verify=False)
        if download_file.split(".")[-1] not in ["tgz", "zip", "tar", "gz"]:
            download_file = os.path.join(target_path, download_file)
        with open(download_file, "wb") as f:
            for chunk in res.iter_content(chunk_size=512):
                if chunk:
                    f.write(chunk)
    if download_file.endswith("zip"):
        z = zipfile.ZipFile(download_file, "r")
        z.extractall(path=target_path)
        z.close()
    if download_file.endswith(".tar.gz") or download_file.endswith(".tar") or download_file.endswith(".tgz"):
        t = tarfile.open(download_file)
        names = t.getnames()
        for name in names:
            t.extract(name, target_path)
        t.close()
    print("The {} file is downloaded and saved in the path {} after processing".format(os.path.basename(url), target_path))

download_dataset("https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/transform.jpg", "./datasets/convert_dataset_to_mindrecord/images/")
if not os.path.exists("./datasets/convert_dataset_to_mindrecord/data_to_mindrecord/"):
    os.makedirs("./datasets/convert_dataset_to_mindrecord/data_to_mindrecord/")<jupyter_output><empty_output><jupyter_text>下载的图片数据文件的目录结构如下：```text./datasets/convert_dataset_to_mindrecord/images/└── transform.jpg``` 导入文件写入工具类`FileWriter`。<jupyter_code>from mindspore.mindrecord import FileWriter<jupyter_output><empty_output><jupyter_text>创建FileWriter对象，传入文件名及分片数量，进行覆盖写。<jupyter_code>data_record_path = './datasets/convert_dataset_to_mindrecord/data_to_mindrecord/test.mindrecord'
writer = FileWriter(file_name=data_record_path, shard_num=4, overwrite=True)<jupyter_output><empty_output><jupyter_text>定义数据集结构文件Schema，调用`write_raw_data`接口写入数据，最后调用`commit`接口生成本地数据文件。Schema文件主要包含字段名`name`、字段数据类型`type`和字段各维度维数`shape`：- 字段名：字段的引用名称，可以包含字母、数字和下划线。- 字段数据类型：包含int32、int64、float32、float64、string、bytes。- 字段维数：一维数组用[-1]表示，更高维度可表示为[m, n, …]，其中m、n为各维度维数。> 如果字段有属性`shape`，则用户传入`write_raw_data`接口的数据必须为`numpy.ndarray`类型，对应数据类型必须为int32、int64、float32、float64。<jupyter_code># 定义schema
data_schema = {"file_name": {"type": "string"}, "label": {"type": "int32"}, "data": {"type": "bytes"}}
writer.add_schema(data_schema, "test_schema")

# 数据准备
file_name = "./datasets/convert_dataset_to_mindrecord/images/transform.jpg"
with open(file_name, "rb") as f:
    bytes_data = f.read()
data = [{"file_name": "transform.jpg", "label": 1, "data": bytes_data}]

indexes = ["file_name", "label"]
writer.add_index(indexes)

# 数据写入
writer.write_raw_data(data)

# 生成本地数据
writer.commit()<jupyter_output><empty_output><jupyter_text>该示例会生成8个文件，成为MindRecord数据集。`test.mindrecord0`和`test.mindrecord0.db`称为1个MindRecord文件，其中`test.mindrecord0`为数据文件，`test.mindrecord0.db`为索引文件，生成的文件为：```text./datasets/convert_dataset_to_mindrecord/data_to_mindrecord/├── test.mindrecord0├── test.mindrecord0.db├── test.mindrecord1├── test.mindrecord1.db├── test.mindrecord2├── test.mindrecord2.db├── test.mindrecord3└── test.mindrecord3.db0 directories, 8 files``` 读取MindRecord数据集导入读取类`mindspore.dataset`。<jupyter_code>import mindspore.dataset as ds<jupyter_output><empty_output><jupyter_text>首先使用`MindDataset`读取MindRecord数据集，然后对数据创建字典迭代器，并通过迭代器读取一条数据记录。<jupyter_code>file_name = './datasets/convert_dataset_to_mindrecord/data_to_mindrecord/test.mindrecord0'
# 创建MindDataset
define_data_set = ds.MindDataset(dataset_files=file_name)
# 创建字典迭代器并通过迭代器读取数据记录
count = 0
for item in define_data_set.create_dict_iterator(output_numpy=True):
    print("sample: {}".format(item))
    count += 1
print("Got {} samples".format(count))<jupyter_output>sample: {'data': array([255, 216, 255, ..., 159, 255, 217], dtype=uint8), 'file_name': array(b'transform.jpg', dtype='|S13'), 'label': array(1, dtype=int32)}
Got 1 samples
{'fraction_non_alphanumeric': 0.148250162022035, 'fraction_numerical': 0.024951393389500972, 'mean_word_length': 7.585535465924895, 'pattern_counts': {'":': 9, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '83934', 'license': 'Apache-2.0', 'n_tokens_mistral': 2931, 'n_tokens_neox': 2993, 'n_words': 205, 'path': 'tutorials/source_zh_cn/intermediate/data.ipynb', 'repo_name': 'mindspore-ai/docs', 'size': '6172'}