<jupyter_start><jupyter_text>数据集预处理 使用的数据集是DeepFashion中的`in-shop_clothes_retrieval_benchmark`数据集，在预处理前，需要制作数据集软链接到`../data/input`目录1. DeeFashion子数据集 in-shop 目录结构如下：```shell.├── Anno│   ├── list_attr_cloth.txt             衣服所有的属性列表│   ├── list_attr_items.txt             每个商品id对应的属性列表│   ├── list_bbox_inshop.txt            每张图片对应的bbox│   ├── list_description_inshop.json    每个商品id详细信息文字描述│   ├── list_item_inshop.txt            所有的商品id列表│   └── list_landmarks_inshop.txt       每张图片对应的关键点位置├── Eval│   └── list_eval_partition.txt         每张图片的train、val、test标签├── Img│   └── img│       ├── MEN│       │   ├── Denim│       │   │    ├── id_0000xxx│       │   │    ├── ...│       │   │    └── id_0000xxx│       │   ├── ...│       │   └── Tees_Tanks│       └── WOMEN│           ├── Blouses_Shirts│           ├── ...│           └── Tees_Tanks├── README.txt├── test.csv                            预处理后的test数据集├── train.csv                           预处理后的train数据集└── val.csv                             预处理后的val数据集```2. `../data/output/`文件结构如下：```shell.├── cache                               缓存文件夹├── logs                                日志文件夹│   └── fashionnet                      使用的网络名称│       ├── events.out.tfevents.1554190389.mysd-desktop   中间结果tensorboard文件│       └── fashionnet.csv              中间结果csv文件├── models                              模型文件│   ├── fashionnet                      使用的网络名称，指定网络保存在该文件夹下│   └── resnet50└── submits                             预测结果保存的文件夹```  dependency<jupyter_code>import re
from pathlib import Path
from multiprocessing.pool import Pool

from PIL import Image
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import imgaug as ia
from imgaug import augmenters as iaa
from tqdm import tqdm_notebook as tqdm
LANDMARKS_INSHOP = '../data/input/Anno/list_landmarks_inshop.txt'
EVAL_PARTITION = '../data/input/Eval/list_eval_partition.txt'
ATTR_ITEMS = '../data/input/Anno/list_attr_items.txt'

TRAIN_DF = '../data/input/train.csv'
VAL_DF = '../data/input/val.csv'
TEST_DF = '../data/input/test.csv'

CATEGORY = ['Denim',
            'Jackets_Vests',
            'Pants',
            'Shirts_Polos',
            'Shorts',
            'Suiting',
            'Sweaters',
            'Sweatshirts_Hoodies',
            'Tees_Tanks',
            'Blouses_Shirts',
            'Cardigans',
            'Denim',
            'Dresses',
            'Graphic_Tees',
            'Jackets_Coats',
            'Leggings',
            'Pants',
            'Rompers_Jumpsuits',
            'Shorts',
            'Skirts',
            'Sweaters',
            'Sweatshirts_Hoodies',
            'Tees_Tanks'
           ]

num_attr = 463<jupyter_output><empty_output><jupyter_text>读取数据集<jupyter_code>landmarks_inshop = pd.read_csv(
    LANDMARKS_INSHOP,
    sep=r' +',
    header=1,
    engine='python'
)
eval_partition = pd.read_csv(
    EVAL_PARTITION,
    sep=r' +',
    header=1,
    engine='python'
)
attr_items = pd.read_csv(
    ATTR_ITEMS,
    sep=r' ',
    header=None,
    skiprows=[0,1],
    names=['item_id', ]+list(range(num_attr)),
    engine='python'
)
# merge all attribute into one string
attr_items['attribute_labels'] = ''
for i in range(num_attr):
    attr_items['attribute_labels'] += (' ' + attr_items[i].map(str))
attr_items['attribute_labels'] = attr_items['attribute_labels'].map(lambda x: x[1:])
attr_items = attr_items.drop(columns=range(463))

eval_dict = dict([(m, c)for _, m, i, c in eval_partition.to_records()])<jupyter_output><empty_output><jupyter_text>编辑数据集<jupyter_code>def edit_attrimg(index):
    """transform the No.index item's attribute_labels of attr_items.
    Transform the label 1 or -1 into the index of attribute labels
    
    Arguments
        index: the index of attr_items
    
    Return
        None
    """
    labels = attr_items['attribute_labels'].loc[index]
    try:
        ll = labels
        labels = re.split(r' +', labels)
    except TypeError as e:
        raise TypeError(f'{e}, {ll}')
#     labels = [str(item[0]) for item in enumerate(labels) if item[1] == '1']
    labels = np.array(labels)
    labels = np.argwhere(labels == '1').flatten().astype(np.str)
    
    labels = ' '.join(labels)
    return labels


pool = Pool()
labels = pool.map(edit_attrimg, range(len(attr_items)))
attr_items['attribute_labels'] = labels
pool.close()
pool.join()<jupyter_output><empty_output><jupyter_text>合并数据集<jupyter_code>df = eval_partition\
    .merge(landmarks_inshop, on='image_name')\
    .merge(attr_items, on='item_id')\
    .drop(columns=['clothes_type', 'variation_type'])\
    .fillna(0)
df['category_label'] = df['image_name'].map(lambda x: CATEGORY.index(x.split('/')[-3]))

df.head()<jupyter_output><empty_output><jupyter_text>分割数据集<jupyter_code>df['evaluation_status'].value_counts()
# split the dataframe by evaluation_status
train_df = df[df['evaluation_status'] == 'train']\
            .drop(columns=['evaluation_status'])\
            .reset_index(drop=True)
val_df = df[df['evaluation_status'] == 'query']\
            .drop(columns=['evaluation_status'])\
            .reset_index(drop=True)
test_df = df[df['evaluation_status'] == 'gallery']\
            .drop(columns=['evaluation_status'])\
            .reset_index(drop=True)

len(train_df), len(val_df), len(test_df)
# save the dataframe
train_df.to_csv(TRAIN_DF, index=False)
val_df.to_csv(VAL_DF, index=False)
test_df.to_csv(TEST_DF, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14246374150908758, 'fraction_numerical': 0.007527079126124472, 'mean_word_length': 2.598414795244386, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '533992', 'license': 'MIT', 'n_tokens_mistral': 2120, 'n_tokens_neox': 2131, 'n_words': 334, 'path': 'src/DatasetPreprocessing.ipynb', 'repo_name': 'Jasonsey/FashionNet', 'size': '5447'}