<jupyter_start><jupyter_text>数据准备<jupyter_code># 下载supervise-to-darknet的转换脚本
!git clone https://github.com/SuyueLiu/supervisely-to-darknet.git
cd supervisely-to-darknet/
# 查看总的数据集
import os
data_path = './org-hands'
datasets = sorted(os.listdir(data_path))[1:]
print(datasets)
os.makedirs('ego-hands', exist_ok=True)
# get the images that contains objects in each data subset

from super2darknet import *
convertor = Convertor()
total = 0
for dataset in datasets:
    img_num = convertor.get_num_of_data(os.path.join(data_path, dataset))  # the number of images with objects
    total += img_num
    print(f"{dataset}: ", img_num)
print('total number: ', total)<jupyter_output>100%|██████████| 1005/1005 [00:00<00:00, 2567.64it/s]
100%|██████████| 238/238 [00:00<00:00, 3446.27it/s]
 25%|██▍       | 242/985 [00:00<00:00, 2405.93it/s]<jupyter_text>划分数据集* 在NG的Machine Learning中有提到，对于小数据集，建议train:val:test = 6:2:2.* 对于ego-hands这个数据集，P表示不同的拍摄者，在划分数据集的时候有两个方案：    1. 打乱所有拍摄者的数据，按比例随机抽取；    2. 按P来划分，例如P01-P08做为训练集，P12-P17做为验证集，其余的为测试集。* 来说第二种划分方法更好一点，因为打乱所有的照片，在测试的时候可能会遇到和训练集中同属于一个拍摄这的照片。而对于第二种的话，测试集则为完全不同的拍摄这，这对于评估模型的能力更为客观。reference: https://arxiv.org/abs/1905.00742 根据比例，将P01 ~ P08划为训练集，P10, P12 ~ P16划为验证集，P19， P19 ～ P23划分为测试集<jupyter_code># split training set
for dataset in datasets[0:26]:
    org_dir = os.path.join(data_path, dataset)
    convertor.convert(org_dir, './ego-hands', mode='train')


# split val set
for dataset in datasets[26: 26+11]:
    org_dir = os.path.join(data_path, dataset)
    convertor.convert(org_dir, './ego-hands', mode='val')

# add files
classes, _ = convertor.get_classes(org_dir)
convertor.add_extra_files('./ego-hands', classes)
# split test set, stored in the directory ./ego-hands_test/iamges/val
for dataset in datasets[37:]:
    org_dir = os.path.join(data_path, dataset)
    convertor.convert(org_dir, './ego-hands_test', mode='val')
    
# add files
classes, _ = convertor.get_classes(org_dir)
convertor.make_dirs('./ego-hands_test', 'train') # since there is only val dir
convertor.add_extra_files('./ego-hands_test', classes)<jupyter_output>loading annotations: 100%|██████████| 313/313 [00:00<00:00, 2245.96it/s]
moving images: 100%|██████████| 256/256 [00:00<00:00, 392.20it/s]
loading annotations: 100%|██████████| 273/273 [00:00<00:00, 1922.12it/s]
moving images: 100%|██████████| 269/269 [00:00<00:00, 303.34it/s]
loading annotations: 100%|██████████| 426/426 [00:00<00:00, 1989.79it/s]
moving images: 100%|██████████| 382/382 [00:01<00:00, 324.46it/s]
loading annotations: 100%|██████████| 1138/1138 [00:00<00:00, 1933.13it/s]
moving images: 100%|██████████| 978/978 [00:03<00:00, 276.86it/s]
loading annotations: 100%|██████████| 853/853 [00:00<00:00, 2042.56it/s]
moving images: 100%|██████████| 648/648 [00:02<00:00, 298.05it/s]
loading annotations: 100%|██████████| 184/184 [00:00<00:00, 2398.26it/s]
moving images: 100%|██████████| 147/147 [00:00<00:00, 375.21it/s]<jupyter_text>验证转换后的标签根据图片尺寸调整转换后的边界框（中心坐标和宽高），然后在原图上画出来看是否符合原始标记。<jupyter_code>%matplotlib inline
from verify_labels import verify_boxes
img_path = './ego-hands/images/train/P01_02_0000000181.jpg'
label_path = './ego-hands/labels/train/P01_02_0000000181.txt'
meta_path = './org-hands/P01_02'
verify_boxes(img_path, label_path, meta_path)<jupyter_output>Image: './ego-hands/images/train/P01_02_0000000181.jpg'
{'fraction_non_alphanumeric': 0.19333533473431402, 'fraction_numerical': 0.13209246472530772, 'mean_word_length': 6.181034482758621, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 10, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '255751', 'license': 'MIT', 'n_tokens_mistral': 1916, 'n_tokens_neox': 1609, 'n_words': 232, 'path': 'prepare_data.ipynb', 'repo_name': 'SuyueLiu/musical-octo-memory', 'size': '3331'}