<jupyter_start><jupyter_text>YOLOv3 training based on Repo ultralytics/yolov3> References:- https://github.com/ultralytics/yolov3What I would do in this Jupyter Notebook:![](./images/yolov3-training.png) 我实现了前两部，但是VSCODE我反复`Ctrl+S`竟然没保存？？？中途Git提交代码了也没有保存进度？？？反正代码很简单，就是将ego-hand训练集前16张图片单独取出来做出新的数据集，也是训练集和验证集，所以data里面train和valid指向一样的txt，names文件都一样。然后重新设置了两个YOLOv3的model cfg文件（直接从`yolov3-1cls.cfg`修改）：- `yolov3-hand.cfg`：保证yolo层classes=1，前一个conv层filters=18- `yolov3-hand-anchors.cfg`：除了上面一个cfg的内容，将yolo层的anchors修改为自己计算的。另外加上了几句训练的语句，反正本地也没跑出来，明天直接上Colab跑。这反复保存还没保存一定是bug！还好代码比较简单，这次就不重写了。  Small dataset: hand64 https://colab.research.google.com/drive/1Oe6opsYZwAhIL04AD4o-OCHo-C2n-M7N?usp=sharing 训练结果![](https://user-images.githubusercontent.com/45664104/86968130-8f667100-c163-11ea-88d7-f84634e965ba.png) 训练发生了很大的问题：训练效果很差，precisi和recall都是0，mAP保持在0.02以下，所以我现在想看一下是哪里导致了这些问题- 检查数据集标注有没有问题- 查看issues看类似问题的思路- 模型配置文件是不是错了- 数据集是不是太小了- 学习速率是不是太小了  检查数据集<jupyter_code>PATH = '../datasets/hand64'
from func import verify_dataset
verify_dataset(PATH)<jupyter_output><empty_output><jupyter_text>差得不多，再检查一下标注。重新自己用一个专门的[yolov3标注工具](https://github.com/developer0hye/Yolo_Label)标注了一张图片，显示标注差不多，说明数据集应该没问题：![image](https://user-images.githubusercontent.com/45664104/87072216-538ae480-c213-11ea-8479-1df7894840cf.png)  检查issues现在检查一下issue里面有没有提示吧，看前人有没有类似的问题。 https://github.com/ultralytics/yolov3/issues/898issuecomment-623174650- 更多图片- 输入size 先检查一下cfg文件，发现没什么问题，都是按照规范改的。不过试试默认的cfg，就发现结果更烂，mAP一直为0。![download](https://user-images.githubusercontent.com/45664104/87096459-cc516700-c23a-11ea-97ff-1dcf5a1f2279.png)> 为什么看mAP？https://github.com/ultralytics/yolov3/issues/1252issuecomment-635795279 然后试试更大的学习速率。在cfg文件里把它改成0.01（原本是0.001）。没有太大改进。![download (1)](https://user-images.githubusercontent.com/45664104/87098448-379d3800-c23f-11ea-9340-f198b5408667.png)<jupyter_code># 在BlueCrystal Phase 4上训练yolov3模型
# 此处代码均为命令行命令<jupyter_output><empty_output><jupyter_text>首先得将anaconda的lib加入DLL环境变量。https://www.jinhang.work/tech/use-uob-supercomputers/contents:libstdc%E9%97%AE%E9%A2%98```shellgit clone https://github.com/ultralytics/yolov3cd yolov3module add languages/anaconda3/3.7export LD_LIBRARY_PATH=/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib:$LD_LIBRARY_PATHpython train.py --cfg ../scratch/ego-hand/yolov3-hand-anchors.cfg --data ../scratch/ego-hand/ego-hand.data --epochs 100 --batch-size 8 --weights '' && mv results.txt results_egohand_egoanc.txtfrom utils import utilsutils.plot_results()``` ```python train.py --cfg ../scratch/ego-hand/yolov3-hand-anchors.cfg --data ../scratch/ego-hand/ego-hand.data --epochs 100 --batch-size 8 --weights weights/last.pt && mv results.txt results_egohand_egoanc.txt``` 目前碰到的困境：拿Colab训练很慢，12小时就没了，100个迭代期只训练到了第19，设置训练完成保存checkpoint也没法保存，只能重新训练，每一个迭代期保存，而且也运用上了之前小数据集训练的checkpoint来resume训练，等待结果中。以为BC4这个超算会更快吧，现在正在进行每一个迭代期保存checkpoint的训练，但问题是似乎速度很慢，是中间不保存的Colab训练时间的两倍多，最烦的是VPN时效只有12小时，到时候还得重新点一次恢复训练。 这个困境造成的最大问题是：没有前面的算法来产生手的边界框，就很难做接下来的边界框分类任务。但是我思考一点：一定要用自己训练的边界框产生模型来做第一步吗？可以先用训练好的yolo模型来做，得到边界框直接跳到做第二步。  使用一个预训练的YOLOv3模型 准备过程- 训练P30 100迭代期：https://colab.research.google.com/drive/14UP63JCr-JPAjUkcWAdhnjaiTHCmz0vY?usp=sharing- 在100迭代期基础上恢复训练P30：https://drive.google.com/file/d/148r6qyyvCZKd3nTOZPUivRhYHjh9IkVv/view?usp=sharing- 在BC4训练ego-hand 100迭代期，每个迭代期保存。<jupyter_code># 观察到在COCO数据集上训练的yolov3-spp模型很显然不适合提取手的边界框，不过！COCO上训练出来的模型可以输出person包括手臂，如果拿专门的手臂+手数据集是不是更准确呢？不过那是不是很麻烦，手臂加手的数据集直接就可以识别左右手了。
# 在P30数据集上训练的YOLOv3-spp
# 观察一批图片的检测结果，发现其实差一点的模型在自己的验证集检测结果也不太好，识别常常出错。
# - 检测不出来手的几率比较高
# - 出现两只同样标签的手
# - 左右手标签相反
# 训练结果好点的模型：在验证集表现
# - 检测出手的几率高于weak的
# - 同标签/相反标签几率还是较高，但比weak的低

# 需要更多轮的训练<jupyter_output><empty_output><jupyter_text>epoch4 ```Namespace(augment=False, batch_size=16, cfg='../datasets/hand64/yolov3-hand-anchors.cfg', conf_thres=0.001, data='../datasets/ego-hand/ego-hand.data', device='', img_size=416, iou_thres=0.6, save_json=False, single_cls=False, task='test', weights='weights/yolov3-egohand-epoch4.pt')Using CUDA device0 _CudaDeviceProperties(name='Tesla T4', total_memory=15079MB)Model Summary: 222 layers, 6.15237e+07 parameters, 6.15237e+07 gradientsFusing layers...Model Summary: 150 layers, 6.14974e+07 parameters, 6.14974e+07 gradientsCaching labels (3199 found, 0 missing, 0 empty, 0 duplicate, for 3199 images): 100% 3199/3199 [00:05<00:00, 537.20it/s]               Class    Images   Targets         P         R   mAP@0.5        F1: 100% 200/200 [11:16<00:00,  3.38s/it]                 all   3.2e+03  4.86e+03     0.782     0.824     0.848     0.802Speed: 11.1/1.7/12.8 ms inference/NMS/total per 416x416 image at batch-size 16```epoch5```Namespace(augment=False, batch_size=16, cfg='../datasets/hand64/yolov3-hand-anchors.cfg', conf_thres=0.001, data='../datasets/ego-hand/ego-hand.data', device='', img_size=416, iou_thres=0.6, save_json=False, single_cls=False, task='test', weights='weights/yolov3-egohand-epoch5.pt')Using CUDA device0 _CudaDeviceProperties(name='Tesla K80', total_memory=11441MB)Model Summary: 222 layers, 6.15237e+07 parameters, 6.15237e+07 gradientsFusing layers...Model Summary: 150 layers, 6.14974e+07 parameters, 6.14974e+07 gradientsCaching labels (3199 found, 0 missing, 0 empty, 0 duplicate, for 3199 images): 100% 3199/3199 [00:08<00:00, 378.29it/s]               Class    Images   Targets         P         R   mAP@0.5        F1: 100% 200/200 [11:23<00:00,  3.42s/it]                 all   3.2e+03  4.86e+03     0.745     0.808      0.82     0.775Speed: 28.1/2.8/30.9 ms inference/NMS/total per 416x416 image at batch-size 16```<jupyter_code># 可以观察到检测效果也还是不赖的，不过置信度还是不够高，需要更多轮次的训练，现在使用的checkpoint仅仅训练了5个迭代期，不过用来做第二阶段的工具还是OK的。

# 目前来看，egohand的epoch4权重实现的hand检测的mAP为0.848，而P30的100best权重实现的mAP为0.847，后者还在继续训练中，也将会有更好的表现。前者在充分训练后也肯定会更好。目前就采用egohand的权重来做第二阶段任务。后面训练的好就用谁。<jupyter_output><empty_output><jupyter_text>epichands- `left_hand`- `right_hand`  错误的一次训练 ![results.png](https://i.loli.net/2020/08/10/NHSpsvmiVkcuZbe.png) 评估- JOBID: 3955391 ```Namespace(augment=False, batch_size=16, cfg='../scratch/epichands/yolov3-spp-epichands.cfg', conf_thres=0.001, data='../scratch/epichands/epichands.data', device='', img_size=416, iou_thres=0.6, save_json=False, single_cls=False, task='test', weights='weights/epichands-fixed-imgsize.pt')Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)           device1 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)Model Summary: 225 layers, 6.25787e+07 parameters, 6.25787e+07 gradientsFusing layers...Model Summary: 152 layers, 6.25519e+07 parameters, 6.25519e+07 gradientsCaching labels ../scratch/ego-hand/labels.npy (3195 found, 0 missing, 0 empty, 0 duplicate, for 3195 images): 100%|██████████| 3195/3195 [00:00<00:00, 19004.05it/s]               Class    Images   Targets         P         R   mAP@0.5        F1                 all   3.2e+03  4.85e+03     0.887     0.842     0.868     0.864           left_hand   3.2e+03  4.85e+03     0.887     0.842     0.868     0.864Speed: 15.4/3.6/18.9 ms inference/NMS/total per 416x416 image at batch-size 16``` 出现了两个问题：1. 对于所有的图片，YOLOv3-spp模型只会识别出一个标签，这个应该是配置的bug，见- [Wrong result 1315](https://github.com/ultralytics/yolov3/issues/1315issuecomment-649679364)- [Why yolo-tiny predictions on video are all from the first class label? 1387](https://github.com/ultralytics/yolov3/issues/1387)结合第二个issue中提供的思路，**检查.npy文件**。我转向打印的输出，发现用的竟然是之前训练的缓存，而之前训练的是单类的数据集ego-hand：```Caching labels ../scratch/ego-hand/labels.npy (57 found, 0 missing, 0 empty, 0 duplicate, for 8166 images):   1%|          | 57/8166 [00:00<00:14, 564.37it/s]```又发现subset文件里的ego-hand全部没有替换成新名称。把所有缓存删掉，改名。2. 测试的mAP是训练的接近两倍（这个和第一个有关，标签只有一个说明另一个标签的验证都错了，正确率就减少一半了）- 见 [The mAP of each epoch in train.py is inconsistent with that in test.py 1091](https://github.com/ultralytics/yolov3/issues/1091)  用P30预训练模型训练，无flip，混合子集<jupyter_code>cd yolov3/
from IPython.display import Image
!python detect.py --cfg cfg/yolov3-spp-P30.cfg --names data/epichands.names --source data/samples/ --output output/noflip-mix-pret --weights weights/yolov3-noflip-mix.pt --save-txt<jupyter_output>Namespace(agnostic_nms=False, augment=False, cfg='cfg/yolov3-spp-P30.cfg', classes=None, conf_thres=0.3, device='', fourcc='mp4v', half=False, img_size=512, iou_thres=0.6, names='data/epichands.names', output='output/noflip-mix-pret', save_txt=True, soft_nms=False, soft_thres=0.05, source='data/samples/', view_img=False, weights='weights/yolov3-noflip-mix.pt')
Using CPU

Model Summary: 225 layers, 6.25787e+07 parameters, 6.25787e+07 gradients
image 1/9 data\samples\0000003541.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.690s)
image 2/9 data\samples\0000004831.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.690s)
image 3/9 data\samples\0000005791.jpg: 320x512 1 right_hands, Done. (1.629s)
image 4/9 data\samples\0000006061.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.826s)
image 5/9 data\samples\P01_06_0000002101.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.601s)
image 6/9 data\samples\P10_01_0000041521.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.556s)
image 7[...]<jupyter_text>用P30预训练模型训练，无flip，不混合子集<jupyter_code>!python detect.py --cfg cfg/yolov3-spp-epichands.cfg --names data/epichands.names --source data/samples/ --output output/noflip-nomix --weights weights/yolov3-noflip-nomix.pt --save-txt<jupyter_output>Namespace(agnostic_nms=False, augment=False, cfg='cfg/yolov3-spp-epichands.cfg', classes=None, conf_thres=0.3, device='', fourcc='mp4v', half=False, img_size=512, iou_thres=0.6, names='data/epichands.names', output='output/noflip-nomix', save_txt=True, soft_nms=False, soft_thres=0.05, source='data/samples/', view_img=False, weights='weights/yolov3-noflip-nomix.pt')
Using CPU

Model Summary: 222 layers, 6.15291e+07 parameters, 6.15291e+07 gradients
image 1/9 data\samples\Blur.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.643s)
image 2/9 data\samples\Close Hands.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.686s)
image 3/9 data\samples\Cross Arms.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.660s)
image 4/9 data\samples\Hand Overlap 1.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.758s)
image 5/9 data\samples\Hand Overlap 2.jpg: 320x512 1 right_hands, Done. (1.784s)
image 6/9 data\samples\Normal Case.jpg: 320x512 1 left_hands, 1 right_hands, Done. (1.743s)
image 7/9 da[...]<jupyter_text>test<jupyter_code>%cd yolov3
!python test.py --cfg cfg/yolov3-spp-epichands.cfg --data ../hand64/hand64.data --weights weights/yolov3-noflip-nomix.pt --batch-size 1 --img-size 416<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13678844519966016, 'fraction_numerical': 0.11488718965354479, 'mean_word_length': 5.5924082140634725, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 29, 'https://': 17, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '662468', 'license': 'MIT', 'n_tokens_mistral': 5875, 'n_tokens_neox': 5442, 'n_words': 623, 'path': 'yolov3-training.ipynb', 'repo_name': 'JinhangZhu/project-diary', 'size': '10593'}