devbox
小丑西瓜9
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

YOLOv8-Detect训练CoCo数据集+自己的数据集_yolov8 验证集 目录 及 文件格式

作者：小丑西瓜9 | 2024-02-22 16:51:39

赞

踩

yolov8 验证集 目录 及 文件格式

目录

0、引言

1、环境准备

2、数据集准备

2.1、创建数据集存放的文件夹

2.2、准备用于YOLOv8-Detect训练的txt

2.2.1 将CoCo数据集Json格式转为Xml

2.2.2 将CoCo数据集Xml整合到一个大的txt文件

2.2.3 将自己标注的xml整合到一个大的txt文件

2.2.4 将生成的txt合并分成训练集和训练集

2.2.5 将大的txt分解成多个小txt，txt以图片名字命名

2.2.6 将大的txt中对应的路径的图片复制到images/train

2.2.7 验证生成的txt是否准确

3、创建配置文件

3.1、设置官方配置文件：default.yaml，可自行修改。

3.2、设置data.yaml

3.3、设置yolov8s.yaml

4、进行训练

5、验证模型

6、总结

0、引言

本文是使用YOLOv8-Detect训练自己的数据集+CoCo数据集，数据集包含COCO数据集的人猫狗数据以及自己制作的人猫狗数据集，类别为0:person、1:cat、2:dog三类，大家可根据自己的数据集类别进行调整。

1、环境准备

可以参考这篇博客：深度学习环境搭建-CSDN博客

本文环境：

  • Windows10
  • python:3.10
  • cuda:11.6
  • pytorch:1.12.0
  • torchvision:0.13.0

2、数据集准备

2.1、创建数据集存放的文件夹

  1. mydata
  2. ______images
  3. ____________train
  4. _________________001.jpg
  5. ____________val
  6. _________________002.jpg
  7. ______labels
  8. ____________train
  9. _________________001.txt
  10. ____________val
  11. _________________002.txt

本人的数据都存放在mydata文件夹中（自定义命名）

​

目录结构如下：images存放训练集和验证集图片，labels存放训练集和验证集txt

​

​

​

2.2、准备用于YOLOv8-Detect训练的txt

2.2.1 将CoCo数据集Json格式转为Xml

从官网下载CoCo数据集的json文件

  1. import argparse, json
  2. import cytoolz
  3. from lxml import etree, objectify
  4. import os, re
  5.  
  6. def instance2xml_base(anno):
  7.     E = objectify.ElementMaker(annotate=False)
  8.     anno_tree = E.annotation(
  9.         E.folder('VOC2014_instance/{}'.format(anno['category_id'])),
  10.         E.filename(anno['file_name']),
  11.         E.source(
  12.             E.database('MS COCO 2014'),
  13.             E.annotation('MS COCO 2014'),
  14.             E.image('Flickr'),
  15.             E.url(anno['coco_url'])
  16.         ),
  17.         E.size(
  18.             E.width(anno['width']),
  19.             E.height(anno['height']),
  20.             E.depth(3)
  21.         ),
  22.         E.segmented(0),
  23.     )
  24.     return anno_tree
  25.  
  26. def instance2xml_bbox(anno, bbox_type='xyxy'):
  27.     """bbox_type: xyxy (xmin, ymin, xmax, ymax); xywh (xmin, ymin, width, height)"""
  28.     assert bbox_type in ['xyxy', 'xywh']
  29.     if bbox_type == 'xyxy':
  30.         xmin, ymin, w, h = anno['bbox']
  31.         xmax = xmin+w
  32.         ymax = ymin+h
  33.     else:
  34.         xmin, ymin, xmax, ymax = anno['bbox']
  35.     E = objectify.ElementMaker(annotate=False)
  36.     anno_tree = E.object(
  37.         E.name(anno['category_id']),
  38.         E.bndbox(
  39.             E.xmin(xmin),
  40.             E.ymin(ymin),
  41.             E.xmax(xmax),
  42.             E.ymax(ymax)
  43.         ),
  44.         E.difficult(anno['iscrowd'])
  45.     )
  46.     return anno_tree
  47.  
  48.  
  49. def parse_instance(content, outdir):
  50.     # print('11111',content)
  51.     categories = {d['id']: d['name'] for d in content['categories']}
  52.     # merge images and annotations: id in images vs image_id in annotations
  53.     merged_info_list = list(map(cytoolz.merge, cytoolz.join('id', content['images'], 'image_id', content['annotations'])))
  54.  
  55.     # print('111111111111',merged_info_list)
  56.     # convert category id to name
  57.     for instance in merged_info_list:
  58.         instance['category_id'] = categories[instance['category_id']]
  59.     # group by filename to pool all bbox in same file
  60.     for name, groups in cytoolz.groupby('file_name', merged_info_list).items():
  61.         anno_tree = instance2xml_base(groups[0])
  62.         # if one file have multiple different objects, save it in each category sub-directory
  63.         filenames = []
  64.         # print(groups)
  65.         for group in groups:
  66.             filenames.append(os.path.join(outdir, re.sub(" ", "_", group['category_id']), os.path.splitext(name)[0] + ".xml"))
  67.             anno_tree.append(instance2xml_bbox(group, bbox_type='xyxy'))
  68.         for filename in filenames:
  69.             etree.ElementTree(anno_tree).write(filename, pretty_print=True)
  70.         print("Formating instance xml file {} done!".format(name))
  71.  
  72. def main(args):
  73.     if not os.path.exists(args.output_dir):
  74.         os.makedirs(args.output_dir)
  75.     content = json.load(open(args.anno_file, 'r'))
  76.     if args.type == 'instance':
  77.         # make subdirectories
  78.         sub_dirs = [re.sub(" ", "_", cate['name']) for cate in content['categories']]
  79.         for sub_dir in sub_dirs:
  80.             sub_dir = os.path.join(args.output_dir, str(sub_dir))
  81.             if not os.path.exists(sub_dir):
  82.                 os.makedirs(sub_dir)
  83.         parse_instance(content, args.output_dir)
  84.  
  85.  
  86.  
  87. if __name__ == "__main__":
  88.     parser = argparse.ArgumentParser()
  89.     parser.add_argument("--anno_file", help="annotation file for object instance/keypoint", default=r'D:\CoCoData\annotations_14_17\instances_train2017.json')
  90.     parser.add_argument("--type", type=str, default='instance', help="object instance or keypoint", choices=['instance', 'keypoint'])
  91.     parser.add_argument("--output_dir", help="output directory for voc annotation xml file", default=r'D:\CoCoData\CoCoXml')
  92.     args = parser.parse_args()
  93.     main(args)

修改文件存放的路径和保存xml文件的路径。

​

运行后得到80个文件夹，分别为80类的xml文件：

​

​

再将需要类别的xml文件整合，得到训练集和验证集xml

​

2.2.2 将CoCo数据集Xml整合到一个大的txt文件
  1. import os
  2. import random
  3. import xml.etree.ElementTree as ET
  4.  
  5. import glob
  6.  
  7.  
  8. classes = ['person', 'cat', 'dog']
  9.  
  10. def convert(size, box):
  11.     # print('size--',size)
  12.     dw = 1. / size[0]
  13.     dh = 1. / size[1]
  14.     x = (box[0] + box[2]) / 2.0
  15.     y = (box[1] + box[3]) / 2.0
  16.     w = min(size[0],box[2] - box[0])
  17.     h = min(size[1],box[3] - box[1])
  18.     # print(x,y,w,h)
  19.     x = x * dw
  20.     w = w * dw
  21.     y = y * dh
  22.     h = h * dh
  23.     return (x, y, w, h)
  24. def convert_annotation(xml, list_file):
  25.     in_file = open(os.path.join(xml), encoding='utf-8')
  26.     tree = ET.parse(in_file)
  27.     root = tree.getroot()
  28.  
  29.  
  30.     xmlsize = root.find('size')
  31.  
  32.     w = int(xmlsize.find('width').text)
  33.     h = int(xmlsize.find('height').text)
  34.     # print(w,h)
  35.  
  36.     for obj in root.iter('object'):
  37.         difficult = 0
  38.         if obj.find('difficult') != None:
  39.             difficult = obj.find('difficult').text
  40.         cls = obj.find('name').text
  41.         if cls not in classes or int(difficult) == 1:
  42.             # print(cls, "------------------------------------\n")
  43.             continue
  44.         cls_id = classes.index(cls)
  45.         # print(cls, cls_id)
  46.         xmlbox = obj.find('bndbox')
  47.         x0 = float(xmlbox.find('xmin').text)
  48.         y0 = float(xmlbox.find('ymin').text)
  49.         x1 = float(xmlbox.find('xmax').text)
  50.         y1 = float(xmlbox.find('ymax').text)
  51.         xmin = min(x0, x1)
  52.         ymin = min(y0, y1)
  53.         xmax = max(x0, x1)
  54.         ymax = max(y0, y1)
  55.  
  56.         # b = (int(xmin), int(ymin), int(xmax), int(ymax))
  57.         b = (float(xmin), float(ymin), float(xmax), float(ymax))
  58.         # print((w, h))
  59.         # if w==0 or h == 0 :
  60.         #     print('11111111111')
  61.         bb = convert((w, h), b)
  62.         # print(bb)
  63.  
  64.         list_file.write(" "+str(cls_id) +"," + ",".join([str(a) for a in bb]))
  65.  
  66.  
  67. if __name__ == "__main__":
  68.     random.seed(0)
  69.  
  70.     # 图片路径
  71.     cocoImgPath = r'I:\allShare\CoCoData\train2017'
  72.     # Xml路径
  73.     cocoXmlPath = r'G:\CoCopcd_xml\cocotrain_pcd'
  74.     #txt保存路径
  75.     txtsavepath = r'G:\Yolov8\ultralytics-main\datasets\mydata\coco_v8txt'
  76.     fileWriteTxt = txtsavepath + '\\'+ cocoImgPath.split('\\')[-1] + '_v8.txt'
  77.     xmls = glob.glob(os.path.join(cocoXmlPath, '*.xml'))
  78.     list_file = open(fileWriteTxt, 'w', encoding='utf-8')
  79.  
  80.     for xml in xmls:
  81.         img = xml.replace(cocoXmlPath, cocoImgPath).replace('.xml', '.jpg')
  82.         if not os.path.exists(img):
  83.             print(img, ' is not exit')
  84.             continue
  85.         # print(img)
  86.         list_file.write(img)
  87.         convert_annotation(xml, list_file)
  88.         list_file.write('\n')
  89.     list_file.close()
  90.

分别运行train2017和val2017后得到：

2.2.3 将自己标注的xml整合到一个大的txt文件

批量处理多个文件夹：

  1. import os
  2. import random
  3. import xml.etree.ElementTree as ET
  4.  
  5. import glob
  6.  
  7.  
  8. classes = ['person', 'cat', 'dog']
  9.  
  10. def convert(size, box):
  11.     # print('size--',size)
  12.     dw = 1. / size[0]
  13.     dh = 1. / size[1]
  14.     x = (box[0] + box[2]) / 2.0
  15.     y = (box[1] + box[3]) / 2.0
  16.     w = min(size[0],box[2] - box[0])
  17.     h = min(size[1],box[3] - box[1])
  18.     # print(x,y,w,h)
  19.     x = x * dw
  20.     w = w * dw
  21.     y = y * dh
  22.     h = h * dh
  23.     return (x, y, w, h)
  24. def convert_annotation(xml, list_file):
  25.     in_file = open(os.path.join(xml), encoding='utf-8')
  26.     tree = ET.parse(in_file)
  27.     root = tree.getroot()
  28.  
  29.  
  30.     xmlsize = root.find('size')
  31.  
  32.     w = int(xmlsize.find('width').text)
  33.     h = int(xmlsize.find('height').text)
  34.     # print(w,h)
  35.  
  36.     for obj in root.iter('object'):
  37.         difficult = 0
  38.         if obj.find('difficult') != None:
  39.             difficult = obj.find('difficult').text
  40.         cls = obj.find('name').text
  41.         if cls not in classes or int(difficult) == 1:
  42.             # print(cls, "------------------------------------\n")
  43.             continue
  44.         cls_id = classes.index(cls)
  45.         # print(cls, cls_id)
  46.         xmlbox = obj.find('bndbox')
  47.         x0 = float(xmlbox.find('xmin').text)
  48.         y0 = float(xmlbox.find('ymin').text)
  49.         x1 = float(xmlbox.find('xmax').text)
  50.         y1 = float(xmlbox.find('ymax').text)
  51.         xmin = min(x0, x1)
  52.         ymin = min(y0, y1)
  53.         xmax = max(x0, x1)
  54.         ymax = max(y0, y1)
  55.  
  56.         # b = (int(xmin), int(ymin), int(xmax), int(ymax))
  57.         b = (float(xmin), float(ymin), float(xmax), float(ymax))
  58.         # print((w, h))
  59.         # if w==0 or h == 0 :
  60.         #     print('11111111111')
  61.         bb = convert((w, h), b)
  62.         # print(bb)
  63.  
  64.         list_file.write(" "+str(cls_id) +"," + ",".join([str(a) for a in bb]))
  65.  
  66.  
  67. if __name__ == "__main__":
  68.     random.seed(0)
  69.  
  70.     pathdir = r'G:\dataset\selfXml'
  71.     for path, dddddd, file_name in os.walk(pathdir):
  72.         print(path)
  73.         # 图片路径
  74.         cocoImgPath = path
  75.         # Xml路径
  76.         cocoXmlPath = path
  77.         #txt保存路径
  78.         txtsavepath = r'G:\ultralytics-main\datasets\mydata\self_v8txt'
  79.         fileWriteTxt = txtsavepath + '\\'+ cocoImgPath.split('\\')[-1] + '_v8.txt'
  80.         xmls = glob.glob(os.path.join(cocoXmlPath, '*.xml'))
  81.         list_file = open(fileWriteTxt, 'w', encoding='utf-8')
  82.  
  83.         for xml in xmls:
  84.             img = xml.replace(cocoXmlPath, cocoImgPath).replace('.xml', '.jpg')
  85.             if not os.path.exists(img):
  86.                 print(img, ' is not exit')
  87.                 continue
  88.             # print(img)
  89.             list_file.write(img)
  90.             convert_annotation(xml, list_file)
  91.             list_file.write('\n')
  92.         list_file.close()
  93.

此代码运行后会得到一个空的txt，删除即可

运行以上代码可得到txt，内容如下：图片路径，classID，xmin，ymin，xmax，ymax  （归一化）

2.2.4 将生成的txt合并分成训练集和训练集

将以上步骤得到的txt放在同一文件夹下

运行以下代码合并分割训练集和验证集，自行调整 ratio

  1. import random
  2. import os
  3. import glob
  4.  
  5.  
  6. # 转化数据
  7. record_jpg_fold_root = r'G:\Yolov8\ultralytics-main\datasets\mydata\self_v8txt'
  8.  
  9. ratio = 0.05
  10. # ratio = 1
  11. record_jpg_files = os.listdir(record_jpg_fold_root)
  12. record_jpg_files = glob.glob(os.path.join(record_jpg_fold_root, '*.txt'))
  13.  
  14. train_files = []
  15. valid_files = []
  16. datas = []
  17. for record_file in record_jpg_files:
  18.     record_file_path = os.path.join(record_jpg_fold_root, record_file)
  19.     f = open(record_file_path, 'r')
  20.     datas_temp = f.readlines()
  21.     total_num = len(datas_temp)
  22.     valid_datas_num = int(total_num * ratio)
  23.     valid_datas = random.sample(datas_temp, valid_datas_num)
  24.     datas += datas_temp # all datas
  25.     valid_files += valid_datas # valid datas
  26.  
  27. # print(datas)
  28. # datas = datas.split('\n')
  29. # print(datas)
  30. random.shuffle(datas)
  31. # print(files)
  32. # total_num = len(datas)
  33. # valid_datas_num = int(total_num * 0.1)
  34. # valid_datas = random.sample(datas, valid_datas_num)
  35.  
  36. traintxtdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\merge_v8txt\merge_CoCo_pcdtrain_01.txt'
  37. valtxtdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\merge_v8txt\merge_CoCo_pcdval_01.txt'
  38.  
  39. with open(traintxtdir, 'w') as ft, \
  40.         open(valtxtdir, 'w') as fv:
  41.     for file in datas:
  42.         if file in valid_files:
  43.             fv.write(file)
  44.         else:
  45.             ft.write(file)
  46.  
  47.

2.2.5 将大的txt分解成多个小txt，txt以图片名字命名

注意检查保存路径

  1.  
  2. import glob
  3. import os
  4.  
  5.  
  6.  
  7. txtpathdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\merge_v8txt\merge_CoCo_pcdtrain_01.txt'
  8. #保存txt路径
  9. savetxtpath = r'G:\Yolov8\ultralytics-main\datasets\mydata\labels\train'
  10.  
  11.  
  12. file = open(txtpathdir,'r',encoding='utf-8')
  13. lines = file.readlines()
  14. for line in lines:
  15.     line = line.split('\n')[0]
  16.     # print(line)
  17.     imgdir = line.split(' ')[0]
  18.     bboxinfo = line.split(' ')[1:]
  19.     # print('imgdir',imgdir)
  20.     savetxtdir = os.path.join(savetxtpath,imgdir.split('\\')[-1].replace('.jpg','.txt'))
  21.     file = open(savetxtdir, 'w', encoding='utf-8')
  22.     # print(savetxtdir)
  23.     for i in range(len(bboxinfo)):
  24.         # print(bboxinfo[i])
  25.         info = bboxinfo[i].split(',')
  26.         # print(info)
  27.         info1 = ' '.join(info)
  28.         print(info1)
  29.         file.write(info1+'\n')
  30.

运行后得到多个训练和验证txt：

txt的内容要与官方格式一致：

分别代表：类别、框的中心点xy坐标以及框的宽高（进行了归一化处理）

2.2.6 将大的txt中对应的路径的图片复制到images/train
  1.  
  2. import glob
  3. import os
  4. import shutil
  5.  
  6. txtpathdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\merge_v8txt\merge_CoCo_pcdtrain_01.txt'
  7. saveimgpath = r'G:\Yolov8\ultralytics-main\datasets\mydata\images\train'
  8. file = open(txtpathdir,'r',encoding='utf-8')
  9. lines = file.readlines()
  10. for line in lines:
  11.     line = line.split('\n')[0]
  12.     imgdir = line.split(' ')[0]
  13.     # print('imgdir',imgdir)
  14.     saveimgdir = os.path.join(saveimgpath,imgdir.split('\\')[-1])
  15.     # print(saveimgdir)
  16.  
  17.     shutil.copy(imgdir,saveimgdir)
  18. print('end.....')
  19.

2.2.7 验证生成的txt是否准确
  1.  
  2. import os
  3. import glob
  4. import cv2
  5. imgdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\images\val'
  6. txtdir = r'G:\Yolov8\ultralytics-main\datasets\mydata\labels\val'
  7. list1 = glob.glob(os.path.join(imgdir,'*.jpg'))
  8. list2 = glob.glob(os.path.join(txtdir,'*.txt'))
  9. idx = 0
  10. # print(len(list1))
  11. # print(len(list2))
  12. while (idx < len(list1)):
  13.  
  14.     imgpath = list1[idx]
  15.     txtpath = list2[idx]
  16.     print(imgpath)
  17.     print(txtpath)
  18.     f = open(txtpath,'r')
  19.     lines = f.readlines()
  20.     img = cv2.imread(imgpath)
  21.     h, w, c = img.shape
  22.     colors = [[0, 0, 255],[0, 255, 0], [0, 255, 255]]
  23.     # print(lines)
  24.     for line in lines:
  25.         # print(line)
  26.         l = line.split(' ')
  27.         # print(len(l))
  28.         label = l[0]
  29.         cx = float(l[1]) * w
  30.         cy = float(l[2]) * h
  31.         weight = float(l[3]) * w
  32.         height = float(l[4]) * h
  33.         xmin = cx - weight/2
  34.         ymin = cy - height/2
  35.         xmax = cx + weight/2
  36.         ymax = cy + height/2
  37.         print(label,(xmin,ymin),(xmax,ymax))
  38.         color = colors[int(label)]
  39.         cv2.putText(img,label,(int(xmin),int(ymin)),cv2.FONT_HERSHEY_SIMPLEX,0.8,color,1,cv2.LINE_AA)
  40.         cv2.rectangle(img,(int(xmin),int(ymin)),(int(xmax),int(ymax)),color,2)
  41.         kpts = []
  42.  
  43.     img = cv2.resize(img, None, fx=1, fy=1)
  44.     cv2.imshow('1',img)
  45.     key = cv2.waitKey(0)
  46.     if key == ord('q'):
  47.         break
  48.     if key == ord('z'):
  49.         idx -=1
  50.     else:
  51.         idx += 1

将txt信息可视化：

完成以上步骤即将训练和验证的数据集准备完成。

3、创建配置文件

3.1、设置官方配置文件：default.yaml，可自行修改。

所有参数参见：Configuration - Ultralytics YOLOv8 Docs

3.2、设置data.yaml

根据自己的数据集位置进行修改和配置。

  1. path: D:\Yolov8\ultralytics-main\datasets\mydata  # dataset root dir
  2. train: images/train  # train images (relative to 'path') 118287 images
  3. val: images/val  # val images (relative to 'path') 5000 images
  4. #test: test-dev2017.txt  # 20288 of 40670 images, submit to https://competitions.codalab.org/competitions/20794
  5.  
  6. # Classes
  7. names:
  8.   0: person
  9.   1: cat
  10.   2: dog
  11. nc: 3

3.3、设置yolov8s.yaml

根据自己想使用的权重进行选择，我这里采用的是yolov8s.pt进行训练，类别为3。

  1. # Ultralytics YOLO 声明：本文内容由网友自发贡献，转载请注明出处：【wpsshop】
    推荐阅读
    相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。