devbox
2023面试高手
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

YOLOV5从训练到部署测试NCNN安卓端部署保姆级教程_requirements: tensorboard>=2.4.1 not found and is

作者：2023面试高手 | 2024-05-14 18:47:20

赞

踩

requirements: tensorboard>=2.4.1 not found and is required by yolov5, attemp

一、基础配置

  • 为防止环境差异，造成教程中部分步骤出现差异，造成部署失败，下面将组件版本列举出来。

组件

版本

anaconda

4.8.3

python

3.8.13

yolov5

6.0

System

Ubuntu 22.04.1 x86_64

System Core

5.10.102.1

gcc

11.2.0

g++

11.2.0

其中System、System Core、gcc、g++版本也许其他低版本也可以，但需自测。建议在docker中进行环境搭建。

  • 初始化系统配置
  1. sudo apt update
  2. sudo apt install git gcc g++ vim curl wget cmake -y

1、Anaconda

conda config --set show_channel_urls yes #在用户目录下生成.condarc文件
  • 为了加快下载速度，我们可以配置大陆境内conda加速镜像。打开用户目录下的.condarc文件，编辑内部信息
  1. channels:
  2.   - defaults
  3. show_channel_urls: true
  4. default_channels:
  5.   - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  6.   - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  7.   - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
  8. custom_channels:
  9.   conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  10.   msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  11.   bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  12.   menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  13.   pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  14.   pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  15.   simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/clou
  • 创建YOLO基础环境
conda create -n yolo python=3.8 # 创建一个名为yolo的python3.8环境

2、YoloV5

  1. git clone https://github.com/ultralytics/yolov5.git # 下载yolov5源码
  2. cd yolov5
  3. git checkout v6.0 -b v6.0 # 切换为6.0版本
  4. git checkout
  5. conda activate yolo # 切换到前面创建的conda环境
  • 打开requirements.txt文件，修改其中内容
  1. # pip install -r requirements.txt
  2.  
  3. # Base ----------------------------------------
  4. matplotlib>=3.2.2
  5. numpy>=1.18.5
  6. opencv-python>=4.1.2
  7. Pillow>=7.1.2
  8. PyYAML>=5.3.1
  9. requests>=2.23.0
  10. scipy>=1.4.1
  11. torch==1.7.1
  12. torchvision==0.8.1
  13. tqdm>=4.41.0
  14.  
  15. # Logging -------------------------------------
  16. tensorboard==2.4.1
  17. # wandb
  18.  
  19. # Plotting ------------------------------------
  20. pandas>=1.1.4
  21. seaborn>=0.11.0
  22.  
  23. # Export --------------------------------------
  24. # coremltools>=4.1  # CoreML export
  25. # onnx>=1.9.0  # ONNX export
  26. # onnx-simplifier>=0.3.6  # ONNX simplifier
  27. # scikit-learn==0.19.2  # CoreML quantization
  28. # tensorflow>=2.4.1  # TFLite export
  29. # tensorflowjs>=3.9.0  # TF.js export
  30.  
  31. # Extras --------------------------------------
  32. # albumentations>=1.0.3
  33. # Cython  # for pycocotools https://github.com/cocodataset/cocoapi/issues/172
  34. # pycocotools>=2.0  # COCO mAP
  35. # roboflow
  36. thop  # FLOPs computation
  • 安装Yolo基础环境
  1. pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
  2. conda install onnx
  3. pip install onnxsim
  4. pip install onnx-simplifier
  5. pip install onnxruntime

二、训练样本

  • 准备labelimg环境
  1. conda create -n labelimg python=3.8
  2. conda activate labelimg
  3. conda install labelimg
  4. labelimg

此时，系统会打开labelimg工具

  • 创建样本
  1. 我们在系统系统目录下创建metadata目录，在目录下分别创建images、labels文件夹。
  2. 将包含需要识别目标的图片全部放入到images文件夹下。
  3. 打开labelimg，点击打开文件夹（打开目录），选择images所在文件夹。
  4. 点击改变存放目录，选择labels所在文件夹
  5. 点击标记模式为yolo模式，如下图所示。

点击查看 -> 自动保存模式，即打开自动保存模式。

  • 开始标注

labelimg快捷键：W，开启标注框。A，上一张。D，下一张。将全部图片标注完毕。（准备的单个图片不需要包含所有需要识别的类别）

  • 创建配置 xxx.yml
  1. # 训练集，yolo会读取此文件夹下所有图片进行训练,有条件建议切分样本或者分批标注训练集、测试集、验证集。此处路径使用了相对路径，因此yolo会以 train.py 所在目录为根目录进行样本的搜索
  2. train: ./metadata/images
  3. # 验证集
  4. val: ./metadata/images
  5. # 验证集
  6. test: ./metadata/images
  7. # 类别个数,标注的时候共涉及到了几个类别，就填几，跟下面names下的个数相等
  8. nc: 3
  9. # 类别列表, 枚举所有类别, 可以直接复制 labels/classes.txt 下的所有类别，粘贴到此处，记得带引号,与classes.txt 顺序不要错
  10. names: ["a", "b", "c"]

此处我们将训练集、验证集、测试集放到了相同位置，可能造成结果与样本拟合，我们也可以将训练集、测试集、验证集按照下面的目录结构放置样本及配置

  1. ── images // 存放所有图像数据
  2.    ├── test // 测试集图片存放位置
  3.    │   ├── xxx.jpg
  4.  
  5.    ├── train // 训练集图片存放位置
  6.    │   ├── xxx.jpg
  7.  
  8.    └── val // 验证集图片存放位置
  9.        ├── xxx.jpg
  10.  
  11. ── labels // 所有标注完毕后的数据
  12.     ├── test // 测试集标注标签文件所在位置
  13.     │   ├── xxx.txt
  14.  
  15.     ├── train// 训练集标注标签文件所在位置
  16.     │   ├── xxx.txt
  17.    
  18.     └── val// 验证集标注标签文件所在位置
  19.         ├── xxx.txt

注意：images下样本集各自文件夹下的样本名称必须与labels下各自文件夹下的样本配置名称相同。

此处提供一个可以自动切分样本的py脚本

  1. import os
  2. import random
  3. import argparse
  4. import shutil
  5. # 默认会从指定的目录的上级目录下的labels下查找。例如sourcepath指定为 ./dataset/images 则会认为 ./dataset/labels中存放的是所有标注完毕后的标签
  6. LABELS_PATH = "./labels"
  7. CLASSES_PROPERTIES = LABELS_PATH + "/classes.txt"
  8. parser = argparse.ArgumentParser()
  9. # 未切分的数据集地址
  10. parser.add_argument('--source_path', type=str, help='input images path')
  11. # 数据集划分完毕后存放的位置，不存在会自己创建
  12. parser.add_argument('--save_path', default='./save_path', type=str, help='new train dataset path')
  13. # parser.add_argument("--val_path", default="./val", type=str, help="new verify dataset path")
  14. # parser.add_argument("--test_path", default="./test", type=str, help="new verify dataset path")
  15. # 分割比例
  16. parser.add_argument("--dataset_split_rate", default="7:2:1", type=str, help="dataset split rate, default: 7:2:1 as train:val:test")
  17. opt = parser.parse_args()
  18.  
  19. source_path = opt.source_path
  20.  
  21. train_path = os.path.join(opt.save_path, "./images/train")
  22. val_path = os.path.join(opt.save_path, "./images/val")
  23. test_path = os.path.join(opt.save_path, "./images/test")
  24.  
  25. dataset_split_rate = opt.dataset_split_rate
  26.  
  27. train_dataset_rate = 0
  28. val_dataset_rate = 0
  29. test_dataset_rate = 0
  30.  
  31. if not os.path.exists(source_path):
  32. 	raise Exception(source_path + " not found exists")
  33.  
  34. if not os.path.exists(train_path):
  35. 	os.makedirs(train_path)
  36.  
  37. if not os.path.exists(val_path):
  38. 	os.makedirs(val_path)
  39.  
  40. if not os.path.exists(test_path):
  41. 	os.makedirs(test_path)
  42.  
  43. dataset_split_rate = dataset_split_rate.split(":")
  44. if len(dataset_split_rate) != 3:
  45. 	raise Exception("dataset_split_rate must be third numbers exp: 7:2:1")
  46.  
  47. train_dataset_rate = dataset_split_rate[0]
  48. val_dataset_rate = dataset_split_rate[1]
  49. test_dataset_rate = dataset_split_rate[2]
  50.  
  51. if not train_dataset_rate.isdigit():
  52. 	raise Exception("dataset_split_rate must be third numbers exp: 7:2:1")
  53.  
  54. if not val_dataset_rate.isdigit():
  55. 	raise Exception("dataset_split_rate must be third numbers exp: 7:2:1")
  56.  
  57. if not test_dataset_rate.isdigit():
  58. 	raise Exception("dataset_split_rate must be third numbers exp: 7:2:1")
  59.  
  60. train_dataset_rate = int(train_dataset_rate)
  61. val_dataset_rate = int(val_dataset_rate)
  62. test_dataset_rate = int(test_dataset_rate)
  63.  
  64. if train_dataset_rate + val_dataset_rate + test_dataset_rate != 10:
  65. 	raise Exception("dataset sum rate must be 10, current val : " + str(train_dataset_rate + val_dataset_rate + test_dataset_rate))
  66.  
  67. train_dataset_rate = train_dataset_rate * 0.1
  68. val_dataset_rate = val_dataset_rate * 0.1
  69. test_dataset_rate = test_dataset_rate * 0.1
  70.  
  71. # 读取源数据路径
  72. all_dataset = {}
  73.  
  74. for image in os.listdir(source_path):
  75. 	image_path = os.path.abspath(os.path.join(source_path, image))
  76. 	fname = os.path.splitext(image)[-2]
  77. 	label_path = os.path.abspath(os.path.join(source_path, "../" + LABELS_PATH + "/" + fname + '.txt'))
  78. 	all_dataset[image_path] = label_path
  79.  
  80. if len(all_dataset) < 3:
  81. 	raise Exception("dataset images number must > 3...")
  82.  
  83. image_orders = list(all_dataset.keys())
  84.  
  85. random.shuffle(image_orders)
  86.  
  87. train_dataset = {}
  88. val_dataset = {}
  89. test_dataset = {}
  90.  
  91. # 分割样本
  92. train_dataset_max_index = int(train_dataset_rate * len(image_orders))
  93. val_dataset_max_index = int(val_dataset_rate * len(image_orders))
  94. test_dataset_max_index = int(test_dataset_rate * len(image_orders))
  95.  
  96. for k in image_orders:
  97. 	if len(train_dataset) >= train_dataset_max_index:
  98. 		if len(val_dataset) >= val_dataset_max_index:
  99. 			test_dataset[k] = all_dataset[k]
  100. 		else:
  101. 			val_dataset[k] = all_dataset[k]
  102. 	else:
  103. 		train_dataset[k] = all_dataset[k]
  104.  
  105. # 保存训练集
  106.  
  107. for f in train_dataset.keys():
  108. 	f = str(f)
  109. 	shutil.copy(f, train_path)
  110. 	clas_path = os.path.abspath(os.path.join(train_path, "../../" + LABELS_PATH + "/train"))
  111. 	if not os.path.exists(clas_path):
  112. 		os.makedirs(clas_path)
  113. 	shutil.copy(train_dataset[f], clas_path)
  114.  
  115.  
  116. for f in val_dataset.keys():
  117. 	f = str(f)
  118. 	shutil.copy(f, val_path)
  119. 	clas_path = os.path.abspath(os.path.join(train_path, "../../" + LABELS_PATH + "/val"))
  120. 	if not os.path.exists(clas_path):
  121. 		os.makedirs(clas_path)
  122. 	shutil.copy(val_dataset[f], clas_path)
  123.  
  124. for f in test_dataset.keys():
  125. 	f = str(f)
  126. 	shutil.copy(f, test_path)
  127. 	clas_path = os.path.abspath(os.path.join(train_path, "../../" + LABELS_PATH + "/test"))
  128. 	if not os.path.exists(clas_path):
  129. 		os.makedirs(clas_path)
  130. 	shutil.copy(test_dataset[f], clas_path)

执行分割

  1. # source_path 所有的样本图片所在目录。save_path：分割完毕后的图片存放目录。data_split_rate：样本切分比例(训练集:验证集:测试集)
  2. python xxx.py --source_path ./metadata/images --save_path ./dataset --dataset_split_rate 7:2:1

样本被保存到了 dataset下，我们在dataset下创建xxx.yaml配置文件。

  1. # 例如根目录名称为dataset
  2. train: ./dataset/images/train
  3. # 验证集
  4. val: ./dataset/images/val
  5. # 测试集
  6. test: ./dataset/images/test
  7. # 类别个数,标注的时候共涉及到了几个类别，就填几
  8. nc: 3 
  9. # 类别列表, 枚举所有类别, 可以直接复制 labels/classes.txt 下的所有类别，粘贴到此处，记得带引号,与ckasses.txt 顺序不要错
  10. names: ["a", "b", "c"]
  • 训练样本
  1. conda activate yolo
  2.  # 进入yolov5源代码目录
  3. cd yolov5
  4. # 手动下载基础模型，指定为6.0版本
  5. wget https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt
  6. # 测试模型，执行完毕后,控制台会输出推理结果，Results saved to runs/detect/xxx 打开此文件夹，查看推理完毕后的结果是否正常
  7. python detect.py --weights yolov5s.pt --img 640 --conf 0.25 --source data/images/
  8. # 开始训练, 注意 --data 后参数，指向上面创建的yaml文件, 训练完毕后，系统会提示样本保存的目录：Optimizer stripped from runs/train/exp/weights/xxx.pt 
  9. python train.py --img 640 --batch 50 --epochs 100 --data ../metadata/xxx.yaml --weights yolov5s.pt --nosave --cache
  10. # 如果出现 requirements: tensorboard>=2.4.1 not found and is required by YOLOv5，手动执行下面的命令
  11. pip install tensorboard==2.4.1
  12. # 测试模型是否正常 --weights 为训练完毕后的模型文件， --img 同训练时 img参数 --conf 最低置信度，当分值低于此分数时，不在图片中标注。--source 需要被推理的图片所在目录
  13. python detect.py --weights runs/train/exp/weights/xxx.pt --img 640 --conf 0.25 --source data/images/
  • 检出为ncnn可用模型
  1. # 检出为onnx模型,注意weights指向上一步训练出来的模型
  2. python export.py --weights ./run/train/exp/weights/xxx.pt --img 640 640 --batch 1 --train --simplify --include onnx --opset 11
  3. # 简化onnx模型
  4. python -m onnxsim ./run/train/exp/weights/xxx.onnx ./run/train/exp/weights/yolov5s-sim.onnx
  5. # 转换为ncnn模型，可以打开此网站 https://convertmodel.com/ 选择onnx转ncnn，如果自行编译了ncnn，可以打开install/bin 执行下边的命令
  6. ./onnx2ncnn ./run/train/exp/weights/yolov5s-sim.onnx ./run/train/exp/weights/yolov5s.param ./run/train/exp/weights/yolov5s.bin
  • 修改配置

打开生成的yolov5s.param文件,修改Reshape参数，将0=6400、1600、400全部改为 0=-1

 如下图

 记录三组Permute数值的倒数第二列信息，如下图，例如我的样本配置结果为 "output", "365", "385"。如果你的Permute倒数第二列参数不是纯数值（除output之外），此时即代表后面所有的步骤都可能出现无法控制的结果，请重新开始训练，及检查你的yolo版本。

 三、安卓端部署

  • 打开Android Studio项目，创建或打开一个项目，切换项目显示模式为project

  • 创建模块，模块类型选择Android Native Library，注意选择c++11

  • 点击Android Studio工具栏 Tools -> SDK Manager,检查cmake 3.10.2是否已安装，如果没有安装，请安装此工具

  • 修改插件目录下 src -> main -> cpp -> CMakeLists.txt文件。
  1. project(yolov5ncnn) # 记录此名称，建议设置为插件名称
  2.  
  3. cmake_minimum_required(VERSION 3.4.1)
  4.  
  5. set(ncnn_DIR ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/lib/cmake/ncnn)
  6. find_package(ncnn REQUIRED)
  7. # yolov5ncnn 参数与上名称对应，yolov5ncnn_jni.cpp即为模块创建时，自动创建在与CMakeLists.txt 平级的cpp文件，当然也可以后面手动创建
  8. add_library(yolov5ncnn SHARED yolov5ncnn_jni.cpp)
  9. # yolov5ncnn同上，下边两个参数不变
  10. target_link_libraries(yolov5ncnn
  11.     ncnn
  12.  
  13.     jnigraphics
  14. )

其中ncnnvulkan需要自行下载编译，或到此下载发行包，选择ncnn-xxx-android-vulkan.zip。如果github打不开，请在此下载（希望大家在此处下载，创作不易，支持一下）。

将ncnnvulkan依赖放入与CMakeLists.txt平级目录，结构如下

  1. ncnnvulkan
  2.     ├─arm64-v8a
  3.     │  ├─include
  4.     │  │  ├─glslang
  5.     │  │  │  ├─Include
  6.     │  │  │  ├─MachineIndependent
  7.     │  │  │  │  └─preprocessor
  8.     │  │  │  ├─Public
  9.     │  │  │  └─SPIRV
  10.     │  │  └─ncnn
  11.     │  └─lib
  12.     │      └─cmake
  13.     │          └─ncnn
  14.     ├─armeabi-v7a
  15.     │  ├─include
  16.     │  │  ├─glslang
  17.     │  │  │  ├─Include
  18.     │  │  │  ├─MachineIndependent
  19.     │  │  │  │  └─preprocessor
  20.     │  │  │  ├─Public
  21.     │  │  │  └─SPIRV
  22.     │  │  └─ncnn
  23.     │  └─lib
  24.     │      └─cmake
  25.         │  │  ├─Include
  26.         │  │  ├─MachineIndependent
  27.         │  │  │  └─preprocessor
  28.         │  │  ├─Public
  29.         │  │  └─SPIRV
  30.         │  └─ncnn
  31.         └─lib
  32.             └─cmake
  33.                 └─ncnn
  34. CMakeLists.txt
  • 修改与CMakeLists.txt平级的yolov5ncnn_jni.cpp文件（CMakeLists.txt中指定的cpp文件）
  1. // Tencent is pleased to support the open source community by making ncnn available.
  2. //
  3. // Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved.
  4. //
  5. // Licensed under the BSD 3-Clause License (the "License"); you may not use this file except
  6. // in compliance with the License. You may obtain a copy of the License at
  7. //
  8. // https://opensource.org/licenses/BSD-3-Clause
  9. //
  10. // Unless required by applicable law or agreed to in writing, software distributed
  11. // under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
  12. // CONDITIONS OF ANY KIND, either express or implied. See the License for the
  13. // specific language governing permissions and limitations under the License.
  14.  
  15. #include <android/asset_manager_jni.h>
  16. #include <android/bitmap.h>
  17. #include <android/log.h>
  18.  
  19. #include <jni.h>
  20.  
  21. #include <string>
  22. #include <vector>
  23.  
  24. // ncnn
  25. #include "layer.h"
  26. #include "net.h"
  27. #include "benchmark.h"
  28.  
  29. static ncnn::UnlockedPoolAllocator g_blob_pool_allocator;
  30. static ncnn::PoolAllocator g_workspace_pool_allocator;
  31.  
  32. static ncnn::Net yolov5;
  33.  
  34. class YoloV5Focus : public ncnn::Layer
  35. {
  36. public:
  37.     YoloV5Focus()
  38.     {
  39.         one_blob_only = true;
  40.     }
  41.  
  42.     virtual int forward(const ncnn::Mat& bottom_blob, ncnn::Mat& top_blob, const ncnn::Option& opt) const
  43.     {
  44.         int w = bottom_blob.w;
  45.         int h = bottom_blob.h;
  46.         int channels = bottom_blob.c;
  47.  
  48.         int outw = w / 2;
  49.         int outh = h / 2;
  50.         int outc = channels * 4;
  51.  
  52.         top_blob.create(outw, outh, outc, 4u, 1, opt.blob_allocator);
  53.         if (top_blob.empty())
  54.             return -100;
  55.  
  56.         #pragma omp parallel for num_threads(opt.num_threads)
  57.         for (int p = 0; p < outc; p++)
  58.         {
  59.             const float* ptr = bottom_blob.channel(p % channels).row((p / channels) % 2) + ((p / channels) / 2);
  60.             float* outptr = top_blob.channel(p);
  61.  
  62.             for (int i = 0; i < outh; i++)
  63.             {
  64.                 for (int j = 0; j < outw; j++)
  65.                 {
  66.                     *outptr = *ptr;
  67.  
  68.                     outptr += 1;
  69.                     ptr += 2;
  70.                 }
  71.  
  72.                 ptr += w;
  73.             }
  74.         }
  75.  
  76.         return 0;
  77.     }
  78. };
  79.  
  80. DEFINE_LAYER_CREATOR(YoloV5Focus)
  81.  
  82. struct Object
  83. {
  84.     float x;
  85.     float y;
  86.     float w;
  87.     float h;
  88.     int label;
  89.     float prob;
  90. };
  91.  
  92. static inline float intersection_area(const Object& a, const Object& b)
  93. {
  94.     if (a.x > b.x + b.w || a.x + a.w < b.x || a.y > b.y + b.h || a.y + a.h < b.y)
  95.     {
  96.         // no intersection
  97.         return 0.f;
  98.     }
  99.  
  100.     float inter_width = std::min(a.x + a.w, b.x + b.w) - std::max(a.x, b.x);
  101.     float inter_height = std::min(a.y + a.h, b.y + b.h) - std::max(a.y, b.y);
  102.  
  103.     return inter_width * inter_height;
  104. }
  105.  
  106. static void qsort_descent_inplace(std::vector<Object>& faceobjects, int left, int right)
  107. {
  108.     int i = left;
  109.     int j = right;
  110.     float p = faceobjects[(left + right) / 2].prob;
  111.  
  112.     while (i <= j)
  113.     {
  114.         while (faceobjects[i].prob > p)
  115.             i++;
  116.  
  117.         while (faceobjects[j].prob < p)
  118.             j--;
  119.  
  120.         if (i <= j)
  121.         {
  122.             // swap
  123.             std::swap(faceobjects[i], faceobjects[j]);
  124.  
  125.             i++;
  126.             j--;
  127.         }
  128.     }
  129.  
  130.     #pragma omp parallel sections
  131.     {
  132.         #pragma omp section
  133.         {
  134.             if (left < j) qsort_descent_inplace(faceobjects, left, j);
  135.         }
  136.         #pragma omp section
  137.         {
  138.             if (i < right) qsort_descent_inplace(faceobjects, i, right);
  139.         }
  140.     }
  141. }
  142.  
  143. static void qsort_descent_inplace(std::vector<Object>& faceobjects)
  144. {
  145.     if (faceobjects.empty())
  146.         return;
  147.  
  148.     qsort_descent_inplace(faceobjects, 0, faceobjects.size() - 1);
  149. }
  150.  
  151. static void nms_sorted_bboxes(const std::vector<Object>& faceobjects, std::vector<int>& picked, float nms_threshold)
  152. {
  153.     picked.clear();
  154.  
  155.     const int n = faceobjects.size();
  156.  
  157.     std::vector<float> areas(n);
  158.     for (int i = 0; i < n; i++)
  159.     {
  160.         areas[i] = faceobjects[i].w * faceobjects[i].h;
  161.     }
  162.  
  163.     for (int i = 0; i < n; i++)
  164.     {
  165.         const Object& a = faceobjects[i];
  166.  
  167.         int keep = 1;
  168.         for (int j = 0; j < (int)picked.size(); j++)
  169.         {
  170.             const Object& b = faceobjects[picked[j]];
  171.  
  172.             // intersection over union
  173.             float inter_area = intersection_area(a, b);
  174.             float union_area = areas[i] + areas[picked[j]] - inter_area;
  175.             // float IoU = inter_area / union_area
  176.             if (inter_area / union_area > nms_threshold)
  177.                 keep = 0;
  178.         }
  179.  
  180.         if (keep)
  181.             picked.push_back(i);
  182.     }
  183. }
  184.  
  185. static inline float sigmoid(float x)
  186. {
  187.     return static_cast<float>(1.f / (1.f + exp(-x)));
  188. }
  189.  
  190. static void generate_proposals(const ncnn::Mat& anchors, int stride, const ncnn::Mat& in_pad, const ncnn::Mat& feat_blob, float prob_threshold, std::vector<Object>& objects)
  191. {
  192.     const int num_grid = feat_blob.h;
  193.  
  194.     int num_grid_x;
  195.     int num_grid_y;
  196.     if (in_pad.w > in_pad.h)
  197.     {
  198.         num_grid_x = in_pad.w / stride;
  199.         num_grid_y = num_grid / num_grid_x;
  200.     }
  201.     else
  202.     {
  203.         num_grid_y = in_pad.h / stride;
  204.         num_grid_x = num_grid / num_grid_y;
  205.     }
  206.  
  207.     const int num_class = feat_blob.w - 5;
  208.  
  209.     const int num_anchors = anchors.w / 2;
  210.  
  211.     for (int q = 0; q < num_anchors; q++)
  212.     {
  213.         const float anchor_w = anchors[q * 2];
  214.         const float anchor_h = anchors[q * 2 + 1];
  215.  
  216.         const ncnn::Mat feat = feat_blob.channel(q);
  217.  
  218.         for (int i = 0; i < num_grid_y; i++)
  219.         {
  220.             for (int j = 0; j < num_grid_x; j++)
  221.             {
  222.                 const float* featptr = feat.row(i * num_grid_x + j);
  223.  
  224.                 // find class index with max class score
  225.                 int class_index = 0;
  226.                 float class_score = -FLT_MAX;
  227.                 for (int k = 0; k < num_class; k++)
  228.                 {
  229.                     float score = featptr[5 + k];
  230.                     if (score > class_score)
  231.                     {
  232.                         class_index = k;
  233.                         class_score = score;
  234.                     }
  235.                 }
  236.  
  237.                 float box_score = featptr[4];
  238.  
  239.                 float confidence = sigmoid(box_score) * sigmoid(class_score);
  240.  
  241.                 if (confidence >= prob_threshold)
  242.                 {
  243.                     // yolov5/models/yolo.py Detect forward
  244.                     // y = x[i].sigmoid()
  245.                     // y[..., 0:2] = (y[..., 0:2] * 2. - 0.5 + self.grid[i].to(x[i].device)) * self.stride[i]  # xy
  246.                     // y[..., 2:4] = (y[..., 2:4] * 2) ** 2 * self.anchor_grid[i]  # wh
  247.  
  248.                     float dx = sigmoid(featptr[0]);
  249.                     float dy = sigmoid(featptr[1]);
  250.                     float dw = sigmoid(featptr[2]);
  251.                     float dh = sigmoid(featptr[3]);
  252.  
  253.                     float pb_cx = (dx * 2.f - 0.5f + j) * stride;
  254.                     float pb_cy = (dy * 2.f - 0.5f + i) * stride;
  255.  
  256.                     float pb_w = pow(dw * 2.f, 2) * anchor_w;
  257.                     float pb_h = pow(dh * 2.f, 2) * anchor_h;
  258.  
  259.                     float x0 = pb_cx - pb_w * 0.5f;
  260.                     float y0 = pb_cy - pb_h * 0.5f;
  261.                     float x1 = pb_cx + pb_w * 0.5f;
  262.                     float y1 = pb_cy + pb_h * 0.5f;
  263.  
  264.                     Object obj;
  265.                     obj.x = x0;
  266.                     obj.y = y0;
  267.                     obj.w = x1 - x0;
  268.                     obj.h = y1 - y0;
  269.                     obj.label = class_index;
  270.                     obj.prob = confidence;
  271.  
  272.                     objects.push_back(obj);
  273.                 }
  274.             }
  275.         }
  276.     }
  277. }
  278.  
  279. extern "C" {
  280.  
  281. // FIXME DeleteGlobalRef is missing for objCls
  282. static jclass objCls = NULL;
  283. static jmethodID constructortorId;
  284. static jfieldID xId;
  285. static jfieldID yId;
  286. static jfieldID wId;
  287. static jfieldID hId;
  288. static jfieldID labelId;
  289. static jfieldID probId;
  290.  
  291. JNIEXPORT jint JNI_OnLoad(JavaVM* vm, void* reserved)
  292. {
  293.     __android_log_print(ANDROID_LOG_DEBUG, "YoloV5Ncnn", "JNI_OnLoad");
  294.  
  295.     ncnn::create_gpu_instance();
  296.  
  297.     return JNI_VERSION_1_4;
  298. }
  299.  
  300. JNIEXPORT void JNI_OnUnload(JavaVM* vm, void* reserved)
  301. {
  302.     __android_log_print(ANDROID_LOG_DEBUG, "YoloV5Ncnn", "JNI_OnUnload");
  303.  
  304.     ncnn::destroy_gpu_instance();
  305. }
  306.  
  307. // public native boolean Init(AssetManager mgr);
  308. JNIEXPORT jboolean JNICALL Java_com_tencent_yolov5ncnn_YoloV5Ncnn_Init(JNIEnv* env, jobject thiz, jobject assetManager)
  309. {
  310.     ncnn::Option opt;
  311.     opt.lightmode = true;
  312.     opt.num_threads = 4;
  313.     opt.blob_allocator = &g_blob_pool_allocator;
  314.     opt.workspace_allocator = &g_workspace_pool_allocator;
  315.     opt.use_packing_layout = true;
  316.  
  317.     // use vulkan compute
  318.     if (ncnn::get_gpu_count() != 0)
  319.         opt.use_vulkan_compute = true;
  320.  
  321.     AAssetManager* mgr = AAssetManager_fromJava(env, assetManager);
  322.  
  323.     yolov5.opt = opt;
  324.  
  325.     yolov5.register_custom_layer("YoloV5Focus", YoloV5Focus_layer_creator);
  326.  
  327.     // init param
  328.     {
  329.         int ret = yolov5.load_param(mgr, "yolov5s.param");
  330.         if (ret != 0)
  331.         {
  332.             __android_log_print(ANDROID_LOG_DEBUG, "YoloV5Ncnn", "load_param failed");
  333.             return JNI_FALSE;
  334.         }
  335.     }
  336.  
  337.     // init bin
  338.     {
  339.         int ret = yolov5.load_model(mgr, "yolov5s.bin");
  340.         if (ret != 0)
  341.         {
  342.             __android_log_print(ANDROID_LOG_DEBUG, "YoloV5Ncnn", "load_model failed");
  343.             return JNI_FALSE;
  344.         }
  345.     }
  346.  
  347.     // init jni glue
  348.     jclass localObjCls = env->FindClass("com/tencent/yolov5ncnn/YoloV5Ncnn$Obj");
  349.     objCls = reinterpret_cast<jclass>(env->NewGlobalRef(localObjCls));
  350.  
  351.     constructortorId = env->GetMethodID(objCls, "<init>", "(Lcom/tencent/yolov5ncnn/YoloV5Ncnn;)V");
  352.  
  353.     xId = env->GetFieldID(objCls, "x", "F");
  354.     yId = env->GetFieldID(objCls, "y", "F");
  355.     wId = env->GetFieldID(objCls, "w", "F");
  356.     hId = env->GetFieldID(objCls, "h", "F");
  357.     labelId = env->GetFieldID(objCls, "label", "Ljava/lang/String;");
  358.     probId = env->GetFieldID(objCls, "prob", "F");
  359.  
  360.     return JNI_TRUE;
  361. }
  362.  
  363. // public native Obj[] Detect(Bitmap bitmap, boolean use_gpu);
  364. JNIEXPORT jobjectArray JNICALL Java_com_tencent_yolov5ncnn_YoloV5Ncnn_Detect(JNIEnv* env, jobject thiz, jobject bitmap, jboolean use_gpu)
  365. {
  366.     if (use_gpu == JNI_TRUE && ncnn::get_gpu_count() == 0)
  367.     {
  368.         return NULL;
  369.         //return env->NewStringUTF("no vulkan capable gpu");
  370.     }
  371.  
  372.     double start_time = ncnn::get_current_time();
  373.  
  374.     AndroidBitmapInfo info;
  375.     AndroidBitmap_getInfo(env, bitmap, &info);
  376.     const int width = info.width;
  377.     const int height = info.height;
  378.     if (info.format != ANDROID_BITMAP_FORMAT_RGBA_8888)
  379.         return NULL;
  380.  
  381.     // ncnn from bitmap
  382.     const int target_size = 640;
  383.  
  384.     // letterbox pad to multiple of 32
  385.     int w = width;
  386.     int h = height;
  387.     float scale = 1.f;
  388.     if (w > h)
  389.     {
  390.         scale = (float)target_size / w;
  391.         w = target_size;
  392.         h = h * scale;
  393.     }
  394.     else
  395.     {
  396.         scale = (float)target_size / h;
  397.         h = target_size;
  398.         w = w * scale;
  399.     }
  400.  
  401.     ncnn::Mat in = ncnn::Mat::from_android_bitmap_resize(env, bitmap, ncnn::Mat::PIXEL_RGB, w, h);
  402.  
  403.     // pad to target_size rectangle
  404.     // yolov5/utils/datasets.py letterbox
  405.     int wpad = (w + 31) / 32 * 32 - w;
  406.     int hpad = (h + 31) / 32 * 32 - h;
  407.     ncnn::Mat in_pad;
  408.     ncnn::copy_make_border(in, in_pad, hpad / 2, hpad - hpad / 2, wpad / 2, wpad - wpad / 2, ncnn::BORDER_CONSTANT, 114.f);
  409.  
  410.     // yolov5
  411.     std::vector<Object> objects;
  412.     {
  413.         const float prob_threshold = 0.25f;
  414.         const float nms_threshold = 0.45f;
  415.  
  416.         const float norm_vals[3] = {1 / 255.f, 1 / 255.f, 1 / 255.f};
  417.         in_pad.substract_mean_normalize(0, norm_vals);
  418.  
  419.         ncnn::Extractor ex = yolov5.create_extractor();
  420.  
  421.         ex.set_vulkan_compute(use_gpu);
  422.  
  423.         ex.input("images", in_pad);
  424.  
  425.         std::vector<Object> proposals;
  426.  
  427.         // anchor setting from yolov5/models/yolov5s.yaml
  428.  
  429.         // stride 8
  430.         {
  431.             ncnn::Mat out;
  432.             ex.extract("output", out);
  433.  
  434.             ncnn::Mat anchors(6);
  435.             anchors[0] = 10.f;
  436.             anchors[1] = 13.f;
  437.             anchors[2] = 16.f;
  438.             anchors[3] = 30.f;
  439.             anchors[4] = 33.f;
  440.             anchors[5] = 23.f;
  441.  
  442.             std::vector<Object> objects8;
  443.             generate_proposals(anchors, 8, in_pad, out, prob_threshold, objects8);
  444.  
  445.             proposals.insert(proposals.end(), objects8.begin(), objects8.end());
  446.         }
  447.  
  448.         // stride 16
  449.         {
  450.             ncnn::Mat out;
  451.             ex.extract("365", out);
  452.  
  453.             ncnn::Mat anchors(6);
  454.             anchors[0] = 30.f;
  455.             anchors[1] = 61.f;
  456.             anchors[2] = 62.f;
  457.             anchors[3] = 45.f;
  458.             anchors[4] = 59.f;
  459.             anchors[5] = 119.f;
  460.  
  461.             std::vector<Object> objects16;
  462.             generate_proposals(anchors, 16, in_pad, out, prob_threshold, objects16);
  463.  
  464.             proposals.insert(proposals.end(), objects16.begin(), objects16.end());
  465.         }
  466.  
  467.         // stride 32
  468.         {
  469.             ncnn::Mat out;
  470.             ex.extract("385", out);
  471.  
  472.             ncnn::Mat anchors(6);
  473.             anchors[0] = 116.f;
  474.             anchors[1] = 90.f;
  475.             anchors[2] = 156.f;
  476.             anchors[3] = 198.f;
  477.             anchors[4] = 373.f;
  478.             anchors[5] = 326.f;
  479.  
  480.             std::vector<Object> objects32;
  481.             generate_proposals(anchors, 32, in_pad, out, prob_threshold, objects32);
  482.  
  483.             proposals.insert(proposals.end(), objects32.begin(), objects32.end());
  484.         }
  485.  
  486.         // sort all proposals by score from highest to lowest
  487.         qsort_descent_inplace(proposals);
  488.  
  489.         // apply nms with nms_threshold
  490.         std::vector<int> picked;
  491.         nms_sorted_bboxes(proposals, picked, nms_threshold);
  492.  
  493.         int count = picked.size();
  494.  
  495.         objects.resize(count);
  496.         for (int i = 0; i < count; i++)
  497.         {
  498.             objects[i] = proposals[picked[i]];
  499.  
  500.             // adjust offset to original unpadded
  501.             float x0 = (objects[i].x - (wpad / 2)) / scale;
  502.             float y0 = (objects[i].y - (hpad / 2)) / scale;
  503.             float x1 = (objects[i].x + objects[i].w - (wpad / 2)) / scale;
  504.             float y1 = (objects[i].y + objects[i].h - (hpad / 2)) / scale;
  505.  
  506.             // clip
  507.             x0 = std::max(std::min(x0, (float)(width - 1)), 0.f);
  508.             y0 = std::max(std::min(y0, (float)(height - 1)), 0.f);
  509.             x1 = std::max(std::min(x1, (float)(width - 1)), 0.f);
  510.             y1 = std::max(std::min(y1, (float)(height - 1)), 0.f);
  511.  
  512.             objects[i].x = x0;
  513.             objects[i].y = y0;
  514.             objects[i].w = x1 - x0;
  515.             objects[i].h = y1 - y0;
  516.         }
  517.     }
  518.  
  519.     // objects to Obj[]
  520.     static const char* class_names[] = {
  521.             "bag", "task_info_area", "task_single_trigger", "task_success"
  522.     };
  523.  
  524.     jobjectArray jObjArray = env->NewObjectArray(objects.size(), objCls, NULL);
  525.  
  526.     for (size_t i=0; i<objects.size(); i++)
  527.     {
  528.         jobject jObj = env->NewObject(objCls, constructortorId, thiz);
  529.  
  530.         env->SetFloatField(jObj, xId, objects[i].x);
  531.         env->SetFloatField(jObj, yId, objects[i].y);
  532.         env->SetFloatField(jObj, wId, objects[i].w);
  533.         env->SetFloatField(jObj, hId, objects[i].h);
  534.         env->SetObjectField(jObj, labelId, env->NewStringUTF(class_names[objects[i].label]));
  535.         env->SetFloatField(jObj, probId, objects[i].prob);
  536.  
  537.         env->SetObjectArrayElement(jObjArray, i, jObj);
  538.     }
  539.  
  540.     double elasped = ncnn::get_current_time() - start_time;
  541.     __android_log_print(ANDROID_LOG_DEBUG, "YoloV5Ncnn", "%.2fms   detect", elasped);
  542.  
  543.     return jObjArray;
  544. }
  545.  
  546. }
  1. 注意 432、451、470行，在第 “二” 步中，“修改配置”步骤中Permute的值，将三处参数依次修改为上同参数值，即 432行修改为 **ex.extract("output", out);** 451行修改为 **ex.extract("365", out);** 470行修改为"ex.extract("385", out);"。
  2. 修改521行列表，与样本集中 labels下的classes.txt列表对应。
  3. 修改351行 constructortorId = env->GetMethodID(objCls, "", "(Lcom/tencent/yolov5ncnn/YoloV5Ncnn;)V");，将“com/tencent/yolov5ncnn/YoloV5Ncnn” 修改为自定义java类，注意前部的“L”和尾部的“V”保留（或者直接按照此包结构在模块下的src/main/java下创建此包和类文件）
  4. 修改348行 jclass localObjCls = env->FindClass("com/tencent/yolov5ncnn/YoloV5Ncnn$Obj"); 中 "com/tencent/yolov5ncnn/YoloV5Ncnn$Obj" 为自定义java类，同上，但不可丢掉后面的$Obj。
  • 创建反射类
  1. // Tencent is pleased to support the open source community by making ncnn available.
  2. //
  3. // Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved.
  4. //
  5. // Licensed under the BSD 3-Clause License (the "License"); you may not use this file except
  6. // in compliance with the License. You may obtain a copy of the License at
  7. //
  8. // https://opensource.org/licenses/BSD-3-Clause
  9. //
  10. // Unless required by applicable law or agreed to in writing, software distributed
  11. // under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
  12. // CONDITIONS OF ANY KIND, either express or implied. See the License for the
  13. // specific language governing permissions and limitations under the License.
  14.  
  15. package com.tencent.yolov5ncnn;
  16.  
  17. import android.content.res.AssetManager;
  18. import android.graphics.Bitmap;
  19.  
  20. public class YoloV5Ncnn
  21. {
  22.     public native boolean Init(AssetManager mgr);
  23.  
  24.     public class Obj {
  25.         // x
  26.         public float x;
  27.         // y
  28.         public float y;
  29.         // w
  30.         public float w;
  31.         // h
  32.         public float h;
  33.         // 标签
  34.         public String label;
  35.         // 置信度
  36.         public float prob;
  37.  
  38.         @Override
  39.         public String toString() {
  40.             return "Obj{" +
  41.                     "x=" + x +
  42.                     ", y=" + y +
  43.                     ", w=" + w +
  44.                     ", h=" + h +
  45.                     ", label='" + label + '\'' +
  46.                     ", prob=" + prob +
  47.                     '}';
  48.         }
  49.     }
  50.  
  51.     public native Obj[] Detect(Bitmap bitmap, boolean use_gpu);
  52.  
  53.     static {
  54.         // 为CMakeLists.txt中配置的project名称
  55.         System.loadLibrary("yolov5ncnn");
  56.     }
  57. }
  • 修改模块的build.grade配置
  1. plugins {
  2.     id 'com.android.library'
  3. }
  4.  
  5. android {
  6.     namespace 'cn.tiktok.ncnn'
  7.     compileSdk 24
  8.  
  9.     defaultConfig {
  10.         minSdk 24
  11.         archivesBaseName = "$applicationId"
  12.         testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
  13.         consumerProguardFiles "consumer-rules.pro"
  14.         externalNativeBuild {
  15.             cmake {
  16.                 cppFlags "-std=c++11"
  17.             }
  18.         }
  19.         ndk {
  20.             // 可减少平台支持，减小包体积
  21.             abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
  22.         }
  23.     }
  24.  
  25.     buildTypes {
  26.         release {
  27.             minifyEnabled false
  28.             proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
  29.         }
  30.     }
  31.     externalNativeBuild {
  32.         cmake {
  33.             // 此处指向CMakeLists.txt 所在文件夹
  34.             path "src/main/jni/CMakeLists.txt"
  35.             version "3.10.2"
  36.         }
  37.     }
  38.     compileOptions {
  39.         sourceCompatibility JavaVersion.VERSION_1_8
  40.         targetCompatibility JavaVersion.VERSION_1_8
  41.     }
  42. }
  43.  
  44. dependencies {
  45.     implementation 'androidx.appcompat:appcompat:1.4.1'
  46.     implementation 'com.google.android.material:material:1.5.0'
  47.     testImplementation 'junit:junit:4.13.2'
  48.     androidTestImplementation 'androidx.test.ext:junit:1.1.3'
  49.     androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
  50. }
  • 部署模型

在模块的src/main目录下创建 assets 文件夹，然后检查文件夹属性。如果被创建为了普通文件夹，如下图所示。

则在此文件夹上右键，选择 Mark Directory as，选择Resources root。

我们在第“二”步的“检出为ncnn可用模型”步骤，生成了 yolov5s.param 和 yolov5s.bin两个文件，将两个文件复制到assets文件夹下。

然后在app中引入此模块，并使用，注意修改为模块名称

 最后选择Android Studio菜单栏 File -> Sync Project wth Gradle FIles，等待执行完毕。

四、测试使用

  1. public class MainActivity extends Activity{
  2.     private YoloV5Ncnn yolov5ncnn = new YoloV5Ncnn();
  3.     
  4.     /** Called when the activity is first created. */
  5.     @Override
  6.     public void onCreate(Bundle savedInstanceState){
  7.         super.onCreate(savedInstanceState);
  8.         // 初始化
  9.         boolean ret_init = yolov5ncnn.Init(getAssets());
  10.         if (!ret_init){
  11.             Log.e("MainActivity", "yolov5ncnn Init failed");
  12.         }
  13.         // 图片、是否使用GPU
  14.         YoloV5Ncnn.Obj[] objects = yolov5ncnn.Detect(imageBitmap, false);
  15.         // 根据注释解析YoloV5Ncnn.Obj
  16.         ...
  17.     }
  18. }

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/2023面试高手/article/detail/569618
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号