devbox
小蓝xlanll
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

手把手教学YOLOV5在RK3568的部署应用及代码实现

作者：小蓝xlanll | 2024-03-13 16:40:56

赞

踩

手把手教学YOLOV5在RK3568的部署应用及代码实现

引言

RK3568支持NPU，提供0.8Tops的算力，可以用于部署深度学习项目。本篇文章介绍Yolov5代码开发、模型转化、部署。

RKNN-Toolkit2环境安装

RKNN-Toolkit2是用来把pytorch、tf等训练模型导出为rknn模型，供后续NPU加速使用。

1.RKNN-Toolkit2下载

下载地址：

https://github.com/rockchip-linux/rknn-toolkit2

2.安装

建议使用conda虚拟环境，找到对应的packages进行安装RKNN-Toolkit2，具体参考doc/Rockchip_Quick_Start_RKNN_Toolkit2_CN-1.4.0.pdf

packages地址：https://github.com/rockchip-linux/rknn-toolkit2/tree/master/packages

doc地址：https://github.com/rockchip-linux/rknn-toolkit2/tree/master/doc

安装部署

安装所需python版本的conda虚拟环境

安装相关依赖

sudo apt-get install libxslt1-dev zlib1g zlib1g-dev libglib2.0-0 libsm6 \ libgl1-mesa-glx libprotobuf-dev gcc

3.获取 RKNN-Toolkit2 安装包，然后执行以下步骤：

安装 Python 依赖：

pip3 install -r doc/requirements*.txt

进入 package 目录：

cd package/

安装 RKNN-Toolkit2

sudo pip3 install rknn_toolkit2*.whl

检查 RKNN-Toolkit2 是否安装成功

  1.  python3
  2.  from rknn.api import RKNN
  3.

如果导入 RKNN 模块没有失败，说明安装成功。

YOLOV5模型转化为RKNN模型

YOLOV5代码下载：

git clone https://github.com/ultralytics/yolov5.git

2.修改models/yolo.py--line56行

  1.     def forward(self, x):
  2.         z = []  # inference output
  3.         for i in range(self.nl):
  4.             x[i] = self.m[i](x[i])  # conv
  5.             bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
  6.             if self.export:
  7.                return x
  8.             x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
  9.  
  10.             if not self.training:  # inference
  11.                 if self.dynamic or self.grid[i].shape[2:4] != x[i].shape[2:4]:
  12.                     self.grid[i], self.anchor_grid[i] = self._make_grid(nx, ny, i)
  13.  
  14.                 if isinstance(self, Segment):  # (boxes + masks)
  15.                     xy, wh, conf, mask = x[i].split((2, 2, self.nc + 1, self.no - self.nc - 5), 4)
  16.                     xy = (xy.sigmoid() * 2 + self.grid[i]) * self.stride[i]  # xy
  17.                     wh = (wh.sigmoid() * 2) ** 2 * self.anchor_grid[i]  # wh
  18.                     y = torch.cat((xy, wh, conf.sigmoid(), mask), 4)
  19.                 else:  # Detect (boxes only)
  20.                     xy, wh, conf = x[i].sigmoid().split((2, 2, self.nc + 1), 4)
  21.                     xy = (xy * 2 + self.grid[i]) * self.stride[i]  # xy
  22.                     wh = (wh * 2) ** 2 * self.anchor_grid[i]  # wh
  23.                     y = torch.cat((xy, wh, conf), 4)
  24.                 z.append(y.view(bs, self.na * nx * ny, self.no))
  25.  
  26.         return x if self.training else (torch.cat(z, 1),) if self.export else (torch.cat(z, 1), x)

3.onnx模型导出

python export.py --weights xxx.pt --include onnx --simplify --opset 12

4.onnx2rknn

  1. import os
  2. import urllib
  3. import traceback
  4. import time
  5. import sys
  6. import numpy as np
  7. import cv2
  8. from rknn.api import RKNN
  9.  
  10. ONNX_MODEL = 'yolov5s.onnx'
  11. RKNN_MODEL = 'yolov5s.rknn'
  12. IMG_PATH = './bus.jpg'
  13. DATASET = './dataset.txt'
  14.  
  15. QUANTIZE_ON = True
  16.  
  17. OBJ_THRESH = 0.25
  18. NMS_THRESH = 0.45
  19. IMG_SIZE = 640
  20.  
  21. CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light",
  22.            "fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant",
  23.            "bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite",
  24.            "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ",
  25.            "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa",
  26.            "pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ",
  27.            "oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")
  28.  
  29.  
  30. def sigmoid(x):
  31.     return 1 / (1 + np.exp(-x))
  32.  
  33.  
  34. def xywh2xyxy(x):
  35.     # Convert [x, y, w, h] to [x1, y1, x2, y2]
  36.     y = np.copy(x)
  37.     y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left x
  38.     y[:, 1] = x[:, 1] - x[:, 3] / 2  # top left y
  39.     y[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right x
  40.     y[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right y
  41.     return y
  42.  
  43.  
  44. def process(input, mask, anchors):
  45.  
  46.     anchors = [anchors[i] for i in mask]
  47.     grid_h, grid_w = map(int, input.shape[0:2])
  48.  
  49.     box_confidence = sigmoid(input[..., 4])
  50.     box_confidence = np.expand_dims(box_confidence, axis=-1)
  51.  
  52.     box_class_probs = sigmoid(input[..., 5:])
  53.  
  54.     box_xy = sigmoid(input[..., :2])*2 - 0.5
  55.  
  56.     col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)
  57.     row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)
  58.     col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  59.     row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  60.     grid = np.concatenate((col, row), axis=-1)
  61.     box_xy += grid
  62.     box_xy *= int(IMG_SIZE/grid_h)
  63.  
  64.     box_wh = pow(sigmoid(input[..., 2:4])*2, 2)
  65.     box_wh = box_wh * anchors
  66.  
  67.     box = np.concatenate((box_xy, box_wh), axis=-1)
  68.  
  69.     return box, box_confidence, box_class_probs
  70.  
  71.  
  72. def filter_boxes(boxes, box_confidences, box_class_probs):
  73.     """Filter boxes with box threshold. It's a bit different with origin yolov5 post process!
  74.     # Arguments
  75.         boxes: ndarray, boxes of objects.
  76.         box_confidences: ndarray, confidences of objects.
  77.         box_class_probs: ndarray, class_probs of objects.
  78.     # Returns
  79.         boxes: ndarray, filtered boxes.
  80.         classes: ndarray, classes for boxes.
  81.         scores: ndarray, scores for boxes.
  82.     """
  83.     boxes = boxes.reshape(-1, 4)
  84.     box_confidences = box_confidences.reshape(-1)
  85.     box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])
  86.  
  87.     _box_pos = np.where(box_confidences >= OBJ_THRESH)
  88.     boxes = boxes[_box_pos]
  89.     box_confidences = box_confidences[_box_pos]
  90.     box_class_probs = box_class_probs[_box_pos]
  91.  
  92.     class_max_score = np.max(box_class_probs, axis=-1)
  93.     classes = np.argmax(box_class_probs, axis=-1)
  94.     _class_pos = np.where(class_max_score >= OBJ_THRESH)
  95.  
  96.     boxes = boxes[_class_pos]
  97.     classes = classes[_class_pos]
  98.     scores = (class_max_score* box_confidences)[_class_pos]
  99.  
  100.     return boxes, classes, scores
  101.  
  102.  
  103. def nms_boxes(boxes, scores):
  104.     """Suppress non-maximal boxes.
  105.     # Arguments
  106.         boxes: ndarray, boxes of objects.
  107.         scores: ndarray, scores of objects.
  108.     # Returns
  109.         keep: ndarray, index of effective boxes.
  110.     """
  111.     x = boxes[:, 0]
  112.     y = boxes[:, 1]
  113.     w = boxes[:, 2] - boxes[:, 0]
  114.     h = boxes[:, 3] - boxes[:, 1]
  115.  
  116.     areas = w * h
  117.     order = scores.argsort()[::-1]
  118.  
  119.     keep = []
  120.     while order.size > 0:
  121.         i = order[0]
  122.         keep.append(i)
  123.  
  124.         xx1 = np.maximum(x[i], x[order[1:]])
  125.         yy1 = np.maximum(y[i], y[order[1:]])
  126.         xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])
  127.         yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])
  128.  
  129.         w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)
  130.         h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)
  131.         inter = w1 * h1
  132.  
  133.         ovr = inter / (areas[i] + areas[order[1:]] - inter)
  134.         inds = np.where(ovr <= NMS_THRESH)[0]
  135.         order = order[inds + 1]
  136.     keep = np.array(keep)
  137.     return keep
  138.  
  139.  
  140. def yolov5_post_process(input_data):
  141.     masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
  142.     anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],
  143.                [59, 119], [116, 90], [156, 198], [373, 326]]
  144.  
  145.     boxes, classes, scores = [], [], []
  146.     for input, mask in zip(input_data, masks):
  147.         b, c, s = process(input, mask, anchors)
  148.         b, c, s = filter_boxes(b, c, s)
  149.         boxes.append(b)
  150.         classes.append(c)
  151.         scores.append(s)
  152.  
  153.     boxes = np.concatenate(boxes)
  154.     boxes = xywh2xyxy(boxes)
  155.     classes = np.concatenate(classes)
  156.     scores = np.concatenate(scores)
  157.  
  158.     nboxes, nclasses, nscores = [], [], []
  159.     for c in set(classes):
  160.         inds = np.where(classes == c)
  161.         b = boxes[inds]
  162.         c = classes[inds]
  163.         s = scores[inds]
  164.  
  165.         keep = nms_boxes(b, s)
  166.  
  167.         nboxes.append(b[keep])
  168.         nclasses.append(c[keep])
  169.         nscores.append(s[keep])
  170.  
  171.     if not nclasses and not nscores:
  172.         return None, None, None
  173.  
  174.     boxes = np.concatenate(nboxes)
  175.     classes = np.concatenate(nclasses)
  176.     scores = np.concatenate(nscores)
  177.  
  178.     return boxes, classes, scores
  179.  
  180.  
  181. def draw(image, boxes, scores, classes):
  182.     """Draw the boxes on the image.
  183.     # Argument:
  184.         image: original image.
  185.         boxes: ndarray, boxes of objects.
  186.         classes: ndarray, classes of objects.
  187.         scores: ndarray, scores of objects.
  188.         all_classes: all classes name.
  189.     """
  190.     for box, score, cl in zip(boxes, scores, classes):
  191.         top, left, right, bottom = box
  192.         print('class: {}, score: {}'.format(CLASSES[cl], score))
  193.         print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))
  194.         top = int(top)
  195.         left = int(left)
  196.         right = int(right)
  197.         bottom = int(bottom)
  198.  
  199.         cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)
  200.         cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),
  201.                     (top, left - 6),
  202.                     cv2.FONT_HERSHEY_SIMPLEX,
  203.                     0.6, (0, 0, 255), 2)
  204.  
  205.  
  206. def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
  207.     # Resize and pad image while meeting stride-multiple constraints
  208.     shape = im.shape[:2]  # current shape [height, width]
  209.     if isinstance(new_shape, int):
  210.         new_shape = (new_shape, new_shape)
  211.  
  212.     # Scale ratio (new / old)
  213.     r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
  214.  
  215.     # Compute padding
  216.     ratio = r, r  # width, height ratios
  217.     new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
  218.     dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh padding
  219.  
  220.     dw /= 2  # divide padding into 2 sides
  221.     dh /= 2
  222.  
  223.     if shape[::-1] != new_unpad:  # resize
  224.         im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
  225.     top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
  226.     left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
  227.     im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add border
  228.     return im, ratio, (dw, dh)
  229.  
  230.  
  231. if __name__ == '__main__':
  232.  
  233.     # Create RKNN object
  234.     rknn = RKNN(verbose=True)
  235.  
  236.     # pre-process config
  237.     print('--> Config model')
  238.     rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]])
  239.     print('done')
  240.  
  241.     # Load ONNX model
  242.     print('--> Loading model')
  243.     ret = rknn.load_onnx(model=ONNX_MODEL)
  244.     if ret != 0:
  245.         print('Load model failed!')
  246.         exit(ret)
  247.     print('done')
  248.  
  249.     # Build model
  250.     print('--> Building model')
  251.     ret = rknn.build(do_quantization=QUANTIZE_ON, dataset=DATASET)
  252.     if ret != 0:
  253.         print('Build model failed!')
  254.         exit(ret)
  255.     print('done')
  256.  
  257.     # Export RKNN model
  258.     print('--> Export rknn model')
  259.     ret = rknn.export_rknn(RKNN_MODEL)
  260.     if ret != 0:
  261.         print('Export rknn model failed!')
  262.         exit(ret)
  263.     print('done')
  264.  
  265.     # Init runtime environment
  266.     print('--> Init runtime environment')
  267.     ret = rknn.init_runtime()
  268.     # ret = rknn.init_runtime('rk3566')
  269.     if ret != 0:
  270.         print('Init runtime environment failed!')
  271.         exit(ret)
  272.     print('done')
  273.  
  274.     # Set inputs
  275.     img = cv2.imread(IMG_PATH)
  276.     # img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))
  277.     img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  278.     img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
  279.  
  280.     # Inference
  281.     print('--> Running model')
  282.     outputs = rknn.inference(inputs=[img])
  283.     np.save('./onnx_yolov5_0.npy', outputs[0])
  284.     np.save('./onnx_yolov5_1.npy', outputs[1])
  285.     np.save('./onnx_yolov5_2.npy', outputs[2])
  286.     print('done')
  287.  
  288.     # post process
  289.     input0_data = outputs[0]
  290.     input1_data = outputs[1]
  291.     input2_data = outputs[2]
  292.  
  293.     input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))
  294.     input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))
  295.     input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))
  296.  
  297.     input_data = list()
  298.     input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))
  299.     input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))
  300.     input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))
  301.  
  302.     boxes, classes, scores = yolov5_post_process(input_data)
  303.  
  304.     img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
  305.     if boxes is not None:
  306.         draw(img_1, boxes, scores, classes)
  307.     # show output
  308.     # cv2.imshow("post process result", img_1)
  309.     # cv2.waitKey(0)
  310.     # cv2.destroyAllWindows()
  311.  
  312.     rknn.release()

至此YOLOV5在RK3568部署完成，但推理部分为python端

C++代码Inference

  1. #include "zql-app-yolov5.hpp"
  2. #include <atomic>
  3. #include <mutex>
  4. #include <queue>
  5. #include <condition_variable>
  6. #include "zql-producer.hpp"
  7. #include "zql-infer-rknn.hpp"
  8. #include "zql-logger.hpp"
  9.  
  10. namespace yolov5
  11. {
  12.     using namespace std;
  13.     using namespace cv;
  14.     using namespace zql;
  15.     const int anchor0[6] = {10, 13, 16, 30, 33, 23};
  16.     const int anchor1[6] = {30, 61, 62, 45, 59, 119};
  17.     const int anchor2[6] = {116, 90, 156, 198, 373, 326};
  18.     // const int anchor0[6] = {12, 16, 19, 36, 40, 28};
  19.     // const int anchor1[6] = {36, 75, 76, 55, 72, 146};
  20.     // const int anchor2[6] = {142, 110, 192, 243, 459, 401};
  21.     //       - [12,16, 19,36, 40,28]  # P3/8
  22.     //   - [36,75, 76,55, 72,146]  # P4/16
  23.     //   - [142,110, 192,243, 459,401]  # P5/32
  24.     static float iou(const zql::Box &a, const zql::Box &b)
  25.     {
  26.         float cleft = max(a.left, b.left);
  27.         float ctop = max(a.top, b.top);
  28.         float cright = min(a.right, b.right);
  29.         float cbottom = min(a.bottom, b.bottom);
  30.  
  31.         float c_area = max(cright - cleft, 0.0f) * max(cbottom - ctop, 0.0f);
  32.         if (c_area == 0.0f)
  33.             return 0.0f;
  34.  
  35.         float a_area = max(0.0f, a.right - a.left) * max(0.0f, a.bottom - a.top);
  36.         float b_area = max(0.0f, b.right - b.left) * max(0.0f, b.bottom - b.top);
  37.         return c_area / (a_area + b_area - c_area);
  38.     }
  39.     static void cpu_nms(zql::BoxArray &boxes, zql::BoxArray &output, float threshold)
  40.     {
  41.  
  42.         std::sort(boxes.begin(), boxes.end(), [](zql::BoxArray::const_reference a, zql::BoxArray::const_reference b)
  43.                   { return a.confidence > b.confidence; });
  44.  
  45.         output.clear();
  46.         vector<bool> remove_flags(boxes.size());
  47.         for (int i = 0; i < boxes.size(); ++i)
  48.         {
  49.  
  50.             if (remove_flags[i])
  51.                 continue;
  52.  
  53.             auto &a = boxes[i];
  54.             output.emplace_back(a);
  55.  
  56.             for (int j = i + 1; j < boxes.size(); ++j)
  57.             {
  58.                 if (remove_flags[j])
  59.                     continue;
  60.  
  61.                 auto &b = boxes[j];
  62.                 if (b.class_label == a.class_label)
  63.                 {
  64.                     if (iou(a, b) >= threshold)
  65.                         remove_flags[j] = true;
  66.                 }
  67.             }
  68.         }
  69.     }
  70.     static float sigmoid(float x)
  71.     {
  72.         return 1.0 / (1.0 + expf(-x));
  73.     }
  74.  
  75.     static float unsigmoid(float y)
  76.     {
  77.         return -1.0 * logf((1.0 / y) - 1.0);
  78.     }
  79.  
  80.     inline static int32_t __clip(float val, float min, float max)
  81.     {
  82.         float f = val <= min ? min : (val >= max ? max : val);
  83.         return f;
  84.     }
  85.  
  86.     static int8_t qnt_f32_to_affine(float f32, int32_t zp, float scale)
  87.     {
  88.         float dst_val = (f32 / scale) + zp;
  89.         int8_t res = (int8_t)__clip(dst_val, -128, 127);
  90.         return res;
  91.     }
  92.  
  93.     static float deqnt_affine_to_f32(int8_t qnt, int32_t zp, float scale) { return ((float)qnt - (float)zp) * scale; }
  94.  
  95.     static int process(int8_t *input, int *anchor, int grid_h, int grid_w, int height, int width, int stride,
  96.                        std::vector<float> &boxes, std::vector<float> &objProbs, std::vector<int> &classId,
  97.                        float threshold, int32_t zp, float scale, int num_class)
  98.     {
  99.  
  100.         int validCount = 0;
  101.         int grid_len = grid_h * grid_w;
  102.         float thres = unsigmoid(threshold);
  103.         int8_t thres_u8 = qnt_f32_to_affine(thres, zp, scale);
  104.  
  105.         for (int a = 0; a < 3; a++)
  106.         {
  107.             for (int i = 0; i < grid_h; i++)
  108.             {
  109.                 for (int j = 0; j < grid_w; j++)
  110.                 {
  111.                     int8_t box_confidence = input[(num_class * a + 4) * grid_len + i * grid_w + j];
  112.                     if (box_confidence >= thres_u8)
  113.                     {
  114.                         int offset = (num_class * a) * grid_len + i * grid_w + j;
  115.                         int8_t *in_ptr = input + offset;
  116.                         int8_t maxClassProbs = in_ptr[5 * grid_len];
  117.                         int maxClassId = 0;
  118.                         for (int k = 1; k < num_class - 5; ++k)
  119.                         {
  120.                             int8_t prob = in_ptr[(5 + k) * grid_len];
  121.                             if (prob > maxClassProbs)
  122.                             {
  123.                                 maxClassId = k;
  124.                                 maxClassProbs = prob;
  125.                             }
  126.                         }
  127.                         float final_score = sigmoid(deqnt_affine_to_f32(maxClassProbs, zp, scale)) * sigmoid(deqnt_affine_to_f32(box_confidence, zp, scale));
  128.  
  129.                         if (final_score > threshold)
  130.                         {
  131.                             float box_x = sigmoid(deqnt_affine_to_f32(*in_ptr, zp, scale)) * 2.0 - 0.5;
  132.                             float box_y = sigmoid(deqnt_affine_to_f32(in_ptr[grid_len], zp, scale)) * 2.0 - 0.5;
  133.                             float box_w = sigmoid(deqnt_affine_to_f32(in_ptr[2 * grid_len], zp, scale)) * 2.0;
  134.                             float box_h = sigmoid(deqnt_affine_to_f32(in_ptr[3 * grid_len], zp, scale)) * 2.0;
  135.                             box_x = (box_x + j) * (float)stride;
  136.                             box_y = (box_y + i) * (float)stride;
  137.                             box_w = box_w * box_w * (float)anchor[a * 2];
  138.                             box_h = box_h * box_h * (float)anchor[a * 2 + 1];
  139.                             box_x -= (box_w / 2.0);
  140.                             box_y -= (box_h / 2.0);
  141.                             boxes.emplace_back(box_x);
  142.                             boxes.emplace_back(box_y);
  143.                             boxes.emplace_back(box_w);
  144.                             boxes.emplace_back(box_h);
  145.  
  146.                             objProbs.emplace_back(final_score);
  147.                             classId.emplace_back(maxClassId);
  148.                             validCount++;
  149.                         }
  150.                     }
  151.                 }
  152.             }
  153.         }
  154.         return validCount;
  155.     }
  156.  
  157.     inline static int clamp(float val, int min, int max)
  158.     {
  159.         return val > min ? (val < max ? val : max) : min;
  160.     }
  161.  
  162.     using ControllerImpl = zql::Producer<
  163.         Mat,                            // input
  164.         zql::BoxArray,                  // output
  165.         std::tuple<std::string, string> // start param
  166.         >;
  167.  
  168.     class InferImpl : public Infer, public ControllerImpl
  169.     {
  170.     public:
  171.         /** 要求在InferImpl里面执行stop，而不是在基类执行stop **/
  172.         virtual ~InferImpl()
  173.         {
  174.             stop();
  175.         }
  176.  
  177.         virtual bool startup(
  178.             const std::string &engine_file, const std::string &engine_label,
  179.             float confidence_threshold = 0.25f, float nms_threshold = 0.5f)
  180.         {
  181.             confidence_threshold_ = confidence_threshold;
  182.             nms_threshold_ = nms_threshold;
  183.             std::tuple<string, string> param_ = make_tuple(engine_file, engine_label);
  184.             return ControllerImpl::startup(param_);
  185.         }
  186.  
  187.         virtual void worker(promise<bool> &result) override
  188.         {
  189.  
  190.             string model_file = std::get<0>(start_param_);
  191.             string lable_file = std::get<1>(start_param_);
  192.             bool use_float = false;
  193.             auto engine = rknn::load_infer(model_file, lable_file, use_float);
  194.             if (engine == nullptr)
  195.             {
  196.                 INFOE("Engine %s load failed", model_file.c_str());
  197.                 result.set_value(false);
  198.                 return;
  199.             }
  200.             engine->print();
  201.             auto input = engine->input();
  202.             num_class = engine->output()->size(1) / 3;
  203.             input_width_ = input->size(2);
  204.             input_height_ = input->size(1);
  205.             m_lables = engine->get_lables();
  206.             result.set_value(true);
  207.  
  208.             Job fetch_job;
  209.             zql::BoxArray keep_output_objs, output_objs, proposals;
  210.             cv::Size target_size(input_width_, input_height_);
  211.             // cv::Mat input_image(input_height_, input_width_, CV_8UC3, input->cpu());
  212.  
  213.             keep_output_objs.reserve(100);
  214.             output_objs.reserve(100);
  215.  
  216.             float sx = 0;
  217.             float sy = 0;
  218.             while (get_job_and_wait(fetch_job))
  219.             {
  220.  
  221.  
  222.                 float scale_letterbox;
  223.                 int resize_rows;
  224.                 int resize_cols;
  225.                 std::vector<float> filterBoxes;
  226.                 std::vector<float> objProbs;
  227.                 std::vector<int> classId;
  228.                 int img_width = fetch_job.input.cols;
  229.                 int img_height = fetch_job.input.rows;
  230.                 if (fetch_job.input.size() != target_size)
  231.                 {
  232.  
  233.                     if ((input_height_ * 1.0 / img_height) < (input_width_ * 1.0 / img_width))
  234.                     {
  235.                         scale_letterbox = input_height_ * 1.0 / img_height;
  236.                     }
  237.                     else
  238.                     {
  239.                         scale_letterbox = input_width_ * 1.0 / img_width;
  240.                     }
  241.                     resize_cols = int(scale_letterbox * img_width);
  242.                     resize_rows = int(scale_letterbox * img_height);
  243.  
  244.                     cv::Mat input_image;
  245.                     cv::cvtColor(fetch_job.input, input_image, COLOR_BGR2RGB);
  246.  
  247.                     cv::resize(input_image, input_image, cv::Size(resize_cols, resize_rows));
  248.  
  249.                     // Generate a gray image for letterbox using opencv
  250.                     cv::Mat img_new(input_height_, input_width_, CV_8UC3, cv::Scalar(114, 114, 114));
  251.                     int top = (input_height_ - resize_rows) / 2;
  252.                     int bot = (input_height_ - resize_rows + 1) / 2;
  253.                     int left = (input_width_ - resize_cols) / 2;
  254.                     int right = (input_width_ - resize_cols + 1) / 2;
  255.                     // Letterbox filling
  256.                     cv::copyMakeBorder(input_image, img_new, top, bot, left, right, cv::BORDER_CONSTANT, cv::Scalar(0, 0, 0));
  257.                     memcpy(input->cpu(), img_new.data, input->bytes());
  258.  
  259.                 }
  260.                 else
  261.                 {
  262.                     memcpy(input->cpu(), fetch_job.input.data, input->bytes());
  263.                     // resize_scale = 1.f;
  264.                 }
  265.  
  266.                 if (!engine->forward())
  267.                 {
  268.                     fetch_job.pro->set_value({});
  269.                     continue;
  270.                 }
  271.  
  272.                 // stride 8
  273.                 int stride0 = 8;
  274.                 int grid_h0 = input_height_ / stride0;
  275.                 int grid_w0 = input_width_ / stride0;
  276.                 int validCount0 = 0;
  277.                 validCount0 = process(engine->output(0).get()->cpu<int8_t>(), (int *)anchor0, grid_h0, grid_w0, input_height_, input_width_,
  278.                                       stride0, filterBoxes, objProbs, classId, confidence_threshold_, engine->output(0).get()->get_zp(), engine->output(0).get()->get_scale(), num_class);
  279.  
  280.                 // stride 16
  281.                 int stride1 = 16;
  282.                 int grid_h1 = input_height_ / stride1;
  283.                 int grid_w1 = input_width_ / stride1;
  284.                 int validCount1 = 0;
  285.                 validCount1 = process(engine->output(1).get()->cpu<int8_t>(), (int *)anchor1, grid_h1, grid_w1, input_height_, input_width_,
  286.                                       stride1, filterBoxes, objProbs, classId, confidence_threshold_, engine->output(1).get()->get_zp(), engine->output(1).get()->get_scale(), num_class);
  287.  
  288.                 // stride 32
  289.                 int stride2 = 32;
  290.                 int grid_h2 = input_height_ / stride2;
  291.                 int grid_w2 = input_width_ / stride2;
  292.                 int validCount2 = 0;
  293.                 validCount2 = process(engine->output(2).get()->cpu<int8_t>(), (int *)anchor2, grid_h2, grid_w2, input_height_, input_width_,
  294.                                       stride2, filterBoxes, objProbs, classId, confidence_threshold_, engine->output(2).get()->get_zp(), engine->output(2).get()->get_scale(), num_class);
  295.  
  296.                 int validCount = validCount0 + validCount1 + validCount2;
  297.  
  298.                 int tmp_h = (input_height_ - resize_rows) / 2;
  299.                 int tmp_w = (input_width_ - resize_cols) / 2;
  300.  
  301.                 float ratio_x = (float)img_height / resize_rows;
  302.                 float ratio_y = (float)img_width / resize_cols;
  303.                 for (int n = 0; n < validCount; ++n)
  304.                 {
  305.  
  306.  
  307.                     float x1 = filterBoxes[n * 4 + 0];
  308.                     float y1 = filterBoxes[n * 4 + 1];
  309.                     float x2 = x1 + filterBoxes[n * 4 + 2];
  310.                     float y2 = y1 + filterBoxes[n * 4 + 3];
  311.                     int id = classId[n];
  312.                     float obj_conf = objProbs[n];
  313.  
  314.                     x1 = (x1 - tmp_w) * ratio_x;
  315.                     y1 = (y1 - tmp_h) * ratio_y;
  316.                     x2 = (x2 - tmp_w) * ratio_x;
  317.                     y2 = (y2 - tmp_h) * ratio_y;
  318.                     x1 = std::max(std::min(x1, (float)(img_width- 1)), 0.f);
  319.                     y1 = std::max(std::min(y1, (float)(img_height - 1)), 0.f);
  320.                     x2 = std::max(std::min(x2, (float)(img_width - 1)), 0.f);
  321.                     y2 = std::max(std::min(y2, (float)(img_height - 1)), 0.f);
  322.                     output_objs.emplace_back(x1, y1, x2, y2, obj_conf, id, m_lables[id]);
  323.  
  324.                 }
  325.                 cpu_nms(output_objs, keep_output_objs, nms_threshold_);
  326.                 fetch_job.pro->set_value(keep_output_objs);
  327.  
  328.                 keep_output_objs.clear();
  329.                 output_objs.clear();
  330.                 // keep_output_objs.clear();
  331.             }
  332.             INFO("Engine destroy.");
  333.         }
  334.  
  335.         virtual bool preprocess(Job &job, const Mat &image) override
  336.         {
  337.             job.input = image;
  338.  
  339.             return !image.empty();
  340.         }
  341.  
  342.         virtual std::shared_future<zql::BoxArray> commit(const Mat &image) override
  343.         {
  344.             return ControllerImpl::commit(image);
  345.         }
  346.  
  347.     private:
  348.         int input_width_ = 0;
  349.         int input_height_ = 0;
  350.         int num_class = 0;
  351.         int gpu_ = 0;
  352.         float confidence_threshold_ = 0;
  353.         float nms_threshold_ = 0;
  354.         std::vector<string> m_lables;
  355.     };
  356.  
  357.     shared_ptr<Infer> create_infer(
  358.         const std::string &engine_file, const std::string &engine_label,
  359.         float confidence_threshold, float nms_threshold)
  360.     {
  361.  
  362.         shared_ptr<InferImpl> instance(new InferImpl());
  363.         if (!instance->startup(
  364.                 engine_file, engine_label, confidence_threshold,
  365.                 nms_threshold))
  366.         {
  367.             instance.reset();
  368.         }
  369.         return instance;
  370.     }
  371. }
  372.

代码部分进行了封装，披露了重要的代码块，如预处理、后处理、坐标映射。model_forward部分API使用请参考官方文档doc/Rockchip_User_Guide_RKNN_Toolkit2_CN-1.4.0.pdf。

结语

至此，YOLOV5已经完全部署移植到RK3568上，有问题请随时留言，下边会介绍如何移植分类模型如repvgg,姿势识别如litehrnet、动作序列识别如tsm算法。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小蓝xlanll/article/detail/229592
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号