devbox
IT小白
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

多线程异步提高RK3588的NPU占用率，进而提高yolov5s帧率_rk3588 多线程

作者：IT小白 | 2024-03-09 01:58:03

赞

踩

rk3588 多线程

前言

        作者手头上有一块香橙派5，其搭载有一颗三核心6TOPS算力的NPU, 由于发布时间不长，社区的资料还是比较匮乏, 所以在这里写一下关于如何提高NPU使用率的教程

        文章和代码使用yolov5s进行讲解, 其他模型如resnet之类的同理,稍作修改就可以使用。 由于已经有很多人,如蓝灵风， 孙启尧等做了如何通过修改模型提高视频推理帧率的教程, 这里我就主要讲另外一种性能的方法——多线程异步

模型修改原理

        yolov5s模型激活函数为silu, 此激活函数量化类型为float16, 导致推理过程中使用CPU进行计算, 量化效果较糟。 将激活函数换为relu, 可以在牺牲一点精度的情况下获得巨大性能提升, 目前测试约为80 - 83帧, c++优化后或许有上百? 详情可看蓝灵风大佬的演示视频

多线程异步原理

        查看NPU占用率的命令

sudo cat /sys/kernel/debug/rknpu/load

        在运行官方demo时我们可以发现，推理过程中NPU使用率较低。

        翻阅官方手册后得知，尽管rk官方有提供函数rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)，使得RKNN模型能使用三核心进行推演，但NPU的使用率仍处于一个较低的水平，如果简单的使用以上函数初始化多个使用多核的rknn模型，在推理过程中就会发生内存泄漏/越界导致系统崩溃

        然而使用以下函数初始化多个使用单核的模型却可以完美运行，那么我们接下来的目标就是通过这一方法初始化自己的rknn线程池

  1. rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0)
  2. rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_1)
  3. rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_2)

首先设计一个初始化rknn模型的函数

  1. def initRKNN(rknnModel="./rknnModel/yolov5s.rknn", id=0):
  2.     rknn_lite = RKNNLite()
  3.     ret = rknn_lite.load_rknn(rknnModel)
  4.     if ret != 0:
  5.         print("Load RKNN rknnModel failed")
  6.         exit(ret)
  7.     if id == 0:
  8.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0)
  9.     elif id == 1:
  10.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_1)
  11.     elif id == 2:
  12.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_2)
  13.     elif id == -1:
  14.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
  15.     else:
  16.         ret = rknn_lite.init_runtime()
  17.     if ret != 0:
  18.         print("Init runtime environment failed")
  19.         exit(ret)
  20.     print(rknnModel, "\t\tdone")
  21.     return rknn_lite

初始化多个rknn模型为一个rknn对象列表，以备后面调用

  1. def initRKNNs(rknnModel="./rknnModel/yolov5s.rknn", TPEs=1):
  2.     rknn_list = []
  3.     for i in range(TPEs):
  4.         rknn_list.append(initRKNN(rknnModel, i % 3))
  5.     return rknn_list

最后结合python官方库里的线程池写一个自己的rknn池

  1. class rknnPoolExecutor():
  2.     def __init__(self, rknnModel, TPEs, func):
  3.         self.TPEs = TPEs
  4.         self.queue = Queue()
  5.         self.rknnPool = initRKNNs(rknnModel, TPEs)
  6.         self.pool = ThreadPoolExecutor(max_workers=TPEs)
  7.         self.func = func
  8.         self.num = 0
  9.  
  10.     def put(self, frame):
  11.         self.queue.put(self.pool.submit(
  12.             self.func, self.rknnPool[self.num % self.TPEs], frame))
  13.         self.num += 1
  14.  
  15.     def get(self):
  16.         if self.queue.empty():
  17.             return None, False
  18.         temp = []
  19.         temp.append(self.queue.get())
  20.         for frame in as_completed(temp):
  21.             return frame.result(), True
  22.  
  23.     def release(self):
  24.         self.pool.shutdown()
  25.         for rknn_lite in self.rknnPool:
  26.             rknn_lite.release()

由于我们要处理的是视频，一个有着严格时间循序的对象，所以我们这里需要提前向线程池中输入几帧，以达到异步的操作(这里的myFunc是一个函数对象，对输入进行处理、推演和绘制目标框体，最后返回目标图像)

  1. # 线程数
  2. TPEs = 6
  3. # 初始化rknn池
  4. pool = rknnPoolExecutor(
  5.     rknnModel=modelPath,
  6.     TPEs=TPEs,
  7.     func=myFunc)
  8.  
  9. # 初始化异步所需要的帧
  10. if (cap.isOpened()):
  11.     for i in range(TPEs + 1):
  12.         ret, frame = cap.read()
  13.         if not ret:
  14.             cap.release()
  15.             del pool
  16.             exit(-1)
  17.         pool.put(frame)

然后便可以对视频进行正常推理

  1. frames, loopTime, initTime = 0, time.time(), time.time()
  2. while (cap.isOpened()):
  3.     frames += 1
  4.     ret, frame = cap.read()
  5.     if not ret:
  6.         break
  7.     pool.put(frame)
  8.     frame, flag = pool.get()
  9.     if flag == False:
  10.         break
  11.     cv2.imshow('test', frame)
  12.     if cv2.waitKey(1) & 0xFF == ord('q'):
  13.         break
  14.     if frames % 30 == 0:
  15.         print("30帧平均帧率:\t", 30 / (time.time() - loopTime), "帧")
  16.         loopTime = time.time()

  最后释放所有资源,防止NPU内存问题

  1. print("总平均帧率\t", frames / (time.time() - initTime))
  2. # 释放cap和rknn线程池
  3. cap.release()
  4. cv2.destroyAllWindows()
  5. pool.release()

        这是在不同线程数下视频推理的帧率:

        测试模型来源 yolov5s，激活函数为silu（非relu优化版本）

        resnet18_for_rk3588, resnet26, resnet50

        测试视频为 新宝岛

 

123456789
yolov5s13.415926.083235.881438.172943.211745.554945.240145.581946.4229
resnet18288.91483.83577.60
resnet26233.16394.83420.10
resnet50186.17259.88284.49

        resnet50在9线程下的NPU占用率:       

        可以看到此时NPU的性能发挥到近乎极致，其推理性能约为280帧 

  • yolov5s在6线程下NPU利用率仅有50 - 60%左右, 性能劣化原因猜想：
    1. python的GIL为伪多线程, 换为c++或许在8线程前仍有较大提升
    2. rk3588的CPU性能跟不上, 对OpenCV绘框部分做c++优化或许有提升

完整代码

        可移步rknn多线程获取yolov5s, resnet26, resnet50的rknn模型、完整代码和演示视频

main.py

  1. import cv2
  2. import time
  3. from rknnpool import rknnPoolExecutor
  4. # 图像处理函数，实际应用过程中需要自行修改
  5. from func import myFunc
  6.  
  7. cap = cv2.VideoCapture('./video/islandBenchmark.mp4')
  8. # cap = cv2.VideoCapture(0)
  9. modelPath = "./rknnModel/yolov5s.rknn"
  10. # 线程数
  11. TPEs = 6
  12. # 初始化rknn池
  13. pool = rknnPoolExecutor(
  14.     rknnModel=modelPath,
  15.     TPEs=TPEs,
  16.     func=myFunc)
  17.  
  18. # 初始化异步所需要的帧
  19. if (cap.isOpened()):
  20.     for i in range(TPEs + 1):
  21.         ret, frame = cap.read()
  22.         if not ret:
  23.             cap.release()
  24.             del pool
  25.             exit(-1)
  26.         pool.put(frame)
  27.  
  28. frames, loopTime, initTime = 0, time.time(), time.time()
  29. while (cap.isOpened()):
  30.     frames += 1
  31.     ret, frame = cap.read()
  32.     if not ret:
  33.         break
  34.     pool.put(frame)
  35.     frame, flag = pool.get()
  36.     if flag == False:
  37.         break
  38.     cv2.imshow('test', frame)
  39.     if cv2.waitKey(1) & 0xFF == ord('q'):
  40.         break
  41.     if frames % 30 == 0:
  42.         print("30帧平均帧率:\t", 30 / (time.time() - loopTime), "帧")
  43.         loopTime = time.time()
  44.  
  45. print("总平均帧率\t", frames / (time.time() - initTime))
  46. # 释放cap和rknn线程池
  47. cap.release()
  48. cv2.destroyAllWindows()
  49. pool.release()

rknnpool.py

  1. from queue import Queue
  2. from rknnlite.api import RKNNLite
  3. from concurrent.futures import ThreadPoolExecutor, as_completed
  4.  
  5.  
  6. def initRKNN(rknnModel="./rknnModel/yolov5s.rknn", id=0):
  7.     rknn_lite = RKNNLite()
  8.     ret = rknn_lite.load_rknn(rknnModel)
  9.     if ret != 0:
  10.         print("Load RKNN rknnModel failed")
  11.         exit(ret)
  12.     if id == 0:
  13.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0)
  14.     elif id == 1:
  15.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_1)
  16.     elif id == 2:
  17.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_2)
  18.     elif id == -1:
  19.         ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
  20.     else:
  21.         ret = rknn_lite.init_runtime()
  22.     if ret != 0:
  23.         print("Init runtime environment failed")
  24.         exit(ret)
  25.     print(rknnModel, "\t\tdone")
  26.     return rknn_lite
  27.  
  28.  
  29. def initRKNNs(rknnModel="./rknnModel/yolov5s.rknn", TPEs=1):
  30.     rknn_list = []
  31.     for i in range(TPEs):
  32.         rknn_list.append(initRKNN(rknnModel, i % 3))
  33.     return rknn_list
  34.  
  35.  
  36. class rknnPoolExecutor():
  37.     def __init__(self, rknnModel, TPEs, func):
  38.         self.TPEs = TPEs
  39.         self.queue = Queue()
  40.         self.rknnPool = initRKNNs(rknnModel, TPEs)
  41.         self.pool = ThreadPoolExecutor(max_workers=TPEs)
  42.         self.func = func
  43.         self.num = 0
  44.  
  45.     def put(self, frame):
  46.         self.queue.put(self.pool.submit(
  47.             self.func, self.rknnPool[self.num % self.TPEs], frame))
  48.         self.num += 1
  49.  
  50.     def get(self):
  51.         if self.queue.empty():
  52.             return None, False
  53.         temp = []
  54.         temp.append(self.queue.get())
  55.         for frame in as_completed(temp):
  56.             return frame.result(), True
  57.  
  58.     def release(self):
  59.         self.pool.shutdown()
  60.         for rknn_lite in self.rknnPool:
  61.             rknn_lite.release()

        func.py

  1. #以下代码改自https://github.com/rockchip-linux/rknn-toolkit2/tree/master/examples/onnx/yolov5
  2. import cv2
  3. import numpy as np
  4. from rknnlite.api import RKNNLite
  5.  
  6. QUANTIZE_ON = True
  7.  
  8. OBJ_THRESH, NMS_THRESH, IMG_SIZE = 0.25, 0.45, 640
  9.  
  10. CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light",
  11.            "fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant",
  12.            "bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite",
  13.            "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ",
  14.            "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa",
  15.            "pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ",
  16.            "oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")
  17.  
  18.  
  19. def sigmoid(x):
  20.     return 1 / (1 + np.exp(-x))
  21.  
  22.  
  23. def xywh2xyxy(x):
  24.     # Convert [x, y, w, h] to [x1, y1, x2, y2]
  25.     y = np.copy(x)
  26.     y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left x
  27.     y[:, 1] = x[:, 1] - x[:, 3] / 2  # top left y
  28.     y[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right x
  29.     y[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right y
  30.     return y
  31.  
  32.  
  33. def process(input, mask, anchors):
  34.  
  35.     anchors = [anchors[i] for i in mask]
  36.     grid_h, grid_w = map(int, input.shape[0:2])
  37.  
  38.     box_confidence = sigmoid(input[..., 4])
  39.     box_confidence = np.expand_dims(box_confidence, axis=-1)
  40.  
  41.     box_class_probs = sigmoid(input[..., 5:])
  42.  
  43.     box_xy = sigmoid(input[..., :2])*2 - 0.5
  44.  
  45.     col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)
  46.     row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)
  47.     col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  48.     row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  49.     grid = np.concatenate((col, row), axis=-1)
  50.     box_xy += grid
  51.     box_xy *= int(IMG_SIZE/grid_h)
  52.  
  53.     box_wh = pow(sigmoid(input[..., 2:4])*2, 2)
  54.     box_wh = box_wh * anchors
  55.  
  56.     box = np.concatenate((box_xy, box_wh), axis=-1)
  57.  
  58.     return box, box_confidence, box_class_probs
  59.  
  60.  
  61. def filter_boxes(boxes, box_confidences, box_class_probs):
  62.     """Filter boxes with box threshold. It's a bit different with origin yolov5 post process!
  64.     # Arguments
  65.         boxes: ndarray, boxes of objects.
  66.         box_confidences: ndarray, confidences of objects.
  67.         box_class_probs: ndarray, class_probs of objects.
  69.     # Returns
  70.         boxes: ndarray, filtered boxes.
  71.         classes: ndarray, classes for boxes.
  72.         scores: ndarray, scores for boxes.
  73.     """
  74.     boxes = boxes.reshape(-1, 4)
  75.     box_confidences = box_confidences.reshape(-1)
  76.     box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])
  77.  
  78.     _box_pos = np.where(box_confidences >= OBJ_THRESH)
  79.     boxes = boxes[_box_pos]
  80.     box_confidences = box_confidences[_box_pos]
  81.     box_class_probs = box_class_probs[_box_pos]
  82.  
  83.     class_max_score = np.max(box_class_probs, axis=-1)
  84.     classes = np.argmax(box_class_probs, axis=-1)
  85.     _class_pos = np.where(class_max_score >= OBJ_THRESH)
  86.  
  87.     boxes = boxes[_class_pos]
  88.     classes = classes[_class_pos]
  89.     scores = (class_max_score * box_confidences)[_class_pos]
  90.  
  91.     return boxes, classes, scores
  92.  
  93.  
  94. def nms_boxes(boxes, scores):
  95.     """Suppress non-maximal boxes.
  97.     # Arguments
  98.         boxes: ndarray, boxes of objects.
  99.         scores: ndarray, scores of objects.
  101.     # Returns
  102.         keep: ndarray, index of effective boxes.
  103.     """
  104.     x = boxes[:, 0]
  105.     y = boxes[:, 1]
  106.     w = boxes[:, 2] - boxes[:, 0]
  107.     h = boxes[:, 3] - boxes[:, 1]
  108.  
  109.     areas = w * h
  110.     order = scores.argsort()[::-1]
  111.  
  112.     keep = []
  113.     while order.size > 0:
  114.         i = order[0]
  115.         keep.append(i)
  116.  
  117.         xx1 = np.maximum(x[i], x[order[1:]])
  118.         yy1 = np.maximum(y[i], y[order[1:]])
  119.         xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])
  120.         yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])
  121.  
  122.         w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)
  123.         h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)
  124.         inter = w1 * h1
  125.  
  126.         ovr = inter / (areas[i] + areas[order[1:]] - inter)
  127.         inds = np.where(ovr <= NMS_THRESH)[0]
  128.         order = order[inds + 1]
  129.     keep = np.array(keep)
  130.     return keep
  131.  
  132.  
  133. def yolov5_post_process(input_data):
  134.     masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
  135.     anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],
  136.                [59, 119], [116, 90], [156, 198], [373, 326]]
  137.  
  138.     boxes, classes, scores = [], [], []
  139.     for input, mask in zip(input_data, masks):
  140.         b, c, s = process(input, mask, anchors)
  141.         b, c, s = filter_boxes(b, c, s)
  142.         boxes.append(b)
  143.         classes.append(c)
  144.         scores.append(s)
  145.  
  146.     boxes = np.concatenate(boxes)
  147.     boxes = xywh2xyxy(boxes)
  148.     classes = np.concatenate(classes)
  149.     scores = np.concatenate(scores)
  150.  
  151.     nboxes, nclasses, nscores = [], [], []
  152.     for c in set(classes):
  153.         inds = np.where(classes == c)
  154.         b = boxes[inds]
  155.         c = classes[inds]
  156.         s = scores[inds]
  157.  
  158.         keep = nms_boxes(b, s)
  159.  
  160.         nboxes.append(b[keep])
  161.         nclasses.append(c[keep])
  162.         nscores.append(s[keep])
  163.  
  164.     if not nclasses and not nscores:
  165.         return None, None, None
  166.  
  167.     boxes = np.concatenate(nboxes)
  168.     classes = np.concatenate(nclasses)
  169.     scores = np.concatenate(nscores)
  170.  
  171.     return boxes, classes, scores
  172.  
  173.  
  174. def draw(image, boxes, scores, classes):
  175.     for box, score, cl in zip(boxes, scores, classes):
  176.         top, left, right, bottom = box
  177.         # print('class: {}, score: {}'.format(CLASSES[cl], score))
  178.         # print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))
  179.         top = int(top)
  180.         left = int(left)
  181.         right = int(right)
  182.         bottom = int(bottom)
  183.  
  184.         cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)
  185.         cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),
  186.                     (top, left - 6),
  187.                     cv2.FONT_HERSHEY_SIMPLEX,
  188.                     0.6, (0, 0, 255), 2)
  189.  
  190.  
  191. def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
  192.     shape = im.shape[:2]  # current shape [height, width]
  193.     if isinstance(new_shape, int):
  194.         new_shape = (new_shape, new_shape)
  195.  
  196.     r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
  197.  
  198.     ratio = r, r  # width, height ratios
  199.     new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
  200.     dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - \
  201.         new_unpad[1]  # wh padding
  202.  
  203.     dw /= 2  # divide padding into 2 sides
  204.     dh /= 2
  205.  
  206.     if shape[::-1] != new_unpad:  # resize
  207.         im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
  208.     top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
  209.     left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
  210.     im = cv2.copyMakeBorder(im, top, bottom, left, right,
  211.                             cv2.BORDER_CONSTANT, value=color)  # add border
  212.     return im, ratio, (dw, dh)
  213.  
  214. def myFunc(rknn_lite, IMG):
  215.     img = cv2.cvtColor(IMG, cv2.COLOR_BGR2RGB)
  216.     img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
  217.     outputs = rknn_lite.inference(inputs=[img])
  218.  
  219.     input0_data = outputs[0]
  220.     input1_data = outputs[1]
  221.     input2_data = outputs[2]
  222.  
  223.     input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))
  224.     input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))
  225.     input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))
  226.  
  227.     input_data = list()
  228.     input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))
  229.     input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))
  230.     input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))
  231.  
  232.     boxes, classes, scores = yolov5_post_process(input_data)
  233.  
  234.     img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
  235.     if boxes is not None:
  236.         draw(img_1, boxes, scores, classes)
  237.     return img_1

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/IT小白/article/detail/213320?site
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号