devbox
一键难忘520
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

RK3568笔记二十八：CRNN车牌识别部署_lprnet rknn

作者：一键难忘520 | 2024-06-27 20:55:19

赞

踩

lprnet rknn

若该文为原创文章，转载请注明原文出处。

想在RK3568上实现车牌识别，想到的方法是使用LPRNet网络识别或CRNN识别，本篇记录使用CRNN识别车牌，也可以换成LPRNet模型，原理一样。

一、平台介绍

1、训练平台：Autodl

2、运行板子：ATK-DLRK3568

二、环境搭建

使用Autodl平台的GPU训练测试

1、创建虚拟环境

conda create -n cnn_plate_env python=3.8

2、激活

conda activate cnn_plate_env

3、安装依赖项

  1. pip install easydict -i https://pypi.tuna.tsinghua.edu.cn/simple
  2. pip install pyyaml -i https://pypi.tuna.tsinghua.edu.cn/simple
  3. pip install opencv-python -i https://pypi.tuna.tsinghua.edu.cn/simple
  4. pip install tensorboardX -i https://pypi.tuna.tsinghua.edu.cn/simple
  5. pip install imgaug -i https://pypi.tuna.tsinghua.edu.cn/simple
  6. pip install torch -i https://pypi.tuna.tsinghua.edu.cn/simple
  7. pip install torchvision -i https://pypi.tuna.tsinghua.edu.cn/simple
  8. pip install tensorboardX -i https://pypi.tuna.tsinghua.edu.cn/simple

4、下载源码

  1. # 源码地址
  2. https://github.com/we0091234/crnn_plate_recognition

5、准备数据集

车牌识别数据集CCPD+CRPD

  1. 从CCPD和CRPD截下来的车牌小图以及我自己收集的一部分车牌 

  2. 数据集打上标签,生成train.txt和val.txt

如有需要，联系博主。

然后执行如下命令，得到train.txt和val.txt

  1. python plateLabel.py --image_path your/train/img/path/ --label_file datasets/train.txt
  2. python plateLabel.py --image_path your/val/img/path/ --label_file datasets/val.txt

数据格式如下：

6、修改配置文件

将train.txt val.txt路径写入lib/config/360CC_config.yaml 中

JSON_FILE: {'train': '/root/crnn_plate_recognition/datasets/train.txt', 'val': '/root/crnn_plate_recognition/datasets/val.txt'}

注意路径

7、修改trian.py

由于安装的ymal版本过高，会导致无法加载。所以需要修改加载方式。

出错：TypeError: load() missing 1 required positional argument: 'Loader'

处理：TypeError: load() missing 1 required positional argument: ‘Loader‘-CSDN博客

8、训练

 python train.py --cfg lib/config/360CC_config.yaml

默认100轮，计时2-3小时，结果保存再output文件夹中

9、测试

python demo.py --model_path saved_model/best.pth --image_path images/test.jpg

三、导出ONNX并推理测试

导出需要安装onnx

  1. pip install onnx
  2. pip install onnxsim

1、导出ONNX

python export.py --weights saved_model/best.pth --save_path saved_model/best.onnx  --simplify

2、推理测试

python onnx_infer.py --onnx_file saved_model/best.onnx  --image_path images/test.jpg

四、转成RKNN

转成RKNN和部署需要用到rknn-toolkit2-1.5.0

把训练转换后的onn拷贝到同级目录下，执行下面转换程序，即可导出RKNN

recongniton_cov_rknn.py

  1. import numpy as np
  2. import cv2
  3. from rknn.api import RKNN
  4. from PIL import Image, ImageDraw, ImageFont
  5.  
  6.  
  7.  
  8. if __name__ == '__main__':
  9.  
  10.     # Create RKNN object
  11.  
  12.     rknn2 = RKNN(verbose=False)
  13.     # pre-process config
  14.     print('--> Config model')
  15.     rknn2.config(mean_values=[[150.54, 150.54, 150.54]], std_values=[[49.215, 49.215, 49.215]], target_platform="rk3568")
  16.     print('done')
  17.  
  18.     # Load ONNX model
  19.     print('--> Loading model')
  20.     ret = rknn2.load_onnx(model="./best.onnx")
  21.     if ret != 0:
  22.         print('Load model failed!')
  23.         exit(ret)
  24.     print('done')
  25.  
  26.     # Build model
  27.     print('--> Building model')
  28.     ret = rknn2.build(do_quantization=False)
  29.     if ret != 0:
  30.         print('Build model failed!')
  31.         exit(ret)
  32.     print('done')
  33.  
  34.     # Export RKNN model
  35.     print('--> Export rknn model')
  36.     ret = rknn2.export_rknn("./best.rknn")
  37.     if ret != 0:
  38.         print('Export rknn model failed!')
  39.         exit(ret)
  40.     print('done')
  41.  
  42.     # Init runtime environment
  43.     print('--> Init runtime environment')
  44.     ret = rknn2.init_runtime()
  45.     if ret != 0:
  46.         print('Init runtime environment failed!')
  47.         exit(ret)
  48.     print('done')
  49.  
  50.     rknn2.release()

五、部署测试

把转换后的RKNN模型拷贝到虚拟机内。

测试RKNN模型

  1. import numpy as np
  2. import cv2
  3. from rknn.api import RKNN
  4. from PIL import Image, ImageDraw, ImageFont
  5.  
  6. plate_chr = "#京沪津渝冀晋蒙辽吉黑苏浙皖闽赣鲁豫鄂湘粤桂琼川贵云藏陕甘青宁新学警港澳挂使领民航危0123456789ABCDEFGHJKLMNPQRSTUVWXYZ险品"
  7.  
  8.  
  9. OBJ_THRESH = 0.25
  10. NMS_THRESH = 0.45
  11. IMG_SIZE = 640
  12.  
  13. CLASSES = ("plate", )
  14.  
  15. def decodePlate(preds):  # 识别后处理  
  16.     pre = 0
  17.     newPreds = []
  18.     for i in range(len(preds[0])):
  19.         if (preds[0][i] != 0).all() and (preds[0][i] != pre).all():
  20.             newPreds.append(preds[0][i])
  21.         pre = preds[0][i]
  22.     plate = ""
  23.     for i in newPreds:
  24.         plate += plate_chr[int(i)]
  25.     print(plate)
  26.     return plate
  27.  
  28.  
  29. def cv2ImgAddText(img, text, left, top, textColor=(0, 255, 0), textSize=20):
  30.     if (isinstance(img, np.ndarray)):  # 判断是否OpenCV图片类型
  31.         img = Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
  32.     # 创建一个可以在给定图像上绘图的对象
  33.     draw = ImageDraw.Draw(img)
  34.     # 字体的格式   
  35.     fontStyle = ImageFont.truetype(
  36.         "simsun.ttc", textSize, encoding="utf-8")
  37.     # 绘制文本
  38.     draw.text((left, top), text, textColor, font=fontStyle)
  39.     # 转换回OpenCV格式
  40.     return cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR)
  41.  
  42. def sigmoid(x):
  43.     # return 1 / (1 + np.exp(-x))
  44.     return x
  45.  
  46. def xywh2xyxy(x):
  47.     # Convert [x, y, w, h] to [x1, y1, x2, y2]
  48.     y = np.copy(x)
  49.     y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left x
  50.     y[:, 1] = x[:, 1] - x[:, 3] / 2  # top left y
  51.     y[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right x
  52.     y[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right y
  53.     return y
  54.  
  55.  
  56. def process(input, mask, anchors):
  57.  
  58.     anchors = [anchors[i] for i in mask]
  59.     grid_h, grid_w = map(int, input.shape[0:2])
  60.  
  61.     box_confidence = sigmoid(input[..., 4])
  62.     box_confidence = np.expand_dims(box_confidence, axis=-1)
  63.  
  64.     box_class_probs = sigmoid(input[..., 5:])
  65.  
  66.     box_xy = sigmoid(input[..., :2])*2 - 0.5
  67.  
  68.     col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)
  69.     row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)
  70.     col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  71.     row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
  72.     grid = np.concatenate((col, row), axis=-1)
  73.     box_xy += grid
  74.     box_xy *= int(IMG_SIZE/grid_h)
  75.  
  76.     box_wh = pow(sigmoid(input[..., 2:4])*2, 2)
  77.     box_wh = box_wh * anchors
  78.  
  79.     box = np.concatenate((box_xy, box_wh), axis=-1)
  80.  
  81.     return box, box_confidence, box_class_probs
  82.  
  83.  
  84. def filter_boxes(boxes, box_confidences, box_class_probs):
  85.     """Filter boxes with box threshold. It's a bit different with origin yolov5 post process!
  87.     # Arguments
  88.         boxes: ndarray, boxes of objects.
  89.         box_confidences: ndarray, confidences of objects.
  90.         box_class_probs: ndarray, class_probs of objects.
  92.     # Returns
  93.         boxes: ndarray, filtered boxes.
  94.         classes: ndarray, classes for boxes.
  95.         scores: ndarray, scores for boxes.
  96.     """
  97.     boxes = boxes.reshape(-1, 4)
  98.     box_confidences = box_confidences.reshape(-1)
  99.     box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])
  100.  
  101.     _box_pos = np.where(box_confidences >= OBJ_THRESH)
  102.     boxes = boxes[_box_pos]
  103.     box_confidences = box_confidences[_box_pos]
  104.     box_class_probs = box_class_probs[_box_pos]
  105.  
  106.     class_max_score = np.max(box_class_probs, axis=-1)
  107.     classes = np.argmax(box_class_probs, axis=-1)
  108.     _class_pos = np.where(class_max_score >= OBJ_THRESH)
  109.  
  110.     boxes = boxes[_class_pos]
  111.     classes = classes[_class_pos]
  112.     scores = (class_max_score* box_confidences)[_class_pos]
  113.  
  114.     return boxes, classes, scores
  115.  
  116.  
  117. def nms_boxes(boxes, scores):
  118.     """Suppress non-maximal boxes.
  120.     # Arguments
  121.         boxes: ndarray, boxes of objects.
  122.         scores: ndarray, scores of objects.
  124.     # Returns
  125.         keep: ndarray, index of effective boxes.
  126.     """
  127.     x = boxes[:, 0]
  128.     y = boxes[:, 1]
  129.     w = boxes[:, 2] - boxes[:, 0]
  130.     h = boxes[:, 3] - boxes[:, 1]
  131.  
  132.     areas = w * h
  133.     order = scores.argsort()[::-1]
  134.  
  135.     keep = []
  136.     while order.size > 0:
  137.         i = order[0]
  138.         keep.append(i)
  139.  
  140.         xx1 = np.maximum(x[i], x[order[1:]])
  141.         yy1 = np.maximum(y[i], y[order[1:]])
  142.         xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])
  143.         yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])
  144.  
  145.         w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)
  146.         h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)
  147.         inter = w1 * h1
  148.  
  149.         ovr = inter / (areas[i] + areas[order[1:]] - inter)
  150.         inds = np.where(ovr <= NMS_THRESH)[0]
  151.         order = order[inds + 1]
  152.     keep = np.array(keep)
  153.     return keep
  154.  
  155.  
  156. def yolov5_post_process(input_data):
  157.     masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
  158.     anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],
  159.                [59, 119], [116, 90], [156, 198], [373, 326]]
  160.  
  161.     boxes, classes, scores = [], [], []
  162.     for input, mask in zip(input_data, masks):
  163.         b, c, s = process(input, mask, anchors)
  164.         b, c, s = filter_boxes(b, c, s)
  165.         boxes.append(b)
  166.         classes.append(c)
  167.         scores.append(s)
  168.  
  169.     boxes = np.concatenate(boxes)
  170.     boxes = xywh2xyxy(boxes)
  171.     classes = np.concatenate(classes)
  172.     scores = np.concatenate(scores)
  173.  
  174.     nboxes, nclasses, nscores = [], [], []
  175.     for c in set(classes):
  176.         inds = np.where(classes == c)
  177.         b = boxes[inds]
  178.         c = classes[inds]
  179.         s = scores[inds]
  180.  
  181.         keep = nms_boxes(b, s)
  182.  
  183.         nboxes.append(b[keep])
  184.         nclasses.append(c[keep])
  185.         nscores.append(s[keep])
  186.  
  187.     if not nclasses and not nscores:
  188.         return None, None, None
  189.  
  190.     boxes = np.concatenate(nboxes)
  191.     classes = np.concatenate(nclasses)
  192.     scores = np.concatenate(nscores)
  193.  
  194.     return boxes, classes, scores
  195.  
  196.  
  197. def draw(image, boxes, scores, classes):
  198.     """Draw the boxes on the image.
  200.     # Argument:
  201.         image: original image.
  202.         boxes: ndarray, boxes of objects.
  203.         classes: ndarray, classes of objects.
  204.         scores: ndarray, scores of objects.
  205.         all_classes: all classes name.
  206.     """
  207.     for box, score, cl in zip(boxes, scores, classes):
  208.         top, left, right, bottom = box
  209.         # print('class: {}, score: {}'.format(CLASSES[cl], score))
  210.         # print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))
  211.         top = int(top)
  212.         left = int(left)
  213.         right = int(right)
  214.         bottom = int(bottom)
  215.  
  216.         cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)
  217.         cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),
  218.                     (top, left - 6),
  219.                     cv2.FONT_HERSHEY_SIMPLEX,
  220.                     0.6, (0, 0, 255), 2)
  221.  
  222.  
  223. def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
  224.     # Resize and pad image while meeting stride-multiple constraints
  225.     shape = im.shape[:2]  # current shape [height, width]
  226.     if isinstance(new_shape, int):
  227.         new_shape = (new_shape, new_shape)
  228.  
  229.     # Scale ratio (new / old)
  230.     r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
  231.  
  232.     # Compute padding
  233.     ratio = r, r  # width, height ratios
  234.     new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
  235.     dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh padding
  236.  
  237.     dw /= 2  # divide padding into 2 sides
  238.     dh /= 2
  239.  
  240.     if shape[::-1] != new_unpad:  # resize
  241.         im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
  242.     top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
  243.     left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
  244.     im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add border
  245.     return im, ratio, (dw, dh)
  246.  
  247.  
  248. if __name__ == '__main__':
  249.  
  250.     # Create RKNN object
  251.     rknn1 = RKNN(verbose=False)
  252.     rknn2 = RKNN(verbose=False)
  253.     # pre-process config
  254.     print('--> Config model')
  255.     rknn1.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform="rk3568")
  256.     rknn2.config(mean_values=[[150.54, 150.54, 150.54]], std_values=[[49.215, 49.215, 49.215]], target_platform="rk3568")
  257.     print('done')
  258.  
  259.     # Load ONNX model
  260.     print('--> Loading model')
  261.     ret = rknn1.load_onnx(model="./yolov5n.onnx")
  262.     ret = rknn2.load_onnx(model="./best.onnx")
  263.     if ret != 0:
  264.         print('Load model failed!')
  265.         exit(ret)
  266.     print('done')
  267.  
  268.     # Build model
  269.     print('--> Building model')
  270.     ret = rknn1.build(do_quantization=True, dataset="dataset.txt")
  271.     ret = rknn2.build(do_quantization=False)
  272.     if ret != 0:
  273.         print('Build model failed!')
  274.         exit(ret)
  275.     print('done')
  276.  
  277.     # Export RKNN model
  278.     print('--> Export rknn model')
  279.     ret = rknn1.export_rknn("./yolov5n.rknn")
  280.     ret = rknn2.export_rknn("./best.rknn")
  281.     if ret != 0:
  282.         print('Export rknn model failed!')
  283.         exit(ret)
  284.     print('done')
  285.  
  286.     # Init runtime environment
  287.     print('--> Init runtime environment')
  288.     ret = rknn1.init_runtime()
  289.     ret = rknn2.init_runtime()
  290.     if ret != 0:
  291.         print('Init runtime environment failed!')
  292.         exit(ret)
  293.     print('done')
  294.  
  295.     # Set inputs
  296.     img = cv2.imread("./car.jpg")
  297.     # img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))
  298.     img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  299.     img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
  300.  
  301.     # Inference
  302.     print('--> Running model')
  303.     outputs = rknn1.inference(inputs=[img])
  304.     print('done')
  305.  
  306.     # post process
  307.     input0_data = outputs[0]
  308.     input1_data = outputs[1]
  309.     input2_data = outputs[2]
  310.  
  311.     input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))
  312.     input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))
  313.     input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))
  314.  
  315.     input_data = list()
  316.     input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))
  317.     input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))
  318.     input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))
  319.  
  320.     boxes, classes, scores = yolov5_post_process(input_data)
  321.  
  322.     img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
  323.     if boxes is not None:
  324.         print(boxes)
  325.         # draw(img_1, boxes, scores, classes)
  326.         for box in boxes:
  327.             x1, y1, x2, y2 = box
  328.             x1 = int(x1)
  329.             x2 = int(x2)
  330.             y1 = int(y1)
  331.             y2 = int(y2)
  332.             cv2.rectangle(img_1, (x1, y1), (x2, y2), (0, 255, 0), 1, 1)
  333.             roi = img[y1:y2, x1:x2]
  334.             try:
  335.                 roi = cv2.resize(roi, (168, 48))
  336.                 print('333')
  337.                 output = rknn2.inference(inputs=[roi])
  338.                 input_data = np.swapaxes(output[0], 1, 2)
  339.                 index = np.argmax(input_data, axis=1)
  340.                 plate_no = decodePlate(index)
  341.                 img_1 = cv2ImgAddText(img_1, str(plate_no), x1, y1-30, (0, 255, 0), 30)
  342.                 print(str(plate_no))
  343.             except:
  344.                 continue
  345.     # show output
  346.     cv2.imshow("post process result", img_1)
  347.     cv2.waitKey(0)
  348.     cv2.destroyAllWindows()
  349.  
  350.     rknn1.release()
  351.     rknn2.release()

程序大概思路是：

1、读取图片 

2、加载模型

3、检测车牌

4、识别车牌

5、显示图片 

上面是在虚拟机模拟运行，结果正常。

六、C++板载部署

自行部署，参考讯为电子的程序，可以部署成功。

如有侵权，或需要完整代码，请及时联系博主。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/一键难忘520/article/detail/763721
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号