当前位置:   article > 正文

triton部署yolov5笔记(四)_triton部署pytorch yolov5s

triton部署pytorch yolov5s

直达链接

Triton部署YOLOV5笔记(一)
Triton部署YOLOV5笔记(二)
triton部署yolov5笔记(三)
github链接

前言

triton部署yolov5笔记(三)已经将模型部署到服务器,通过客户端代码进行推理,本篇将模型推理部分放到了python backend部分,即将模型推理写进服务端,客户端只需要调用接口,传输内容为输入图片路径,输出内容为输出图片路径。为了降低传输图片造成的网络延时,将图片文件夹images挂载到了容器内,服务端只需要接收客户端传来的图片路径,即可调用本地图片进行推理,返回给客户端输出图片的路径。输入输出路径是写死的。

文件放置目录结构如下

tensorrt-server
├── client.py
├── images
│   ├── input_img
│   │   └── 4.jpg
│   └── output_img
│       ├── 20220816
│       │   ├── 48c5a660-0fbd-4e21-9667-f668d3b8148e.jpg
│       │   ├── 7aee8a1a-5599-4d3d-a5e4-bfda3f35443b.jpg
│       │   └── ae09a6a3-da83-4b50-85af-7be13f1d3a7d.jpg
│       └── 20220817
│           ├── 3f086bb3-7137-40f6-890a-7871ae43817b.jpg
│           ├── 573ae2d4-e6d8-4a61-8360-bebb3eb3252c.jpg
│           ├── 7563cb88-3be0-4248-be81-a7d379ff29a5.jpg
│           ├── ac01b9f3-0f8c-428d-886a-587a5b9c92fc.jpg
│           └── cacab491-67ee-4cfc-9790-983143b1283c.jpg
├── models
│   ├── custom_model
│   │   ├── 1
│   │   │   ├── model.py
│   │   │   └── __pycache__
│   │   │       ├── hat_utils.cpython-38.pyc
│   │   │       └── model.cpython-38.pyc
│   │   ├── config.pbtxt
│   │   └── triton_server.tar.gz
│   └── model_0
│       └── 1
│           └── model.plan
├── plugins
│   └── libmyplugins.so
└── readme.txt
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31

custom_model目录

其中custom_model目录下为python backend部分,我们再model.py文件中进行模型的推理工作。这里需要将模型推理过程中需要的依赖库打包在custom_model文件夹下,即triton_server.tar.gz压缩包,并在配置文件config.pbtxt文件中最后说明。
triton_server.tar.gz文件的创建方法可以参考Triton部署YOLOV5笔记(二)文章最后的使用本地环境,需要什么依赖库在conda虚拟环境中安装玩然后打包即可。
config.pbtxt配置文件如下:

name: "custom_model"
backend: "python"
input [
  {
    name: "input0"
    data_type: TYPE_STRING
    dims: [1]
  }
]
output [
  {
    name: "output0"
    data_type: TYPE_STRING
    dims: [1]
  }
]
instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]
parameters: {
  key: "EXECUTION_ENV_PATH",
  value: {string_value: "$$TRITON_MODEL_DIRECTORY/triton_server.tar.gz"}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26

模型的输入为一个字符串,输出为一个字符串,即图片路径。
model.py文件如下:

# model.py
import json
import numpy as np
import triton_python_backend_utils as pb_utils
import cv2
import random
import math
from enum import Enum
from torch.utils.dlpack import from_dlpack
import time
import uuid
import os

_LINE_THICKNESS_SCALING = 500.0
_TEXT_THICKNESS_SCALING = 700.0
_TEXT_SCALING = 520.0

np.random.seed(69)
RAND_COLORS = np.random.randint(10, 255, (80, 3), "int")  # used for class visu

class COCOLabels(Enum):
    hat = 0
    no_hat = 1

class BoundingBox:
    def __init__(self, classID, confidence, x1, x2, y1, y2, image_width, image_height):
        self.classID = classID
        self.confidence = confidence
        self.x1 = x1
        self.x2 = x2
        self.y1 = y1
        self.y2 = y2
        self.u1 = x1 / image_width
        self.u2 = x2 / image_width
        self.v1 = y1 / image_height
        self.v2 = y2 / image_height
    
    def box(self):
        return (self.x1, self.y1, self.x2, self.y2)
        
    def width(self):
        return self.x2 - self.x1
    
    def height(self):
        return self.y2 - self.y1

    def center_absolute(self):
        return (0.5 * (self.x1 + self.x2), 0.5 * (self.y1 + self.y2))
    
    def center_normalized(self):
        return (0.5 * (self.u1 + self.u2), 0.5 * (self.v1 + self.v2))
    
    def size_absolute(self):
        return (self.x2 - self.x1, self.y2 - self.y1)
    
    def size_normalized(self):
        return (self.u2 - self.u1, self.v2 - self.v1)

class TritonPythonModel:

    def initialize(self, args):
        self.model_config = model_config = json.loads(args['model_config'])
        output0_config = pb_utils.get_output_config_by_name(model_config, "output0")
        # output1_config = pb_utils.get_output_config_by_name(model_config, "output1")
        self.output0_dtype = pb_utils.triton_string_to_numpy(output0_config['data_type'])
        # self.output1_dtype = pb_utils.triton_string_to_numpy(output1_config['data_type'])

    def execute(self, requests):
        output0_dtype = self.output0_dtype
        # output1_dtype = self.output1_dtype
        responses = []
        for request in requests:
            in_0 = pb_utils.get_input_tensor_by_name(request, 'input0')
            # in_1 = pb_utils.get_input_tensor_by_name(request, 'input1')
            in_0 = in_0.as_numpy() #获取输入字符串的numpy格式数据
            text = in_0[0].decode("utf-8") #输入的第0个字符,即传入的数据,该数据是客户端编码的格式,需要解码。
            # in_1 = in_1.as_numpy()
            t1 = time.time()
            img = self._recognize(text)
            t2 = time.time()
            print('inference time is: {}ms'.format(1000 * (t2 - t1)))
            # out_1 = in_1
            # 文件夹管理(根据日期建立文件夹,根据时间输出图片)
            time_now = time.strftime("%Y%m%d", time.localtime())
            output_path = '/images/output_img/' + time_now
            output_path_0 = '/' + time_now
            if not os.path.exists(output_path):
                os.makedirs(output_path) 
            # out_pic_name = output_path + '/' + time.strftime("%H_%M_%S", time.localtime()) + '.jpg'
            uuidFour = str(uuid.uuid4())
            out_pic_name = output_path + '/' + uuidFour + '.jpg'
            out_pic_name_0 = output_path_0 + '/' + uuidFour + '.jpg'
            cv2.imwrite(out_pic_name, img.astype(np.uint8))
            out_pic_name = np.array(out_pic_name_0)
            out_tensor_0 = pb_utils.Tensor('output0', out_pic_name.astype(output0_dtype))
            inference_response = pb_utils.InferenceResponse(output_tensors=[out_tensor_0])
            responses.append(inference_response)
        return responses

    def finalize(self):
        print('Cleaning up...')

    def _recognize(self,draw_path):
        con_thres = 0.25 # 置信度阈值
        iou_thres = 0.45 # 非极大值抑制阈值 iou
        img_size = [640, 640]

        # input_image = draw.copy()
        # input_image_buffer = self.preprocess(input_image, img_size)
        # draw = draw.copy()
        draw = cv2.imread(draw_path)
        # src_size = draw.shape[:2]
        # 图片填充并进行归一化
        # img = self.letterbox(draw,img_size,stride=32)[0]

        # img = img[:, :, ::-1].transpose(2, 0, 1)  # BGR to RGB, to 3x416x416
        # img = np.ascontiguousarray(img)

        # 归一化
        # img=img.astype(dtype=np.float32)
        # img/=255.0
        # 维度扩张
        # input_image_buffer=np.expand_dims(img,axis=0).astype(np.float32)
        
        input_image_buffer = self.preprocess(draw, img_size)
        input_image_buffer = np.expand_dims(input_image_buffer, axis=0)
        # print(input_image_buffer.shape)
        # 维度扩张
        # input_image_buffer = np.expand_dims(input_image_buffer, axis=0)

        inference_request = pb_utils.InferenceRequest(
            model_name = 'model_0',
            requested_output_names=['prob'],
            inputs=[pb_utils.Tensor('data', input_image_buffer)]
        )
        inference_response = inference_request.exec()
        # result = pb_utils.get_output_tensor_by_name(inference_response, 'prob')
        # result = result.as_numpy()
        result = self.pb_tensor_to_numpy(pb_utils.get_output_tensor_by_name(inference_response, 'prob'))
        # print(result.shape)
        detected_objects = self.postprocess(result, draw.shape[1], draw.shape[0], img_size, con_thres, iou_thres)
        # print(detected_objects)
        for box in detected_objects:
            # print(f"{COCOLabels(box.classID).name}: {box.confidence}")
            draw = self.plot_one_box(box.box(), draw,color=tuple(RAND_COLORS[box.classID % 64].tolist()), label=f"{COCOLabels(box.classID).name}:{box.confidence:.2f}",) 
        return draw

    def letterbox(self, img, new_shape=(640, 640), color=(114, 114, 114), auto=False, scaleFill=False, scaleup=True,
                  stride=32):
        '''图片归一化'''
        # Resize and pad image while meeting stride-multiple constraints
        shape = img.shape[:2]  # current shape [height, width]
        if isinstance(new_shape, int):
            new_shape = (new_shape, new_shape)

        # Scale ratio (new / old)
        r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
        if not scaleup:  # only scale down, do not scale up (for better test mAP)
            r = min(r, 1.0)

        # Compute padding
        ratio = r, r  # width, height ratios

        new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
        dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh padding

        if auto:  # minimum rectangle
            dw, dh = np.mod(dw, stride), np.mod(dh, stride)  # wh padding
        elif scaleFill:  # stretch
            dw, dh = 0.0, 0.0
            new_unpad = (new_shape[1], new_shape[0])
            ratio = new_shape[1] / shape[1], new_shape[0] / shape[0]  # width, height ratios

        dw /= 2  # divide padding into 2 sides
        dh /= 2

        if shape[::-1] != new_unpad:  # resize
            img = cv2.resize(img, new_unpad, interpolation=cv2.INTER_LINEAR)

        top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
        left, right = int(round(dw - 0.1)), int(round(dw + 0.1))

        img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add border
        return img, ratio, (dw, dh)
    
    def pb_tensor_to_numpy(self,pb_tensor):
        '''pb_tensor转换为numpy格式'''
        if pb_tensor.is_cpu():
            return pb_tensor.as_numpy()
        else:
            pytorch_tensor = from_dlpack(pb_tensor.to_dlpack())
            return pytorch_tensor.cpu().numpy()
    
    def preprocess(self, raw_bgr_image, input_shape):
    
        input_w, input_h = input_shape
        image_raw = raw_bgr_image
        h, w, c = image_raw.shape
        image = cv2.cvtColor(image_raw, cv2.COLOR_BGR2RGB)
        # Calculate widht and height and paddings
        r_w = input_w / w
        r_h = input_h / h
        if r_h > r_w:
            tw = input_w
            th = int(r_w * h)
            tx1 = tx2 = 0
            ty1 = int((input_h - th) / 2)
            ty2 = input_h - th - ty1
        else:
            tw = int(r_h * w)
            th = input_h
            tx1 = int((input_w - tw) / 2)
            tx2 = input_w - tw - tx1
            ty1 = ty2 = 0
        # Resize the image with long side while maintaining ratio
        image = cv2.resize(image, (tw, th))
        # Pad the short side with (128,128,128)
        image = cv2.copyMakeBorder(
            image, ty1, ty2, tx1, tx2, cv2.BORDER_CONSTANT, (128, 128, 128)
        )
        image = image.astype(np.float32)
        # Normalize to [0,1]
        image /= 255.0
        # HWC to CHW format:
        image = np.transpose(image, [2, 0, 1])
        return image

    def postprocess(self, output, origin_w, origin_h, input_shape, conf_th=0.5, nms_threshold=0.5, letter_box=False):
        """Postprocess TensorRT outputs.
        # Args
            output: list of detections with schema 
            [num_boxes,cx,cy,w,h,conf,cls_id, cx,cy,w,h,conf,cls_id, ...] 
            conf_th: confidence threshold
            letter_box: boolean, referring to _preprocess_yolo()
        # Returns
            list of bounding boxes with all detections above threshold and after nms, see class BoundingBox
        """
    
        # Get the num of boxes detected
        # Here we use the first row of output in that batch_size = 1
        output = output[0]
        num = int(output[0])
        # Reshape to a two dimentional ndarray
        pred = np.reshape(output[1:], (-1, 6))[:num, :]

        # Do nms
        boxes = self.non_max_suppression(pred, origin_h, origin_w, input_shape[0], input_shape[1], conf_thres=conf_th, nms_thres=nms_threshold)
        result_boxes = boxes[:, :4] if len(boxes) else np.array([])
        result_scores = boxes[:, 4] if len(boxes) else np.array([])
        result_classid = boxes[:, 5].astype(np.int) if len(boxes) else np.array([])
        
        detected_objects = []
        for box, score, label in zip(result_boxes, result_scores, result_classid):
            detected_objects.append(BoundingBox(label, score, box[0], box[2], box[1], box[3], origin_h, origin_w))
        return detected_objects

    def non_max_suppression(self, prediction, origin_h, origin_w, input_w, input_h, conf_thres=0.5, nms_thres=0.4):
        """
        description: Removes detections with lower object confidence score than 'conf_thres' and performs
        Non-Maximum Suppression to further filter detections.
        param:
            prediction: detections, (x1, y1, x2, y2, conf, cls_id)
            origin_h: original image height
            origin_w: original image width
            conf_thres: a confidence threshold to filter detections
            nms_thres: a iou threshold to filter detections
        return:
            boxes: output after nms with the shape (x1, y1, x2, y2, conf, cls_id)
        """
        # Get the boxes that score > CONF_THRESH
        boxes = prediction[prediction[:, 4] >= conf_thres]
        # print(boxes)
        # Trandform bbox from [center_x, center_y, w, h] to [x1, y1, x2, y2]
        boxes[:, :4] = self.xywh2xyxy(boxes[:, :4], origin_h, origin_w, input_w, input_h )
        # clip the coordinates
        boxes[:, 0] = np.clip(boxes[:, 0], 0, origin_w -1)
        boxes[:, 2] = np.clip(boxes[:, 2], 0, origin_w -1)
        boxes[:, 1] = np.clip(boxes[:, 1], 0, origin_h -1)
        boxes[:, 3] = np.clip(boxes[:, 3], 0, origin_h -1)
        # Object confidence
        confs = boxes[:, 4]
        # Sort by the confs
        boxes = boxes[np.argsort(-confs)]
        # Perform non-maximum suppression
        keep_boxes = []
        while boxes.shape[0]:
            large_overlap = self.bbox_iou(np.expand_dims(boxes[0, :4], 0), boxes[:, :4]) > nms_thres
            label_match = boxes[0, -1] == boxes[:, -1]
            # Indices of boxes with lower confidence scores, large IOUs and matching labels
            invalid = large_overlap & label_match
            keep_boxes += [boxes[0]]
            boxes = boxes[~invalid]
        boxes = np.stack(keep_boxes, 0) if len(keep_boxes) else np.array([])
        return boxes
    
    def xywh2xyxy(self, x, origin_h, origin_w, input_w, input_h):
        """
        description:    Convert nx4 boxes from [x, y, w, h] to [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right
        param:
            origin_h:   height of original image
            origin_w:   width of original image
            x:          A boxes numpy, each row is a box [center_x, center_y, w, h]
        return:
            y:          A boxes numpy, each row is a box [x1, y1, x2, y2]
        """
        y = np.zeros_like(x)
        r_w = input_w / origin_w
        r_h = input_h / origin_h
        if r_h > r_w:
            y[:, 0] = x[:, 0] - x[:, 2] / 2
            y[:, 2] = x[:, 0] + x[:, 2] / 2
            y[:, 1] = x[:, 1] - x[:, 3] / 2 - (input_h - r_w * origin_h) / 2
            y[:, 3] = x[:, 1] + x[:, 3] / 2 - (input_h - r_w * origin_h) / 2
            y /= r_w
        else:
            y[:, 0] = x[:, 0] - x[:, 2] / 2 - (input_w - r_h * origin_w) / 2
            y[:, 2] = x[:, 0] + x[:, 2] / 2 - (input_w - r_h * origin_w) / 2
            y[:, 1] = x[:, 1] - x[:, 3] / 2
            y[:, 3] = x[:, 1] + x[:, 3] / 2
            y /= r_h

        return y

    def bbox_iou(self, box1, box2, x1y1x2y2=True):
        """
        description: compute the IoU of two bounding boxes
        param:
            box1: A box coordinate (can be (x1, y1, x2, y2) or (x, y, w, h))
            box2: A box coordinate (can be (x1, y1, x2, y2) or (x, y, w, h))            
            x1y1x2y2: select the coordinate format
        return:
            iou: computed iou
        """
        if not x1y1x2y2:
            # Transform from center and width to exact coordinates
            b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2
            b1_y1, b1_y2 = box1[:, 1] - box1[:, 3] / 2, box1[:, 1] + box1[:, 3] / 2
            b2_x1, b2_x2 = box2[:, 0] - box2[:, 2] / 2, box2[:, 0] + box2[:, 2] / 2
            b2_y1, b2_y2 = box2[:, 1] - box2[:, 3] / 2, box2[:, 1] + box2[:, 3] / 2
        else:
            # Get the coordinates of bounding boxes
            b1_x1, b1_y1, b1_x2, b1_y2 = box1[:, 0], box1[:, 1], box1[:, 2], box1[:, 3]
            b2_x1, b2_y1, b2_x2, b2_y2 = box2[:, 0], box2[:, 1], box2[:, 2], box2[:, 3]

        # Get the coordinates of the intersection rectangle
        inter_rect_x1 = np.maximum(b1_x1, b2_x1)
        inter_rect_y1 = np.maximum(b1_y1, b2_y1)
        inter_rect_x2 = np.minimum(b1_x2, b2_x2)
        inter_rect_y2 = np.minimum(b1_y2, b2_y2)
        # Intersection area
        inter_area = np.clip(inter_rect_x2 - inter_rect_x1 + 1, 0, None) * \
                    np.clip(inter_rect_y2 - inter_rect_y1 + 1, 0, None)
        # Union Area
        b1_area = (b1_x2 - b1_x1 + 1) * (b1_y2 - b1_y1 + 1)
        b2_area = (b2_x2 - b2_x1 + 1) * (b2_y2 - b2_y1 + 1)

        iou = inter_area / (b1_area + b2_area - inter_area + 1e-16)

        return iou

    def render_box(self, img, box, color=(200, 200, 200)):
        """
        Render a box. Calculates scaling and thickness automatically.
        :param img: image to render into
        :param box: (x1, y1, x2, y2) - box coordinates
        :param color: (b, g, r) - box color
        :return: updated image
        """
        x1, y1, x2, y2 = box
        thickness = int(
            round(
                (img.shape[0] * img.shape[1])
                / (_LINE_THICKNESS_SCALING * _LINE_THICKNESS_SCALING)
            )
        )
        thickness = max(1, thickness)
        img = cv2.rectangle(
            img,
            (int(x1), int(y1)),
            (int(x2), int(y2)),
            color,
            thickness=thickness
        )
        return img

    def render_filled_box(self, img, box, color=(200, 200, 200)):
        """
        Render a box. Calculates scaling and thickness automatically.
        :param img: image to render into
        :param box: (x1, y1, x2, y2) - box coordinates
        :param color: (b, g, r) - box color
        :return: updated image
        """
        x1, y1, x2, y2 = box
        img = cv2.rectangle(
            img,
            (int(x1), int(y1)),
            (int(x2), int(y2)),
            color,
            thickness=cv2.FILLED
        )
        return img

    def get_text_size(self, img, text, normalised_scaling=1.0):
        """
        Get calculated text size (as box width and height)
        :param img: image reference, used to determine appropriate text scaling
        :param text: text to display
        :param normalised_scaling: additional normalised scaling. Default 1.0.
        :return: (width, height) - width and height of text box
        """
        thickness = int(
            round(
                (img.shape[0] * img.shape[1])
                / (_TEXT_THICKNESS_SCALING * _TEXT_THICKNESS_SCALING)
            )
            * normalised_scaling
        )
        thickness = max(1, thickness)
        scaling = img.shape[0] / _TEXT_SCALING * normalised_scaling
        return cv2.getTextSize(text, cv2.FONT_HERSHEY_SIMPLEX, scaling, thickness)[0]


    def render_text(self, img, text, pos, color=(200, 200, 200), normalised_scaling=1.0):
        """
        Render a text into the image. Calculates scaling and thickness automatically.
        :param img: image to render into
        :param text: text to display
        :param pos: (x, y) - upper left coordinates of render position
        :param color: (b, g, r) - text color
        :param normalised_scaling: additional normalised scaling. Default 1.0.
        :return: updated image
        """
        x, y = pos
        thickness = int(
            round(
                (img.shape[0] * img.shape[1])
                / (_TEXT_THICKNESS_SCALING * _TEXT_THICKNESS_SCALING)
            )
            * normalised_scaling
        )
        thickness = max(2, thickness)
        scaling = img.shape[0] / _TEXT_SCALING * normalised_scaling
        size = self.get_text_size(img, text, normalised_scaling)
        cv2.putText(
            img,
            text,
            (int(x), int(y + size[1])),
            cv2.FONT_HERSHEY_SIMPLEX,
            scaling,
            color,
            thickness=thickness,
        )
        return img

    def plot_one_box(self, x, img, color=None, label=None, line_thickness=None):
        """
        description: Plots one bounding box on image img,
                    this function comes from YoLov5 project.
        param: 
            x:      a box likes [x1,y1,x2,y2]
            img:    a opencv image object
            color:  color to draw rectangle, such as (0,255,0)
            label:  str
            line_thickness: int
        return:
            no return

        """
        tl = (
            line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1
        )  # line/font thickness
        if color == None:
            color = [np.random.randint(0, 255) for _ in range(3)]
        c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))
        
        cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)
        if label:
            tf = max(tl - 1, 1)  # font thickness
            t_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]
            c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3
            cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA)  # filled
            cv2.putText(
                img,
                label,
                (c1[0], c1[1] - 2),
                0,
                tl / 3,
                [225, 255, 255],
                thickness=tf,
                lineType=cv2.LINE_AA,
            )
        return img
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224
  • 225
  • 226
  • 227
  • 228
  • 229
  • 230
  • 231
  • 232
  • 233
  • 234
  • 235
  • 236
  • 237
  • 238
  • 239
  • 240
  • 241
  • 242
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • 253
  • 254
  • 255
  • 256
  • 257
  • 258
  • 259
  • 260
  • 261
  • 262
  • 263
  • 264
  • 265
  • 266
  • 267
  • 268
  • 269
  • 270
  • 271
  • 272
  • 273
  • 274
  • 275
  • 276
  • 277
  • 278
  • 279
  • 280
  • 281
  • 282
  • 283
  • 284
  • 285
  • 286
  • 287
  • 288
  • 289
  • 290
  • 291
  • 292
  • 293
  • 294
  • 295
  • 296
  • 297
  • 298
  • 299
  • 300
  • 301
  • 302
  • 303
  • 304
  • 305
  • 306
  • 307
  • 308
  • 309
  • 310
  • 311
  • 312
  • 313
  • 314
  • 315
  • 316
  • 317
  • 318
  • 319
  • 320
  • 321
  • 322
  • 323
  • 324
  • 325
  • 326
  • 327
  • 328
  • 329
  • 330
  • 331
  • 332
  • 333
  • 334
  • 335
  • 336
  • 337
  • 338
  • 339
  • 340
  • 341
  • 342
  • 343
  • 344
  • 345
  • 346
  • 347
  • 348
  • 349
  • 350
  • 351
  • 352
  • 353
  • 354
  • 355
  • 356
  • 357
  • 358
  • 359
  • 360
  • 361
  • 362
  • 363
  • 364
  • 365
  • 366
  • 367
  • 368
  • 369
  • 370
  • 371
  • 372
  • 373
  • 374
  • 375
  • 376
  • 377
  • 378
  • 379
  • 380
  • 381
  • 382
  • 383
  • 384
  • 385
  • 386
  • 387
  • 388
  • 389
  • 390
  • 391
  • 392
  • 393
  • 394
  • 395
  • 396
  • 397
  • 398
  • 399
  • 400
  • 401
  • 402
  • 403
  • 404
  • 405
  • 406
  • 407
  • 408
  • 409
  • 410
  • 411
  • 412
  • 413
  • 414
  • 415
  • 416
  • 417
  • 418
  • 419
  • 420
  • 421
  • 422
  • 423
  • 424
  • 425
  • 426
  • 427
  • 428
  • 429
  • 430
  • 431
  • 432
  • 433
  • 434
  • 435
  • 436
  • 437
  • 438
  • 439
  • 440
  • 441
  • 442
  • 443
  • 444
  • 445
  • 446
  • 447
  • 448
  • 449
  • 450
  • 451
  • 452
  • 453
  • 454
  • 455
  • 456
  • 457
  • 458
  • 459
  • 460
  • 461
  • 462
  • 463
  • 464
  • 465
  • 466
  • 467
  • 468
  • 469
  • 470
  • 471
  • 472
  • 473
  • 474
  • 475
  • 476
  • 477
  • 478
  • 479
  • 480
  • 481
  • 482
  • 483
  • 484
  • 485
  • 486
  • 487
  • 488
  • 489
  • 490
  • 491
  • 492
  • 493

model_0目录

放置训练好的模型。model.plan文件就是我们训练好的模型,可以参考triton部署yolov5笔记(三)来获取model.plan文件,或者按你自己的方法来获取tensorrt加速部署,最终得到该文件。

plugins目录

放置tensorrt模型转换过程中生成的.so文件,libmyplugins.so。

images目录

我在这个目录下建立了两个目录,一个是input_img,放置输入图片,一个是output_img,放置输出图片。在输出图片文件夹中,会根据当前时间自动建立按日期命名的文件夹,生成的图片名称为uuid随机命名。

Docker Run

运行docker容器,开启服务

sudo docker run --gpus all -it --rm --name tensorrt-server --shm-size=1g -p8080:8000 -p8081:8001 -p8082:8002 -v $(pwd)/models:/models -v $(pwd)/plugins:/plugins -v $(pwd)/images:/images --env LD_PRELOAD=/plugins/libmyplugins.so nvcr.io/nvidia/tritonserver:21.09-py3
tritonserver --model-repository=/models --strict-model-config=false --log-verbose 1
  • 1
  • 2

客户端

运行client.py文件

import numpy as np
import cv2
import tritonclient.http as httpclient
import time

if __name__ == '__main__':
    triton_client = httpclient.InferenceServerClient(url='127.0.0.1:8080')
    img_path = '/images/input_img/4.jpg'
    # img_path = 'https://pic.rmb.bdstatic.com/bjh/down/e0b159fdbd9eba47bbfbdff212492bd4.jpeg'
    input_data0 = np.array([img_path.encode("utf-8")],dtype=np.object_)
    print(input_data0)
    inputs = []
    inputs.append(httpclient.InferInput('input0', [1], "BYTES"))
    inputs[0].set_data_from_numpy(input_data0, binary_data=True)
    outputs = []
    outputs.append(httpclient.InferRequestedOutput('output0', binary_data=False)
    t1 = time.time()
    results = triton_client.infer('custom_model', inputs=inputs, outputs=outputs)
    t2 = time.time()
    print('inference time is: {}ms'.format(1000 * (t2 - t1)))
    output_data0 = results.as_numpy('output0')
    print(output_data0)
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/小丑西瓜9/article/detail/443533
推荐阅读
相关标签
  

闽ICP备14008679号