赞
踩
C++ and Python with three threads, read, predict and write · ZouJiu1/ultralytics@edd67f0
支持C++和python模块的,共三个线程合作,每个线程独立工作,就像消费者和生产者。
读取视频的线程是一个生产者,对视频帧做inference的线程是消费者,对视频帧做inference的线程会给出predict的结果,现在它是一个生产者,绘制和保存线程在视频帧上绘制矩形框,并且保存到视频中,它是一个消费者,所以第二个线程既是生产者也是消费者。要实现这些功能,需要线程安全的队列。Python标准库提供了一个线程安全的队列库Queue,是queue库内的一个类。C++没有提供这样的线程安全的队列。所以我自己写了一个线程安全的队列。参考是这本书以及Python文档,
C++ Concurrency in Action PRACTICAL MULTITHREADING,
Queue 在python标准库的queue内, 以及书内的codes
https://github.com/ZouJiu1/multithread_Cplusplus。
自己写好的C++线程安全的队列,可以设置队列的最大容量,达到最大容量时就会等待直到容量小于最大容量,不会一直等待,只要队列不空就可以get,只要队列不满就可以put,还实现了front,empty,full,qsize,join,task_done等功能,front, empty, full, qsize都是常见的功能,也就是front队列的前端,empty是否空,full是否满,qsize队列当前的大小,不同的线程可以相同合作,消费者get,生产者put。
下面就是实现的线程安全的队列,C++ yolov8的多线程运行codes,以及 python yolov8的多线程运行codes,都在下面的附录里面,可以转到后面看的。
需要使用C++20的标准来编译才能通过,标准太低编译不能通过的
/* ############# ###ZouJiu ###20240421 ###1069679911@qq.com #https://zoujiu.blog.csdn.net/ #https://zhihu.com/people/zoujiu1 #https://github.com/ZouJiu1 ############# */ #ifndef THREAD_SAFE_QUEUE_HPP #define THREAD_SAFE_QUEUE_HPP #include <queue> #include <condition_variable> #include <mutex> #include <iostream> template<typename T> class ThreadQueue : public std::queue<T> { private: std::mutex mut; std::condition_variable condition; std::queue<T> queue_data; int maxsize = -1; public: int unfinished_tasks = 0; int finished_tasks = 0; ThreadQueue() {} ~ThreadQueue() {} ThreadQueue(int max_size) : maxsize(max_size) { } void task_done() { std::lock_guard<std::mutex> lock_(mut); int unfinished = unfinished_tasks - 1; if (unfinished <= 0) { if (unfinished < 0) { std::cerr << "error, task_done() called too many times, exit -1." << std::endl; exit(-1); } //condition.notify_one(); } unfinished_tasks = unfinished; } void join() { std::unique_lock<std::mutex> lock_(mut); condition.wait(lock_, [this] {return unfinished_tasks == 0; }); } int qsize() { std::lock_guard<std::mutex> lock_(mut); return queue_data.size(); } T front() { std::unique_lock<std::mutex> lock_(mut); condition.wait(lock_, [this] {return !empty(); }); return queue_data.front(); } bool empty() { return queue_data.size() == 0; } bool full() { return this->maxsize >= 0 && this->maxsize <= queue_data.size(); } void put(T&& a) { std::unique_lock<std::mutex> lock_(mut); condition.wait(lock_, [this] {return !full(); }); queue_data.push(a); unfinished_tasks++; condition.notify_one(); } void get(T& a) { std::unique_lock<std::mutex> lock_(mut); condition.wait(lock_, [this] {return !empty(); }); a = queue_data.front(); queue_data.pop(); condition.notify_one(); } }; #endif
该程序既可以运行window10也可以运行在Ubuntu。
导出的方式是
from ultralytics import YOLO
model = YOLO('yolov8n.pt')
model.export(format=r"onnx", simplify=True)
1)根据
opencv get-started, 下载window的release版本,像
windows.exe。然后安装好就行
下面你需要加入一些路径到环境变量,setting设置->system系统->about关于->high system set高级系统设置->environment variable环境变量->sys variable系统变量->Path->create.
C:\ruanjian\opencv\opencv\build\x64\vc16\lib
C:\ruanjian\opencv\opencv\build\include
C:\ruanjian\opencv\opencv\build\x64\vc16\bin
3) Microsoft Visual studio–> project项目 --> YOLOv8-Threads-Cpp-Python-Video and property 属性–> VC++ directory
IncludePath 包括目录: C:\ruanjian\opencv\opencv\build\include
LibraryPath 库目录: C:\ruanjian\opencv\opencv\build\x64\vc16\lib
Microsoft Visual studio–> project项目 --> YOLOv8-Threads-Cpp-Python-Video and property --> linker链接器 --> input输入 --> AdditionalDependencies 附加额外依赖
opencv_world*.lib
Microsoft Visual studio–> project 项目–> YOLOv8-Threads-Cpp-Python-Video and property属性–> common常规 --> C++ language standard 语言标准 --> C++20. CXX_STANDARD 应该大于或者等于 C++17.
应该选择Release而不是Debug,因 opencv_world.lib静态库不支持debug. 若想debug,需要从opencv的source编译,并且选择BUILD_opencv_world=False,不用world lib。然后就可以产出很多静态库,加入到VS的库目录输入内,就可以debug了的。
在编译和生成codes以后,就可以运行程序了,可能需要修改文件multi_thread_read_predict_write.cpp中的一些路径,导出的Onnx文件的batch_size应该和code内的batch_size保持一致不然不能运行起来。
若电脑还有cuda,那么在编译Opencv的时候就可以开启选项cuda,并且设置code内的GPU=true。
GPU
batch_size
projectBasePath
Onnx_path
video_path
输入的视频可以从下面的网址下载
https://motchallenge.net/vis/MOT16-09
https://motchallenge.net/vis/MOT16-06
https://www.alipan.com/s/p9MQVG1wrCr
1). 安装opencv
git clone https://github.com/opencv/opencv.git cd opencv git clone https://github.com/opencv/opencv_contrib.git mkdir build cd build cmake -DCMAKE_BUILD_TYPE=RELEASE -DWITH_FFMPEG=ON -DWITH_OPENCL=ON -DWITH_CUDA=ON -DOPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules .. make -j10 make install vim ~/.bashrc # add below to .bashrc tail 加入到环境变量中 ... export PATH=/usr/local/lib:/usr/local/include/opencv4:$PATH export LIB_PATH=/usr/local/lib:$LIB_PATH export LD_LIB_PATH=/usr/local/lib:$LD_LIB_PATH ... source ~/.bashrc
git clone https://github.com/ultralytics/ultralytics.git
cd examples/YOLOv8-Threads-Cpp-Python-Video
mkdir build
cd build
cmake ..
make
./YOLOv8ThreadsCppPythonVideo
若C++程序不能正常运行,可能是输入的mp4格式不支持,可以改用avi格式的输入视频,可以正常运行的,然后修改输出的codes。
string output_path = video_path.substr(0, video_path.size() - 4) + "_output.avi";
const int fourcc = VideoWriter::fourcc('X', 'V', 'I', 'D'); // for avi
// cv2.VideoWriter_fourcc('I', '4', '2', '0') // for avi
// cv2.VideoWriter_fourcc('P', 'I', 'M', 'I') // for avi
window或者Linux都可以正常运行的,pip install ultralytics安装库,python multi_thread_read_predict_write.py运行的,也需要做些修改的。
C++ Concurrency in Action PRACTICAL MULTITHREADING
https://github.com/ZouJiu1/multithread_Cplusplus.
OpenCV - Open Computer Vision Library
python yolov8的多线程运行codes
############# ###ZouJiu ###20240421 ###1069679911@qq.com #https://zoujiu.blog.csdn.net/ #https://zhihu.com/people/zoujiu1 #https://github.com/ZouJiu1 ############# import cv2 from queue import Queue from ultralytics import YOLO from threading import Thread def predict_image(model, batch): images = [] while True: image = 0 while len(images) < batch: image = image_que.get() if isinstance(image, int): image_que.task_done() break images.append(image) if len(images)==0: break # result = model.predict(images, batch = batch) result = model.track(images, batch = batch, persist=True) result_que.put(result) for _ in range(len(images)): image_que.task_done() images = [] if isinstance(image, int): break result_que.put(0) def get_image(pth): cap = cv2.VideoCapture(pth) if not cap.isOpened(): cap.release() cv2.destroyAllWindows() image_que.put(0) informa_que.put((-1, -1, -1)) return fps = cap.get(cv2.CAP_PROP_FPS) frame_width = cap.get(cv2.CAP_PROP_FRAME_WIDTH) frame_height = cap.get(cv2.CAP_PROP_FRAME_HEIGHT) informa_que.put((fps, int(frame_width), int(frame_height))) success, frame = cap.read() while success: image_que.put(frame) success, frame = cap.read() cap.release() cv2.destroyAllWindows() image_que.put(0) def write_video(write_path): fps, frame_width, frame_height = informa_que.get() if fps < 0 and frame_width < 0 and frame_height < 0: return # fourcc = cv2.VideoWriter_fourcc(*"mp4v") fourcc = cv2.VideoWriter_fourcc('X', 'V', 'I', 'D') video_writer = cv2.VideoWriter(write_path, fourcc, fps, (frame_width, frame_height)) informa_que.task_done() informa_que.join() while True: result = result_que.get() if isinstance(result, int): result_que.task_done() video_writer.release() cv2.destroyAllWindows() break for ind, _ in enumerate(result): # frame = result[ind].plot() frame = result[ind].orig_img video_writer.write(frame) result_que.task_done() if __name__ == "__main__": # model = YOLO('yolov8n.pt') # model = YOLO('yolov8n-seg.pt') model = YOLO('yolov8n-pose.pt') batch_size = 3 Image_in_queue_maxsize = 900 Result_in_queue_maxsize = 900 video_path = r'C:\Users\10696\Desktop\CV\MOT16-06-raw.mp4' outpath = video_path.replace('.mp4', "_output.avi") image_que = Queue(maxsize = Image_in_queue_maxsize) result_que = Queue(maxsize = Result_in_queue_maxsize) informa_que = Queue(1) t0 = Thread(target=get_image, args = (video_path,)) t1 = Thread(target=predict_image, args=(model, batch_size)) t2 = Thread(target=write_video, args=(outpath,)) t0.setDaemon(True) t1.setDaemon(True) t2.setDaemon(True) t0.start() t1.start() t2.start() t0.join() t1.join() t2.join() image_que.join() result_que.join()
C++ yolov8的多线程运行codes
/* ############# ###ZouJiu ###20240421 ###1069679911@qq.com #https://zoujiu.blog.csdn.net/ #https://zhihu.com/people/zoujiu1 #https://github.com/ZouJiu1 ############# */ #include <opencv2/dnn.hpp> #include <opencv2/imgproc.hpp> #include <opencv2/imgcodecs.hpp> #include <opencv2/core.hpp> #include <fstream> #include <sstream> #include <vector> #include <iostream> #include <opencv2/highgui.hpp> #include <thread> #include "thread_safe_Queue.hpp" using namespace std; using namespace cv; using namespace cv::dnn; struct infor { double fps; int width; int height; }; struct Result { vector<Mat> outs; Mat img; }; struct letterbox{ float paddingValue = 116 - 2; bool swapRB = false; int inpWidth = 640; int inpHeight = 640; Scalar scale = 1 / 255.0; Scalar mean = 0.0; float conf_threshold = 0.25; float iou_threshold = 0.7; }; vector<string> classes; bool GPU = false; int batch_size = 2; float conf_threshold = 0.25; float iou_threshold = 0.7; string projectBasePath = "C:\\Users\\10696\\Desktop\\CV\\ultralytics\\examples\\YOLOv8-CPP-Inference"; // Set your ultralytics base path string Onnx_path = projectBasePath + "\\yolov8n_2.onnx"; string classes_path = projectBasePath + "\\classes.txt"; string video_path = projectBasePath + "\\MOT16-06-raw.mp4"; string output_path = video_path.substr(0, video_path.size() - 4) + "_output.mp4"; // string output_path = video_path.substr(0, video_path.size() - 4) + "_output.avi"; string ModelType = "yolov8"; int output_length = 1; int Image_in_queue_maxsize = 390; int Result_in_queue_maxsize = 390; ThreadQueue<cv::Mat> image_que = ThreadQueue<cv::Mat>(Image_in_queue_maxsize); ThreadQueue<vector<Result>> result_que = ThreadQueue<vector<Result>>(Image_in_queue_maxsize); ThreadQueue<infor> informa_que = ThreadQueue<infor>((int)1); void yoloPostProcessing( std::vector<Mat>&& outs, std::vector<int>& keep_classIds, std::vector<float>& keep_confidences, std::vector<Rect2d>& keep_boxes, float conf_threshold = 0.25, float iou_threshold = 0.7, const std::string& modeltype = "yolov8") { // C:\ruanjian\opencv\opencv\sources\modules\dnn\test\test_onnx_importer.cpp // Retrieve std::vector<int> classIds; std::vector<float> confidences; std::vector<Rect2d> boxes; if (modeltype == "yolov8") { cv::transposeND(outs[0], { 0, 2, 1 }, outs[0]); } // each row is [cx, cy, w, h, conf_obj, conf_class1, ..., conf_class80] for (auto preds : outs) { preds = preds.reshape(1, preds.size[1]); // [1, 8400, 85] -> [8400, 85] for (int i = 0; i < preds.rows; ++i) { // filter out non objects float obj_conf = (modeltype != "yolov8") ? preds.at<float>(i, 4) : 1.0f; if (obj_conf < conf_threshold) continue; Mat scores = preds.row(i).colRange((modeltype != "yolov8") ? 5 : 4, preds.cols); double conf; Point maxLoc; minMaxLoc(scores, 0, &conf, 0, &maxLoc); conf = (modeltype != "yolov8") ? conf * obj_conf : conf; if (conf < conf_threshold) continue; // get bbox coords float* det = preds.ptr<float>(i); double cx = det[0]; double cy = det[1]; double w = det[2]; double h = det[3]; // [x1, y1, x2, y2] boxes.push_back(Rect2d(cx - 0.5 * w, cy - 0.5 * h, cx + 0.5 * w, cy + 0.5 * h)); classIds.push_back(maxLoc.x); confidences.push_back(conf); } } // NMS std::vector<int> keep_idx; dnn::NMSBoxes(boxes, confidences, conf_threshold, iou_threshold, keep_idx); for (auto i : keep_idx) { keep_classIds.push_back(classIds[i]); keep_confidences.push_back(confidences[i]); keep_boxes.push_back(boxes[i]); } } void drawPrediction(int classId, float conf, int left, int top, int right, int bottom, Mat& frame) { //C:\ruanjian\opencv\opencv\sources\samples\dnn\object_detection.cpp rectangle(frame, Point(left, top), Point(right, bottom), Scalar(0, 255, 0)); string label = cv::format("%.2f", conf); if (!classes.empty()) { CV_Assert(classId < (int)classes.size()); label = classes[classId] + ": " + label; } int baseLine; Size labelSize = getTextSize(label, FONT_HERSHEY_SIMPLEX, 0.5, 1, &baseLine); top = max(top, labelSize.height); rectangle(frame, Point(left, top - labelSize.height), Point(left + labelSize.width, top + baseLine), Scalar::all(255), FILLED); putText(frame, label, Point(left, top), FONT_HERSHEY_SIMPLEX, 0.5, Scalar()); } void predict_image(Net&& net, const int batch_size) { cout << "in predict_image function" << endl; ImagePaddingMode paddingMode = static_cast<ImagePaddingMode>(DNN_PMODE_LETTERBOX); letterbox ltr; Size size(ltr.inpWidth, ltr.inpHeight); Image2BlobParams imgParams( ltr.scale, size, ltr.mean, ltr.swapRB, CV_32F, DNN_LAYOUT_NCHW, paddingMode, ltr.paddingValue); Mat inp, img; vector<Mat> outs, batch; vector<MatShape> inLayerShapes; vector<MatShape> outLayerShapes; net.getLayerShapes(MatShape(), 0, inLayerShapes, outLayerShapes); MatShape kk = inLayerShapes[0]; if (inLayerShapes[0][0] != batch_size) { cerr<<""; } int cnt = 0; vector<Range> ranges(3, Range::all()); vector<Result> ret; int img_width = 0; while (true) { ret.clear(); if (batch_size > 1) { batch.clear(); } for (int i = 0; i < batch_size; i++) { image_que.get(img); img_width = img.size[1]; cout << "predict..." << cnt++ <<" " << img_width << endl; if (img_width < 9) { image_que.task_done(); if (batch_size > 1) { for (int j = 0; j < batch.size(); j++) { image_que.task_done(); } } result_que.put({}); return; } if (batch_size > 1) { batch.push_back(img); } } if (batch_size > 1) { inp = blobFromImagesWithParams(batch, imgParams); } else { inp = blobFromImageWithParams(img, imgParams); } net.setInput(inp); net.forward(outs, net.getUnconnectedOutLayersNames()); if (batch_size > 1) { for (int ic = 0; ic < batch.size(); ic++) { ranges[0].start = ic; ranges[0].end = ic + 1; ret.push_back({ {outs[0](ranges)}, std::move(batch[ic]) }); image_que.task_done(); } } else { ret.push_back({ std::move(outs), std::move(img)}); image_que.task_done(); } result_que.put(std::move(ret)); } } void read_video_image(string video_path) { Mat img; bool success; VideoCapture cap = VideoCapture(video_path); MatShape tensor(3, 1); img.create(tensor, 0); if (!cap.isOpened()) { cap.release(); image_que.put(std::move(img)); informa_que.put({ -1, -1, -1 }); return; } double fps = cap.get(CAP_PROP_FPS); double frame_width = cap.get(CAP_PROP_FRAME_WIDTH); double frame_height = cap.get(CAP_PROP_FRAME_HEIGHT); infor tp = { fps, (int)(frame_width), (int)(frame_height) }; informa_que.put(std::move(tp)); success = cap.read(img); int cnt = 0; //string imr = "C:\\Users\\10696\\Desktop\\CV\\ultralytics\\examples\\YOLOv8-CPP-Inference\\tmp"; while (success) { cout << "read..." << cnt++ << " " << (int)success << endl; //imwrite(imr + "\\" + to_string(cnt) + "_.jpg", img); image_que.put(std::move(img.clone())); success = cap.read(img); //if (cnt == 31) break; } cap.release(); img.create(tensor, 0); image_que.put(std::move(img)); } void write_to_video(string output_path) { cout << "in write_to_video function" << endl; int frame_width, frame_height; double fps; infor inf; informa_que.get(inf); fps = inf.fps; frame_width = inf.width; frame_height = inf.height; if (fps < 0 && frame_width < 0 && frame_height < 0) { return; } cout << fps << " " << frame_width << " " << frame_height << endl; const int fourcc = VideoWriter::fourcc('m', 'p', '4', 'v'); // const int fourcc = VideoWriter::fourcc('X', 'V', 'I', 'D'); // for avi // cv2.VideoWriter_fourcc('I', '4', '2', '0') // for avi // cv2.VideoWriter_fourcc('P', 'I', 'M', 'I') // for avi VideoWriter video_writer = VideoWriter(output_path, fourcc, fps, Size(frame_width, frame_height)); if (!video_writer.isOpened()) { cout << "video output writer is wrong......" << endl; exit(-1); } cout << "in video_writer" << endl; informa_que.task_done(); informa_que.join(); Mat inp; vector<int> keep_classIds; vector<float> keep_confidences; vector<Rect2d> keep_boxes; vector<Rect> boxes; // rescale boxes back to original image Image2BlobParams paramNet; letterbox ltr; Size size(ltr.inpWidth, ltr.inpHeight); ImagePaddingMode paddingMode = static_cast<ImagePaddingMode>(DNN_PMODE_LETTERBOX); paramNet.scalefactor = ltr.scale; paramNet.size = size; paramNet.mean = ltr.mean; paramNet.swapRB = ltr.swapRB; paramNet.paddingmode = paddingMode; vector<Result> result; int cnt = 0; //string imr = "C:\\Users\\10696\\Desktop\\CV\\ultralytics\\examples\\YOLOv8-CPP-Inference\\tmp"; while (true) { result.clear(); result_que.get(result); cout << "result_que.unfinished_tasks�� " << result.size() << endl; if (result.size() == 0) { result_que.task_done(); video_writer.release(); break; } for (int i = 0; i < result.size(); i++) { // Retrieve keep_boxes.clear(); keep_confidences.clear(); boxes.clear(); keep_classIds.clear(); yoloPostProcessing(std::move(result[i].outs), keep_classIds, keep_confidences, keep_boxes, conf_threshold, iou_threshold, ModelType); for (auto& box : keep_boxes) { boxes.push_back(Rect(cvFloor(box.x), cvFloor(box.y), cvFloor(box.width - box.x), cvFloor(box.height - box.y))); } paramNet.blobRectsToImageRects(boxes, boxes, result[i].img.size()); for (size_t idx = 0; idx < boxes.size(); ++idx) { Rect box = boxes[idx]; drawPrediction(keep_classIds[idx], keep_confidences[idx], box.x, box.y, box.width + box.x, box.height + box.y, result[i].img); } cout << "write frame......." << cnt++ << endl; video_writer.write(result[i].img); //imwrite(imr +"\\" + to_string(cnt) + ".jpg", result[i].img); } result_que.task_done(); } cout << "post break" << endl; } /* ThreadQueue<int> trr = ThreadQueue<int>(100); void putput() { for (int i = 0; i < 10000000; i++) { trr.put(i); int a = 0; } } void getget() { int a; for (int i = 0; i < 10; i++) { a = trr.get(); int k = 0; } } */ void YoLoForward() { /* thread tt(getget); thread tt1(putput); tt.join(); tt1.join(); exit(0); */ if (classes.empty()) { ifstream ifs(classes_path.c_str()); if (!ifs.is_open()) CV_Error(Error::StsError, "File " + classes_path + " not found"); string line; while (getline(ifs, line)) { classes.push_back(line); } ifs.close(); } Net net = readNet(Onnx_path, "", "onnx"); if (GPU) { net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); } else { net.setPreferableBackend(DNN_BACKEND_DEFAULT); net.setPreferableTarget(DNN_TARGET_CPU); } thread t0(read_video_image, video_path); thread t1(predict_image, std::move(net), batch_size); thread t2(write_to_video, output_path); t0.join(); t1.join(); t2.join(); image_que.join(); result_que.join(); cout << "\n\nAll finished......" << endl; return; } int main(int argc, char** argv) { YoLoForward(); return EXIT_SUCCESS; }
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。