零基础入门Jetson Nano——MediaPipe双版本（CPU+GPU）的安装与使用_mediapipe gpu

---

前言

MediaPipe 是一款由 Google Research 开发并开源的多媒体机器学习模型应用框架，可以直接调用其API完成目标检测、人脸检测以及关键点检测等。本文将详细介绍MediaPipe在嵌入式平台Jeston Nano上的安装与使用。由于GPU版需要更改许多文件，打开文件，“CTRL+F"可以搜索文件。

---

一、准备工作

1、下载MediaPipe

git clone -b v0.8.5 https://github.com/google/mediapipe
1

2、安装官方编译器 bazel4.0.0
安装基本环境

sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python3
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt install openjdk-11-jdk
1
2
3
4

从github下载bazel4.0.0

新建文件夹 bazel-4.0.0并将下载的文件移动到该文件夹下解压
如果下载速度慢的话，可以直接用我下面分享的百度网盘里的压缩包

mkdir bazel-4.0.0&&cd bazel-4.0.0
mv  ~/bazel-4.0.0-dist-zip bazel-4.0.0
unzip bazel-4.0.0-dist.zip
bash ./compile.sh
sudo cp output/bazel /usr/local/bin
1
2
3
4
5

安装完成后运行下述代码检查版本信息，不报错即安装成功

3、安装MediaPipe所需依赖

sudo apt install -y python3-dev
sudo apt install -y cmake
1
2

4、安装编译器protobuf-compiler

cd ~
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.19.1/protoc-3.19.1-linux-aarch_64.zip
unzip protoc-3.19.1-linux-aarch_64.zip -d protoc3.19.1
1
2
3

将protobuf-compilerv3.19.1中的“bin”和include下的“google”复制到/mediapipe中。

二、CPU版本

1、修改mediapipe/setup.py

cd mediapipe
nano setup.py    # 打开setup.py文件
1
2

定位到

protoc_command = [self._protoc, ‘-I.’, ‘–python_out=.’, source]

修改为

protoc_command = [self._protoc, ‘-I.’, ‘-I/usr/local/include’, ‘–python_out=.’, source]

2、删除不必要的OpenCV模块和链接器标志

sed -i -e "/\"imgcodecs\"/d;/\"calib3d\"/d;/\"features2d\"/d;/\"highgui\"/d;/\"video\"/d;/\"videoio\"/d" third_party/BUILD
sed -i -e "/-ljpeg/d;/-lpng/d;/-ltiff/d;/-lImath/d;/-lIlmImf/d;/-lHalf/d;/-lIex/d;/-lIlmThread/d;/-lrt/d;/-ldc1394/d;/-lavcodec/d;/-lavformat/d;/-lavutil/d;/-lswscale/d;/-lavresample/d" third_party/BUILD
1
2

3、修改mediapipe/third_party/BUILD

nano third_party/BUILD
1

定位到

“WITH_ITT”: “OFF”,
“WITH_JASPER”: “OFF”,
“WITH_WEBP”: “OFF”,

修改为

“WITH_ITT”: “OFF”,
“WITH_JASPER”: “OFF”,
“WITH_WEBP”: “OFF”,
“ENABLE_NEON”: “OFF”,
“WITH_TENGINE”: “OFF”,

4、升级gcc
查看gcc版本

gcc -v
g++ -v
1
2

更新gcc版本号

sudo apt-get update
sudo apt-get install gcc-8
sudo apt-get install g++-8
cd /usr/bin
sudo rm gcc g++
sudo ln -s gcc-8 gcc
sudo ln -s g++-8 g++
1
2
3
4
5
6
7

查看版本

gcc -v
1

5、编译

python3 setup.py gen_protos && python3 setup.py bdist_wheel
1

编译成功后，mediapipe文件夹下出现一个名为dist的新文件夹，里面存放mediapipe的.whl文件。

6、安装MediaPipe

python3 -m pip install cython
python3 -m pip install numpy
python3 -m pip install pillow
python3 -m pip install mediapipe/dist/mediapipe-0.8-cp36-cp36-linux_aarch64.whl
1
2
3
4

7、编译运行GPU版Holistic示例
在mediapipe文件夹下，运行下述命令编译Holistic GPU版：

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/holistic_tracking:holistic_tracking_gpu
1

编译成功后，运行下述命令运行生成的程序：

export GLOG_logtostderr=1
./bazel-bin/mediapipe/examples/desktop/holistic_tracking/holistic_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/holistic_tracking/holistic_tracking_gpu.pbtxt
1
2

如果你的摄像头是USB的，那么运行官方示例，摄像头会直接打开，如果你是CSI摄像头，在mediapipe\examples\desktop\demo_run_graph_main_gpu.cc中设置正确的GStream Pipeline，否则无法打开摄像头。

三、GPU版本

我们不难发现，即便我们安装的是CPU版本的MediaPipe,我们也可以通过终端编译运行GPU版本的示例。但是当我们运行自己的程序，直接“import mediapipe”时，会发现Jetson Nano调用的是CPU版本的模型，即便我们在终端指定了用GPU来运行程序，CPU版本MediaPipe的检测帧率十分感人，完全无法满足检测要求。Jetson Nano区别于树莓派的一个特点就是，它有媲美一般显卡的CUDA数量，所以我们要想在Jetson Nano上实现MediaPipe检测，必须要充分发挥其特点，用GPU加速来实现MediaPipe检测。
我们定位到MediaPipe文件夹下调用模型的地方，发现模型文件中用到的所有参数都是CPU，在mediapipe/python/solutions中的所有.py文件调用的模型都是CPU版的模型。所以我们要想MediaPipe GPU版在Jetson Nano的实现，就需要一步一步的更改官方文件的内容。下面我将详细说明我们应该如何更改官方文件的内容。

1、配置CUDA路径

cd mediapipe
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
sudo ldconfig
export TF_CUDA_PATHS=/usr/local/cuda:/usr/lib/aarch64-linux-gnu:/usr/include
1
2
3
4

2、修改.bazelrc文件

nano .bazelrc
1

在最后添加下面的内容

build:using_cuda --define=using_cuda=true
build:using_cuda --action_env TF_NEED_CUDA=1
build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain
build --define=tensorflow_enable_mlir_generated_gpu_kernels=0
build:using_cuda --define=tensorflow_enable_mlir_generated_gpu_kernels=1
build:cuda --config=using_cuda
build:cuda --define=using_cuda_nvcc=true

3、配置protoc编译器

sudo mv protoc3.19.1/bin/* /usr/local/bin/
sudo mv protoc3.19.1/include/* /usr/local/include/
sudo chown user /usr/local/bin/protoc
sudo chown -R user /usr/local/include/google
1
2
3
4

注意：user是自己的用户名

4、更改requirements.txt文件

sed -i -e "s/numpy/numpy==1.19.4/g" requirements.txt
sed -i -e "s/opencv-contrib-python/opencv-python/g" requirements.txt
1
2

5、更改setup.py文件

nano setup.py
1

将

version = ‘0.8’

改为

version = ‘0.8.5_cuda102’

将

bazel_command = [
‘bazel’,
‘build’,
‘–compilation_mode=opt’,
‘–define=MEDIAPIPE_DISABLE_GPU=1’,
‘–action_env=PYTHON_BIN_PATH=’ + _normalize_path(sys.executable),
os.path.join(‘mediapipe/modules/’, graph_path),
]

改为

bazel_command = [
‘bazel’,
‘build’,
‘–compilation_mode=opt’,
‘–config=cuda’,
‘–spawn_strategy=local’,
‘–define=no_gcp_support=true’,
‘–define=no_aws_support=true’,
‘–define=no_nccl_support=true’,
‘–copt=-DMESA_EGL_NO_X11_HEADERS’,
‘–copt=-DEGL_NO_X11’,
‘–local_ram_resources=4096’,
‘–local_cpu_resources=3’,
‘–action_env=PYTHON_BIN_PATH=’ + _normalize_path(sys.executable),
os.path.join(‘mediapipe/modules/’, graph_path),
]

将

bazel_command = [
‘bazel’,
‘build’,
‘–compilation_mode=opt’,
‘–define=MEDIAPIPE_DISABLE_GPU=1’,
‘–action_env=PYTHON_BIN_PATH=’ + _normalize_path(sys.executable),
str(ext.bazel_target + ‘.so’),
]

改为

bazel_command = [
‘bazel’,
‘build’,
‘–compilation_mode=opt’,
‘–config=cuda’,
‘–spawn_strategy=local’,
‘–define=no_gcp_support=true’,
‘–define=no_aws_support=true’,
‘–define=no_nccl_support=true’,
‘–copt=-DMESA_EGL_NO_X11_HEADERS’,
‘–copt=-DEGL_NO_X11’,
‘–local_ram_resources=4096’,
‘–local_cpu_resources=3’,
‘–action_env=PYTHON_BIN_PATH=’ + _normalize_path(sys.executable),
str(ext.bazel_target + ‘.so’),
]

将

def run(self):
_check_bazel()
binary_graphs = [
‘face_detection/face_detection_front_cpu’,
‘face_landmark/face_landmark_front_cpu’,
‘hand_landmark/hand_landmark_tracking_cpu’,
‘holistic_landmark/holistic_landmark_cpu’, ‘objectron/objectron_cpu’,
‘pose_landmark/pose_landmark_cpu’
]

改为

def run(self):
_check_bazel()
binary_graphs = [
‘face_detection/face_detection_front_gpu’,
‘face_landmark/face_landmark_front_gpu’,
‘hand_landmark/hand_landmark_tracking_gpu’,
‘holistic_landmark/holistic_landmark_gpu’,
‘objectron/objectron_gpu’,
‘pose_landmark/pose_landmark_gpu’
]

6、更改mediapipe/framework/tool/BUILD

cc_binary(
name = “encode_as_c_string”,
srcs = [“encode_as_c_string.cc”],
visibility = [“//visibility:public”],
deps = [
“@com_google_absl//absl/strings”,
],
linkopts = [“-lm”],
)

7、更改mediapipe/python/BUILD
将

cc_library(
name = “builtin_calculators”,
deps = [
“//mediapipe/calculators/core:gate_calculator”,
“//mediapipe/calculators/core:pass_through_calculator”,
“//mediapipe/calculators/core:side_packet_to_stream_calculator”,
“//mediapipe/calculators/core:split_normalized_landmark_list_calculator”,
“//mediapipe/calculators/core:string_to_int_calculator”,
“//mediapipe/calculators/image:image_transformation_calculator”,
“//mediapipe/calculators/util:detection_unique_id_calculator”,
“//mediapipe/modules/face_detection:face_detection_front_cpu”,
“//mediapipe/modules/face_landmark:face_landmark_front_cpu”,
“//mediapipe/modules/hand_landmark:hand_landmark_tracking_cpu”,
“//mediapipe/modules/holistic_landmark:holistic_landmark_cpu”,
“//mediapipe/modules/objectron:objectron_cpu”,
“//mediapipe/modules/palm_detection:palm_detection_cpu”,
“//mediapipe/modules/pose_detection:pose_detection_cpu”,
“//mediapipe/modules/pose_landmark:pose_landmark_by_roi_cpu”,
“//mediapipe/modules/pose_landmark:pose_landmark_cpu”,
“//mediapipe/modules/selfie_segmentation:selfie_segmentation_cpu”,
],
)

改为

cc_library(
name = “builtin_calculators”,
deps = [
“//mediapipe/calculators/core:gate_calculator”,
“//mediapipe/calculators/core:pass_through_calculator”,
“//mediapipe/calculators/core:side_packet_to_stream_calculator”,
“//mediapipe/calculators/core:split_normalized_landmark_list_calculator”,
“//mediapipe/calculators/core:string_to_int_calculator”,
“//mediapipe/calculators/image:image_transformation_calculator”,
“//mediapipe/calculators/util:detection_unique_id_calculator”,
“//mediapipe/modules/face_detection:face_detection_front_cpu”,
“//mediapipe/modules/face_detection:face_detection_front_gpu”,
“//mediapipe/modules/face_landmark:face_landmark_front_cpu”,
“//mediapipe/modules/face_landmark:face_landmark_front_gpu”,
“//mediapipe/modules/hand_landmark:hand_landmark_tracking_gpu”,
“//mediapipe/modules/holistic_landmark:holistic_landmark_cpu”,
“//mediapipe/modules/holistic_landmark:holistic_landmark_gpu”,
#“//mediapipe/modules/objectron:objectron_cpu”,
“//mediapipe/modules/objectron:objectron_gpu”,
“//mediapipe/modules/palm_detection:palm_detection_gpu”,
“//mediapipe/modules/pose_detection:pose_detection_gpu”,
“//mediapipe/modules/pose_landmark:pose_landmark_by_roi_gpu”,
“//mediapipe/modules/pose_landmark:pose_landmark_cpu”,
“//mediapipe/modules/pose_landmark:pose_landmark_gpu”,
“//mediapipe/modules/selfie_segmentation:selfie_segmentation_cpu”,
“//mediapipe/modules/selfie_segmentation:selfie_segmentation_gpu”,
“//mediapipe/gpu:image_frame_to_gpu_buffer_calculator”,
“//mediapipe/calculators/image:color_convert_calculator”,
],
)

8、修改mediapipe/modules/holistic_landmark/holistic_landmark_gpu.pbtxt
将

#Predicts pose landmarks.
node {
calculator: “PoseLandmarkGpu”
input_stream: “IMAGE:image”
input_side_packet: “MODEL_COMPLEXITY:model_complexity”
input_side_packet: “SMOOTH_LANDMARKS:smooth_landmarks”
output_stream: “LANDMARKS:pose_landmarks”
output_stream: “ROI_FROM_LANDMARKS:pose_landmarks_roi”
output_stream: “DETECTION:pose_detection”
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}

node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Predicts pose landmarks.
node {
calculator: “PoseLandmarkGpu”
input_stream: “IMAGE:image”
input_side_packet: “MODEL_COMPLEXITY:model_complexity”
input_side_packet: “SMOOTH_LANDMARKS:smooth_landmarks”
output_stream: “LANDMARKS:pose_landmarks”
output_stream: “ROI_FROM_LANDMARKS:pose_landmarks_roi”
output_stream: “DETECTION:pose_detection”

将

#Predicts left and right hand landmarks based on the initial pose landmarks.
node {
calculator: “HandLandmarksLeftAndRightGpu”
input_stream: “IMAGE:image”
input_stream: “POSE_LANDMARKS:pose_landmarks”
output_stream: “LEFT_HAND_LANDMARKS:left_hand_landmarks”
output_stream: “RIGHT_HAND_LANDMARKS:right_hand_landmarks”
}

改为

#Predicts left and right hand landmarks based on the initial pose landmarks.
node {
calculator: “HandLandmarksLeftAndRightGpu”
input_stream: “IMAGE:image_gpu”
input_stream: “POSE_LANDMARKS:pose_landmarks”
output_stream: “LEFT_HAND_LANDMARKS:left_hand_landmarks”
output_stream: “RIGHT_HAND_LANDMARKS:right_hand_landmarks”
}

将

#Predicts face landmarks based on the initial pose landmarks.
node {
calculator: “FaceLandmarksFromPoseGpu”
input_stream: “IMAGE:image”
input_stream: “FACE_LANDMARKS_FROM_POSE:face_landmarks_from_pose”
output_stream: “FACE_LANDMARKS:face_landmarks”
}

改为

#Predicts face landmarks based on the initial pose landmarks.
node {
calculator: “FaceLandmarksFromPoseGpu”
input_stream: “IMAGE:image_gpu”
input_stream: “FACE_LANDMARKS_FROM_POSE:face_landmarks_from_pose”
output_stream: “FACE_LANDMARKS:face_landmarks”
}

9、修改mediapipe/python/solutions/holistic.py

将

BINARYPB_FILE_PATH = ‘mediapipe/modules/holistic_landmark/holistic_landmark_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/holistic_landmark/holistic_landmark_gpu.binarypb’

=============================================================

_download_oss_pose_landmark_model(model_complexity)
super().__init__(
    binary_graph_path=BINARYPB_FILE_PATH,
    side_inputs={
        'model_complexity': model_complexity,
        'smooth_landmarks': smooth_landmarks and not static_image_mode,
    },
    calculator_params={
        'poselandmarkcpu__ConstantSidePacketCalculator.packet': [
            constant_side_packet_calculator_pb2
            .ConstantSidePacketCalculatorOptions.ConstantSidePacket(
                bool_value=not static_image_mode)
        ],
        'poselandmarkcpu__posedetectioncpu__TensorsToDetectionsCalculator.min_score_thresh':
            min_detection_confidence,
        'poselandmarkcpu__poselandmarkbyroicpu__ThresholdingCalculator.threshold':
            min_tracking_confidence,
    },
    outputs=[
        'pose_landmarks', 'left_hand_landmarks', 'right_hand_landmarks',
        'face_landmarks'
    ])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

↓

_download_oss_pose_landmark_model(model_complexity)
super().__init__(
    binary_graph_path=BINARYPB_FILE_PATH,
    side_inputs={
        'model_complexity': model_complexity,
        'smooth_landmarks': smooth_landmarks and not static_image_mode,
    },
    calculator_params={
        'poselandmarkgpu__ConstantSidePacketCalculator.packet': [
            constant_side_packet_calculator_pb2
            .ConstantSidePacketCalculatorOptions.ConstantSidePacket(
                bool_value=not static_image_mode)
        ],
        'poselandmarkgpu__posedetectiongpu__TensorsToDetectionsCalculator.min_score_thresh':
            min_detection_confidence,
        'poselandmarkgpu__poselandmarkbyroigpu__ThresholdingCalculator.threshold':
            min_tracking_confidence,
    },
    outputs=[
        'pose_landmarks', 'left_hand_landmarks', 'right_hand_landmarks',
        'face_landmarks'
    ])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

10、修改mediapipe/modules/pose_landmark/pose_landmark_gpu.pbtxt

将

#Calculates size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “SIZE:image_size”
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Calculates size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “SIZE:image_size”
}

将

#round of pose detection.
node {
calculator: “GateCalculator”
input_stream: “image”
input_stream: “image_size”
input_stream: “DISALLOW:prev_pose_rect_from_landmarks_is_present”
output_stream: “image_for_pose_detection”

改为

#round of pose detection.
node {
calculator: “GateCalculator”
input_stream: “image_gpu”
input_stream: “image_size”
input_stream: “DISALLOW:prev_pose_rect_from_landmarks_is_present”
output_stream: “image_for_pose_detection”

将

node {
calculator: “PoseLandmarkByRoiGpu”
input_side_packet: “MODEL_COMPLEXITY:model_complexity”
input_stream: “IMAGE:image”
input_stream: “ROI:pose_rect”
output_stream: “LANDMARKS:unfiltered_pose_landmarks”
output_stream: “AUXILIARY_LANDMARKS:unfiltered_auxiliary_landmarks”

改为

node {
calculator: “PoseLandmarkByRoiGpu”
input_side_packet: “MODEL_COMPLEXITY:model_complexity”
input_stream: “IMAGE:image_gpu”
input_stream: “ROI:pose_rect”
output_stream: “LANDMARKS:unfiltered_pose_landmarks”
output_stream: “AUXILIARY_LANDMARKS:unfiltered_auxiliary_landmarks”

将

#timestamp bound update occurs to jump start the feedback loop.
node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image”
input_stream: “LOOP:pose_rect_from_landmarks”
input_stream_info: {
tag_index: “LOOP”

改为

#timestamp bound update occurs to jump start the feedback loop.
node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image_gpu”
input_stream: “LOOP:pose_rect_from_landmarks”
input_stream_info: {
tag_index: “LOOP”

11、修改mediapipe/python/solutions/pose.py

BINARYPB_FILE_PATH = ‘mediapipe/modules/pose_landmark/pose_landmark_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/pose_landmark/pose_landmark_gpu.binarypb’

class Pose(SolutionBase):
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘poselandmarkcpu__posedetectioncpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘poselandmarkcpu__poselandmarkbyroicpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘pose_landmarks’])
↓
class Pose(SolutionBase):
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘poselandmarkgpu__posedetectiongpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘poselandmarkgpu__poselandmarkbyroigpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘pose_landmarks’])

12、修改mediapipe/modules/hand_landmark/hand_landmark_tracking_gpu.pbtxt

将

#Drops the incoming image if enough hands have already been identified from the
#previous image. Otherwise, passes the incoming image through to trigger a new
#round of palm detection.
node {
calculator: “GateCalculator”
input_stream: “image”
input_stream: “DISALLOW:prev_has_enough_hands”
output_stream: “palm_detection_image”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Drops the incoming image if enough hands have already been identified from the
#previous image. Otherwise, passes the incoming image through to trigger a new
#round of palm detection.
node {
calculator: “GateCalculator”
input_stream: “image_gpu”
input_stream: “DISALLOW:prev_has_enough_hands”
output_stream: “palm_detection_image”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

将

#Extracts image size.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “SIZE:image_size”
}

改为

#Extracts image size.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “SIZE:image_size”
}

将

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:hand_rects”
input_stream: “CLONE:0:image”
input_stream: “CLONE:1:image_size”
output_stream: “ITEM:single_hand_rect”
output_stream: “CLONE:0:image_for_landmarks”
output_stream: “CLONE:1:image_size_for_landmarks”
output_stream: “BATCH_END:hand_rects_timestamp”
}

改为

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:hand_rects”
input_stream: “CLONE:0:image_gpu”
input_stream: “CLONE:1:image_size”
output_stream: “ITEM:single_hand_rect”
output_stream: “CLONE:0:image_for_landmarks”
output_stream: “CLONE:1:image_size_for_landmarks”
output_stream: “BATCH_END:hand_rects_timestamp”
}

将

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image”
input_stream: “LOOP:hand_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_hand_rects_from_landmarks”
}

改为

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image_gpu”
input_stream: “LOOP:hand_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_hand_rects_from_landmarks”
}

13、修改mediapipe/python/solutions/hands.py

BINARYPB_FILE_PATH = ‘mediapipe/modules/hand_landmark/hand_landmark_tracking_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/hand_landmark/hand_landmark_tracking_gpu.binarypb’

将

calculator_params={
‘ConstantSidePacketCalculator.packet’: [
constant_side_packet_calculator_pb2
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘palmdetectioncpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘handlandmarkcpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘multi_hand_landmarks’, ‘multi_handedness’])

改为

calculator_params={
‘ConstantSidePacketCalculator.packet’: [
constant_side_packet_calculator_pb2
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘palmdetectiongpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘handlandmarkgpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘multi_hand_landmarks’, ‘multi_handedness’])

14、修改mediapipe/modules/selfie_segmentation/selfie_segmentation_gpu.pbtxt

将

#Resizes the input image into a tensor with a dimension desired by the model.
node {
calculator: “SwitchContainer”
input_side_packet: “SELECT:model_selection”
input_stream: “IMAGE_GPU:image”
output_stream: “TENSORS:input_tensors”
options: {
[mediapipe.SwitchContainerOptions.ext] {

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Resizes the input image into a tensor with a dimension desired by the model.
node {
calculator: “SwitchContainer”
input_side_packet: “SELECT:model_selection”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “TENSORS:input_tensors”
options: {
[mediapipe.SwitchContainerOptions.ext] {

将

#Retrieves the size of the input image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “SIZE:input_size”
}

改为

#Retrieves the size of the input image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “SIZE:input_size”
}

将

#Processes the output tensors into a segmentation mask that has the same size
#as the input image into the graph.
node {
calculator: “TensorsToSegmentationCalculator”
input_stream: “TENSORS:output_tensors”
input_stream: “OUTPUT_SIZE:input_size”
output_stream: “MASK:mask_image”
options: {
[mediapipe.TensorsToSegmentationCalculatorOptions.ext] {
activation: NONE
gpu_origin: TOP_LEFT
}
}
}
#Converts the incoming Image into the corresponding GpuBuffer type.
node: {
calculator: “FromImageCalculator”
input_stream: “IMAGE:mask_image”
output_stream: “IMAGE_GPU:segmentation_mask”
}

改为

#Processes the output tensors into a segmentation mask that has the same size
#as the input image into the graph.
node {
calculator: “TensorsToSegmentationCalculator”
input_stream: “TENSORS:output_tensors”
input_stream: “OUTPUT_SIZE:input_size”
output_stream: “MASK:mask_image”
options: {
[mediapipe.TensorsToSegmentationCalculatorOptions.ext] {
activation: NONE
gpu_origin: TOP_LEFT
}
}
}
#Converts the incoming Image into the corresponding GpuBuffer type.
node: {
calculator: “FromImageCalculator”
input_stream: “IMAGE:mask_image”
output_stream: “IMAGE_CPU:segmentation_mask”
}

15、修改mediapipe/python/solutions/selfie_segmentation.py

BINARYPB_FILE_PATH = ‘mediapipe/modules/selfie_segmentation/selfie_segmentation_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/selfie_segmentation/selfie_segmentation_gpu.binarypb’

16、修改mediapipe/modules/objectron/objectron_gpu.pbtxt

将

#Input/Output streams and input side packets.
#Note that the input image is assumed to have aspect ratio 3:4 (width:height).
input_stream: “IMAGE_GPU:image”
#Allowed category labels, e.g. Footwear, Coffee cup, Mug, Chair, Camera

改为

#Input/Output streams and input side packets.
#Note that the input image is assumed to have aspect ratio 3:4 (width:height).
input_stream: “IMAGE_GPU:image”
#Path to TfLite model for 3D bounding box landmark prediction
input_side_packet: “MODEL_PATH:box_landmark_model_path”
#Allowed category labels, e.g. Footwear, Coffee cup, Mug, Chair, Camera

将

output_stream: “FRAME_ANNOTATION:detected_objects”
#Defines whether landmarks from the previous video frame should be used to help

改为

output_stream: “FRAME_ANNOTATION:detected_objects”
#Collection of box landmarks. (NormalizedLandmarkList)
output_stream: “MULTI_LANDMARKS:multi_box_landmarks”
#Crop rectangles derived from bounding box landmarks.
output_stream: “NORM_RECTS:multi_box_rects”
#Defines whether landmarks from the previous video frame should be used to help

将

#Defines whether landmarks from the previous video frame should be used to help

改为

#Loads the file in the specified path into a blob.
node {
calculator: “LocalFileContentsCalculator”
input_side_packet: “FILE_PATH:0:box_landmark_model_path”
output_side_packet: “CONTENTS:0:box_landmark_model_blob”
}
#Converts the input blob into a TF Lite model.
node {
calculator: “TfLiteModelCalculator”
input_side_packet: “MODEL_BLOB:box_landmark_model_blob”
output_side_packet: “MODEL:box_landmark_model”
}
#Defines whether landmarks from the previous video frame should be used to help

将

#Drops the incoming image if BoxLandmarkSubgraph was able to identify box
#presence in the previous image. Otherwise, passes the incoming image through
#to trigger a new round of box detection in ObjectDetectionOidV4Subgraph.
node {
calculator: “GateCalculator”
input_stream: “image”
input_stream: “DISALLOW:prev_has_enough_objects”
output_stream: “detection_image”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Drops the incoming image if BoxLandmarkSubgraph was able to identify box
#presence in the previous image. Otherwise, passes the incoming image through
#to trigger a new round of box detection in ObjectDetectionOidV4Subgraph.
node {
calculator: “GateCalculator”
input_stream: “image_gpu”
input_stream: “DISALLOW:prev_has_enough_objects”
output_stream: “detection_image”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

将

#Extracts image size from the input images.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “SIZE:image_size”
}

改为

#Extracts image size from the input images.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “SIZE:image_size”
}

将

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:box_rects”
input_stream: “CLONE:image”
output_stream: “ITEM:single_box_rect”
output_stream: “CLONE:landmarks_image”
output_stream: “BATCH_END:box_rects_timestamp”
}

改为

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:box_rects”
input_stream: “CLONE:image_gpu”
output_stream: “ITEM:single_box_rect”
output_stream: “CLONE:landmarks_image”
output_stream: “BATCH_END:box_rects_timestamp”
}

将

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image”
input_stream: “LOOP:box_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_box_rects_from_landmarks”
}

改为

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image_gpu”
input_stream: “LOOP:box_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_box_rects_from_landmarks”
}

将

#Subgraph that localizes box landmarks.
node {
calculator: “BoxLandmarkSubgraph”
input_stream: “IMAGE:landmarks_image”
input_stream: “NORM_RECT:single_box_rect”
output_stream: “NORM_LANDMARKS:single_box_landmarks”
}

改为

#Subgraph that localizes box landmarks.
node {
calculator: “BoxLandmarkSubgraph”
input_stream: “IMAGE:landmarks_image”
input_side_packet: “MODEL:box_landmark_model”
input_stream: “NORM_RECT:single_box_rect”
output_stream: “NORM_LANDMARKS:single_box_landmarks”
}

将

#Performs association between NormalizedRect vector elements from previous
#image and rects based on object detections from the current image. This
#calculator ensures that the output box_rects vector doesn’t contain
#overlapping regions based on the specified min_similarity_threshold.
node {
calculator: “AssociationNormRectCalculator”
input_stream: “box_rects_from_detections”
input_stream: “gated_prev_box_rects_from_landmarks”
output_stream: “box_rects”
options: {
[mediapipe.AssociationCalculatorOptions.ext] {
min_similarity_threshold: 0.2
}
}
}
#Outputs each element of box_rects at a fake timestamp for the rest of the
#graph to process. Clones image and image size packets for each
#single_box_rect at the fake timestamp. At the end of the loop, outputs the
#BATCH_END timestamp for downstream calculators to inform them that all
#elements in the vector have been processed.
node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:box_rects”
input_stream: “CLONE:image”
output_stream: “ITEM:single_box_rect”
output_stream: “CLONE:landmarks_image”
output_stream: “BATCH_END:box_rects_timestamp”
}

改为

#Performs association between NormalizedRect vector elements from previous
#image and rects based on object detections from the current image. This
#calculator ensures that the output box_rects vector doesn’t contain
#overlapping regions based on the specified min_similarity_threshold.
node {
calculator: “AssociationNormRectCalculator”
input_stream: “box_rects_from_detections”
input_stream: “gated_prev_box_rects_from_landmarks”
output_stream: “multi_box_rects”
options: {
[mediapipe.AssociationCalculatorOptions.ext] {
min_similarity_threshold: 0.2
}
}
}
#Outputs each element of box_rects at a fake timestamp for the rest of the
#graph to process. Clones image and image size packets for each
#single_box_rect at the fake timestamp. At the end of the loop, outputs the
#BATCH_END timestamp for downstream calculators to inform them that all
#elements in the vector have been processed.
node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:multi_box_rects”
input_stream: “CLONE:image”
output_stream: “ITEM:single_box_rect”
output_stream: “CLONE:landmarks_image”
output_stream: “BATCH_END:box_rects_timestamp”
}

17、修改mediapipe/modules/objectron/box_landmark_gpu.pbtxt

将

input_stream: “IMAGE:image”
input_stream: “NORM_RECT:box_rect”
output_stream: “NORM_LANDMARKS:box_landmarks”

改为

input_stream: “IMAGE:image”
input_stream: “NORM_RECT:box_rect”
input_side_packet: “MODEL:model”
output_stream: “NORM_LANDMARKS:box_landmarks”

将

#Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
#vector of tensors representing, for instance, detection boxes/keypoints and
#scores.
node {
calculator: “InferenceCalculator”
input_stream: “TENSORS:image_tensor”
output_stream: “TENSORS:output_tensors”
options: {
[mediapipe.InferenceCalculatorOptions.ext] {
model_path: “object_detection_3d.tflite”
delegate { gpu {} }
}
}
}

改为

#Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
#vector of tensors representing, for instance, detection boxes/keypoints and
#scores.
node {
calculator: “InferenceCalculator”
input_stream: “TENSORS:image_tensor”
input_side_packet: “MODEL:model”
output_stream: “TENSORS:output_tensors”
options: {
[mediapipe.InferenceCalculatorOptions.ext] {
model_path: “object_detection_3d.tflite”
delegate { gpu {} }
}
}
}

18、修改mediapipe/python/solutions/objectron.py

BINARYPB_FILE_PATH = ‘mediapipe/modules/objectron/objectron_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/objectron/objectron_gpu.binarypb’

19、修改mediapipe/python/solutions/face_mesh.py

BINARYPB_FILE_PATH = ‘mediapipe/modules/face_landmark/face_landmark_front_cpu.binarypb’
↓
BINARYPB_FILE_PATH = ‘mediapipe/modules/face_landmark/face_landmark_front_gpu.binarypb’

将

super().init(
binary_graph_path=BINARYPB_FILE_PATH,
side_inputs={
‘num_faces’: max_num_faces,
},
calculator_params={
‘ConstantSidePacketCalculator.packet’: [
constant_side_packet_calculator_pb2
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘facedetectionfrontcpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘facelandmarkcpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘multi_face_landmarks’])

改为

super().init(
binary_graph_path=BINARYPB_FILE_PATH,
side_inputs={
‘num_faces’: max_num_faces,
},
calculator_params={
‘ConstantSidePacketCalculator.packet’: [
constant_side_packet_calculator_pb2
.ConstantSidePacketCalculatorOptions.ConstantSidePacket(
bool_value=not static_image_mode)
],
‘facedetectionfrontgpu__TensorsToDetectionsCalculator.min_score_thresh’:
min_detection_confidence,
‘facelandmarkgpu__ThresholdingCalculator.threshold’:
min_tracking_confidence,
},
outputs=[‘multi_face_landmarks’])

20、修改mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt
将

#Converts the input GPU image (GpuBuffer) to the multi-backend image type
#(Image).
node: {
calculator: “ToImageCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “IMAGE:multi_backend_image”
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Converts the input GPU image (GpuBuffer) to the multi-backend image type
#(Image).
node: {
calculator: “ToImageCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “IMAGE:multi_backend_image”
}

21、修改mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt

将

#Drops the incoming image if enough faces have already been identified from the
#previous image. Otherwise, passes the incoming image through to trigger a new
#round of face detection.
node {
calculator: “GateCalculator”
input_stream: “image”
input_stream: “DISALLOW:prev_has_enough_faces”
output_stream: “gated_image”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

改为

node: {
calculator: “ColorConvertCalculator”
input_stream: “RGB_IN:image”
output_stream: “RGBA_OUT:image_rgba”
}
node: {
calculator: “ImageFrameToGpuBufferCalculator”
input_stream: “image_rgba”
output_stream: “image_gpu”
}
#Drops the incoming image if enough faces have already been identified from the
#previous image. Otherwise, passes the incoming image through to trigger a new
#round of face detection.
node {
calculator: “GateCalculator”
input_stream: “image_gpu”
input_stream: “DISALLOW:prev_has_enough_faces”
output_stream: “gated_image_gpu”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}
node {
calculator: “GateCalculator”
input_stream: “image”
input_stream: “DISALLOW:prev_has_enough_faces”
output_stream: “gated_image_cpu”
options: {
[mediapipe.GateCalculatorOptions.ext] {
empty_packets_as_allow: true
}
}
}

将

#Detects faces.
node {
calculator: “FaceDetectionFrontGpu”
input_stream: “IMAGE:gated_image”
output_stream: “DETECTIONS:all_face_detections”
}

改为

#Detects faces.
node {
calculator: “FaceDetectionFrontGpu”
input_stream: “IMAGE:gated_image_cpu”
output_stream: “DETECTIONS:all_face_detections”
}

将

#Calculate size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:gated_image”
output_stream: “SIZE:gated_image_size”
}

改为

#Calculate size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:gated_image_gpu”
output_stream: “SIZE:gated_image_size”
}

将

#Calculate size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image”
output_stream: “SIZE:image_size”
}

改为

#Calculate size of the image.
node {
calculator: “ImagePropertiesCalculator”
input_stream: “IMAGE_GPU:image_gpu”
output_stream: “SIZE:image_size”
}

将

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:face_rects”
input_stream: “CLONE:0:image”
input_stream: “CLONE:1:image_size”
output_stream: “ITEM:face_rect”
output_stream: “CLONE:0:landmarks_loop_image”
output_stream: “CLONE:1:landmarks_loop_image_size”
output_stream: “BATCH_END:landmarks_loop_end_timestamp”
}

改为

node {
calculator: “BeginLoopNormalizedRectCalculator”
input_stream: “ITERABLE:face_rects”
input_stream: “CLONE:0:image_gpu”
input_stream: “CLONE:1:image_size”
output_stream: “ITEM:face_rect”
output_stream: “CLONE:0:landmarks_loop_image”
output_stream: “CLONE:1:landmarks_loop_image_size”
output_stream: “BATCH_END:landmarks_loop_end_timestamp”
}

将

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image”
input_stream: “LOOP:face_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_face_rects_from_landmarks”
}

改为

node {
calculator: “PreviousLoopbackCalculator”
input_stream: “MAIN:image_gpu”
input_stream: “LOOP:face_rects_from_landmarks”
input_stream_info: {
tag_index: “LOOP”
back_edge: true
}
output_stream: “PREV_LOOP:prev_face_rects_from_landmarks”
}

22、编译

python3 setup.py gen_protos && python3 setup.py bdist_wheel
1

23、安装MediaPipe

python3 -m pip install cython
python3 -m pip install numpy
python3 -m pip install pillow
python3 -m pip install mediapipe/dist/mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64.whl
1
2
3
4

24、运行GPU版demo

cd ~
git clone https://github.com/Kazuhito00/mediapipe-python-sample && cd mediapipe-python-sample
python3 sample_hand.py
1
2
3

25、运行自己的程序
检测效果如下

四、编译好的.whl文件（CPU和GPU）以及bazel压缩包

百度网盘链接：https://pan.baidu.com/s/1VJ9bDZlcGwwEtEQeQYJR4Q
提取码：52yj

---

总结

本文介绍了MediaPipe在Jetson Nano上的安装与使用，以及通过修改官方文件内容成功编译GPU版的MediaPipe。

参考文章：
Jetson Nano Ubuntu18.04环境下配置Mediapipe
PINTO0309