arm平台安装onnx出现的问题记录_could not find a package configuration file provid

arm平台安装onnx==1.6.0 报错如下.

	  CMake Warning at CMakeLists.txt:394 (find_package):
      By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
      asked CMake to find a package configuration file provided by "pybind11",
      but CMake did not find one.
    
      Could not find a package configuration file provided by "pybind11"
      (requested version 2.2) with any of the following names:
    
        pybind11Config.cmake
        pybind11-config.cmake
    
      Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
      "pybind11_DIR" to a directory containing one of the above files.  If
      "pybind11" provides a separate development package or SDK, be sure it has
      been installed.
      /// 下面的不是重点. 错误在上面.
	--
    -- ******** Summary ********
    --   CMake version         : 3.10.2
    --   CMake command         : /usr/bin/cmake
    --   System                : Linux
    --   C++ compiler          : /usr/bin/c++
    --   C++ compiler version  : 7.5.0
    --   CXX flags             :  -Wnon-virtual-dtor
    --   Build type            : Release
    --   Compile definitions   :
    --   CMAKE_PREFIX_PATH     :
    --   CMAKE_INSTALL_PREFIX  : /usr/local
    --   CMAKE_MODULE_PATH     :
    --
    --   ONNX version          : 1.6.0
    --   ONNX NAMESPACE        : onnx
    --   ONNX_BUILD_TESTS      : OFF
    --   ONNX_BUILD_BENCHMARKS : OFF
    --   ONNX_USE_LITE_PROTO   : OFF
    --   ONNXIFI_DUMMY_BACKEND : OFF
    --   ONNXIFI_ENABLE_EXT    : OFF
    --
    --   Protobuf compiler     : /usr/bin/protoc
    --   Protobuf includes     : /usr/include
    --   Protobuf libraries    : /usr/lib/aarch64-linux-gnu/libprotobuf.so;-lpthread
    --   BUILD_ONNX_PYTHON     : ON
    --     Python version      :
    --     Python executable   : /home/firefly/Desktop/AI1.6.1/vvv/bin/python3.6
    --     Python includes     : /usr/include/python3.6m
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build
    Scanning dependencies of target gen_onnx_proto
    Scanning dependencies of target onnxifi_loader
    Scanning dependencies of target onnxifi_dummy
    [  1%] Running gen_proto.py on onnx/onnx.in.proto
    [  3%] Building C object CMakeFiles/onnxifi_dummy.dir/onnx/onnxifi_dummy.c.o
    [  4%] Building C object CMakeFiles/onnxifi_loader.dir/onnx/onnxifi_loader.c.o
    /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/onnx/onnxifi_dummy.c: In function ‘onnxGetExtensionFunctionAddress’:
    /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/onnx/onnxifi_dummy.c:173:21: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
               *function = &onnxGetExtensionFunctionAddress;
                         ^
    /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/onnx/onnxifi_dummy.c:176:21: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
               *function = &onnxSetIOAndRunGraph;
                         ^
    [  6%] Linking C static library libonnxifi_loader.a
    Processing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/onnx/onnx.in.proto
    Writing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-ml.proto
    Writing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-ml.proto3
    generating /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx_pb.py
    [  8%] Running C++ protocol buffer compiler on /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-ml.proto
    [  9%] Linking C shared library libonnxifi_dummy.so
    [  9%] Built target onnxifi_loader
    Scanning dependencies of target onnxifi_wrapper
    [ 11%] Building C object CMakeFiles/onnxifi_wrapper.dir/onnx/onnxifi_wrapper.c.o
    [ 11%] Built target onnxifi_dummy
    Writing mypy to onnx/onnx_ml_pb2.pyi
    [ 11%] Built target gen_onnx_proto
    [ 13%] Running gen_proto.py on onnx/onnx-operators.in.proto
    Processing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/onnx/onnx-operators.in.proto
    Writing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
    Writing /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-operators-ml.proto3
    generating /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx_operators_pb.py
    [ 14%] Running C++ protocol buffer compiler on /tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
    [ 16%] Linking C shared module libonnxifi.so
    [ 16%] Built target onnxifi_wrapper
    Writing mypy to onnx/onnx_operators_ml_pb2.pyi
    Scanning dependencies of target onnx_proto
    [ 18%] Building CXX object CMakeFiles/onnx_proto.dir/onnx/onnx-operators-ml.pb.cc.o
    [ 19%] Building CXX object CMakeFiles/onnx_proto.dir/onnx/onnx-ml.pb.cc.o
    [ 21%] Linking CXX static library libonnx_proto.a
    [ 24%] Built target onnx_proto
    Scanning dependencies of target onnx
    [ 26%] Building CXX object CMakeFiles/onnx.dir/onnx/common/status.cc.o
    [ 27%] Building CXX object CMakeFiles/onnx.dir/onnx/common/assertions.cc.o
    [ 29%] Building CXX object CMakeFiles/onnx.dir/onnx/common/ir_pb_converter.cc.o
    [ 32%] Building CXX object CMakeFiles/onnx.dir/onnx/common/interned_strings.cc.o
    [ 32%] Building CXX object CMakeFiles/onnx.dir/onnx/checker.cc.o
    [ 34%] Building CXX object CMakeFiles/onnx.dir/onnx/common/model_helpers.cc.o
    [ 36%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/attr_proto_util.cc.o
    [ 37%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/controlflow/defs.cc.o
    [ 40%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/data_type_utils.cc.o
    [ 40%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/controlflow/old.cc.o
    [ 42%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/function.cc.o
    c++: internal compiler error: Killed (program cc1plus)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
    CMakeFiles/onnx.dir/build.make:134: recipe for target 'CMakeFiles/onnx.dir/onnx/common/ir_pb_converter.cc.o' failed
    make[2]: *** [CMakeFiles/onnx.dir/onnx/common/ir_pb_converter.cc.o] Error 4
    make[2]: *** Waiting for unfinished jobs....
    CMakeFiles/Makefile2:279: recipe for target 'CMakeFiles/onnx.dir/all' failed
    make[1]: *** [CMakeFiles/onnx.dir/all] Error 2
    Makefile:129: recipe for target 'all' failed
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/setup.py", line 334, in <module>
        'backend-test-tools = onnx.backend.test.cmd_tools:main',
      File "/home/firefly/Desktop/AI1.6.1/vvv/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/firefly/Desktop/AI1.6.1/vvv/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.6/distutils/command/install.py", line 589, in run
        self.run_command('build')
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/setup.py", line 209, in run
        self.run_command('cmake_build')
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/setup.py", line 203, in run
        subprocess.check_call(build_args)
      File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '.', '--', '-j', '6']' returned non-zero exit status 2.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/firefly/Desktop/AI1.6.1/vvv/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pc9fr36z/onnx_966c7d1f68e74990b7a6797c009e08b1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-39ksjifz/install-record.txt --single-version-externally-managed --compile --install-headers /home/firefly/Desktop/AI1.6.1/vvv/include/site/python3.6/onnx Check the logs for full command output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151

这个狗屎问题, 很是麻烦哦. 百度无果, 从输出信息来看应该是编译的时候少了 pybind11 项目. 于是百度了一下,

后来在这里找到了说明
https://www.cnblogs.com/jourluohua/p/10235317.html

原来必须手动编译, 装好了pybind11之后(源码版本编译ONNX必须手动编译pybind11，使用pip install pybind11并不管用)

然后看看 pybind11 如何编译的, 百度到下面的文章

https://www.cnblogs.com/gaowengang/p/9448723.html

内容如下

pybind11 安装

注意：

1. 安装之前，最好把 /usr/bin/python 指向 python3，安装完再调回去。以后主要使用 python3.

2. 安装之前，最好先安装 boost.

---

Prerequisites:

$ sudo apt-get install python-dev  (or python3-dev)

$ sudo apt-get install cmake

$ sudo pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/  pytest

$ sudo pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/  numpy

$ sudo pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/  scipy

$ sudo pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/  pybind11
1
2
3
4
5
6
7
8
9
10
11

Get pybind11, from website: https://github.com/pybind/pybind11 to get the source package.
从这个网站下载 pybind11 源码

Get Eigen, from website: http://eigen.tuxfamily.org/index.php?title=Main_Page to get the source package.
从这个网站下载 Eigen 源码

for the above two source package, uncompoze it, and,

$ cd  that_folder

$ mkdir build

$ cd build

$ cmake ..                 //# 这个.. 是当前目录的意思无需修改 
$ cmake --build . --config Release --target check
$ make check -j 4  (make and check, not necessary)
$ sudo make install  (maybe should disable python3 in /usr/bin/ if you use python2)

1
2
3
4
5
6
7
8
9
10
11

执行安装

$ pip instal onnx==1.6.0
1

仍然报错, 一开始的错误…

再次百度,
在这里找到了答案, 应该是参数 -Dpybind11_DIR 没有加导致的.
https://stackoverflow.com/questions/63254584/how-to-make-cmake-find-pybind11

cmake … -G “Visual Studio 15 2017” -A x64 `
-Dpybind11_DIR=C:/Users/ben.wolfley/Anaconda3/Library/share/cmake/pybind11/

但问题是我这里install onnx 没有地方写命令传参数Dpybind11_DIR 呀…

还 应该是 pybind11没装好. 参照下文.
https://www.cnblogs.com/jourluohua/p/10235317.html
重要的是这么一个信息, ## (源码版本编译ONNX必须手动编译pybind11，使用pip install pybind11并不管用)

还有一个重要的信息的是, 使用

pip install pybind11

安装的文件默认会安装到/user/bin下面. 所以不会安装到 虚拟环境下. 所以如果是 用虚拟环境venv的朋友要注意了. 虚拟环境下必须用源码编译的方式安装 pybind11

于是开启了痛苦的无边无际的make之坑. 超级郁闷的坑呀. 这里真想报个粗口,
(这是谁TMD设计的模式, make来make去, 就是make不成功. 每次make都有一大堆问题…我心中有 一万头草泥马奔腾…)

下载pybind11源码

git clone https://github.com/pybind/pybind11.git
1

如果你需要eigen库的话，还需要先安装eigen，这里我选择的是无eigen的方式

mkdir build
cd build
cmake ..
make -j 4  # 这里有坑, -j 4 是开启4核并行编译.  要根据机器情况用.
sudo make install
1
2
3
4
5

这里有个问题，如果你同时存在python2.x和python3.x版本的话，默认选择的是python3.x版本的，但是我现在的整个环境是python2.7.12，所以就必须禁用掉Python3.x，大家都说禁用，其实就是让自动寻找方式找不到python3.x。
采用的方式是在/usr/bin中所有找到的python3.x相关的文件都移动到另一个文件夹内，

执行到的时候开始报错然后开始编译 然后报错.

make -j 4

且每次执行的保存内容稍微有些变化, 都是中断在某个.cpp.o文件的时候就崩了.
而且每次执行make 中断的文件不是同一个文件.

(vvv) firefly@firefly:~/Desktop/AI1.6.1/pybind11/build$ make -j 4
[  2%] Building CXX object tests/CMakeFiles/pybind11_cross_module_tests.dir/pybind11_cross_module_tests.cpp.o
[  7%] Built target cross_module_gil_utils
[  9%] Building CXX object tests/CMakeFiles/pybind11_tests.dir/test_chrono.cpp.o
[ 11%] Building CXX object tests/CMakeFiles/pybind11_tests.dir/test_builtin_casters.cpp.o
[ 14%] Building CXX object tests/CMakeFiles/pybind11_tests.dir/test_class.cpp.o
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
tests/CMakeFiles/pybind11_tests.dir/build.make:230: recipe for target 'tests/CMakeFiles/pybind11_tests.dir/test_class.cpp.o' failed
make[2]: *** [tests/CMakeFiles/pybind11_tests.dir/test_class.cpp.o] Error 4
make[2]: *** Waiting for unfinished jobs....
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
tests/CMakeFiles/pybind11_tests.dir/build.make:206: recipe for target 'tests/CMakeFiles/pybind11_tests.dir/test_chrono.cpp.o' failed
make[2]: *** [tests/CMakeFiles/pybind11_tests.dir/test_chrono.cpp.o] Error 4
^Ctests/CMakeFiles/pybind11_tests.dir/build.make:134: recipe for target 'tests/CMakeFiles/pybind11_tests.dir/test_builtin_casters.cpp.o' failed
make[2]: *** [tests/CMakeFiles/pybind11_tests.dir/test_builtin_casters.cpp.o] Interrupt
tests/CMakeFiles/pybind11_cross_module_tests.dir/build.make:62: recipe for target 'tests/CMakeFiles/pybind11_cross_module_tests.dir/pybind11_cross_module_tests.cpp.o' failed
make[2]: *** [tests/CMakeFiles/pybind11_cross_module_tests.dir/pybind11_cross_module_tests.cpp.o] Interrupt
CMakeFiles/Makefile2:129: recipe for target 'tests/CMakeFiles/pybind11_cross_module_tests.dir/all' failed
make[1]: *** [tests/CMakeFiles/pybind11_cross_module_tests.dir/all] Interrupt
CMakeFiles/Makefile2:166: recipe for target 'tests/CMakeFiles/pybind11_tests.dir/all' failed
make[1]: *** [tests/CMakeFiles/pybind11_tests.dir/all] Interrupt
Makefile:129: recipe for target 'all' failed
make: *** [all] Interrupt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

单看报错内容, 共同点似乎都是

recipe for target ‘all’ failed
单单是这个报错信息根本找不到解决方案. 于是我猜测可能是内存太小了? 应该是环境问题.
如果是配置问题, 每次报错应该是一样的. 配置错误有迹可循. 如果是配置错误一般情况下一个文件都不会被编译, 而这里却被有可以编译的文件.
大概率说明不是配置问题, 应该是环境问题. 于是我查看了下
make -j 4

这个命令后面两个参数的意思 原来 -j 4的意思是开启4核并行编译的意思.
我这个ARM芯片哪有这么多核, 改成make单核编译吧. 改成

make

结果能够编译成的cpp.o文件数量也多了起来.

先不管它, 多试几次, 看看能不能最后完成编译. 我感觉可能跟内存有关系, 比较RK3399开发板的内存只有2GB.
现在是每次编译

Building CXX object tests/CMakeFiles/pybind11_tests.dir/test_stl_binders.cpp.o

都编译不过去, 应该是内存不够了.
重启下开发板, 打开监控看看内存使用率.

至此已基本上明白编译失败的两个原因一个是 make 后面的参数要根据自己机器的情况加.

第二个是内存不够. 那怎么办呢? Ubuntu 有虚拟内存, 看看它怎么开.提高一下虚拟内存
加粗样式

1.首先用命令free查看系统内 Swap 分区大小。命令：

$ free -m
1

说明我的系统并未开启虚拟内存
下面查看一下硬盘使用情况

$ df -B G
1

可以看的出我的 /userdata目录是挂载的mmcblklp8 这个板载闪存, 有10GB, 使用率是70% , 也就是说, 还有3G的可用空间…快满了呀.
再分它2个GB吧.

2.创建一个 Swap 文件。

$ cd /userdata
$ mkdir swap
$ cd swap
$ sudo dd if=/dev/zero of=swapfile bs=1024 count=200000
1
2
3
4

在上面命令中的 count 即代表swap文件大小。我们可以自己调整。
命令有点慢, 耐心等待…

3.把生成的文件转换成 Swap 文件

sudo mkswap swapfile
1

4. 激活 Swap 文件：

sudo swapon swapfile
1

5. 再次查看 的结果。

free -m
1

total used free shared buffers cachedMem: 1002 967 34 0 22 410-/+ buffers/cache: 534 467Swap: 1053 32 1021

添加虚拟内存成功。

6. 最后如果需要卸载这个 swap 文件，可以进入建立的 swap 文件目录。执行下列命令。

sudo swapoff swapfile  # 别乱用,这是卸载.
1

如果需要下次开机还一直保持这个 swap ，可以sudo -s换到root然后把它写入 /etc/fstab 文件。

swapfilepath swap swap defaults 0 0
1

然后回到我们的 pybind11目录 再次执行make

$ cd ***/pybind11/build/
$ make
1
2

为了确保没问题

$ make clean # 了一下
$ make
1
2

这些pybind11总算是make成功了. 然后再来

$ sudo make install
1

输出这个不知是否算是正常了. 看上去是成功了.

然后 编译安装onnx 两种方式, 一种是源代码编译, 另外一种是pip安装
pip方式比较简单,

先试试pip方式

pip install onnx==1.6.0
1

源码编译方式如下:

git clone https://github.com/onnx/onnx.git
cd ./onnx
python setup.py build
sudo python setup.py install
1
2
3
4

这里很有可能会出现cmake提示 Protobuf compiler not found 的问题，因此需要安装合适的protobuf编译器

sudo apt-get install  libprotobuf-dev protobuf-compiler
1

竟然成功了… 哇咔咔…

总结一下, 关键是两个点, 一个是这个make的报错内容实在是不怎么人性化, 也不报个内存溢出. 或者内存不够, 害我百度了半天没结果. 被误导了 …
第二个是, 没有足够细心的观察问题点. 输出的日志. 若是早点看出来每次输出的不同. 说不定会更快的解决这个问题.
第三个是, 对命令不太仔细. 不了解命令行的意思就照抄. 结果把自己坑进去了. make -j 4 后面的-j 是几个并发进程的意思.