赞
踩
Impala repo里有C++、Java、Python、Bash的代码,编译过程中可能遇到各种问题。这个文章将作为一个汇总,以后每遇到一个编译问题,解决完后都会更新一下。
在一台CentOS 7.4机器上编译Impala 4.0时遇到错误:
$ (pushd java && mvn clean package -DskipTests) ...... [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Impala Parent POM 4.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Impala Parent POM .......................... FAILURE [0.281s] [INFO] Test Data Generator ............................... SKIPPED [INFO] impala-minimal-s3a-aws-sdk ........................ SKIPPED [INFO] impala-minimal-hive-exec .......................... SKIPPED [INFO] Impala Executor Dependencies ...................... SKIPPED [INFO] Apache Impala External Data Source ................ SKIPPED [INFO] Apache Impala External Data Source API ............ SKIPPED [INFO] Apache Impala External Data Source Sample ......... SKIPPED [INFO] Apache Impala External Data Source Test Library ... SKIPPED [INFO] Apache Impala Query Event Hook API ................ SKIPPED [INFO] YARN Extras ....................................... SKIPPED [INFO] Apache Impala Query Engine Frontend ............... SKIPPED [INFO] Impala Nested Table Flattener ..................... SKIPPED [INFO] test-hive-udfs .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 0.408s [INFO] Finished at: Thu Nov 19 17:35:58 PST 2020 [INFO] Final Memory: 16M/951M [INFO] ------------------------------------------------------------------------ [ERROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.4.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.4.1: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.4.1 from/to central (http://repo.maven.apache.org/maven2): Failed to transfer file: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom. Return code is: 501, ReasonPhrase:HTTPS Required. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
这个跟Impala 4.0无法,主要是maven repo要求用HTTPS的方式访问。而这台机器上的maven版本比较旧,使用的还是HTTP的URL,自己在 ~/.m2/settings.xml 里写个 HTTPS 的 URL 就行了:
<?xml version="1.0" encoding="UTF-8"?>
<settings>
<mirrors>
<mirror>
<id>maven</id>
<name>central maven</name>
<url>https://repo.maven.apache.org/maven2</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
</settings>
国内的服务器使用阿里云之类的镜像会更快,也没有HTTPS的强制限制,可以把 ~/.m2/settings.xml 改为:
<?xml version="1.0" encoding="UTF-8"?>
<settings>
<mirrors>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
</settings>
[INFO] BUILD FAILURE
[ERROR] Plugin net.sourceforge.czt.dev:cup-maven-plugin:1.6-cdh or one of its dependencies could not be resolved: Failed to read artifact descriptor for net.sourceforge.czt.dev:cup-maven-plugin:jar:1.6-cdh: Could not find artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh in cloudera.thirdparty.repo (https://repository.cloudera.com/content/repositories/third-party) -> [Help 1]
diff --git a/impala-parent/pom.xml b/impala-parent/pom.xml index 494749785..28b19128f 100644 --- a/impala-parent/pom.xml +++ b/impala-parent/pom.xml @@ -146,7 +146,7 @@ under the License. </repository> <repository> <id>cdh.rcs.releases.repo</id> - <url>https://repository.cloudera.com/content/groups/cdh-releases-rcs</url> + <url>https://repository.cloudera.com/artifactory/cdh-releases-rcs</url> <name>CDH Releases Repository</name> <snapshots> <enabled>true</enabled> @@ -197,7 +197,7 @@ under the License. </repository> <repository> <id>cloudera.thirdparty.repo</id> - <url>https://repository.cloudera.com/content/repositories/third-party</url> + <url>https://repository.cloudera.com/artifactory/cdh-releases-rcs</url> <name>Cloudera Third Party Repository</name> <snapshots> <enabled>false</enabled> @@ -236,7 +236,7 @@ under the License. <pluginRepositories> <pluginRepository> <id>cloudera.thirdparty.repo</id> - <url>https://repository.cloudera.com/content/repositories/third-party</url> + <url>https://repository.cloudera.com/artifactory/cdh-releases-rcs</url> <name>Cloudera Third Party Repository</name> <snapshots> <enabled>false</enabled>
Makefile和Ninja是两个不同的编译工具,使用了一个之后,没法立马切换到另一个做增量编译。如果需要从make切换到ninja(反方向同理),初次使用buildall.sh时不要加 -noclean 选项,否则会报如下错误:
[100%] Built target loggingsupport /home/quanlong/workspace/Impala/be/src/kudu/util/mutex.h:120: error: undefined reference to 'kudu::Mutex::AssertAcquired() const' ../../build/debug/runtime/libRuntime.a(types.cc.o):types.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' /home/quanlong/workspace/Impala/be/src/udf/udf.cc:154: error: undefined reference to 'fLI::FLAGS_stress_fn_ctx_alloc' /home/quanlong/workspace/Impala/be/src/udf/udf.cc:155: error: undefined reference to 'fLI::FLAGS_stress_fn_ctx_alloc' /home/quanlong/workspace/Impala/be/src/util/thread.cc:304: error: undefined reference to 'fLB::FLAGS_thread_creation_fault_injection' /home/quanlong/workspace/Impala/be/src/catalog/catalog-server.cc:293: error: undefined reference to 'fLI::FLAGS_stress_catalog_startup_delay_ms' /home/quanlong/workspace/Impala/be/src/catalog/catalog-server.cc:294: error: undefined reference to 'fLI::FLAGS_stress_catalog_startup_delay_ms' ../../build/debug/codegen/libCodeGen.a(instruction-counter.cc.o):instruction-counter.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' ../../build/debug/codegen/libCodeGen.a(codegen-callgraph.cc.o):codegen-callgraph.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' ../../build/debug/codegen/libCodeGen.a(codegen-symbol-emitter.cc.o):codegen-symbol-emitter.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' /home/quanlong/workspace/Impala/be/src/exec/exchange-node.cc:97: error: undefined reference to 'fLI::FLAGS_stress_datastream_recvr_delay_ms' /home/quanlong/workspace/Impala/be/src/exec/exchange-node.cc:98: error: undefined reference to 'fLI::FLAGS_stress_datastream_recvr_delay_ms' /home/quanlong/workspace/Impala/be/src/exec/hdfs-scan-node-base.cc:706: error: undefined reference to 'fLB::FLAGS_skip_file_runtime_filtering' /home/quanlong/workspace/Impala/be/src/runtime/io/disk-io-mgr.cc:798: error: undefined reference to 'fLI::FLAGS_stress_scratch_write_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/disk-io-mgr.cc:799: error: undefined reference to 'fLI::FLAGS_stress_scratch_write_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/hdfs-file-reader.cc:114: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/hdfs-file-reader.cc:115: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/local-file-reader.cc:60: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/local-file-reader.cc:61: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/kudu/rpc/messenger.cc:261: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' /home/quanlong/workspace/Impala/be/src/kudu/util/thread_restrictions.h:77: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/thread_restrictions.h:78: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/countdown_latch.h:71: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' /home/quanlong/workspace/Impala/be/src/kudu/rpc/reactor.cc:494: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/rpc/reactor.cc:495: error: undefined reference to 'kudu::ThreadRestrictions::SetIOAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/countdown_latch.h:81: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead collect2: error: ld returned 1 exit status ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead Generated: /home/quanlong/workspace/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.5414ebd990c149b6136b79ddf15467ad.20210922_13_40_49.xml [100%] Built target fesupport be/src/util/CMakeFiles/impala-profile-tool.dir/build.make:212: recipe for target 'be/build/debug/util/impala-profile-tool' failed make[3]: *** [be/build/debug/util/impala-profile-tool] Error 1 CMakeFiles/Makefile2:11043: recipe for target 'be/src/util/CMakeFiles/impala-profile-tool.dir/all' failed make[2]: *** [be/src/util/CMakeFiles/impala-profile-tool.dir/all] Error 2 make[2]: *** Waiting for unfinished jobs.... /home/quanlong/workspace/Impala/be/src/catalog/catalog-server.cc:293: error: undefined reference to 'fLI::FLAGS_stress_catalog_startup_delay_ms' /home/quanlong/workspace/Impala/be/src/catalog/catalog-server.cc:294: error: undefined reference to 'fLI::FLAGS_stress_catalog_startup_delay_ms' /home/quanlong/workspace/Impala/be/src/kudu/util/mutex.h:120: error: undefined reference to 'kudu::Mutex::AssertAcquired() const' ../../build/debug/runtime/libRuntime.a(types.cc.o):types.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' /home/quanlong/workspace/Impala/be/src/udf/udf.cc:154: error: undefined reference to 'fLI::FLAGS_stress_fn_ctx_alloc' /home/quanlong/workspace/Impala/be/src/udf/udf.cc:155: error: undefined reference to 'fLI::FLAGS_stress_fn_ctx_alloc' /home/quanlong/workspace/Impala/be/src/util/thread.cc:304: error: undefined reference to 'fLB::FLAGS_thread_creation_fault_injection' ../../build/debug/codegen/libCodeGen.a(instruction-counter.cc.o):instruction-counter.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' ../../build/debug/codegen/libCodeGen.a(codegen-callgraph.cc.o):codegen-callgraph.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' ../../build/debug/codegen/libCodeGen.a(codegen-symbol-emitter.cc.o):codegen-symbol-emitter.cc:llvm::VerifyDisableABIBreakingChecks: error: undefined reference to 'llvm::DisableABIBreakingChecks' /home/quanlong/workspace/Impala/be/src/exec/exchange-node.cc:97: error: undefined reference to 'fLI::FLAGS_stress_datastream_recvr_delay_ms' /home/quanlong/workspace/Impala/be/src/exec/exchange-node.cc:98: error: undefined reference to 'fLI::FLAGS_stress_datastream_recvr_delay_ms' /home/quanlong/workspace/Impala/be/src/exec/hdfs-scan-node-base.cc:706: error: undefined reference to 'fLB::FLAGS_skip_file_runtime_filtering' /home/quanlong/workspace/Impala/be/src/runtime/io/disk-io-mgr.cc:798: error: undefined reference to 'fLI::FLAGS_stress_scratch_write_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/disk-io-mgr.cc:799: error: undefined reference to 'fLI::FLAGS_stress_scratch_write_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/hdfs-file-reader.cc:114: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/hdfs-file-reader.cc:115: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/local-file-reader.cc:60: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/runtime/io/local-file-reader.cc:61: error: undefined reference to 'fLI::FLAGS_stress_disk_read_delay_ms' /home/quanlong/workspace/Impala/be/src/kudu/rpc/messenger.cc:261: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' /home/quanlong/workspace/Impala/be/src/kudu/util/thread_restrictions.h:77: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/thread_restrictions.h:78: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/countdown_latch.h:71: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' /home/quanlong/workspace/Impala/be/src/kudu/rpc/reactor.cc:494: error: undefined reference to 'kudu::ThreadRestrictions::SetWaitAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/rpc/reactor.cc:495: error: undefined reference to 'kudu::ThreadRestrictions::SetIOAllowed(bool)' /home/quanlong/workspace/Impala/be/src/kudu/util/countdown_latch.h:81: error: undefined reference to 'kudu::ThreadRestrictions::AssertWaitAllowed()' ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead ../../../toolchain/toolchain-packages-gcc7.5.0/openldap-2.4.47/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead collect2: error: ld returned 1 exit status Generated: /home/quanlong/workspace/Impala/logs/extra_junit_xml_logs/generate_junitxml.build.bf16cdccc74efd5090a3231aa745ac89.20210922_13_40_49.xml be/src/service/CMakeFiles/impalad.dir/build.make:213: recipe for target 'be/build/debug/service/impalad' failed make[3]: *** [be/build/debug/service/impalad] Error 1 CMakeFiles/Makefile2:8362: recipe for target 'be/src/service/CMakeFiles/impalad.dir/all' failed make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2
比如报如下的错:
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "/home/quanlong/workspace/Impala/bin/start-impala-cluster.py", line 23, in <module>
import getpass
ImportError: No module named getpass
这是impala的virtualenv有问题,可以重新构建一下virtualenv。修改 bin/impala-python-common.sh,在最后一行加上 “-r” 参数:
diff --git a/bin/impala-python-common.sh b/bin/impala-python-common.sh
index 4026029fa..ae636316c 100644
--- a/bin/impala-python-common.sh
+++ b/bin/impala-python-common.sh
@@ -29,4 +29,4 @@ export LD_LIBRARY_PATH="$(python "$IMPALA_HOME/infra/python/bootstrap_virtualenv
PY_DIR="$(dirname "$0")/../infra/python"
PY_ENV_DIR="${PY_DIR}/env-gcc${IMPALA_GCC_VERSION}"
-python "$PY_DIR/bootstrap_virtualenv.py"
+python "$PY_DIR/bootstrap_virtualenv.py" -r
然后执行该脚本即可
$ bash bin/impala-python-common.sh
INFO:bootstrap_virtualenv:Creating python virtualenv
INFO:bootstrap_virtualenv:Installing setuptools into the virtualenv
INFO:bootstrap_virtualenv:Installing packages into the virtualenv
INFO:bootstrap_virtualenv:Installing Kudu into the virtualenv
bin/impala-python-common.sh 其实就是 infra/python/bootstrap_virtualenv.py 的封装脚本,而这个python脚本有个 -r 参数,可以强制重建virtualenv。
注:修复完后记得把 bin/impala-python-common.sh 改回去。
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。