3.Hive系列之docker-compose部署升级总结_dockercompose升级版本

1. 版本号修改

对于升级而言，我们最先考虑的是docker hub中有的较新的版本，然后我们需要简单了解下hadoop2与hadoop3的区别，首先明确的是端口号有所改变，如下图所示

2. Hive镜像构建

刚刚我们修改了Hive为bde2020/hive:3.1.2-postgresql-metastore，仓库中并没有，我们就要考虑修改Dockerfile自行构建镜像了

现在可以尝试构建镜像了,并启动

docker build -t bde2020/hive:3.1.2-postgresql-metastore .
1

3. apt-get update 404 Not Found

哇偶，构建出错

原因：/etc/apt/sources.list中源存在问题，我们修改为目前可用的源即可

RUN sed -i 's/^.*$/deb http:\/\/deb.debian.org\/debian\/ buster main\ndeb-src http:\/\/deb.debian.org\/debian\/ buster main\ndeb http:\/\/deb.debian.org\/debian-security\/ buster\/updates main\ndeb-src http:\/\/deb.debian.org\/debian-security\/ buster\/updates main/g' /etc/apt/sources.list
1

4. entrypoint.sh文件不存在

再次尝试构建镜像了,并启动，发现如下错误

打开entrypoint.sh文件，发现报错，我们按照提示加上注释即可

5. 版本冲突与格式错误

/opt/hive/conf/hive-env.sh: line 37: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 45: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 46: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 49: $'\r': command not found
/opt/hive/conf/hive-env.sh: line 52: $'\r': command not found
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
1
2
3
4
5
6
7
8
9
10
11
12

我们将hive-env.sh格式改为LF，unix格式

Dockerfile中加入这两段解决log冲突问题

# solve log version conflict
RUN cp /opt/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar /opt/hive/lib/
RUN rm -rf /opt/hive/lib/guava-19.0.jar
1
2
3

再次构建尝试运行

6. url链接问题

看起来日志都很正常了，但是我们执行任何hive操作时，报如下错误:

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
1
2
3

控制台没有输出错误日志，我们查看其他地方是由有hive.log

# find / -name hive.log
/tmp/root/hive.log
1
2

查看日志，可以发现，url非法了哎，这是因为docker在默认创建网络的时候，会给添加_default后缀导致url非法

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: java.net.URISyntaxException Illegal character in hostname at index 49: thrift://docker-hive-hive-metastore-1.docker-hive_default:9083
        at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:168) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:267) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:182) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:94) ~[hive-exec-3.1.2.jar:3.1.2]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_232]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_232]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_232]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_232]
        at org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4299) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4367) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4347) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4603) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:291) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:274) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:435) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:375) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:355) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:331) ~[hive-exec-3.1.2.jar:3.1.2]
        at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:960) ~[hive-exec-3.1.2.jar:3.1.2]
        ... 16 mor
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

我们修改docker-compose.yaml，指定网络名称即可

# 版本由3改为3.5
version: "3.5"
# solve java.net.URISyntaxException Illegal character in hostname at index 49: thrift://docker-hive-hive-metastore-1.docker-hive_default:9083
networks:
  default:
    name: docker-hive-default
1
2
3
4
5
6

先down后up，再次执行show databases;正常了哎, 也可以看到网络docker-hive-default

(base) PS D:\Java\新建文件夹\docker-hive> docker network ls
NETWORK ID     NAME                  DRIVER    SCOPE
b53167dc0bb1   bridge                bridge    local
6ae196a5dfd6   docker-hive-default   bridge    local
1582f3dcbd8b   docker-hive_default   bridge    local
3e85deffd6b1   host                  host      local
8740d8865907   none                  null      local
1
2
3
4
5
6
7

现在即使通过beeline连接也是OK的，完结

欢迎关注公众号算法小生，与我沟通交流