当前位置:   article > 正文

在windows10上安装apache-hive-3.1.3_windows安装hive3

windows安装hive3

一、hive介绍

hive是什么:hive是基于Hadoop的一个数据仓库工具,用来进行数据提取、转化、加载,这是一种可以存储、查询和分析存储在Hadoop中的大规模数据的机制。hive数据仓库工具能将结构化的数据文件映射为一张数据库表,并提供SQL查询功能,能将SQL语句转变成MapReduce任务来执行,hive 是一种底层封装了Hadoop 的数据仓库处理工具,使用类SQL 的hiveSQL 语言实现数据查询。

二、hive下载

https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-3.1.3/

三、配置hive的环境变量

08f2234aa0a3467cad6ee4607caceb0c.png31ea2ede0a9a41fc860b42a215f8ff88.png

四、修改hive中配置文件的参数

4.1、修改hive-env.sh中的参数

  1. # Set HADOOP_HOME to point to a specific hadoop install directory
  2. export HADOOP_HOME=D:\bigdata\hadoop
  3. # Hive Configuration Directory can be controlled by:
  4. export HIVE_CONF_DIR=D:\bigdata\apache-hive-3.1.3-bin\conf
  5. # Folder containing extra libraries required for hive compilation/execution can be controlled by:
  6. export HIVE_AUX_JARS_PATH=D:\bigdata\apache-hive-3.1.3-bin\lib

4.2、在mysql上创建一个hive的数据库,在后面的配置中会用到

4.3、hive是运行在hadoop之上的,需要通过hadoop创建几个文件夹

4.4、修改hive-site.xml文件配置

  1. <property>
  2. <name>hive.metastore.warehouse.dir</name>
  3. <value>/user/hive/warehouse</value>
  4. <description>location of default database for the warehouse</description>
  5. </property>
  6. <!--hive的临时数据目录,指定的位置在hdfs上的目录-->
  7. <property>
  8. <name>hive.exec.scratchdir</name>
  9. <value>/tmp/hive</value>
  10. <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
  11. </property>
  12. <!-- scratchdir 本地目录 -->
  13. <property>
  14. <name>hive.exec.local.scratchdir</name>
  15. <value>D:/bigdata/apache-hive-3.1.3-bin/my_hive/scratch_dir</value>
  16. <description>Local scratch space for Hive jobs</description>
  17. </property>
  18. <!-- resources_dir 本地目录 -->
  19. <property>
  20. <name>hive.downloaded.resources.dir</name>
  21. <value>D:/bigdata/apache-hive-3.1.3-bin/my_hive/resources_dir/${hive.session.id}_resources</value>
  22. <description>Temporary local directory for added resources in the remote file system.</description>
  23. </property>
  24. <!-- querylog 本地目录 -->
  25. <property>
  26. <name>hive.querylog.location</name>
  27. <value>D:/bigdata/apache-hive-3.1.3-bin/my_hive/querylog_dir</value>
  28. <description>Location of Hive run time structured log file</description>
  29. </property>
  30. <!-- operation_logs 本地目录 -->
  31. <property>
  32. <name>hive.server2.logging.operation.log.location</name>
  33. <value>D:/bigdata/apache-hive-3.1.3-bin/my_hive/operation_logs_dir</value>
  34. <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  35. </property>
  36. <!-- 数据库连接地址配置 -->
  37. <property>
  38. <name>javax.jdo.option.ConnectionURL</name>
  39. <value>jdbc:mysql://localhost:3306/hive?serverTimezone=UTC&amp;useSSL=false&amp;allowPublicKeyRetrieval=true</value>
  40. <description>
  41. JDBC connect string for a JDBC metastore.
  42. </description>
  43. </property>
  44. <!-- 数据库驱动配置 -->
  45. <property>
  46. <name>javax.jdo.option.ConnectionDriverName</name>
  47. <value>com.mysql.cj.jdbc.Driver</value>
  48. <description>Driver class name for a JDBC metastore</description>
  49. </property>
  50. <!-- 数据库用户名 -->
  51. <property>
  52. <name>javax.jdo.option.ConnectionUserName</name>
  53. <value>root</value>
  54. <description>Username to use against metastore database</description>
  55. </property>
  56. <!-- 数据库访问密码 -->
  57. <property>
  58. <name>javax.jdo.option.ConnectionPassword</name>
  59. <value>123456</value>
  60. <description>password to use against metastore database</description>
  61. </property>
  62. <!-- 解决 Caused by: MetaException(message:Version information not found in metastore. ) -->
  63. <property>
  64. <name>hive.metastore.schema.verification</name>
  65. <value>false</value>
  66. <description>
  67. Enforce metastore schema version consistency.
  68. True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic
  69. schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
  70. proper metastore schema migration. (Default)
  71. False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
  72. </description>
  73. </property>
  74. <!-- 自动创建全部 -->
  75. <!-- hive Required table missing : "DBS" in Catalog""Schema" 错误 -->
  76. <property>
  77. <name>datanucleus.schema.autoCreateAll</name>
  78. <value>true</value>
  79. <description>Auto creates necessary schema on a startup if one doesn't exist. Set this to false, after creating it once.To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not recommended for production use cases, run schematool command instead.</description>
  80. </property>

五、启动hive

默认情况下现在3.1.3版本是不存在window下的cmd命令。需要下载apache-hive-2.2.0-src.tar.gz这个包,把bin下面对应的cmd拷贝到hive中去

启动hive之前,先需要启动hadoop。然后再hive的bin目录下执行hive.cmd start

在执行命令的之后,出现了这样的情况

  1. Beeline version 3.1.3 by Apache Hive
  2. Hive Session ID = 5f6198d8-1183-4500-9cba-830da0127197
  3. 2023-11-09 01:24:22,203 INFO SessionState: Hive Session ID = 5f6198d8-1183-4500-9cba-830da0127197
  4. Error applying authorization policy on hive configuration: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/用户名/5f6198d8-1183-4500-9cba-830da0127197. Name node is in safe mode.
  5. The reported blocks 49 has reached the threshold 0.9990 of total blocks 49. The minimum number of live datanodes is not required. In safe mode extension. Safe mode will be turned off automatically in 3 seconds. NamenodeHostName:localhost
  6. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1612)
  7. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1599)
  8. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3437)
  9. at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1166)
  10. at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:742)
  11. at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  12. at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
  13. at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
  14. at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
  15. at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1213)
  16. at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1089)
  17. at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1012)
  18. at java.security.AccessController.doPrivileged(Native Method)
  19. at javax.security.auth.Subject.doAs(Subject.java:422)
  20. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
  21. at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3026)

手动关闭安全模式

hadoop dfsadmin -safemode leave
  1. D:\bigdata\hadoop\sbin>hadoop dfsadmin -safemode leave
  2. DEPRECATED: Use of this script to execute hdfs command is deprecated.
  3. Instead use the hdfs command for it.
  4. Safe mode is OFF

启动hive,在hive的bin下面执行hive.cmd start

5800f7e6a6f54ff3b8a1f5bf25c3b66a.png

执行过程中遇到这种先不用管,这个是hive与hadoop版本不匹配的问题 

  1. 2023-11-09 01:28:25,800 INFO session.SessionState: Resetting thread name to main
  2. 2023-11-09 01:28:25,807 INFO conf.HiveConf: Using the default value passed in for log id: 255c0661-6506-4a5f-bbad-f48b74f972dd
  3. 2023-11-09 01:28:25,807 INFO session.SessionState: Updating thread name to 255c0661-6506-4a5f-bbad-f48b74f972dd main
  4. 2023-11-09 01:28:25,836 INFO conf.HiveConf: Using the default value passed in for log id: 255c0661-6506-4a5f-bbad-f48b74f972dd
  5. 2023-11-09 01:28:25,837 INFO session.SessionState: Resetting thread name to main
  6. Beeline version 3.1.3 by Apache Hive
  7. hive>

这样说明启动成功了。

 

 

 

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/weixin_40725706/article/detail/620618
推荐阅读
相关标签
  

闽ICP备14008679号