devbox
盐析白兔
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Hive 搭建_dgqg1223

作者：盐析白兔 | 2024-08-02 18:09:49

赞

踩

dgqg1223

搭建环境

Getting Started Guide

Requirements

  • Java 1.7
    Note: Hive versions 1.2 onward require Java 1.7 or newer. Hive versions 0.14 to 1.1 work with Java 1.6 as well. Users are strongly advised to start moving to Java 1.8 (see HIVE-8607).
  • Hadoop 2.x (preferred), 1.x (not supported by Hive 2.0.0 onward).
    Hive versions up to 0.13 also supported Hadoop 0.20.x, 0.23.x.
  • Hive is commonly used in production Linux and Windows environment. Mac is a commonly used development environment. The instructions in this document are applicable to Linux and Mac. Using it on Windows would require slightly different steps.

三种搭建模式

Hive Metastore Administration.

Introduction

All the metadata for Hive tables and partitions are accessed through the Hive Metastore. Metadata is persisted using JPOX ORM solution (Data Nucleus) so any database that is supported by it can be used by Hive. Most of the commercial relational databases and many open source databases are supported. See the list of supported databases in section below.

You can find an E/R diagram for the metastore here.

There are 2 different ways to setup the metastore server and metastore database using different Hive configurations:

Configuration options for metastore database where metadata is persisted:

Configuration options for metastore server:

准备工作

  1. 安装并配置mysql

    yum install -y mysql-server
service mysqld
mysql
# 设置远程登录权限
grant all privileges on *.* to 'root'@'%' identified by '123' with grant option;
# 删除原有重复登录权限
use mysql;
delete from user where host!='%';
# 修改后需要刷新权限或者重启服务生效
flush privileges;
# 退出后 用密码登录测试

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11

  2. 启动集群

    zkServer.sh start
# 在执行start-all.sh 命令的节点同时也是ResourceManager节点 可以直接启动RM(不包含RM备机)
start-all.sh 
# 启动RM
yarn-daemon.sh start resourcemanager

    • 1
    • 2
    • 3
    • 4
    • 5

  3. 上传 apache-hive-1.2.1-bin.tar.gz 安装包和 mysql-connector-java-5.1.32-bin.jar mysql 驱动包

  4. 安装hive1.2.1

    tar xf apache-hive-1.2.1-bin.tar.gz /opt/

    • 1

  5. 配置环境变量 使用 hive 可以测试是否配置成功

  6. 如果做分离多用户模式 需要拷贝hive到其他节点中

单节点模式搭建

Remote Metastore Database

  1. 配置mysql

Remote Metastore Database

In this configuration, you would use a traditional standalone RDBMS server. The following example configuration will set up a metastore in a MySQL server. This configuration of metastore database is recommended for any real use.

Config ParamConfig ValueComment
javax.jdo.option.ConnectionURLjdbc:mysql:///?createDatabaseIfNotExist=truemetadata is stored in a MySQL server
javax.jdo.option.ConnectionDriverNamecom.mysql.jdbc.DriverMySQL JDBC driver class
javax.jdo.option.ConnectionUserName<user name>user name for connecting to MySQL server
javax.jdo.option.ConnectionPassword<password>password for connecting to MySQL server

Local/Embedded Metastore Server

In local/embedded metastore setup, the metastore server component is used like a library within the Hive Client. Each Hive Client will open a connection to the database and make SQL queries against it. Make sure that the database is accessible from the machines where Hive queries are executed since this is a local store. Also make sure the JDBC client library is in the classpath of Hive Client. This configuration is often used with HiveServer2 (to use embedded metastore only with HiveServer2 add “–hiveconf hive.metastore.uris=’ '” in command line parameters of the hiveserver2 start command or use hiveserver2-site.xml (available in Hive 0.14)).

Config ParamConfig ValueComment
hive.metastore.urisnot needed because this is local store
hive.metastore.localtruethis is local store (removed in Hive 0.10, see configuration description section)
hive.metastore.warehouse.dir<base hdfs path>Points to default location of non-external Hive tables in HDFS.

  • hive-site.xml (hive-default.xml.template 重命名并删除其中默认配置)

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
--><configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://node01/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>123</value>
</property>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive_remote/warehouse</value>
</property>
</configuration>


    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39

  1. mysql 驱动包拷入 hive/lib 下

  2. 删除 hive/lib 或 hadoop-*/share/hadoop/yarn/lib中 版本较低的 jline-*.jar 并将高版本拷入

  3. 运行hive hive

    hive
show tables;
create table tbl(id int,age int);
show tables;
insert into tbl value(1,20);
# 此时可用网页查看yran任务,也可在hdfs后台查看hive <base hdfs path> 目录

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6

  4. mysql 中查看元数据

    mysql -uroot -p
show databases;
use hive_remote;
# hive 为我们创建了大概29个表,
show tables;
select * from TBLS;
select * from COLUMNS_V2;

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

多用户模式-Remote分离搭建

Remote Metastore Server

  1. 服务端配置,同单点模式相同

    Server Configuration Parameters

    The following example uses a Remote Metastore Database.

    Config ParamConfig ValueComment
    javax.jdo.option.ConnectionURLjdbc:mysql:///?createDatabaseIfNotExist=truemetadata is stored in a MySQL server
    javax.jdo.option.ConnectionDriverNamecom.mysql.jdbc.DriverMySQL JDBC driver class
    javax.jdo.option.ConnectionUserName<user name>user name for connecting to MySQL server
    javax.jdo.option.ConnectionPassword<password>password for connecting to MySQL server
    hive.metastore.warehouse.dir<base hdfs path>default location for Hive tables.
    hive.metastore.thrift.bind.host<host_name>Host name to bind the metastore service to. When empty, “localhost” is used. This configuration is available Hive 4.0.0 onwards.

    From Hive 3.0.0 (HIVE-16452) onwards the metastore database stores a GUID which can be queried using the Thrift API get_metastore_db_uuid by metastore clients in order to identify the backend database instance. This API can be accessed by the HiveMetaStoreClient using the method getMetastoreDbUuid().

  • hive-site.xml (注意:ConnectionURL与warehouse.dir 配置值不要与其他hive集群相同)
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://node01/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>root</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>123</value>
</property>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive/warehouse</value>
</property>
</configuration>


    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40

  1. 客户端配置

    Client Configuration Parameters
    Config ParamConfig ValueComment
    hive.metastore.uristhrift://<host_name>:<port>host and port for the Thrift metastore server. If hive.metastore.thrift.bind.host is specified, host should be same as that configuration. Read more about this in dynamic service discovery configuration parameters.
    hive.metastore.localfalseMetastore is remote. Note: This is no longer needed as of Hive 0.10. Setting hive.metastore.uri is sufficient.
    hive.metastore.warehouse.dir<base hdfs path>Points to default location of non-external Hive tables in HDFS.

  • 拷贝服务端hive 文件到所有客户端节点,并配置环境变量

  • hive-site.xml (注意 thrift 默认端口号9083)

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://node02:9083</value>
</property>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/user/hive/warehouse</value>
</property>
</configuration>


    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
  1. 启动服务端 hive --service metastore (启动会阻塞窗口,使用 ss -nal 可以看到9083端口号)
  2. 启动客户端 hive (报错处理 参考单节点模式)
本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号