Hadoop与Hive单节点环境安装文档_单节点部署hive

软件清单

安装步骤

上传安装包至服务器

!注意不要漏传了

将软件拖入如下页面即可上传

基础环境配置

改hosts和hostName

## 改主机名
vi /etc/hostname

hadoop001

## 改ip 主机名称映射
vi /etc/hosts
1
2
3
4
5
6
7

防火墙关闭

systemctl stop firewalld.service 
systemctl disable firewalld.service 
##检查状态
systemctl status firewalld.service 
1
2
3
4

免密登录

ssh-keygen
		#一路回车
		Generating public/private rsa key pair.
		Enter file in which to save the key (/root/.ssh/id_rsa): 
		Enter passphrase (empty for no passphrase): 
		Enter same passphrase again: 
		Your identification has been saved in /root/.ssh/id_rsa.
		Your public key has been saved in /root/.ssh/id_rsa.pub.
		The key fingerprint is:
		df:71:f6:3e:bb:bb:6c:38:91:f4:bc:70:a1:dd:86:a9 root@flower1
		The key's randomart image is:
		+--[ RSA 2048]----+
		|                 |
		|                 |
		|                 |
		|             . . |
		|        S   o Ooo|
		|         . . Oo*o|
		|          . ..=.o|
		|            Eo.= |
		|              o*B|
		+-----------------+
cd ~/.ssh
# 把生成的 id_rsa.pub 文件内容追加到 authorized_keys 中
cat id_rsa.pub  >> authorized_keys

# authorized_keys 授权600，不然ssh node01 还是需要输入密码
chmod 600 authorized_keys
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

基础软件安装

NTP时间同步安装与配置

##执行bash安装ntp
yum install -y ntp
1
2

添加阿里时间同步服务器

# 第一步
vim /etc/ntp.conf

# 在文件中加入阿里时间同步服务器
server ntp.aliyun.com iburst
1
2
3
4
5

# 启动ntp服务
service ntpd start
# 设置开机启动
chkconfig ntpd on
1
2
3
4

JDK安装

##解压到指定目录
tar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/local/

vi /etc/profile

##添加如下
export JAVA_HOME=/usr/local/jdk1.8.0_171
export PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
1
2
3
4
5
6
7
8

##使配置生效
source /etc/profile
##检查java安装是否成功
java -version
1
2
3
4

若成功显示如下：

MySQL安装

##卸载删除原有的mariadb，否则可能会报异常
rpm -qa|grep mariadb
1
2

##  mariadb-xxxxxxxxxxxxxx 为rpm -qa|grep mariadb查询的结果
rpm -e --nodeps  mariadb-xxxxxxxxxxxxxx      

##再次查询，结果为空表示删除干净
rpm -qa|grep mariadb
1
2
3
4
5

##建立文件夹
cd ~
mkdir mysql5.7
##解压到该文件夹中
tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar -C mysql5.7
cd mysql5.7
##安装
sudo rpm -ivh mysql*.rpm --nodeps --force
1
2
3
4
5
6
7
8

• 启动MySQL

##查看mysql状态
systemctl status mysqld
#启动Mysql
systemctl start mysqld
1
2
3
4

• 获取初始密码

grep 'temporary password' /var/log/mysqld.log
1

• 改密码

##初次登录
 mysql -uroot -p
##输入grep 'temporary password' /var/log/mysqld.log 步骤得到的密码
# 修改密码复杂度
set global validate_password_policy=0;
# 修改密码长度
set global validate_password_length=1;
# 修改密码为1234
ALTER USER 'root'@'localhost' IDENTIFIED BY '1234';
# 开启远程访问
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '1234' WITH GRANT OPTION;
##使权限生效
FLUSH PRIVILEGES;
##建立Hive数据库
#这个hive数据库与hive-site.xml中localhost:3306/hive的hive对应，两者名称必须一致，用来保存hive元数据
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

• 设置开机自启

systemctl enable mysqld    ##打开mysql的开机自启
systemctl disable mysqld   ##关闭mysql开机自启
systemctl daemon-reload
1
2
3

• 重启可正常使用mysql

##重启
systemctl restart mysqld
1
2

Hadoop安装

环境变量配置

##创建必要路径 根据实际调整
mkdir -p /opt/bigdata/
mkdir -p /bigdata/hadoop/tmp
mkdir -p /bigdata/hadoop/data/dfs/name
mkdir -p /bigdata/hadoop/data/dfs/data

##解压安装hadoop3


## 以下配置文件均在下述目录
cd /opt/bigdata/hadoop-3.3.1/etc/hadoop
vi /etc/profile
##新增如下
export HADOOP_HOME=/opt/bigdata/hadoop-3.3.1
export PATH=$PATH:%JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

##使变量生效
source /etc/profile 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

hadoop-env.sh 配置

## 编辑该配置文件
vi hadoop-env.sh 

## 增加
export JAVA_HOME=/usr/local/jdk1.8.0_171/
1
2
3
4
5

core-site.xml 配置

## 编辑该配置文件
vi core-site.xml  
1
2

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop001:9000</value>
        <description>hdfs内部通讯访问地址</description>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <!--临时文件目录需要自己建立-->
        <value>/bigdata/hadoop/tmp</value>
    </property>

<property>
	<name>hadoop.proxyuser.root.hosts</name>
	<value>*</value>
</property>

<property>
	<name>hadoop.proxyuser.root.groups</name>
	<value>*</value>
</property>

    <property>
        <name>hadoop.proxyuser.root.users</name>
        <value>*</value>
    </property>

</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

hdfs-site.xml 配置

## 编辑该配置文件
vi hdfs-site.xml  
1
2

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/bigdata/hadoop/data/dfs/name</value>
        <description> namenode 存放name table(fsimage)本地目录需要修改,如果没有需要自己创建文件目录)</description>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/bigdata/hadoop/data/dfs/data</value>
        <description>datanode存放block本地目录（需要修改,如果没有需要自己创建文件目录）</description>
    </property>

    <property>
        	<!--由于只有一台机器,hdfs的副本数就指定为1-->
            <name>dfs.replication</name>
            <value>1</value>
    </property>
</configuration>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

mapred-site.xml 配置

## 编辑该配置文件
vi mapred-site.xml  
1
2

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*
	</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*
	</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/bigdata/hadoop-3.3.1/etc/hadoop:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/common/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/hdfs/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/mapreduce/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop-3.3.1/share/hadoop/yarn/*
	</value>
</property>

</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

yarn-site.xml 配置

## 编辑该配置文件
vi yarn-site.xml 
1
2

<configuration>

<property>
   <name>yarn.scheduler.minimum-allocation-mb</name>
   <value>2048</value>
   <description>default value is 1024</description>
</property>

    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>

<!-- Site specific YARN configuration properties -->

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

</configuration>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

workers 配置

## 编辑该配置文件
vi workers

hadoop001
1
2
3
4

格式化hadoop

#hadoop namenode格式化，看到INFO common.Storage: Storage directory /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/data/dfs/name has been successfully formatted.即为成功格式化
[hadoop@hadoop001 hadoop]$ hadoop namenode -format
1
2

启动hadoop

[hadoop@hadoop001 hadoop] start-all.sh
#启动完之后,可以jps命令查看一下是否启动成功
[hadoop@hadoop001 hadoop] jps
2241 NameNode
2599 NodeManager
2987 Jps
2350 DataNode
2927 ResourceManager
#测试一下hdfs能否正常运行
[hadoop@hadoop001 hadoop]$ hdfs dfs -ls
19/04/09 00:05:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `.': No such file or directory
1
2
3
4
5
6
7
8
9
10
11
12

Hive安装

Hive安装

tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /opt/bigdata

拷贝mysql-connector-java-5.1.47-bin.jar至$Hive_HOME/lib目录下，如果没有jar包的话会报错
1
2
3

Hive环境变量配置

vi /etc/profile
##新增如下
export HIVE_HOME=/opt/bigdata/hive3.1.2
PATH=$HIVE_HOME/bin:$PATH
export PATH

##使变量生效
source /etc/profile 

##改名
cd /opt/bigdata/
mv apache-hive-3.1.2-bin hive3.1.2

### 配置文件位置
cd /opt/bigdata/hive3.1.2/conf


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

hive-env.sh 配置

vi hive-env.sh

export HADOOP_HOME=/opt/bigdata/hadoop-3.3.1/
export HIVE_CONF_DIR=/opt/bigdata/hive3.1.2/conf
export HIVE_AUX_JARS_PATH=/opt/bigdata/hive3.1.2/lib
1
2
3
4
5

hive-site.xml 配置

<configuration>
<property>
        <name>javax.jdo.option.ConnectionURL</name>

        <!--hive_basic为要创建的数据库名,注意字符集设置-->
        <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8&amp;useSSL=false</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <!--MySQL登录账户名-->
        <value>root</value>
</property>

<property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <!--MySQL登录密码-->
        <value>1234</value>
</property>
  <property>
    <!--hive表在hdfs的位置-->
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>
</configuration>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

升级元数据

使用Hive自带的schematool工具升级元数据，也就是把最新的元数据重新写入MySQL数据库中。
可以在终端中执行如下命令（注意，不是在“mysql>”命令提示下执行）：

cd /opt/bigdata/hive3.1.2
./bin/schematool -initSchema -dbType mysql
1
2

异常

拷贝mysql-connector至$Hive_HOME/lib目录下，如果没有jar包的话会报错

1
2
3

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=root
YARN_NODEMANAGER_USER=root
1
2
3
4
5
6
7
8

参考

Linux安装jdk

Hive3.1.3安装和使用指南